Shapes, Matrix Operations, and Derivatives
Reference for the notation, shape rules, and derivative identities used throughout the Deep Learning notes.
Notation used in these notes
| Object | Symbol | Example shape |
|---|---|---|
| Scalar | \(a\) | \(a\in\mathbb{R}\) |
| Vector | \(x\) | \(x\in\mathbb{R}^{d}\) |
| Matrix | \(A\) | \(A\in\mathbb{R}^{m\times n}\) |
| Batch of vectors | \(X\) | \(X\in\mathbb{R}^{N\times d}\) |
| Higher-order tensor | \(T\) | \(T\in\mathbb{R}^{N\times C\times H\times W}\) |
Vectors are treated as column vectors unless a transpose is shown. Batches normally store one example per row, so \(X\in\mathbb{R}^{N\times d}\) contains \(N\) examples with \(d\) features each.
The shape of a gradient matches the shape of the quantity being differentiated:
Matrix operations
Matrix-vector and matrix-matrix products
If \(A\in\mathbb{R}^{m\times n}\) and \(x\in\mathbb{R}^{n}\), then
If \(A\in\mathbb{R}^{m\times n}\) and \(B\in\mathbb{R}^{n\times p}\), then
The inner dimensions must match:
Matrix multiplication is associative, \((AB)C=A(BC)\), but generally not commutative: \(AB\ne BA\).
Inner, outer, and elementwise products
For \(u,v\in\mathbb{R}^d\):
is an inner product, while
is an outer product. The Hadamard product
multiplies corresponding entries and is different from a matrix product.
Shape check: inner and outer products
If \(u\in\mathbb{R}^{K}\) and \(v\in\mathbb{R}^{d}\), then \(u^{\mathsf T}v\) is invalid unless \(K=d\), but \(uv^{\mathsf T}\) is always a valid \(K\times d\) matrix. This is why the weight gradient of a linear layer naturally appears as an outer product.
Transpose rules
Transposition exchanges rows and columns:
The most frequently used identities are
The multiplication order reverses because the transpose must preserve all pairwise inner products. For a scalar,
When an inverse exists,
Batch layouts
The formula for a batch depends on which axis indexes examples. Neither layout is more mathematically correct.
Examples stored as columns
Let \(X_c\in\mathbb{R}^{d\times N}\), where each column is an example. For \(W\in\mathbb{R}^{K\times d}\),
Examples stored as rows
Let \(X_r\in\mathbb{R}^{N\times d}\), where each row is an example. Then
The two representations are transposes of each other:
These notes use row-oriented batches unless a section states otherwise.
Broadcasting and bias
A single-example bias is
For a row-oriented batch, the same bias is added to every row:
The matrix \(\mathbf{1}_Nb^{\mathsf T}\in\mathbb{R}^{N\times K}\) contains \(N\) copies of \(b^{\mathsf T}\). Broadcasting is a compact implementation of this repetition. It does not introduce a different bias for every example.
Broadcasting along the correct axis
If \(Z\in\mathbb{R}^{32\times10}\) contains scores for 32 examples and 10 classes, then \(b\in\mathbb{R}^{10}\) is added across the class axis. A vector of length 32 would instead vary across examples and would not represent one shared class bias.
Gradients, Jacobians, and Hessians
Scalar output
For \(L:\mathbb{R}^d\to\mathbb{R}\), the gradient is a column vector:
The first-order change is
Vector output
For \(y=f(x)\), where \(f:\mathbb{R}^d\to\mathbb{R}^K\), the Jacobian uses output-by-input layout:
Locally,
Second derivatives
For a scalar \(L(x)\), the Hessian is
It describes local curvature and has shape \(d\times d\).
Differentials and product rules
Differentials make matrix derivatives easier to derive because dimensions remain visible. Basic rules include
For a scalar objective, rearrange the differential into one of these forms:
or
The coefficient multiplying \(dx\) or \(dA\) identifies the gradient.
Trace identities
The trace is the sum of diagonal entries. Useful identities are
Factors may be cycled inside a trace, but their order cannot otherwise be rearranged.
Chain rule
For \(x\mapsto y\mapsto z\),
If \(L\) is scalar and \(y=f(x)\), then
This is the vector form of the chain rule used by backpropagation. The incoming gradient \(\nabla_yL\) is multiplied by the transpose of the local Jacobian.
Common derivative identities
Assume \(x\) and \(a\) are column vectors and \(L\) is scalar.
| Function | Gradient with respect to \(x\) |
|---|---|
| \(a^{\mathsf T}x\) | \(a\) |
| \(x^{\mathsf T}x\) | \(2x\) |
| \(\frac12x^{\mathsf T}x\) | \(x\) |
| \(x^{\mathsf T}Ax\) | \((A+A^{\mathsf T})x\) |
| \(\frac12x^{\mathsf T}Ax\), symmetric \(A\) | \(Ax\) |
| \(\frac12\|Ax-b\|_2^2\) | \(A^{\mathsf T}(Ax-b)\) |
| \(\log\sum_i e^{x_i}\) | \(\operatorname{softmax}(x)\) |
For an elementwise function \(y_i=f(x_i)\),
Linear-layer derivatives
One example
Let
and let the incoming gradient be
The differential of the layer is
Substituting into \(dL=g^{\mathsf T}dz\) and matching terms gives
The weight gradient is an outer product with shape
Row-oriented batch
Let
and define \(G=\nabla_ZL\in\mathbb{R}^{N\times K}\). Then
The bias gradient sums contributions from every example because the same bias is shared across the batch. If the loss is a batch mean, its \(1/N\) factor is already included in \(G\).
Shape check: batch linear-layer gradients
Given \(X\in\mathbb{R}^{N\times d}\), \(W\in\mathbb{R}^{K\times d}\), and \(G\in\mathbb{R}^{N\times K}\):
which matches \(W\), while
which matches \(X\).
Shape audit
When a transpose or derivative seems to appear without explanation, check:
- What is the shape of every input, output, and parameter?
- Which axis represents the batch?
- Is the operation a matrix product, outer product, or elementwise product?
- Which quantity is scalar?
- Does each gradient have the same shape as its variable?
- Is a batch contribution summed or averaged?
These checks usually determine where every transpose and reduction must appear.