Skip to content

Shapes, Matrix Operations, and Derivatives

Reference for the notation, shape rules, and derivative identities used throughout the Deep Learning notes.

Notation used in these notes

Object Symbol Example shape
Scalar \(a\) \(a\in\mathbb{R}\)
Vector \(x\) \(x\in\mathbb{R}^{d}\)
Matrix \(A\) \(A\in\mathbb{R}^{m\times n}\)
Batch of vectors \(X\) \(X\in\mathbb{R}^{N\times d}\)
Higher-order tensor \(T\) \(T\in\mathbb{R}^{N\times C\times H\times W}\)

Vectors are treated as column vectors unless a transpose is shown. Batches normally store one example per row, so \(X\in\mathbb{R}^{N\times d}\) contains \(N\) examples with \(d\) features each.

The shape of a gradient matches the shape of the quantity being differentiated:

\[ W\in\mathbb{R}^{K\times d} \quad\Longrightarrow\quad \nabla_W L\in\mathbb{R}^{K\times d}. \]

Matrix operations

Matrix-vector and matrix-matrix products

If \(A\in\mathbb{R}^{m\times n}\) and \(x\in\mathbb{R}^{n}\), then

\[ y=Ax\in\mathbb{R}^{m}, \qquad y_i=\sum_{j=1}^{n}A_{ij}x_j. \]

If \(A\in\mathbb{R}^{m\times n}\) and \(B\in\mathbb{R}^{n\times p}\), then

\[ C=AB\in\mathbb{R}^{m\times p}, \qquad C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}. \]

The inner dimensions must match:

\[ (m\times n)(n\times p)=(m\times p). \]

Matrix multiplication is associative, \((AB)C=A(BC)\), but generally not commutative: \(AB\ne BA\).

Inner, outer, and elementwise products

For \(u,v\in\mathbb{R}^d\):

\[ u^{\mathsf T}v=\sum_i u_iv_i\in\mathbb{R} \]

is an inner product, while

\[ uv^{\mathsf T}\in\mathbb{R}^{d\times d}, \qquad (uv^{\mathsf T})_{ij}=u_iv_j \]

is an outer product. The Hadamard product

\[ u\odot v=(u_1v_1,\ldots,u_dv_d)^{\mathsf T} \]

multiplies corresponding entries and is different from a matrix product.

Shape check: inner and outer products

If \(u\in\mathbb{R}^{K}\) and \(v\in\mathbb{R}^{d}\), then \(u^{\mathsf T}v\) is invalid unless \(K=d\), but \(uv^{\mathsf T}\) is always a valid \(K\times d\) matrix. This is why the weight gradient of a linear layer naturally appears as an outer product.

Transpose rules

Transposition exchanges rows and columns:

\[ A\in\mathbb{R}^{m\times n} \quad\Longrightarrow\quad A^{\mathsf T}\in\mathbb{R}^{n\times m}. \]

The most frequently used identities are

\[ (A^{\mathsf T})^{\mathsf T}=A, \qquad (A+B)^{\mathsf T}=A^{\mathsf T}+B^{\mathsf T}, \]
\[ (AB)^{\mathsf T}=B^{\mathsf T}A^{\mathsf T}, \qquad (ABC)^{\mathsf T}=C^{\mathsf T}B^{\mathsf T}A^{\mathsf T}. \]

The multiplication order reverses because the transpose must preserve all pairwise inner products. For a scalar,

\[ x^{\mathsf T}Ay=(x^{\mathsf T}Ay)^{\mathsf T} =y^{\mathsf T}A^{\mathsf T}x. \]

When an inverse exists,

\[ (AB)^{-1}=B^{-1}A^{-1}, \qquad (A^{\mathsf T})^{-1}=(A^{-1})^{\mathsf T}. \]

Batch layouts

The formula for a batch depends on which axis indexes examples. Neither layout is more mathematically correct.

Examples stored as columns

Let \(X_c\in\mathbb{R}^{d\times N}\), where each column is an example. For \(W\in\mathbb{R}^{K\times d}\),

\[ S_c=WX_c+b\mathbf{1}_N^{\mathsf T} \in\mathbb{R}^{K\times N}. \]

Examples stored as rows

Let \(X_r\in\mathbb{R}^{N\times d}\), where each row is an example. Then

\[ S_r=X_rW^{\mathsf T}+\mathbf{1}_Nb^{\mathsf T} \in\mathbb{R}^{N\times K}. \]

The two representations are transposes of each other:

\[ X_r=X_c^{\mathsf T}, \qquad S_r=S_c^{\mathsf T}. \]

These notes use row-oriented batches unless a section states otherwise.

Broadcasting and bias

A single-example bias is

\[ z=Wx+b, \qquad b\in\mathbb{R}^{K}. \]

For a row-oriented batch, the same bias is added to every row:

\[ Z=XW^{\mathsf T}+\mathbf{1}_Nb^{\mathsf T}. \]

The matrix \(\mathbf{1}_Nb^{\mathsf T}\in\mathbb{R}^{N\times K}\) contains \(N\) copies of \(b^{\mathsf T}\). Broadcasting is a compact implementation of this repetition. It does not introduce a different bias for every example.

Broadcasting along the correct axis

If \(Z\in\mathbb{R}^{32\times10}\) contains scores for 32 examples and 10 classes, then \(b\in\mathbb{R}^{10}\) is added across the class axis. A vector of length 32 would instead vary across examples and would not represent one shared class bias.

Gradients, Jacobians, and Hessians

Scalar output

For \(L:\mathbb{R}^d\to\mathbb{R}\), the gradient is a column vector:

\[ \nabla_xL= \begin{bmatrix} \partial L/\partial x_1\\ \vdots\\ \partial L/\partial x_d \end{bmatrix} \in\mathbb{R}^{d}. \]

The first-order change is

\[ dL=(\nabla_xL)^{\mathsf T}dx. \]

Vector output

For \(y=f(x)\), where \(f:\mathbb{R}^d\to\mathbb{R}^K\), the Jacobian uses output-by-input layout:

\[ J_{y,x}=\frac{\partial y}{\partial x} \in\mathbb{R}^{K\times d}, \qquad (J_{y,x})_{ij}=\frac{\partial y_i}{\partial x_j}. \]

Locally,

\[ dy=J_{y,x}\,dx. \]

Second derivatives

For a scalar \(L(x)\), the Hessian is

\[ H_L=\nabla_x^2L, \qquad (H_L)_{ij}=\frac{\partial^2L}{\partial x_i\partial x_j}. \]

It describes local curvature and has shape \(d\times d\).

Differentials and product rules

Differentials make matrix derivatives easier to derive because dimensions remain visible. Basic rules include

\[ d(A+B)=dA+dB, \qquad d(cA)=c\,dA, \]
\[ d(AB)=(dA)B+A(dB), \qquad d(Ax)=(dA)x+A\,dx. \]

For a scalar objective, rearrange the differential into one of these forms:

\[ dL=(\nabla_xL)^{\mathsf T}dx \]

or

\[ dL=\operatorname{tr}\!\left((\nabla_AL)^{\mathsf T}dA\right). \]

The coefficient multiplying \(dx\) or \(dA\) identifies the gradient.

Trace identities

The trace is the sum of diagonal entries. Useful identities are

\[ \operatorname{tr}(A)=\operatorname{tr}(A^{\mathsf T}), \qquad \operatorname{tr}(AB)=\operatorname{tr}(BA), \]
\[ \operatorname{tr}(ABC)=\operatorname{tr}(BCA)=\operatorname{tr}(CAB). \]

Factors may be cycled inside a trace, but their order cannot otherwise be rearranged.

Chain rule

For \(x\mapsto y\mapsto z\),

\[ J_{z,x}=J_{z,y}J_{y,x}. \]

If \(L\) is scalar and \(y=f(x)\), then

\[ \nabla_xL=J_{y,x}^{\mathsf T}\nabla_yL. \]

This is the vector form of the chain rule used by backpropagation. The incoming gradient \(\nabla_yL\) is multiplied by the transpose of the local Jacobian.

Common derivative identities

Assume \(x\) and \(a\) are column vectors and \(L\) is scalar.

Function Gradient with respect to \(x\)
\(a^{\mathsf T}x\) \(a\)
\(x^{\mathsf T}x\) \(2x\)
\(\frac12x^{\mathsf T}x\) \(x\)
\(x^{\mathsf T}Ax\) \((A+A^{\mathsf T})x\)
\(\frac12x^{\mathsf T}Ax\), symmetric \(A\) \(Ax\)
\(\frac12\|Ax-b\|_2^2\) \(A^{\mathsf T}(Ax-b)\)
\(\log\sum_i e^{x_i}\) \(\operatorname{softmax}(x)\)

For an elementwise function \(y_i=f(x_i)\),

\[ J_{y,x}=\operatorname{diag}(f'(x_1),\ldots,f'(x_d)). \]

Linear-layer derivatives

One example

Let

\[ z=Wx+b, \]

and let the incoming gradient be

\[ g=\nabla_zL\in\mathbb{R}^{K}. \]

The differential of the layer is

\[ dz=(dW)x+W\,dx+db. \]

Substituting into \(dL=g^{\mathsf T}dz\) and matching terms gives

\[ \boxed{\nabla_WL=gx^{\mathsf T}}, \qquad \boxed{\nabla_xL=W^{\mathsf T}g}, \qquad \boxed{\nabla_bL=g}. \]

The weight gradient is an outer product with shape

\[ (K\times1)(1\times d)=K\times d. \]

Row-oriented batch

Let

\[ Z=XW^{\mathsf T}+\mathbf{1}_Nb^{\mathsf T} \]

and define \(G=\nabla_ZL\in\mathbb{R}^{N\times K}\). Then

\[ \boxed{\nabla_XL=GW}, \qquad \boxed{\nabla_WL=G^{\mathsf T}X}, \qquad \boxed{\nabla_bL=G^{\mathsf T}\mathbf{1}_N}. \]

The bias gradient sums contributions from every example because the same bias is shared across the batch. If the loss is a batch mean, its \(1/N\) factor is already included in \(G\).

Shape check: batch linear-layer gradients

Given \(X\in\mathbb{R}^{N\times d}\), \(W\in\mathbb{R}^{K\times d}\), and \(G\in\mathbb{R}^{N\times K}\):

\[ G^{\mathsf T}X:(K\times N)(N\times d)=K\times d, \]

which matches \(W\), while

\[ GW:(N\times K)(K\times d)=N\times d, \]

which matches \(X\).

Shape audit

When a transpose or derivative seems to appear without explanation, check:

  1. What is the shape of every input, output, and parameter?
  2. Which axis represents the batch?
  3. Is the operation a matrix product, outer product, or elementwise product?
  4. Which quantity is scalar?
  5. Does each gradient have the same shape as its variable?
  6. Is a batch contribution summed or averaged?

These checks usually determine where every transpose and reduction must appear.