Math Foundations for Deep Learning
Quick reference for the mathematics used in model construction, backpropagation, loss functions, and optimization. The tabs are organized around the concepts that recur in derivations and implementations.
Shapes are part of the mathematics
A scalar is an element of \(\mathbb{R}\), a vector is in \(\mathbb{R}^n\), and a matrix is in \(\mathbb{R}^{m\times n}\). A tensor adds named axes such as batch, channel, sequence, height, width, and feature. In every expression, check that the dimensions contract as intended.
For \(X\in\mathbb{R}^{n\times d}\), \(W\in\mathbb{R}^{d\times k}\), and \(b\in\mathbb{R}^k\),
The outer product \(uv^{\mathsf T}\) is a matrix; the inner product \(u^{\mathsf T}v\) is a scalar. Broadcasting is shorthand for repeating an array along an axis and must be checked explicitly.
Useful matrix derivatives
With a scalar objective and column-vector convention:
For matrices, the Frobenius inner product is \(\langle A,B\rangle_F=\operatorname{tr}(A^{\mathsf T}B)\). Trace identities are often the cleanest way to derive gradients:
Worked example: least-squares gradient
Let \(L(w)=\frac12\|Xw-y\|_2^2\). Set \(r=Xw-y\). Since \(dr=X\,dw\), the differential is \(dL=r^{\mathsf T}X\,dw\), so \(\nabla_wL=X^{\mathsf T}(Xw-y)\). The result has the same shape as \(w\).
Derivatives of functions of many variables
For \(f:\mathbb{R}^n\to\mathbb{R}\), the gradient is the vector of first partial derivatives:
The directional derivative in direction \(v\) is \(D_vf(x)=\nabla f(x)^{\mathsf T}v\). For a unit vector, it is largest in the gradient direction. A first-order Taylor approximation is:
For \(f:\mathbb{R}^n\to\mathbb{R}^m\), the Jacobian \(J\in\mathbb{R}^{m\times n}\) contains \(J_{ij}=\partial f_i/\partial x_j\). Locally, \(f(x+\Delta x)\approx f(x)+J\Delta x\).
Chain rule and backpropagation
If \(z=g(x)\) and \(y=f(z)\), then \(J_{y,x}=J_{y,z}J_{z,x}\). Backpropagation evaluates this product in reverse order without explicitly constructing every Jacobian. For a scalar loss, vector-Jacobian products are the central operation.
For elementwise nonlinearities, the local Jacobian is diagonal. For softmax, the Jacobian is dense:
Worked example: reverse-mode chain rule
Let \(u=Wx+b\), \(h=\tanh(u)\), and \(L=c^{\mathsf T}h\). If the upstream gradient is \(\nabla_hL=c\), then
Second-order structure
The Hessian of a scalar function is
When the function is twice continuously differentiable, the Hessian is symmetric. The second-order Taylor approximation is:
At a stationary point, a positive definite Hessian indicates a strict local minimum; a negative definite Hessian indicates a strict local maximum; an indefinite Hessian indicates saddle behavior. Positive semidefinite curvature is necessary for a differentiable local minimum, but not always sufficient.
Eigenvalues describe curvature magnitude and direction. A large condition number means some directions are much steeper than others, which can make gradient descent zig-zag or progress slowly.
Hessian-vector products
Deep networks have too many parameters to form a full Hessian. Hessian-vector products \(Hv\) can often be computed with automatic differentiation at a cost comparable to a few gradient evaluations. They support curvature diagnostics and second-order or approximate second-order methods.
Worked example: classifying a stationary point
For \(f(x,y)=x^2-4y^2\), \(\nabla f(0,0)=0\) and
The Hessian has both positive and negative eigenvalues, so the origin is a saddle point rather than a minimum.
Different norms produce different geometry
For \(p\ge1\), the vector \(p\)-norm is
The unit-ball geometry changes with \(p\). \(L_2\) gives round level sets and smooth gradients away from the origin; \(L_1\) gives diamond-shaped level sets and promotes exact zeros; \(L_\infty\) limits the largest coordinate.
Norms are used as penalties, constraints, and measures of parameter or gradient size:
The squared \(L_2\) penalty has gradient \(2\lambda\theta\). The \(L_1\) norm is not differentiable at zero, so optimization uses a subgradient or a proximal operator. For a convex function, a subgradient \(g\) at \(x\) satisfies \(f(y)\ge f(x)+g^{\mathsf T}(y-x)\) for all \(y\).
Also distinguish vector norms from matrix norms. The Frobenius norm is \(\|A\|_F^2=\sum_{ij}A_{ij}^2\), while the spectral norm is the largest singular value.
Worked example: shrinkage from regularization
Consider \(L(w)=\frac12(w-a)^2+\lambda|w|\). The quadratic term pulls \(w\) toward \(a\), while the \(L_1\) term pulls it toward zero. The exact minimizer is the soft-thresholding rule \(w^*=\operatorname{sign}(a)\max(|a|-\lambda,0)\), illustrating why \(L_1\) can create sparsity.
Random variables and estimation
A probability mass function sums to one; a continuous density integrates to one. A density value is not itself a probability. Conditional probability is \(p(x\mid y)=p(x,y)/p(y)\), and Bayes' rule is:
Expectation, variance, and covariance are:
For independent samples, the log-likelihood is a sum:
Maximizing likelihood is equivalent to minimizing negative log-likelihood. This connection explains many supervised learning losses.
Worked example: Gaussian maximum likelihood
If observations are modeled as \(x_i\sim\mathcal{N}(\mu,\sigma^2)\) with known \(\sigma\), the negative log-likelihood differs from \(\frac{1}{2\sigma^2}\sum_i(x_i-\mu)^2\) only by constants. Thus maximum likelihood selects the mean that minimizes squared error.
First-order optimization
An unconstrained problem is \(\min_x f(x)\). Gradient descent uses \(x_{t+1}=x_t-\eta\nabla f(x_t)\). A line search chooses a step using the objective; a fixed learning rate chooses it in advance. Stochastic gradient descent replaces the full gradient with an unbiased or approximately unbiased mini-batch estimate.
Momentum accumulates a moving direction. Adaptive methods rescale coordinates using estimates of first and second moments. These methods change optimization dynamics, not the underlying objective.
For constrained problems, projected gradient descent takes a gradient step and projects back into the feasible set. Lagrange multipliers convert constraints into a saddle-point formulation.
Convexity, strong convexity, and smoothness
A differentiable function is convex if
It is \(\mu\)-strongly convex if the right side includes \(\frac{\mu}{2}\|y-x\|_2^2\). Strong convexity gives a unique minimizer and a curvature lower bound. A function is \(L\)-smooth when its gradient is \(L\)-Lipschitz:
For twice-differentiable functions, convexity corresponds to \(H\succeq0\); strong convexity corresponds to \(H\succeq\mu I\); and smoothness corresponds to \(H\preceq LI\), when these bounds hold globally. Concavity is the reverse inequality: \(H\preceq0\). A function can be convex in one variable and non-convex jointly.
Local versus global behavior
Convexity supplies global guarantees. Deep-network objectives are generally non-convex, so a stationary point need not be globally optimal. Local curvature, saddle points, flat directions, and parameter symmetries are therefore important when interpreting training.
Worked example: curvature controls a step size
For \(f(x)=\frac12cx^2\), the gradient is \(cx\) and the Hessian is \(c\). Gradient descent gives \(x_{t+1}=(1-\eta c)x_t\). If \(0<\eta c<2\), the iterates converge; a larger step can oscillate or diverge.
Entropy, cross-entropy, and KL divergence
Let \(p=(p_1,\ldots,p_k)\) and \(q=(q_1,\ldots,q_k)\) be probability distributions over the same outcomes. Each probability is non-negative and the probabilities sum to one. By convention, \(0\log 0=0\); a zero-probability outcome contributes nothing to entropy or KL divergence.
Entropy: uncertainty in one distribution
Meaning: Entropy measures how uncertain or unpredictable a distribution is.
Formula:
Use: It measures the information content of a random variable and helps describe the uncertainty in a model's predictions.
Important properties: \(H(p)\ge0\); it is zero for a deterministic distribution; for \(k\) outcomes it is largest for the uniform distribution, with maximum \(\log k\); and it is concave in \(p\). The log base determines the units: base 2 gives bits, while natural log gives nats.
Cross-entropy: cost of using the wrong distribution
Meaning: Cross-entropy measures how well a predicted distribution \(q\) represents outcomes generated from the true distribution \(p\).
Formula:
Use: It is the standard classification loss. With a one-hot target, it becomes \(-\log q_y\), so assigning low probability to the correct class produces a large penalty.
Important properties: \(H(p,q)\ge H(p)\), with equality only when \(p=q\); it is not symmetric; and it can become infinite when \(q_i=0\) for an outcome with \(p_i>0\). It is not itself a distance.
KL divergence: information lost by approximation
Meaning: KL divergence measures how different a reference distribution \(p\) is from an approximating distribution \(q\).
Formula:
Use: It is used to compare distributions, regularize model predictions, and measure how much an approximate probability model departs from a target or prior.
Important properties: \(D_{KL}(p\|q)\ge0\), and it equals zero only when \(p=q\); it is not symmetric, so \(D_{KL}(p\|q)\ne D_{KL}(q\|p)\) in general; it does not satisfy the triangle inequality; and it can be infinite when \(q_i=0\) where \(p_i>0\).
Relationship
Since \(H(p)\) is fixed when the target distribution is fixed, minimizing cross-entropy with respect to \(q\) is equivalent to minimizing \(D_{KL}(p\|q)\).
Softmax and numerical stability
Softmax maps logits to probabilities:
In finite precision, compute softmax and log-sum-exp after shifting logits by their maximum:
Stable implementations should avoid explicitly taking \(\log(\operatorname{softmax}(z))\) when a combined log-softmax or cross-entropy operation is available.
Worked example: stable log-sum-exp
Directly computing \(e^{1000}\) overflows. Instead,
which is finite and mathematically equivalent.