Gradient Descent and Backpropagation
Gradient-based learning has a simple loop:
The forward pass evaluates the model. The gradient tells us how changing each parameter changes the loss. The update moves the parameters toward lower loss.
See Shapes, Matrix Operations, and Derivatives for the gradient convention, matrix product rules, and detailed linear-layer shape derivations used below.
The optimization problem
Let \(\theta\) denote all trainable parameters and let \(L(\theta)\) be the objective. Training solves
For a data set with \(N\) examples,
The first term measures data fit. The second term expresses a preference such as small or sparse parameters.
Derivatives and gradients
For a scalar function of one variable,
For \(L:\mathbb{R}^d\to\mathbb{R}\), the gradient is
For a small change \(\Delta\theta\),
The directional derivative is smallest when \(\Delta\theta\) points opposite the gradient. This is why \(-\nabla L\) is the steepest local descent direction under the Euclidean norm.
Gradient descent
The basic update is
where \(\eta>0\) is the learning rate. The gradient is recomputed after each update because the slope changes as the parameters move.
Learning-rate behavior
- If \(\eta\) is too small, training can be extremely slow.
- If \(\eta\) is too large, the loss can oscillate or diverge.
- A useful learning rate depends on curvature, parameterization, and data scale.
- Learning-rate schedules change \(\eta\) during training; they do not change the definition of the loss.
For a quadratic function with Hessian eigenvalues between \(\mu\) and \(L\), stable fixed-step behavior requires a sufficiently small step, with the largest curvature direction usually controlling the limit.
Batch and mini-batch methods
Full-batch gradient descent uses every example:
For a mini-batch \(B_t\) of size \(M\),
The mini-batch gradient is noisy but cheaper to compute. Under random sampling, it is often an estimate of the full gradient. Batch size trades computation, memory, noise, and update frequency.
Worked example: quadratic gradient descent
For \(L(w)=\frac12(w-3)^2\), the gradient is \(w-3\). With \(w_0=0\) and \(\eta=0.5\):
The iterates approach the minimizer \(w=3\). A larger step can reach it faster for this simple function, but can overshoot when it becomes too large.
Deriving a linear-regression update
Let \(X\in\mathbb{R}^{N\times d}\), \(w\in\mathbb{R}^d\), and \(y\in\mathbb{R}^N\). For the half-squared-error objective,
Using the matrix derivative,
The update is therefore
The residual \(Xw-y\) measures prediction error; multiplying by \(X^{\mathsf T}\) attributes that error back to each feature.
Backpropagation
Backpropagation is reverse-mode application of the chain rule. A neural network is a composition:
The forward pass stores intermediate values. The backward pass starts with \(\partial L/\partial h_2\) and propagates derivatives backward, reusing the local derivative of each operation.
If \(u=g(v)\) and the upstream derivative is \(\bar u=\partial L/\partial u\), then
For vectors, this is a vector-Jacobian product. It avoids forming a large Jacobian explicitly.
See the chain rule reference for the Jacobian form and shape convention.
A linear layer
For
and upstream gradient \(\bar z=\partial L/\partial z\), the parameter and input gradients are
For a batch, the weight gradient sums or averages the outer products over examples, according to the loss reduction convention.
Nonlinearities
If \(h=\tanh(z)\), then
If \(h=\sigma(z)\), then
The chain rule multiplies these local derivatives by the upstream gradient. Repeated multiplication through many layers can cause vanishing or exploding gradients.
Worked example: one hidden unit
Let \(z=wx+b\), \(h=\tanh(z)\), and \(L=\frac12(h-y)^2\). Then
Each factor corresponds to one edge in the computation graph.
Jacobians and Hessians
For \(f:\mathbb{R}^n\to\mathbb{R}^m\), the Jacobian is
For a scalar loss, the gradient has one entry per parameter. If the parameters are a matrix, the gradient has the same shape as that matrix; it is not a single scalar.
The Hessian is the derivative of the gradient:
It describes local curvature, but deep networks are usually too large for an explicit Hessian. Hessian-vector products can provide curvature information without storing the full matrix.
Automatic differentiation
Automatic differentiation applies the chain rule to a program made of elementary operations. It is different from symbolic differentiation, which constructs an algebraic expression, and numerical differentiation, which approximates derivatives with finite differences.
Forward-mode propagates derivatives from inputs to outputs and is efficient when there are few inputs. Reverse-mode propagates gradients from a scalar output back to many parameters and is efficient for neural-network training. Backpropagation is reverse-mode automatic differentiation applied to a network and loss.
Optimization beyond basic gradient descent
Momentum
Momentum maintains a velocity:
It smooths noisy gradients and can accelerate movement along consistent directions.
Adaptive methods
Methods such as Adam maintain estimates of first and second gradient moments and rescale coordinates. They can be effective when parameter gradients have very different scales, but their behavior and generalization can differ from plain SGD.
Weight decay
For an \(L_2\) penalty, the objective includes \(\frac\lambda2\|\theta\|_2^2\), adding \(\lambda\theta\) to the gradient. In some optimizers, decoupled weight decay applies the shrinkage directly rather than treating it exactly as part of the loss gradient.
Common failure modes
- Wrong gradient shape: every parameter gradient should match the parameter's shape.
- Wrong reduction: switching between sum and mean changes gradient scale and may require changing the learning rate.
- Numerical instability: use stable log-softmax and cross-entropy implementations.
- Exploding gradients: inspect norms, reduce the step size, normalize inputs, or use gradient clipping when appropriate.
- Vanishing gradients: inspect activation derivatives and the depth of the computation graph.
- Data leakage: optimization can reduce training loss while validation performance gets worse.
- Confusing a stationary point with a global minimum: non-convex neural-network objectives do not generally provide that guarantee.