Linear Classifiers and Loss Functions
This topic connects four objects:
The model describes how inputs become scores. The loss describes what counts as a good set of scores. Learning will later adjust the parameters to reduce that loss.
This page uses row-oriented batches. See Shapes, Matrix Operations, and Derivatives for shape rules, transpose identities, batch layouts, and linear-layer derivatives.
Supervised-learning notation
A supervised data set contains pairs
where \(x_i\in\mathbb{R}^d\) is an input with \(d\) features and \(y_i\) is its target. For classification with \(K\) classes, \(y_i\) is often stored as an integer in \(\{1,\ldots,K\}\) or as a one-hot vector in \(\mathbb{R}^K\).
The parameters are learned from the training data. They are not the same thing as the input features: the input describes one example, while the parameters describe the model shared by all examples.
Linear models
Binary linear classifier
The score for an input is
where \(w\in\mathbb{R}^d\) is a weight vector and \(b\in\mathbb{R}\) is a bias. The sign of the score can define a binary prediction:
The decision boundary is the set of points satisfying \(w^{\mathsf T}x+b=0\). It is a line in two dimensions, a plane in three dimensions, and a hyperplane in higher dimensions. The vector \(w\) is perpendicular to this boundary; the bias moves the boundary without changing its orientation.
Multiclass linear classifier
For \(K\) classes, use a matrix \(W\in\mathbb{R}^{K\times d}\) and bias \(b\in\mathbb{R}^K\):
The result \(s(x)\in\mathbb{R}^K\) contains one score per class. The predicted class is
Each row of \(W\) acts as a template or linear classifier for one class. The boundary between classes \(a\) and \(b\) is where their scores are equal:
Batch form and shapes
For a batch of \(N\) examples stored as rows in \(X\in\mathbb{R}^{N\times d}\),
Some libraries store examples as columns and use \(S=WX+b\). Both conventions are valid; the important rule is to keep the shapes and transpose operations consistent.
See Batch layouts for a direct comparison of the two forms.
Worked example: interpreting a score
Let \(w=(2,-1)\), \(b=-1\), and \(x=(1,2)\). Then
The classifier assigns the negative class. The boundary is \(2x_1-x_2-1=0\), and the vector \((2,-1)\) is normal to it.
Scores and probabilities
A score is not automatically a probability. Scores can be negative, larger than one, and shifted by a constant without changing the predicted class. Probabilities must be non-negative and sum to one.
Sigmoid and binary classification
The sigmoid function maps a scalar score to \((0,1)\):
Interpret it as \(P(Y=1\mid x)\). The inverse relationship is the log-odds:
This is logistic regression: linear in the log-odds, nonlinear in the probability.
Softmax and multiclass classification
Softmax converts scores into a categorical distribution:
Adding the same constant to every score does not change the probabilities. This shift invariance is useful for numerical stability: subtract \(\max_j s_j\) before exponentiating.
Loss functions
The empirical risk is the average loss over the training set:
where \(\theta\) collects the model parameters. A loss is not the same as accuracy: it provides a numerical surface that an optimizer can use to improve the parameters.
Squared error for regression
For a scalar prediction \(\hat y\),
The factor \(1/2\) does not change the minimizer; it makes the derivative simpler. Squared error corresponds to a Gaussian-noise likelihood under common assumptions.
Multiclass hinge loss
For scores \(s\) and correct class \(y\), the multiclass hinge loss is
where \(\Delta>0\) is a margin, often set to 1. The correct class should score at least \(\Delta\) higher than every incorrect class. Once that margin is satisfied, the term contributes zero.
Hinge loss is piecewise linear and non-differentiable at the hinge points. Subgradients or a chosen convention handle those points.
Cross-entropy and negative log-likelihood
For a one-hot target \(y\) and predicted probabilities \(p\),
For a soft target distribution \(q\),
Cross-entropy strongly penalizes assigning a very small probability to the correct class. It is the negative log-likelihood for a categorical model, so minimizing average cross-entropy is maximum likelihood estimation.
For logits passed through softmax, the gradient has a particularly useful form:
The gradient is predicted probability minus target indicator.
Worked example: hinge loss versus cross-entropy
Suppose the correct class has score \(s_y=2\), an incorrect class has score \(s_j=1.5\), and \(\Delta=1\). The hinge term is
The margin is not yet satisfied, so the loss pushes the correct score up or the incorrect score down. Cross-entropy would instead convert all scores to probabilities and penalize the probability assigned to the correct class.
Regularization
Regularization adds a parameter penalty:
Common choices are
The \(L_2\) penalty discourages large weights smoothly. The \(L_1\) penalty can produce exact zeros and therefore sparse parameters, but is non-differentiable at zero. Regularization changes the objective being optimized; it is not merely a post-processing step.
What this model cannot do
A single linear classifier can only create linear decision boundaries in the chosen feature space. If the classes are not linearly separable, changing the optimizer cannot make the model represent a nonlinear boundary. The model must receive better features or gain nonlinear layers.
This limitation motivates neural networks: compose affine transformations with nonlinear activation functions and learn the representation together with the classifier.