Skip to content

Linear Classifiers and Loss Functions

This topic connects four objects:

\[ \text{data} \longrightarrow \text{linear scores} \longrightarrow \text{predictions} \longrightarrow \text{loss}. \]

The model describes how inputs become scores. The loss describes what counts as a good set of scores. Learning will later adjust the parameters to reduce that loss.

This page uses row-oriented batches. See Shapes, Matrix Operations, and Derivatives for shape rules, transpose identities, batch layouts, and linear-layer derivatives.

Supervised-learning notation

A supervised data set contains pairs

\[ \mathcal{D}=\{(x_i,y_i)\}_{i=1}^{N}, \]

where \(x_i\in\mathbb{R}^d\) is an input with \(d\) features and \(y_i\) is its target. For classification with \(K\) classes, \(y_i\) is often stored as an integer in \(\{1,\ldots,K\}\) or as a one-hot vector in \(\mathbb{R}^K\).

The parameters are learned from the training data. They are not the same thing as the input features: the input describes one example, while the parameters describe the model shared by all examples.

Linear models

Binary linear classifier

The score for an input is

\[ s(x)=w^{\mathsf T}x+b, \]

where \(w\in\mathbb{R}^d\) is a weight vector and \(b\in\mathbb{R}\) is a bias. The sign of the score can define a binary prediction:

\[ \hat y= \begin{cases} 1,&s(x)\ge0,\\ 0,&s(x)<0. \end{cases} \]

The decision boundary is the set of points satisfying \(w^{\mathsf T}x+b=0\). It is a line in two dimensions, a plane in three dimensions, and a hyperplane in higher dimensions. The vector \(w\) is perpendicular to this boundary; the bias moves the boundary without changing its orientation.

Geometry of a binary linear classifier Two classes lie on opposite sides of a diagonal decision boundary. The weight vector is perpendicular to the boundary and points toward increasing score. x₁ x₂ wᵀx + b = 0 w positive score negative score class +1 class −1

Multiclass linear classifier

For \(K\) classes, use a matrix \(W\in\mathbb{R}^{K\times d}\) and bias \(b\in\mathbb{R}^K\):

\[ s(x)=Wx+b. \]

The result \(s(x)\in\mathbb{R}^K\) contains one score per class. The predicted class is

\[ \hat y=\arg\max_k s_k(x). \]

Each row of \(W\) acts as a template or linear classifier for one class. The boundary between classes \(a\) and \(b\) is where their scores are equal:

\[ (w_a-w_b)^{\mathsf T}x+(b_a-b_b)=0. \]

Batch form and shapes

For a batch of \(N\) examples stored as rows in \(X\in\mathbb{R}^{N\times d}\),

\[ S=XW^{\mathsf T}+\mathbf{1}b^{\mathsf T}\in\mathbb{R}^{N\times K}. \]

Some libraries store examples as columns and use \(S=WX+b\). Both conventions are valid; the important rule is to keep the shapes and transpose operations consistent.

See Batch layouts for a direct comparison of the two forms.

Worked example: interpreting a score

Let \(w=(2,-1)\), \(b=-1\), and \(x=(1,2)\). Then

\[ s(x)=2(1)-1(2)-1=-1. \]

The classifier assigns the negative class. The boundary is \(2x_1-x_2-1=0\), and the vector \((2,-1)\) is normal to it.

Scores and probabilities

A score is not automatically a probability. Scores can be negative, larger than one, and shifted by a constant without changing the predicted class. Probabilities must be non-negative and sum to one.

Sigmoid and binary classification

The sigmoid function maps a scalar score to \((0,1)\):

\[ \sigma(s)=\frac{1}{1+e^{-s}}. \]

Interpret it as \(P(Y=1\mid x)\). The inverse relationship is the log-odds:

\[ \log\frac{P(Y=1\mid x)}{1-P(Y=1\mid x)}=w^{\mathsf T}x+b. \]

This is logistic regression: linear in the log-odds, nonlinear in the probability.

Softmax and multiclass classification

Softmax converts scores into a categorical distribution:

\[ p_k=\operatorname{softmax}(s)_k =\frac{e^{s_k}}{\sum_{j=1}^{K}e^{s_j}}. \]

Adding the same constant to every score does not change the probabilities. This shift invariance is useful for numerical stability: subtract \(\max_j s_j\) before exponentiating.

Loss functions

The empirical risk is the average loss over the training set:

\[ \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell(f_\theta(x_i),y_i), \]

where \(\theta\) collects the model parameters. A loss is not the same as accuracy: it provides a numerical surface that an optimizer can use to improve the parameters.

Squared error for regression

For a scalar prediction \(\hat y\),

\[ \ell_{\text{MSE}}(\hat y,y)=\frac12(\hat y-y)^2. \]

The factor \(1/2\) does not change the minimizer; it makes the derivative simpler. Squared error corresponds to a Gaussian-noise likelihood under common assumptions.

Multiclass hinge loss

For scores \(s\) and correct class \(y\), the multiclass hinge loss is

\[ \ell_{\text{hinge}}(s,y)=\sum_{j\ne y}\max\left(0,\,s_j-s_y+\Delta\right), \]

where \(\Delta>0\) is a margin, often set to 1. The correct class should score at least \(\Delta\) higher than every incorrect class. Once that margin is satisfied, the term contributes zero.

Hinge loss is piecewise linear and non-differentiable at the hinge points. Subgradients or a chosen convention handle those points.

Cross-entropy and negative log-likelihood

For a one-hot target \(y\) and predicted probabilities \(p\),

\[ \ell_{\text{CE}}(p,y)=-\log p_y. \]

For a soft target distribution \(q\),

\[ H(q,p)=-\sum_{k=1}^{K}q_k\log p_k. \]

Cross-entropy strongly penalizes assigning a very small probability to the correct class. It is the negative log-likelihood for a categorical model, so minimizing average cross-entropy is maximum likelihood estimation.

For logits passed through softmax, the gradient has a particularly useful form:

\[ \frac{\partial \ell_{\text{CE}}}{\partial s_k}=p_k-\mathbf{1}[k=y]. \]

The gradient is predicted probability minus target indicator.

Classification losses as functions of the signed margin Zero-one loss changes abruptly at zero, hinge loss decreases linearly to zero at margin one, and logistic loss decreases smoothly. margin ys loss 0 −2 2 0 1 2 3 logistic hinge 0–1
Worked example: hinge loss versus cross-entropy

Suppose the correct class has score \(s_y=2\), an incorrect class has score \(s_j=1.5\), and \(\Delta=1\). The hinge term is

\[ \max(0,1.5-2+1)=0.5. \]

The margin is not yet satisfied, so the loss pushes the correct score up or the incorrect score down. Cross-entropy would instead convert all scores to probabilities and penalize the probability assigned to the correct class.

Regularization

Regularization adds a parameter penalty:

\[ \mathcal{L}_{\text{total}}(\theta)=\mathcal{L}_{\text{data}}(\theta)+\lambda R(\theta). \]

Common choices are

\[ R_2(W)=\frac12\|W\|_F^2, \qquad R_1(W)=\|W\|_1. \]

The \(L_2\) penalty discourages large weights smoothly. The \(L_1\) penalty can produce exact zeros and therefore sparse parameters, but is non-differentiable at zero. Regularization changes the objective being optimized; it is not merely a post-processing step.

What this model cannot do

A single linear classifier can only create linear decision boundaries in the chosen feature space. If the classes are not linearly separable, changing the optimizer cannot make the model represent a nonlinear boundary. The model must receive better features or gain nonlinear layers.

This limitation motivates neural networks: compose affine transformations with nonlinear activation functions and learn the representation together with the classifier.