What Is Empirical Risk Minimization? Complete 2026 Guide
- 9 hours ago
- 24 min read
Every time a model is trained by "minimizing the loss," it relies on one load-bearing idea: that doing well on the data in front of you is a reasonable stand-in for doing well on data you haven't seen yet. That idea is called empirical risk minimization, and it sits underneath linear regression, logistic regression, support vector machines, and the neural networks behind modern AI — simple enough to state in one line, deep enough to require an entire branch of mathematics to say when it can be trusted.
TL;DR
ERM picks the hypothesis with the lowest average loss on a finite training sample, because the true population risk over the full data distribution can't be computed directly.
ERM is a learning principle, not one algorithm — linear regression, logistic regression, and gradient-boosted trees are all instances of it.
Low training loss doesn't guarantee low loss on new data; that gap is the generalization gap, and controlling it is the central problem of statistical learning theory.
VC dimension and Rademacher complexity explain when and why ERM generalizes, under specific assumptions.
Regularized ERM and structural risk minimization extend plain ERM by trading fit against complexity explicitly.
Even overparameterized deep networks that hit near-zero training loss are still running a form of approximate ERM.
What Is Empirical Risk Minimization?
Empirical risk minimization (ERM) is a machine learning principle that selects a predictor by minimizing its average loss on a finite training sample, used as a computable stand-in for the true expected loss over the full data distribution. ERM underlies most supervised learning algorithms, but low training loss alone doesn't guarantee good performance on new, unseen data.
Table of Contents
Empirical Risk Minimization in One Sentence
Empirical risk minimization is the rule that a learner should choose, from the predictors it's allowed to consider, the one that performs best on average on the training data it actually has — because that data is the only direct evidence available. Everything below refines that sentence: what "performs best" means precisely, why "on the training data" is a compromise, and what has to hold for the compromise to work.
ERM is not a specific piece of code; it's a specification. Linear regression fit by least squares, logistic regression fit by log-loss, a support vector machine fit by hinge loss, and a deep neural network fit by cross-entropy are all instances of the same abstract rule: pick the hypothesis that minimizes average loss on the sample. Shalev-Shwartz and Ben-David introduce ERM, alongside structural risk minimization and minimum description length, as one of the core rules that gives a first rigorous answer to "how can a machine learn?" [1]
The Problem ERM Is Trying to Solve
Supervised learning starts simply: there's an input space of possible examples, an output space of labels, and an unknown process generating input-output pairs. A learner receives a finite training data sample from that process and must produce a rule that works on future examples from the same process.
The difficulty hides in "future examples." A learner never sees the full population of possible inputs; it sees a finite sample, while the space of future cases is enormous or infinite. Stanford's CS229 notes frame supervised learning around exactly this: given a dataset, learn a function that predicts outputs for inputs not yet observed [2]. The whole difficulty of machine learning lives in the gap between what's been measured and what will be asked.
If a learner could directly measure a candidate's performance on every possible future input, it would just pick the best one. It can't — the data-generating distribution is never directly observable, only sampled. ERM responds to this: since true population performance can't be computed, approximate it using the only concrete evidence available, the finite sample, and select a predictor that does well there.
Loss, Population Risk, and Empirical Risk
To make "performs well" precise, statistical learning theory uses a loss function: a rule assigning a number to how wrong a single prediction is — near zero for accurate predictions, larger for worse ones. Loss, in this technical sense, has nothing to do with financial risk; it's simply a quantitative penalty for one wrong prediction.
In standard notation: let 𝒳 be the input space, 𝒴 the output space, and a data point Z = (X, Y) drawn from an unknown distribution P over 𝒳 × 𝒴. A hypothesis h maps inputs to predictions and comes from an allowed hypothesis class 𝓗. The loss of h on (x, y) is ℓ(h(x), y).
Population risk (expected or true risk) is h's average loss over the entire distribution:
R(h) = E_(X,Y)~P [ ℓ(h(X), Y) ]This is the number that actually matters, and it's essentially always impossible to compute directly, since P is unknown [1] [2].
Empirical risk is h's average loss on the finite sample S = {(x₁,y₁),…,(xₙ,yₙ)}, drawn i.i.d. from P:
R̂_n(h) = (1/n) · Σ_{i=1}^{n} ℓ(h(x_i), y_i)Unlike population risk, this is directly computable. In classification under zero-one loss, empirical risk reduces to ordinary training error [3]. With continuous losses like squared error, "empirical risk" and "training error" are related but not interchangeable, since "error" colloquially implies a binary right-or-wrong count.
Table 1. Empirical Risk vs. Population Risk
Property | Empirical Risk R̂_n(h) | Population Risk R(h) |
Defined over | The finite sample S | The entire, unknown distribution P |
Computable? | Yes, a direct average | No, P is unknown |
Depends on n? | Yes; itself a random quantity | No, a fixed property of h and P |
Used for | Selecting a hypothesis during training | What the learner actually wants low |
Reported as, post-training | Training error | Approximated only by test-set loss |
A related distinction: test-set loss is not population risk either. It's a second empirical average, on different data, estimating R(h) for an already-fixed hypothesis. It's more trustworthy than training loss because that data wasn't used to choose the hypothesis, but it's still an estimate with sampling variability, not R(h) itself.
The Formal Definition of ERM
The ERM rule: choose the hypothesis in 𝓗 minimizing empirical risk.
ĥ ∈ argmin_{h ∈ 𝓗} R̂_n(h)"argmin" denotes the set of minimizing hypotheses, written with "∈" rather than "=" because several hypotheses can tie. When that happens, ERM as stated doesn't specify which to return; a real algorithm resolves ties some other way — preferring simplicity, say, or whatever an optimizer converges to first. That tie-break is itself inductive bias [1].
In practice a learner minimizes over a parameterized model h_θ rather than an abstract function set:
θ̂ ∈ argmin_{θ ∈ Θ} (1/n) · Σ_{i=1}^{n} ℓ(h_θ(x_i), y_i)This matters because a parameterized class may only cover part of an abstract hypothesis space, and the same function can sometimes be represented by multiple parameter settings.
Real optimizers rarely find the exact minimizer — they stop early and return a hypothesis whose empirical risk is close to, not equal to, the true minimum. This is approximate ERM; the gap is treated formally later as optimization error.
A Step-by-Step Numerical Example
Consider a tiny binary classification problem: predicting pass (y=1) or fail (y=0) from hours studied (x).
Sample (n=5): (1,0), (2,0), (3,1), (4,1), (5,1) Hypothesis class: threshold classifiers h_θ(x) = 1 if x ≥ θ, restricted to θ ∈ {2, 3, 4} Loss: zero-one loss
x | y | h₂(x) | h₃(x) | h₄(x) |
1 | 0 | 0 ✓ | 0 ✓ | 0 ✓ |
2 | 0 | 1 ✗ | 0 ✓ | 0 ✓ |
3 | 1 | 1 ✓ | 1 ✓ | 0 ✗ |
4 | 1 | 1 ✓ | 1 ✓ | 1 ✓ |
5 | 1 | 1 ✓ | 1 ✓ | 1 ✓ |
R̂₅(h) | 0.20 | 0.00 | 0.20 |
ERM selects h₃, the unique hypothesis with lowest empirical risk. Notice how mechanical this is: once the sample is fixed, empirical risk is just counting and dividing, and finding the minimizer over a small class is comparing three numbers.
What the arithmetic does not tell you is how h₃ performs on a student who studied 2.5 hours, or a different cohort entirely. Zero empirical risk on five points is consistent with h₃ being an excellent general rule, or with it having gotten lucky on a small, unrepresentative sample — the calculation alone can't distinguish those. That's the subject of generalization, below.
ERM in Regression
In regression, y is a real number, and the standard loss is squared error: ℓ(h(x), y) = (h(x) − y)². Linear regression fit by ordinary least squares is the canonical ERM instance — "least squares" is precisely empirical risk under squared loss [2] [12].
Take the sample (1,2), (2,3), (3,5), comparing h_a(x) = x + 1 and h_b(x) = 2x:
x | y | h_a(x) | (h_a−y)² | h_b(x) | (h_b−y)² |
1 | 2 | 2 | 0 | 2 | 0 |
2 | 3 | 3 | 0 | 4 | 1 |
3 | 5 | 4 | 1 | 6 | 1 |
R̂₃(h) | 0.333 | 0.667 |
h_a has the lower empirical risk (0.333 vs. 0.667), so ERM selects it. Ordinary least squares solves this exact minimization analytically over all linear functions rather than comparing a short list, but the objective — minimize average squared residual — is identical [2].
Squared error penalizes large residuals disproportionately, making least-squares sensitive to outliers; other losses trade that sensitivity for different statistical properties. Choosing a loss is a modeling decision, encoding how costly different mistakes are.
ERM in Classification
Classification suggests the zero-one loss: a penalty of 1 for a misclassification, 0 otherwise. Under it, empirical risk is exactly the misclassification rate — classification error on the sample.
The trouble is computational. Zero-one loss is a step function — flat almost everywhere, discontinuous at the boundary — making it non-differentiable and, for most useful hypothesis classes, non-convex. Gradient-based optimizers generally can't minimize it directly, and exact minimization is computationally hard in the worst case [4].
The fix: minimize a different, tractable loss during training, and evaluate actual accuracy with zero-one loss afterward. That tractable stand-in is a surrogate loss.
Loss Functions and Surrogate Losses
A surrogate loss is substituted for the loss you actually care about, chosen because it's easier to optimize while still encouraging the right predictions. The three most common classification surrogates:
Logistic loss, ℓ = log(1 + e^(−y·h(x))), underlying logistic regression — a smooth, probabilistic penalty growing with confident wrong predictions [4].
Hinge loss, ℓ = max(0, 1 − y·h(x)), underlying the support vector machine — penalizing anything within, or on the wrong side of, a margin [2].
Exponential loss, ℓ = e^(−y·h(x)), underlying boosting — growing sharply for confidently wrong predictions.
All three are convex, (sub)differentiable upper bounds on zero-one loss, which is what makes them tractable [4]. This creates a real asymmetry: the loss a model trains on is often not what it's judged on. An AI fraud detection classifier might minimize logistic loss while being reported on using precision, recall, or business cost — quantities logistic loss only approximately tracks.
Table 2. Common Problems and Their Losses
Problem type | Typical loss | Empirical risk minimized |
Regression | Squared / absolute error | Mean squared / absolute error |
Binary classification (target) | Zero-one loss | Misclassification rate |
Binary classification (training) | Logistic / hinge loss | Average log-loss / hinge loss |
Multiclass classification | Cross-entropy | Average negative log-likelihood |
Probabilistic models | Negative log-likelihood | Average negative log-likelihood |
Ranking | Pairwise hinge/exponential | Average pairwise ranking loss |
ERM and Maximum Likelihood
ERM and maximum likelihood estimation are often treated separately, but under one common loss choice they coincide. If a model outputs a probability distribution over labels — as logistic regression does — and the loss is negative log-likelihood, minimizing empirical risk is, term for term, the same optimization as maximizing average log-likelihood. CS229 derives several standard algorithms from exactly this view [2].
The equivalence is a special case, not a universal law. ERM works for any loss over any hypothesis class; maximum likelihood applies specifically to probabilistic-output models under log-loss. Squared-error regression corresponds to maximum likelihood only under an assumed Gaussian noise model — it's an ordinary ERM problem that isn't automatically a likelihood problem.
Worth distinguishing from maximum a posteriori (MAP) estimation, which adds a prior over parameters and maximizes the posterior — equivalent to ERM with a penalty term derived from the negative log-prior, exactly the structure of regularized ERM below.
Hypothesis Classes and Inductive Bias
"The allowed class 𝓗" is doing real work. A learner can't minimize over all conceivable functions — that space is too large, and unrestricted, a rule that memorizes training labels and predicts arbitrarily elsewhere would hit zero empirical risk while being useless on new data. Choosing 𝓗 — linear functions, bounded-depth trees, a given network architecture — is itself a substantive assumption, called inductive bias: a built-in preference for some explanations over others, independent of what the sample alone could prove [1].
This creates a direct trade-off. A restrictive class may be unable to represent the true pattern at all, producing underfitting — high error on both training and future data. A highly expressive class (large neural networks) can fit almost any pattern, including noise, producing overfitting — low training error, poor performance elsewhere.
This decomposes into approximation error — the best hypothesis in 𝓗 still not matching the true optimum — and estimation error — only having a finite sample to find the best hypothesis within 𝓗 [1] [2]. There's no contradiction in fitting training data closely while failing on new data: that's exactly what happens when estimation error is large, the bias-variance tradeoff restated in learning-theoretic language.
Why Low Training Risk May Not Generalize
Define the generalization gap of h as R(h) − R̂_n(h). Small means training performance is trustworthy; large means it isn't.
Here's the subtlety: for any single, fixed hypothesis chosen independently of the data, the law of large numbers guarantees R̂_n(h) converges to R(h) as the sample grows, since it's an average of i.i.d. variables with mean R(h) [2] [3]. But ERM doesn't evaluate one fixed hypothesis — it searches an entire class and returns whichever looked best on this particular sample. That search introduces a selection effect: among many hypotheses, the one that looks best on a finite, noisy sample is more likely than average to have gotten lucky — the way the "best" stock picker among a thousand random guessers this year is more likely lucky than skilled. Evaluating a hypothesis chosen through data-dependent search is fundamentally more subtle than evaluating one fixed in advance — which is exactly why a properly held-out test set or validation set, never reused for tuning, supplies the evidence training loss can't.
Uniform Convergence and the Finite-Class Bound
The tool for formalizing this is uniform convergence: instead of asking whether R̂_n(h) converges to R(h) for one fixed h, ask whether it converges simultaneously, for every h in 𝓗 at once. If empirical risk is guaranteed close to population risk for the worst-case hypothesis in the class, it's automatically close for whichever one ERM selects [1] [2].
For a finite class, this follows from Hoeffding's inequality — bounding how far one empirical average can stray from its expectation [5] — plus a union bound over all |𝓗| hypotheses. Assuming loss bounded in [0,1] and an i.i.d. sample, with probability at least 1 − δ:
sup_{h ∈ 𝓗} | R(h) − R̂_n(h) | ≤ √( log(2|𝓗| / δ) / (2n) )Here n is sample size, |𝓗| the class size, δ the allowed failure probability. The bound shrinks as n grows and grows (slowly) with |𝓗|. Critically, class size enters only logarithmically — doubling |𝓗| barely moves the bound, while doubling n shrinks it meaningfully. This form is representative rather than universal — constants and formulations vary across textbooks, and such bounds can be loose in practice; they guarantee an upper limit, not the exact gap [2] [3].
This yields a guarantee about ERM's actual output. Write ĥ for ERM's result and h* = argmin_{h∈𝓗} R(h) for the (unobservable) best hypothesis by true risk. A short argument — ĥ's empirical risk is at most h*'s by definition of ERM, and both empirical risks are close to their population risks by the bound above — gives:
R(ĥ) − inf_{h ∈ 𝓗} R(h) ≤ 2 · sup_{h ∈ 𝓗} | R(h) − R̂_n(h) |ERM's excess risk — how much worse than the best available hypothesis — is bounded by twice the worst-case generalization gap over the class. If uniform convergence holds tightly, ERM stays close to the best hypothesis in 𝓗, even though R(h) was never observed for any hypothesis directly [1] [2].
VC Dimension, Rademacher Complexity, and PAC Learning
The bound above assumes a finite class, counted by |𝓗|. Most useful classes — all linear functions, all networks of a given architecture — are infinite, so a different capacity notion is needed.
VC dimension, named for Vapnik and Chervonenkis, measures a binary classifier class's capacity by the largest number of points it can "shatter" — realize every possible labeling of. A class with VC dimension d satisfies uniform convergence bounds structurally similar to the finite case, with d replacing log|𝓗| [1] [7]. That's why the finite-class intuition generalizes: what matters isn't the raw hypothesis count, but how much genuinely different behavior the class can express on a given sample. VC dimension suits binary-valued classes specifically; it doesn't directly extend to regression or multiclass without modification.
Rademacher complexity offers a related, sample-dependent measure: how well a class can fit random noise — independent random signs assigned to actual training inputs — rather than counting worst-case shattering patterns. Bartlett and Mendelson show Rademacher and Gaussian complexities yield general risk bounds for broad function classes, including networks and SVMs, often estimable directly from data [9]. Because it adapts to the specific sample and loss class, Rademacher complexity frequently yields tighter, more practical bounds than VC-dimension arguments, though both address the same question from different angles.
PAC learning — "probably approximately correct," introduced by Valiant [10] — ties these measures to sample-size requirements. A class is PAC-learnable if, for any accuracy ε and confidence 1−δ, some sample size lets an algorithm output, with probability ≥1−δ, a hypothesis within ε of the best achievable error. The realizable setting assumes some hypothesis achieves zero error; the agnostic setting doesn't, and asks only for closeness to the best hypothesis in 𝓗 — the setting matching ERM throughout this article. Under standard assumptions, ERM is a PAC-learning algorithm for any class with finite VC dimension: the sample-size requirements for PAC learnability and for ERM's uniform convergence coincide, up to constants [1] [7]. Finite-class counting, VC dimension, Rademacher complexity, and algorithmic stability (below) capture different structural properties of a problem; none is universally tightest.
Approximation, Estimation, and Optimization Error
Decompose the total gap into three sources, each governed by different levers.
Approximation error: the gap between the best predictor overall and the best available inside 𝓗. It reflects modeling choice, not data volume — reducible only by choosing a richer class.
Estimation error: the gap between the best predictor in 𝓗 and what ERM actually selects, caused by finite-sample noise. This is what the uniform-convergence bounds above control, and it shrinks with more data.
Optimization error: the gap between the exact ERM minimizer and what a real, finite-time optimizer returns. For approximate ERM h̃ satisfying R̂_n(h̃) ≤ inf_{h∈𝓗} R̂_n(h) + ε_opt, this slack propagates into the final guarantee.
R(h̃) − R* ≤ [R(h*_𝓗) − R*] + [2·sup_{h∈𝓗}|R(h) − R̂_n(h)|] + [ε_opt]
(approximation) (estimation) (optimization)Here h*_𝓗 is the best hypothesis achievable within 𝓗. This tells you where to intervene: large approximation error calls for a richer class or better feature extraction; large estimation error calls for more data or regularization; large optimization error calls for a better optimizer or more iterations. Exact terminology varies across sources, but this three-way split is a standard organizing lens [1] [2] [7].
Regularized ERM and Structural Risk Minimization
Plain ERM has a known failure mode already implicit above: given an expressive enough class, ERM drives empirical risk to zero by fitting noise. Regularized ERM adds an explicit complexity penalty to the objective:
θ̂ ∈ argmin_θ [ (1/n)·Σᵢ ℓ(h_θ(x_i), y_i) + λ·Ω(θ) ]The first term is ordinary empirical loss; Ω(θ) grows with parameter "complexity" — commonly the L1 norm (encouraging sparsity) or L2 norm (shrinking parameters smoothly). λ ≥ 0 controls the trade-off: λ=0 recovers plain ERM. This is why regularized ERM is related to, but not identical to, plain ERM — it's ERM on a modified objective, not just a modified hypothesis class. Regularization can also be implicit: early-stopped gradient descent on overparameterized models shows a built-in preference for simpler solutions with no explicit penalty [2].
Structural risk minimization (SRM), introduced by Vapnik, is related but distinct: define a nested sequence of hypothesis classes of increasing complexity, run ERM in each, then select among the candidates using a criterion balancing fit against the capacity of the class it came from [6]. Both trade fit against complexity, but SRM is specifically about model selection across a hierarchy of classes, while a regularization penalty modifies the objective within one class.
How ERM Objectives Are Optimized
Writing an ERM objective doesn't make it easy to solve. Ordinary least-squares regression is solvable analytically [2] [12], but most loss-class combinations aren't, and require iteration.
When empirical risk is convex in the parameters (true for squared-loss linear regression, logistic regression, linear SVMs), any local minimum found is also global, making gradient methods reliable. Gradient descent moves parameters in the direction that decreases empirical risk fastest, using the full sample each step; stochastic gradient descent (SGD) approximates that direction from one example or a mini-batch, trading noise for cheap iterations — essential once data no longer fits in memory [1] [2]. Mini-batch training, dominant in deep learning, sits between the extremes, and batch size and learning rate materially affect both speed and the final solution.
For non-convex objectives — the norm for deep networks — gradient methods reach only a stationary point, possibly a local minimum or saddle, not necessarily the global one. Many ERM problems are also worst-case hard to solve exactly, which is why approximate ERM is the practical target.
Input: training data S, model h_θ, loss ℓ
Initialize parameters θ
Repeat until a stopping criterion is met:
Estimate the gradient of average training loss w.r.t. θ
Update θ to decrease the estimated loss
Return the fitted predictor h_θThis sketches one strategy for approximately solving an ERM objective — not a definition of ERM, which is compatible with closed-form solutions, coordinate descent, and other methods too.
ERM in Modern Deep Learning
Regularized and plain ERM remain the objective underneath most modern machine learning systems. Linear and logistic regression minimize squared or log-loss directly. SVMs minimize hinge loss with an L2 penalty — textbook regularized ERM. Networks, including convolutional and transformer architectures behind large language models, minimize cross-entropy via SGD variants, and fine-tuning a pretrained model is mechanically another round of ERM, initialized from learned parameters rather than scratch — also the mechanism behind transfer learning.
What's changed is the relationship between training loss and generalization at extreme scale. Classical theory, built on VC-style capacity measures, predicts that a class expressive enough to hit zero empirical risk should overfit, since it can also memorize noise. Highly overparameterized networks routinely reach near-zero training loss and still generalize well — not a clean fit for the classical U-shaped curve. Belkin, Hsu, Ma, and Mandal document "double descent": as capacity increases past the point of exactly interpolating the data, test error can fall again, beyond the interpolation threshold, rather than keep rising [11]. This doesn't make classical capacity arguments wrong, just incomplete alone — optimization trajectory, architecture, implicit regularization, and early stopping all interact with capacity in ways pure worst-case bounds don't fully capture [2] [11]. Interpolation and "benign overfitting" remain active research, not settled science.
Assumptions, Limitations, and Failure Modes
ERM's guarantees are conditional. The standard analysis assumes i.i.d. training data matching the deployment distribution. Distribution shift breaks this directly; covariate shift (inputs change, input-output relationship doesn't) and label shift (label distribution changes independently) are common cases. Concept drift — the underlying relationship itself changing over time — is related but distinct, common in time-series data where i.i.d. is violated by construction.
Real data is rarely clean. Outliers, label noise, class imbalance, duplicates, and data leakage all degrade the link between low empirical risk and real performance, without necessarily inflating training loss at all. Adaptive data analysis — repeatedly reusing held-out data to guide decisions — silently erodes generalization guarantees, since the "held-out" set stops being independent of the choices being tested against it, which is precisely why disciplined validation practice and a single untouched final test matter.
Finally, minimizing average loss isn't minimizing worst-case or subgroup-specific loss. A model can have low aggregate risk while performing poorly for an underrepresented subgroup — squarely algorithmic bias territory — or remain vulnerable to adversarial inputs, since ERM optimizes an average. ERM faithfully minimizes whatever objective it's handed; a poorly chosen loss, class, or sampling process means the model optimizes the wrong target with mathematical precision — a design and governance problem, not a flaw in ERM itself. Model drift monitoring and model governance exist to catch that gap, formalized in frameworks for AI model risk management.
Table 3. Common ERM Failure Modes
Failure mode | Consequence | Mitigation |
Distribution shift | Low training loss, poor real-world performance | Monitoring, retraining, domain adaptation |
Label noise / leakage | Misleading training loss | Data auditing, pipeline checks |
Excessive class complexity | Overfitting | Regularization, more data, simpler class |
Adaptive validation-set reuse | Overly optimistic estimates | Strict train/validation/test separation |
Average-only optimization | Poor subgroup performance | Subgroup evaluation, robust objectives |
ERM Compared With Related Approaches
Table 4. ERM vs. Related Learning Principles
Principle | Core idea | Relation to plain ERM |
Empirical Risk Minimization | Minimize average loss on the sample | Baseline |
Regularized ERM | Add a complexity penalty | ERM plus a penalty term |
Structural Risk Minimization | Select across nested classes by capacity-aware criterion | Model selection built around repeated ERM |
Maximum Likelihood | Maximize likelihood of observed data | Equivalent to ERM under log-loss |
MAP Estimation | MLE plus a parameter prior | Equivalent to regularized ERM |
Bayesian Inference | Maintain a full posterior over hypotheses | Distinct paradigm, no single point estimate |
Distributionally Robust Optimization | Minimize worst-case loss over plausible distributions | Guards against shift plain ERM ignores |
Online / Regret Minimization | Minimize cumulative loss without i.i.d. assumptions | Different data-arrival model |
A Practical ERM Workflow
Define the prediction target precisely, including what counts as correct.
Understand the deployment distribution and whether it matches the training data's origin.
Select a hypothesis class sized to the problem and data volume.
Choose a loss aligned with real costs, distinct from the final evaluation metric.
Construct the objective, add regularization, and optimize it.
Monitor training against a validation set never touched during fitting.
Evaluate once on a genuinely held-out test set, check subgroup performance, and monitor drift after deployment.
Common Misconceptions
"ERM just means minimizing classification error."
ERM minimizes whatever loss it's given; zero-one error is one choice, often not even what's optimized during training.
"Empirical and true risk are the same."
They're linked by a probabilistic guarantee, not equality — empirical risk is a random estimate of a fixed, unknown quantity.
"Lowest training loss always means best model."
That identifies the ERM-optimal hypothesis within the sample, which can differ sharply from the best on new data.
"ERM automatically prevents overfitting."
Plain ERM has no built-in complexity penalty; preventing overfitting needs regularization, a restricted class, or more data.
"Regularization is unrelated to ERM."
It modifies the ERM objective directly by adding a penalty to the empirical loss.
"Gradient descent and ERM are synonyms."
ERM defines what is minimized; gradient descent is one family of algorithms for how — closed-form solutions solve the same objective.
"ERM only applies to linear models."
It applies to any hypothesis class, including decision trees, random forests, and deep networks.
"ERM is obsolete for overparameterized networks."
Those networks are still trained by minimizing empirical loss; ERM remains the objective even where classical theory needs supplementing.
"A generalization bound predicts exact test error."
It's an upper-confidence guarantee under assumptions, not a point prediction, and can be loose.
"Zero training error means useless memorization."
It's compatible with either excellent or poor generalization, depending on class, data, and optimization dynamics — not diagnostic on its own.
Conclusion
Empirical risk minimization earns its central place in supervised learning by converting an impossible problem — minimize error on data you've never seen — into a solvable one: minimize error on data you have. That substitution is powerful and, by construction, a compromise. The machinery in this article — population versus empirical risk, generalization gaps, uniform convergence, VC dimension and Rademacher complexity, PAC learning, regularization, and double descent — exists to answer one question honestly: under what conditions does doing well on the sample actually mean doing well in the world? Understanding ERM's mathematics is inseparable from understanding the real limits and strengths of every model built on top of it.
FAQ
What is empirical risk minimization in simple terms?
It's the rule of picking whichever available predictor makes the fewest or smallest mistakes on the training data you have, as a practical stand-in for one that would work on future data you can't measure directly.
What is the difference between empirical risk and population risk?
Empirical risk is the average loss on the specific training sample and is directly computable. Population risk is the expected loss over the entire, unknown data distribution and can't be computed directly; empirical risk estimates it.
Is empirical risk minimization the same as gradient descent?
No. ERM defines the objective — minimize average loss over a hypothesis class. Gradient descent and its variants are algorithms for solving that objective; some ERM problems, like ordinary least squares, don't need gradient descent at all.
Does ERM guarantee generalization to new data?
No, not alone. Generalization also depends on the hypothesis class's complexity, sample size, whether i.i.d. holds, and whether the deployment distribution matches training. ERM plus a well-controlled class, backed by uniform convergence, is what supports it.
What is the ERM formula?
ĥ ∈ argmin_{h∈𝓗} (1/n)·Σᵢ ℓ(h(xᵢ), yᵢ), where 𝓗 is the hypothesis class, ℓ the chosen loss, and the sum runs over n training examples.
How does ERM relate to maximum likelihood estimation?
When a model outputs probabilities and the loss is negative log-likelihood, minimizing empirical risk is mathematically identical to maximizing average likelihood of the data. ERM is the more general framework; maximum likelihood is one instance of it.
What is the difference between ERM and regularized ERM?
Plain ERM minimizes only average training loss. Regularized ERM adds an explicit penalty for hypothesis complexity to that same objective, trading fit against simplicity.
What is structural risk minimization, and how does it differ from ERM?
It selects a hypothesis by running ERM across nested classes of increasing complexity and choosing among results using a capacity-aware criterion, rather than fixing one class and running ERM once.
Why does zero training error not always mean overfitting?
Zero training error only says a hypothesis fits the sample perfectly; whether it generalizes depends on the class, data volume, and optimization process. Some very large models reach near-zero training error and still generalize well — "double descent."
What is the relationship between VC dimension and ERM?
VC dimension measures how much genuinely different behavior a class can express. It replaces simple class-size counting in bounds for infinite classes, and finite VC dimension ties closely to whether ERM is guaranteed to generalize under PAC learning.
How is cross-validation related to ERM?
It's a technique for estimating likely performance on new data and choosing among ERM configurations, like different regularization strengths. It's not part of ERM's definition, but a standard tool for choosing which ERM setup to trust.
What is a surrogate loss function, and why is it needed?
A substitute loss, like logistic or hinge loss, used during training instead of a harder-to-optimize target like zero-one error, because the surrogate is convex or differentiable and therefore tractable, while still approximating the true objective.
Does deep learning make classical ERM theory obsolete?
No. Deep networks are still trained by minimizing empirical loss, so ERM remains the objective. What classical capacity theory alone doesn't fully explain is why heavily overparameterized networks that fit training data almost exactly still generalize — active research, not a rejection of ERM.
What is the difference between approximation error and estimation error?
Approximation error is the gap between the best possible predictor and the best one available within the chosen class — it depends on model choice, not sample size. Estimation error is the added gap from having only a finite sample to find that best predictor; it shrinks with more data.
Key Takeaways
ERM converts an unmeasurable goal (minimize error on the true distribution) into a measurable one (minimize error on the sample); learning theory exists to characterize how good that substitution is.
Empirical and population risk are linked by probabilistic guarantees, not equality — treating training performance as real-world performance is the most common misreading of ERM.
The hypothesis class is an active choice with real consequences: it encodes inductive bias and trades approximation error against estimation error.
Surrogate losses like logistic and hinge loss exist because directly minimizing zero-one loss is computationally intractable for most useful classes.
ERM's generalization guarantees rest on uniform convergence, and VC dimension, Rademacher complexity, and PAC learning capture complementary aspects of a class's capacity.
Regularized ERM and structural risk minimization both address overfitting, but through different mechanisms — they aren't interchangeable with plain ERM or each other.
Modern deep learning still runs on ERM, but double descent shows classical capacity intuition alone doesn't fully explain overparameterized generalization.
ERM optimizes exactly what it's given; poor generalization is usually a symptom of a poorly chosen loss, class, or sampling process, not a flaw in the principle.
Actionable Next Steps
Work through this article's numerical examples by hand with a different toy dataset, computing empirical risk for several candidates to build direct intuition.
Before your next project, write down which loss you're training on and which metric you'll be judged on, and confirm whether they diverge.
Study the Hoeffding-plus-union-bound derivation in a primary source until you can reproduce the finite-class bound from scratch.
Audit a current model against Table 3's failure modes — distribution shift, label noise, adaptive reuse, subgroup gaps — rather than trusting aggregate training loss alone.
Track training loss against a genuinely held-out validation curve throughout training, not just at the end, to catch overfitting or drift early.
If you work with large, overparameterized models, read a primary source on double descent before treating near-zero training loss as automatically alarming.
Keep "why does this fit the training data" and "why should this work in production" as two separate questions requiring separate evidence.
Glossary
Approximation Error: The gap between the best possible predictor and the best one available within a chosen hypothesis class.
Bias-Variance Tradeoff: The trade-off between a model too simple to capture structure (high bias) and too sensitive to the training sample (high variance).
Cross-Validation: Repeatedly splitting data into training/validation subsets to estimate performance and compare configurations.
Empirical Risk: The average loss a hypothesis achieves on a specific, finite training sample.
Estimation Error: The gap between the best hypothesis in a class and the one actually selected, from finite-sample noise.
Excess Risk: The difference between a selected hypothesis's population risk and the best possible hypothesis's population risk.
Generalization Gap: The difference between a hypothesis's population risk and its empirical risk.
Gradient Descent: An iterative method updating parameters in the direction that most steeply decreases a loss function.
Hinge Loss: A loss, used in support vector machines, penalizing predictions inside or beyond a margin around the boundary.
Hypothesis Class: The set of candidate predictors a learning algorithm may choose from.
Inductive Bias: Built-in assumptions a learner brings beyond what training data alone determines.
Loss Function: A function quantifying how wrong a single prediction is.
Maximum Likelihood Estimation: Selecting parameters to maximize the probability the model assigns the observed data; equals ERM under negative log-likelihood loss.
Optimization Error: The gap between the empirical risk an optimizer achieves and the true minimum over the class.
Overfitting: Fitting the training data closely, including its noise, at the cost of new-data performance.
PAC Learning: A framework defining what it means for a class to be learnable within stated accuracy and confidence, using bounded data.
Population Risk: A hypothesis's expected loss over the entire, true data-generating distribution.
Rademacher Complexity: A sample-dependent measure of how well a class can fit random noise, used in generalization bounds.
Regularization: Adding a complexity penalty to the training objective to discourage overfitting.
Sample Complexity: The number of examples required to guarantee a target accuracy and confidence.
Stochastic Gradient Descent: Gradient descent that estimates the gradient from one example or a small batch per step.
Structural Risk Minimization: Model selection via ERM across nested classes of increasing complexity, using a capacity-aware criterion.
Surrogate Loss: A loss substituted for a harder-to-optimize target, chosen for tractability while still encouraging correct predictions.
Training Error: The fraction of misclassified training examples; empirical risk under zero-one loss.
Underfitting: A model too simple to capture real structure, producing high error on both training and new data.
Uniform Convergence: Empirical risk staying close to population risk simultaneously across an entire hypothesis class.
VC Dimension: A capacity measure for binary classifiers, equal to the largest point set the class can label every possible way.
Zero-One Loss: A loss assigning penalty 1 to a misclassified example, 0 to a correct one.
Sources & References
Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014. https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf
Andrew Ng and Tengyu Ma, CS229 Lecture Notes, Stanford University, 2023. https://cs229.stanford.edu/main_notes.pdf
Andrew Ng, CS229 Lecture Notes, Part VI: Learning Theory, Stanford University. https://cs229.stanford.edu/summer2019/cs229-notes4.pdf
John C. Duchi, CS229 Supplemental Lecture Notes: Loss Functions, Stanford University. https://cs229.stanford.edu/extra-notes/loss-functions.pdf
John C. Duchi, CS229 Supplemental Lecture Notes: Hoeffding's Inequality, Stanford University. https://cs229.stanford.edu/extra-notes/hoeffding.pdf
Vladimir N. Vapnik, "An Overview of Statistical Learning Theory," IEEE Transactions on Neural Networks, vol. 10, no. 5, pp. 988–999, 1999. https://doi.org/10.1109/72.788640
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of Machine Learning, 2nd ed., MIT Press, 2018. https://cs.nyu.edu/~mohri/mlbook/
Olivier Bousquet and André Elisseeff, "Stability and Generalization," Journal of Machine Learning Research, vol. 2, pp. 499–526, 2002. https://jmlr.org/papers/v2/bousquet02a.html
Peter L. Bartlett and Shahar Mendelson, "Rademacher and Gaussian Complexities: Risk Bounds and Structural Results," Journal of Machine Learning Research, vol. 3, pp. 463–482, 2002. https://jmlr.csail.mit.edu/papers/v3/bartlett02a.html
Leslie G. Valiant, "A Theory of the Learnable," Communications of the ACM, vol. 27, no. 11, pp. 1134–1142, 1984. https://dl.acm.org/doi/10.1145/1968.1972
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal, "Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-Off," Proceedings of the National Academy of Sciences, vol. 116, no. 32, pp. 15849–15854, 2019. https://www.pnas.org/doi/10.1073/pnas.1903070116
Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer, 2009. https://web.stanford.edu/~hastie/ElemStatLearn/