What Is Stochastic Gradient Descent (SGD)?
- 1 day ago
- 28 min read
Every time a neural network learns to recognize a face, translate a sentence, or predict a house price, something has to nudge millions of numbers, one small step at a time, toward values that make fewer mistakes — and for most of the history of modern machine learning, that nudging is done by stochastic gradient descent, an idea that started as a 1951 statistics paper and quietly became the engine room of the deep learning era.
TL;DR
Stochastic gradient descent (SGD) updates a model's parameters using the gradient estimated from one example or a small mini-batch, instead of the whole dataset at once.
SGD traces back to the Robbins–Monro stochastic approximation method from 1951, long before it became the default way to train neural networks [1].
Mini-batch SGD, not strict single-example SGD, is what most frameworks like PyTorch, TensorFlow, and scikit-learn actually run when you call "SGD" today [4][5][6].
The learning rate and batch size are the two hyperparameters that most affect training speed, stability, and final model quality.
Variants like momentum, Nesterov momentum, and Adam build on SGD's core idea to speed up convergence, but none of them guarantee the global minimum on non-convex problems [7][8].
What Is Stochastic Gradient Descent (SGD)?
Stochastic gradient descent (SGD) is an optimization algorithm that trains machine learning models by repeatedly updating their parameters in the direction that reduces prediction error, using the gradient calculated from a randomly selected training example or small batch rather than the entire dataset at once.
Table of Contents
Plain-English Definition
Training a machine learning model is really a search problem. The model has a set of parameters — numbers such as weights and biases — and the goal is to find the values of those parameters that make the model's predictions as close as possible to the correct answers. This search is called optimization, and the object being minimized is usually called a loss function or objective function: a single number that gets smaller as the model gets better.
Stochastic gradient descent is the algorithm most commonly used to do this search. At each step, SGD looks at how the loss would change if each parameter moved slightly in one direction or the other — that information is the gradient — and then moves every parameter a small amount in the direction that reduces the loss. The word "stochastic" refers to the fact that this gradient is not computed from the entire training set every time. Instead, it is estimated from a randomly chosen single example or a small random subset of examples, called a mini-batch. That randomness introduces noise into each individual update, but it also means each update is dramatically cheaper to compute, which is what allows SGD to train models on datasets with millions or billions of examples.
SGD matters because nearly every widely used neural network, from image classifiers to large language models, and many classic linear models, are fitted using some form of stochastic gradient method. Understanding SGD is close to a prerequisite for understanding how modern AI systems are actually trained [2][3].
Essential Foundations
Before going further, a few concepts need to be nailed down, because the rest of the article depends on them.
Model parameters and weights. A model's parameters (often called weights when referring to a neural network's connection strengths) are the internal numbers the model uses to turn inputs into predictions. Training means finding good values for these parameters.
Predictions. Given an input, such as an image or a row of tabular data, the model produces an output: a predicted class, a predicted number, or a probability distribution.
Loss functions and objective functions. A loss function scores how wrong a single prediction was compared to the true answer. Averaging (or summing) the loss over a set of examples gives the objective function that training tries to minimize. Common choices include mean squared error for regression and cross-entropy for classification.
Gradients and partial derivatives. For a function with many inputs (here, the parameters), the gradient is a vector containing the partial derivative of the loss with respect to each parameter. Each entry tells you, if you nudge that one parameter slightly while holding everything else fixed, whether the loss goes up or down, and by roughly how much.
The negative-gradient direction. The gradient points in the direction of steepest increase of the loss. Moving in the opposite direction — the negative gradient — is, locally, the direction that decreases the loss fastest. This is the entire mechanical idea behind gradient-based optimization.
Learning rates. The learning rate is a small positive number that controls how big a step is taken in the negative-gradient direction at each update. Framework defaults (such as 0.01 in several libraries) are implementation choices, not universally correct values.
Training examples and datasets. A training set is a collection of input–output pairs used to fit the model. A validation set and a test set are held out to check how well the model performs on data it did not train on.
Empirical risk minimization. In practice, you cannot compute the true expected loss over all possible data; you only have a finite sample. Empirical risk minimization is the strategy of minimizing the average loss over the training sample as a stand-in for minimizing the true, unknown risk.
The loss surface. If you imagine plotting the loss as a function of the model's parameters, you get a high-dimensional surface with hills, valleys, plateaus, and saddle points. Training is the process of walking across this surface toward a low point. For a simple two-parameter linear model, this surface can literally be drawn as a bowl-shaped or valley-shaped 3D plot; for a deep network with millions of parameters, it exists only mathematically, but the same intuition — walking downhill — still applies.
From Gradient Descent to SGD
Full-Batch Gradient Descent
Full-batch gradient descent computes the gradient of the loss using every example in the training set before making a single parameter update. If the loss for example $i$ is $L_i(\theta)$, where $\theta$ represents all model parameters, the full-batch objective is:
$$J(\theta) = \frac{1}{N}\sum_{i=1}^{N} L_i(\theta)$$
Here, $N$ is the number of training examples and $J(\theta)$ is the average loss. The update rule is:
$$\theta_{t+1} = \theta_t - \eta , \nabla J(\theta_t)$$
$\theta_t$ is the parameter vector at step $t$, $\eta$ (eta) is the learning rate, and $\nabla J(\theta_t)$ is the gradient of the full-batch loss. In plain English: compute the average error direction across the entire dataset, then take one careful step.
Single-Example Stochastic Gradient Descent
Strict single-example SGD instead picks one training example $i$ at random on each iteration and updates using only that example's gradient:
$$\theta_{t+1} = \theta_t - \eta , \nabla L_i(\theta_t)$$
This is dramatically cheaper per step — you touch one data point instead of the whole dataset — but each individual gradient $\nabla L_i(\theta_t)$ is a noisy estimate of the true gradient $\nabla J(\theta_t)$. Under standard assumptions (examples drawn independently and identically from the data distribution), this single-example gradient is an unbiased estimator of the full gradient, meaning its average over many random draws equals the true gradient, even though any one draw can be far off.
Mini-Batch Stochastic Gradient Descent
Mini-batch SGD samples a small batch $B$ of $m$ examples (commonly somewhere between 8 and a few thousand, depending on the task and hardware) and averages their gradients:
$$\theta_{t+1} = \theta_t - \eta , \frac{1}{m}\sum_{i \in B} \nabla L_i(\theta_t)$$
In modern deep-learning practice, when people say "we trained with SGD," they almost always mean mini-batch SGD, not strict single-example SGD. The terminology is used loosely, but the mechanics differ: batch size changes both the variance of the gradient estimate and the computational cost per update, and the two informal usages of "SGD" are not identical algorithms [2][4].
Comparison Table: Batch, Single-Example, and Mini-Batch Approaches
Property | Full-Batch GD | Single-Example SGD | Mini-Batch SGD |
Data used per update | Entire dataset | One example | Small subset (e.g., 32–1,024) |
Computational cost per update | High | Very low | Low to moderate |
Update frequency per epoch | 1 | N (dataset size) | N / batch size |
Gradient variance | None (exact) | High | Moderate, decreases as batch grows |
Memory requirements | High (whole dataset in memory) | Very low | Low to moderate |
Parallelism / hardware use | Good for large batches on accelerators | Poor (hard to vectorize) | Good; matches GPU/TPU throughput |
Stability of the loss curve | Smooth | Noisy, oscillating | Smoother than single-example, noisier than full-batch |
Typical use cases | Small datasets, convex problems, research analysis | Streaming/online learning, extremely constrained memory | Standard deep learning and large-scale linear model training |
How SGD Works Step by Step
A full SGD training run follows this cycle:
Initialize parameters, typically with small random values chosen according to an initialization scheme suited to the network's architecture.
Shuffle the training data so that each pass over the data is not biased by the original ordering.
Select an example or mini-batch from the shuffled data.
Perform the forward pass: feed the inputs through the model to generate predictions.
Calculate the loss by comparing predictions to the true targets.
Compute gradients of the loss with respect to every parameter, usually via backpropagation in neural networks.
Update parameters by moving them a small step in the negative-gradient direction, scaled by the learning rate.
Repeat steps 3–7 across all mini-batches; one full pass through the training data is called an epoch, and training typically runs for many epochs.
Evaluate on a validation set periodically (for example, at the end of each epoch) to track how well the model generalizes to unseen data.
Stop according to a chosen criterion: a fixed number of epochs, a validation metric that stops improving (early stopping), or a computational budget.
initialize parameters theta
for epoch in 1..max_epochs:
shuffle(training_data)
for batch in split_into_batches(training_data, batch_size):
predictions = forward_pass(batch.inputs, theta)
loss = compute_loss(predictions, batch.targets)
gradients = backward_pass(loss, theta)
theta = theta - learning_rate * gradients
val_score = evaluate(theta, validation_data)
if stopping_criterion_met(val_score):
break
return thetaEach pass through the inner loop is one iteration and one optimizer step; a full pass through the outer loop is one epoch. These terms are frequently confused, but they describe different units of training progress.
Worked Numerical Example
To make this concrete, consider single-parameter linear regression through the origin: predict $\hat{y} = w x$, where $w$ is the one parameter being learned (no intercept, to keep the arithmetic simple).
Data (3 points):
$x$ | $y$ |
1 | 2 |
2 | 3 |
3 | 6 |
Model: $\hat{y}_i = w x_i$
Loss (squared error) for one example: $L_i(w) = \tfrac{1}{2}(w x_i - y_i)^2$
Gradient of the loss with respect to $w$: $\dfrac{dL_i}{dw} = (w x_i - y_i), x_i$
Start with $w_0 = 1$ and a learning rate $\eta = 0.1$.
Update 1 — using example $(x=1, y=2)$:
Prediction: $\hat{y} = 1 \times 1 = 1$
Gradient: $(1 - 2) \times 1 = -1$
Update: $w_1 = 1 - 0.1 \times (-1) = 1 + 0.1 = 1.1$
Update 2 — using example $(x=2, y=3)$:
Prediction: $\hat{y} = 1.1 \times 2 = 2.2$
Gradient: $(2.2 - 3) \times 2 = (-0.8) \times 2 = -1.6$
Update: $w_2 = 1.1 - 0.1 \times (-1.6) = 1.1 + 0.16 = 1.26$
Notice the two updates moved $w$ by different amounts (+0.1, then +0.16) because each was computed from a different training example with a different gradient. This is the practical meaning of "stochastic": the direction and size of each step depend on which data point happened to be sampled. Before training, the prediction for $x=1$ was off by 1 unit; after two updates, $w=1.26$ predicts $\hat{y}=1.26$ for $x=1$ (true value 2) and $\hat{y}=2.52$ for $x=2$ (true value 3) — closer on the second point, illustrating that individual updates trade off performance across different examples rather than improving every point uniformly on every step. Running this same process, in shuffled order, across many epochs is what full-batch or mini-batch training generalizes into for models with many parameters.
Why SGD Is Useful
SGD's biggest practical advantage is scalability. Because each update depends only on one example or a small batch, SGD can train on datasets that would never fit in memory all at once, and it can start improving the model after seeing just a fraction of the data rather than waiting to process everything first [2]. This makes it a natural fit for streaming and online-learning settings, where data arrives continuously and the model needs to update incrementally rather than retrain from scratch.
Mini-batch SGD also maps efficiently onto modern hardware: GPUs and TPUs are built to process many examples in parallel, so a well-chosen batch size can keep the hardware busy while still allowing frequent parameter updates. On large and sparse datasets, such as text or click-through data with millions of rare features, SGD's ability to update from a handful of examples at a time, rather than waiting for a full pass, is often the difference between a training run that finishes in hours and one that never finishes.
The noise introduced by sampling is not purely a downside. In non-convex problems, some researchers have argued that gradient noise may help models avoid or escape certain shallow local minima and can interact with the loss landscape in ways that affect which regions of parameter space training tends to settle in — but this is an active area of research, not a settled guarantee, and noise does not reliably escape all local minima or saddle points [2][3].
Balanced against these benefits are real costs: SGD requires more careful tuning of the learning rate than full-batch methods, produces noisier training curves, and its convergence behavior is harder to reason about precisely than deterministic full-batch descent.
Limitations and Failure Modes
Issue | Practical Remedy |
Noisy or oscillating training/validation curves | Reduce learning rate, increase batch size, or apply a moving average to the loss for monitoring |
High sensitivity to the learning rate | Run a small learning-rate sweep or use a warmup-plus-decay schedule |
Slow convergence near the optimum | Apply a decaying learning rate or switch to momentum-based methods |
Poor conditioning of the loss surface | Normalize/standardize input features; consider adaptive optimizers |
Plateaus and saddle regions in non-convex loss surfaces | Use momentum to maintain velocity through flat regions |
Exploding gradients | Apply gradient clipping; check for excessively high learning rates |
Vanishing gradients | Use appropriate activation functions and initialization; consider architectural changes (e.g., residual connections) |
Unscaled features | Standardize or normalize inputs before training |
Bad initialization | Use initialization schemes suited to the architecture (e.g., Xavier/He initialization) |
Data-order effects | Shuffle data every epoch |
Non-representative mini-batches | Use stratified sampling for imbalanced data; increase batch size |
Use class weighting, resampling, or stratified batches | |
Batch-size extremes (too small or too large) | Tune batch size alongside learning rate; monitor gradient variance |
Overfitting and underfitting | Apply regularization, early stopping, or adjust model capacity |
Training instability | Lower the learning rate, add gradient clipping, or check for data leakage |
Premature stopping | Monitor validation performance rather than training loss alone before halting |
Learning Rate and Schedules
Of all SGD's hyperparameters, the learning rate typically has the largest effect on training outcomes. A learning rate that is too high can cause the loss to diverge or oscillate wildly; one that is too low can make training impractically slow or cause it to stall on a plateau long before reaching a good solution [2][3].
Constant learning rate. The simplest choice: $\eta$ stays fixed for the entire run. Easy to reason about, but often suboptimal, since a rate that is good early in training (when large steps are useful) may be too large later (when fine adjustments are needed).
Step decay. The learning rate is multiplied by a fixed factor (e.g., 0.1) every fixed number of epochs.
Exponential decay. The learning rate decreases continuously according to $\eta_t = \eta_0 e^{-kt}$, for some decay constant $k$.
Inverse-time decay. The learning rate follows $\eta_t = \eta_0 / (1 + kt)$, echoing the diminishing-step-size condition from the original stochastic approximation theory, where step sizes shrink over time to guarantee convergence under certain assumptions [1].
Cosine decay. The learning rate follows a cosine curve down to near zero over training, popular in modern deep learning because it tends to produce smooth, well-behaved final convergence.
Warmup. Training starts with a small learning rate that ramps up over the first few epochs before switching to the main schedule. This has been shown to stabilize training in large-batch settings, where a full-size learning rate applied immediately can cause early instability [13].
One-cycle-style schedules. The learning rate rises during an initial phase and then falls, often paired with an inverse pattern for momentum, aiming to combine fast early progress with careful late-stage convergence.
Reduce-on-plateau. The learning rate is cut only when a monitored validation metric stops improving for a set number of epochs, adapting the schedule to the specific training run rather than following a fixed clock.
Theoretical diminishing-step intuition. Classical stochastic approximation theory shows that, under conditions such as $\sum \eta_t = \infty$ and $\sum \eta_t^2 < \infty$, iterative stochastic updates can converge to a solution in probability, which is part of the theoretical motivation for decaying learning rates over time [1][9].
No single schedule is universally best; the right choice depends on the model, dataset size, and available compute budget, and batch size and learning rate interact — larger batches often tolerate or require different learning rates than smaller ones, a relationship discussed further below [13].
Batch Size
Batch size controls how many examples contribute to each gradient estimate, and it interacts with nearly every other part of training.
At one extreme, single-example updates produce the highest gradient variance and the noisiest steps. Large mini-batches average over more examples, producing a smoother, lower-variance gradient estimate, but each update costs more compute and touches the data less frequently per epoch. Full-batch gradient descent has zero sampling variance but is often computationally impractical for large datasets and can behave differently in terms of generalization than smaller batches.
Larger batches generally improve throughput on parallel hardware (more examples processed per second) up to a point, but this comes with increased memory demand and diminishing marginal benefit once hardware is saturated. Claims that larger batches always train faster, or that smaller batches always generalize better, are both oversimplifications; empirical results depend heavily on the model, dataset, and how the learning rate is adjusted alongside batch size [2][13].
One influential empirical heuristic is the linear scaling rule: when batch size is multiplied by some factor $k$, multiply the learning rate by approximately the same factor $k$, combined with a warmup period, as demonstrated in large-scale image classification training [13]. This is a practical heuristic validated in specific settings, not a universal law that applies unconditionally to every architecture and dataset.
Gradient accumulation lets you simulate a larger batch size than fits in memory by summing gradients across several smaller "micro-batches" before performing a single parameter update — useful when a model is too large to fit a full batch on the available hardware.
Distributed training splits a large batch across multiple devices, computing partial gradients on each device and then aggregating them (commonly via an all-reduce operation) before the shared update is applied — an approach directly connected to the large-minibatch scaling research discussed above [13].
Momentum and Variants
Vanilla SGD, as already described, uses only the current gradient: $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$.
SGD with momentum adds a velocity term $v_t$ that accumulates a running average of past gradients:
$$v_{t+1} = \mu v_t + \nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta , v_{t+1}$$
Here $\mu$ (mu) is the momentum coefficient (commonly around 0.9). Momentum helps carry the optimizer through flat regions and small bumps in the loss surface, in the same way a heavy ball rolling downhill keeps moving through small dips because of its accumulated velocity — though unlike a physical ball, momentum in SGD has no true "mass" or energy conservation, and this analogy stops being literal at that point.
Nesterov momentum evaluates the gradient at a position that accounts for the momentum step already taken, effectively looking ahead before computing the correction, which can improve convergence behavior over standard momentum in some settings [7].
Averaged SGD (Polyak–Ruppert averaging) keeps a running average of the parameter iterates themselves, $\bar\theta_T = \frac{1}{T}\sum_{t=1}^{T}\theta_t$, rather than only using the final iterate. Under certain statistical assumptions, this averaged estimate can achieve favorable asymptotic convergence properties compared with using the last iterate alone [9].
Weight decay with SGD adds a term that shrinks parameters toward zero at each step, commonly implemented as $\theta_{t+1} = \theta_t - \eta(\nabla L(\theta_t) + \lambda \theta_t)$, where $\lambda$ is the weight decay coefficient. For plain SGD, this form of weight decay is mathematically equivalent to adding an L2 penalty to the loss; that equivalence does not hold in the same simple form once adaptive optimizers like Adam are involved, which motivated decoupled weight decay approaches discussed later [12].
Variance-reduced methods such as SAG, SAGA, and SVRG modify the stochastic gradient estimate itself — for example, by periodically recomputing a full-batch gradient and correcting the stochastic estimate against it — to reduce gradient variance and, under certain conditions, achieve faster convergence rates than plain SGD on specific problem classes, though they are less commonly used in large-scale deep learning than in classical convex optimization.
Comparison Table: SGD Variants
Variant | What It Changes | Main Benefit | Main Trade-off |
Vanilla SGD | Nothing extra; uses raw gradient | Simple, low memory | Can be slow through flat or oscillating regions |
SGD with momentum | Adds accumulated velocity term | Smoother, often faster progress | One extra hyperparameter ($\mu$) and one extra state buffer |
Nesterov momentum | Looks ahead before computing gradient | Can improve convergence over standard momentum | Slightly more complex update; sensitive to tuning |
Averaged SGD (Polyak–Ruppert) | Averages parameter iterates over time | Favorable theoretical convergence in some regimes | Averaging window and start point need tuning |
SGD with weight decay | Shrinks parameters each step | Helps control overfitting | Must be tuned jointly with learning rate |
Variance-reduced (SAG/SAGA/SVRG) | Corrects the stochastic gradient estimate | Reduced gradient variance on suitable problems | Extra memory/computation; less common in large-scale deep learning |
SGD vs Other Optimizers
Adaptive optimizers such as AdaGrad, RMSProp, and Adam maintain per-parameter statistics of past gradients and use them to scale each parameter's effective learning rate individually, rather than applying one global rate to every parameter as plain SGD does [8][10][11].
AdaGrad accumulates the sum of squared past gradients for each parameter and divides the learning rate by the square root of that sum, which gives infrequently updated parameters (common with sparse features) relatively larger effective steps [10].
RMSProp modifies this idea by using a decaying (exponentially weighted) moving average of squared gradients instead of an ever-growing sum, addressing AdaGrad's tendency for the effective learning rate to shrink too aggressively over long training runs [11].
Adam combines a momentum-like moving average of the gradient itself with an RMSProp-like moving average of squared gradients, then uses both to compute an adaptive, bias-corrected update for each parameter [8].
AdamW decouples weight decay from the adaptive gradient update, addressing the fact that combining L2 regularization with Adam's per-parameter scaling does not behave the same way it does with plain SGD [12].
Comparison Table: Optimizer Landscape
Optimizer | Per-Parameter Adaptive Rates | Extra Optimizer State | Memory Overhead | Tuning Sensitivity | Sparse-Gradient Behavior | Common Use Cases | Potential Strengths | Potential Weaknesses |
Full-batch GD | No | None | Low | Moderate | Poor for very sparse data | Small/medium convex problems, research analysis | Exact gradient, simple theory | Impractical at large scale |
Mini-batch SGD | No | None (momentum optional) | Low | High (learning rate critical) | Moderate | Linear models, large-scale deep learning | Simple, well understood, often strong final generalization | Requires careful tuning, slower early progress on some problems |
SGD + momentum | No | One velocity buffer | Low–moderate | Moderate–high | Moderate | Computer vision, many production deep-learning pipelines | Faster progress through flat regions | Extra hyperparameter, still learning-rate sensitive |
AdaGrad | Yes | Per-parameter accumulator | Moderate | Lower for sparse problems | Strong (favors rare features) | Sparse, high-dimensional problems (e.g., text) | Good early behavior on sparse data | Learning rate can shrink too much over long runs |
RMSProp | Yes | Per-parameter moving average | Moderate | Lower than AdaGrad | Strong | RNNs and other non-stationary objectives | Handles non-stationary objectives well | Extra hyperparameters; less theoretical grounding than Adam |
Adam | Yes | Two per-parameter moving averages | Moderate–high | Generally lower | Strong | Wide range of deep learning tasks, fast prototyping | Often fast, robust default across many tasks | Can generalize worse than well-tuned SGD in some settings; not universally superior |
AdamW | Yes | Two per-parameter moving averages | Moderate–high | Generally lower | Strong | Transformer training, modern large models | Cleaner weight-decay behavior than Adam | Same core sensitivities as Adam otherwise |
No optimizer in this table is a universal winner; the right choice depends on data sparsity, model architecture, available compute, and how much tuning time is available.
Practical Implementation
The following examples show SGD in four contexts: a from-scratch NumPy version, PyTorch's built-in optimizer, TensorFlow/Keras's built-in optimizer, and scikit-learn's linear estimator. The NumPy and PyTorch/Keras examples illustrate optimizer-level SGD as used to train models such as neural networks; the scikit-learn example is a distinct linear estimator that is fit via SGD rather than a general-purpose neural-network optimizer.
From-Scratch NumPy Implementation
import numpy as np
rng = np.random.default_rng(seed=42)
X = rng.normal(size=(200, 1))
true_w, true_b = 3.0, -1.0
y = true_w * X[:, 0] + true_b + rng.normal(scale=0.1, size=200)
w, b = 0.0, 0.0
lr = 0.05
epochs = 20
batch_size = 16
for epoch in range(epochs):
perm = rng.permutation(len(X))
X_shuffled, y_shuffled = X[perm], y[perm]
for start in range(0, len(X), batch_size):
xb = X_shuffled[start:start + batch_size, 0]
yb = y_shuffled[start:start + batch_size]
preds = w * xb + b
error = preds - yb
grad_w = np.mean(error * xb)
grad_b = np.mean(error)
w -= lr * grad_w
b -= lr * grad_b
print(f"Learned w={w:.3f}, b={b:.3f}")This script generates a noisy linear dataset with a known true slope and intercept, then runs mini-batch SGD by hand: each inner loop iteration samples one mini-batch, computes the mean-squared-error gradient with respect to w and b, and updates both parameters. The random seed makes results reproducible.
PyTorch: torch.optim.SGD
import torch
import torch.nn as nn
torch.manual_seed(42)
model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
loss_fn = nn.MSELoss()
X = torch.randn(200, 1)
y = 3.0 * X + (-1.0) + 0.1 * torch.randn(200, 1)
for epoch in range(20):
optimizer.zero_grad()
predictions = model(X)
loss = loss_fn(predictions, y)
loss.backward()
optimizer.step()
print(f"Final loss: {loss.item():.4f}")torch.optim.SGD is constructed with the model's parameters, a learning rate, and an optional momentum term; optimizer.zero_grad() clears old gradients before each step, loss.backward() computes new gradients via automatic differentiation, and optimizer.step() applies the update rule described earlier [4]. PyTorch's momentum implementation differs slightly in scaling convention from the original Sutskever et al. formulation, so mixing hyperparameters copied from other frameworks without checking the current documentation can produce different behavior [4][7].
TensorFlow/Keras: keras.optimizers.SGD
import tensorflow as tf
tf.random.set_seed(42)
model = tf.keras.Sequential([tf.keras.layers.Dense(1, input_shape=(1,))])
optimizer = tf.keras.optimizers.SGD(learning_rate=0.05, momentum=0.9, nesterov=True)
model.compile(optimizer=optimizer, loss="mse")
X = tf.random.normal((200, 1))
y = 3.0 * X + (-1.0) + 0.1 * tf.random.normal((200, 1))
model.fit(X, y, epochs=20, batch_size=16, verbose=0)
print("Training complete")Here, keras.optimizers.SGD accepts learning_rate, momentum, and a nesterov flag directly; model.compile wires the optimizer to the mean-squared-error loss, and model.fit runs the full mini-batch training loop internally, handling shuffling and batching for you [5].
scikit-learn: SGDRegressor
import numpy as np
from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
X = rng.normal(size=(200, 1))
y = 3.0 * X[:, 0] - 1.0 + rng.normal(scale=0.1, size=200)
X_scaled = StandardScaler().fit_transform(X)
model = SGDRegressor(max_iter=1000, learning_rate="invscaling", eta0=0.05, random_state=42)
model.fit(X_scaled, y)
print(f"Coefficient: {model.coef_[0]:.3f}, Intercept: {model.intercept_[0]:.3f}")SGDRegressor is a linear model fit via stochastic gradient descent rather than a generic neural-network optimizer; it is distinct from torch.optim.SGD or keras.optimizers.SGD, which optimize arbitrary differentiable models. StandardScaler is applied first because scikit-learn's SGD-based estimators are noted as sensitive to feature scaling [6].
Practical Tuning Workflow
Normalize or standardize input features so that no single feature dominates the gradient due to scale differences.
Build a simple baseline (e.g., a small model with default hyperparameters) before optimizing further.
Select an initial learning rate using a short range test or established defaults for the framework and task.
Select a batch size based on available memory and hardware parallelism, then revisit the learning rate if the batch size changes substantially.
Choose a momentum value (commonly in the 0.8–0.99 range for many neural-network tasks) if using SGD with momentum.
Apply a learning-rate schedule appropriate to the training budget (e.g., step decay, cosine decay, or warmup for large batches).
Add weight decay or other regularization if overfitting is observed on the validation set.
Apply gradient clipping if gradients are unstable or exploding, particularly in recurrent architectures.
Monitor training and validation curves together, not training loss alone, to detect overfitting or instability early.
Use early stopping based on validation performance rather than a fixed epoch count when practical.
Fix random seeds for reproducibility during development and debugging.
Run a small hyperparameter search (grid, random, or Bayesian) over the most influential settings — usually learning rate first, then batch size and momentum.
Log and checkpoint training runs so experiments can be compared and recovered.
Diagnostic Table
Symptom | Likely Cause | What to Inspect | Possible Fix |
Loss explodes to NaN | Learning rate too high, exploding gradients | Gradient magnitudes, learning rate | Lower learning rate; add gradient clipping |
Loss barely moves | Learning rate too low, poor initialization | Learning rate, initialization scheme | Raise learning rate; check initialization |
Training loss low, validation loss high | Overfitting | Validation vs. training curves | Add regularization, more data, or early stopping |
Very noisy training curve | Batch size too small, learning rate too high | Batch size, gradient variance | Increase batch size or reduce learning rate |
Progress stalls mid-training | Plateau or poor conditioning | Learning-rate schedule, feature scaling | Add momentum or a decay schedule; standardize features |
Good results only with specific seed | Fragile initialization or unstable optimization | Multiple seeds, variance across runs | Average results over several seeds; stabilize hyperparameters |
When to Use SGD
Plain SGD or SGD with momentum can be a strong choice for large linear models on sparse, high-dimensional data such as text classification, where scikit-learn's SGDClassifier and SGDRegressor are specifically designed for this regime [6]. It is also well suited to online-learning settings where data arrives continuously and the model needs incremental updates rather than full retraining.
In computer vision, SGD with momentum has a long track record of training convolutional and residual networks to strong final accuracy, and some practitioners favor it when predictable optimizer behavior and strong final generalization matter more than fastest possible early-epoch progress [2][13]. For general neural network training where rapid prototyping and robustness across architectures matter more, an adaptive optimizer such as Adam or AdamW is often more convenient, particularly for transformer-style models, though neither choice is universally superior across all tasks.
Common Misconceptions
"SGD always uses exactly one sample." In modern usage, "SGD" usually refers to mini-batch SGD, which uses small groups of examples per update, not strictly one [2][4].
"Stochastic means the optimizer moves randomly." The optimizer still moves in a gradient-informed direction; only the specific data used to estimate that gradient at each step is randomly sampled.
"SGD always finds the global minimum." On non-convex loss surfaces, such as those of deep neural networks, SGD is only guaranteed to converge to a stationary point under certain conditions, not the global minimum [2][3].
"More noise is always better." Gradient noise can sometimes help avoid certain shallow local minima, but excessive noise (from a learning rate that is too high or a batch size that is too small) commonly destabilizes training rather than helping it.
"A larger batch is always faster." Larger batches increase per-step compute and memory use; total wall-clock speedup depends on hardware utilization and whether the learning rate is adjusted accordingly [13].
"A smaller batch always generalizes better." Empirical results on this question vary by architecture, dataset, and how carefully the learning rate is tuned; it is an active research question, not a fixed rule [2].
"Momentum and learning rate do the same thing." The learning rate scales the raw gradient step; momentum accumulates a running average of past gradients. They interact but control different aspects of the update.
"Adam always beats SGD." Adam often converges faster in early training, but well-tuned SGD with momentum has, in various studies and practical settings, matched or exceeded Adam's final generalization performance on some tasks; neither is a universal winner [8].
"One epoch means one optimizer update." One epoch is one full pass through the training data, which typically corresponds to many mini-batch updates, not just one.
"A decreasing training loss proves the model generalizes." Training loss reflects fit to the training data only; generalization must be checked on a separate validation set or test set.
Advanced Perspective
A single stochastic gradient is best thought of as a random estimator of the true gradient: it has some expected value (ideally the true gradient, under standard sampling assumptions) and some variance around that expected value. The behavior of SGD over many steps can be analyzed in terms of expected descent — how much the loss is expected to decrease, on average, given the current gradient estimate's bias and variance [1][2].
Convergence guarantees differ sharply between convex and non-convex objectives. For convex or strongly convex problems, classical stochastic approximation theory provides conditions under which iterates converge in probability to the optimum, given appropriately shrinking step sizes [1][9]. For the non-convex objectives typical of deep neural networks, the standard results are weaker: under smoothness assumptions, SGD can be shown to converge to a stationary point — a point where the gradient is near zero, which could be a local minimum, saddle point, or other flat region — rather than provably reaching the global minimum [2][3].
It is also important to separate optimization from generalization. Minimizing training loss is an optimization problem; achieving good performance on new, unseen data is a generalization problem. A model can drive training loss very low while still generalizing poorly, which is why validation and test sets remain essential even when the training loss curve looks excellent [3].
Finally, modern distributed and large-batch training changes what a "batch" practically means: a single logical batch may be split across many accelerators, gradients aggregated across devices, and the effective batch size scaled up by orders of magnitude compared to single-machine training — all while relying on the same core mini-batch SGD update rule underneath, adjusted with techniques like the linear scaling rule and warmup discussed earlier [13].
FAQ
What is stochastic gradient descent in simple terms?
Stochastic gradient descent is a method for training machine learning models by repeatedly nudging the model's parameters in the direction that reduces prediction error, using the gradient computed from a small, randomly chosen sample of the training data rather than the entire dataset at once.
Is SGD the same as gradient descent?
No. Gradient descent is the general family of algorithms that move parameters opposite to the gradient; full-batch gradient descent uses the entire dataset per update, while SGD uses a randomly sampled example or mini-batch, making SGD one specific, more scalable member of that family.
Does SGD always use one training example per update?
In its strict historical form, yes, but modern deep-learning frameworks and most practitioners use "SGD" to mean mini-batch SGD, which updates using a small batch of examples (commonly 32 to a few thousand) rather than a single example [2][4].
What is the difference between an epoch, a batch, and an iteration?
An epoch is one full pass through the entire training dataset. A batch (or mini-batch) is a subset of the data used for a single parameter update. An iteration (or optimizer step) is one such update; the number of iterations per epoch equals the dataset size divided by the batch size.
How do I choose a learning rate for SGD?
There is no single universally correct value; common practice is to try a range of learning rates on a small experiment or use a short learning-rate range test, then refine using a schedule such as step decay or cosine decay based on how training behaves.
What is the best batch size for SGD?
There is no single best batch size; it depends on available memory, hardware parallelism, and how the learning rate is adjusted alongside it. Smaller batches give noisier but more frequent updates; larger batches give smoother but less frequent updates and require more memory per step.
Why does my SGD training loss look noisy or jump around?
This is often expected behavior, since each update is based on a randomly sampled batch rather than the whole dataset. If the noise is extreme, it can indicate a learning rate that is too high or a batch size that is too small relative to the task.
What is momentum in SGD, and why use it?
Momentum accumulates a running average of past gradients so that the optimizer keeps moving through flat regions and small bumps in the loss surface, generally speeding up convergence compared to vanilla SGD, though it adds one extra hyperparameter to tune [7].
Is Adam always better than SGD?
No. Adam often converges faster in early training and requires less tuning, but well-tuned SGD with momentum has matched or outperformed Adam's final generalization on various tasks in different studies; the better choice depends on the specific model, dataset, and available tuning time [8].
Can SGD get stuck in a local minimum?
On non-convex problems such as deep neural network training, SGD can converge to a stationary point that is a local minimum, saddle point, or flat region rather than the global minimum; gradient noise may sometimes help it move past shallow traps, but this is not guaranteed [2][3].
What causes exploding or vanishing gradients in SGD training, and how do I fix them?
Exploding gradients often stem from a learning rate that is too high, poor initialization, or unstable architectures, and can be addressed with gradient clipping and lower learning rates; vanishing gradients often stem from deep architectures with poorly chosen activation functions or initializations, addressed through better initialization schemes or architectural changes such as residual connections.
Do I need to scale or normalize my features before using SGD?
Yes, in most cases. Unscaled features can distort the loss surface's conditioning, making training slower and less stable; both neural-network optimizers and scikit-learn's SGD-based estimators are documented as sensitive to feature scaling [6].
What is the difference between SGD in PyTorch/TensorFlow and SGDRegressor in scikit-learn?
torch.optim.SGD and keras.optimizers.SGD are general-purpose optimizers that can train any differentiable model, including deep neural networks. SGDRegressor and SGDClassifier in scikit-learn are specific linear model estimators fitted via stochastic gradient descent; they are not general-purpose optimizers for arbitrary architectures [4][5][6].
Key Takeaways
SGD estimates the gradient from a single example or small mini-batch rather than the full dataset, trading exact accuracy for dramatically lower per-step cost.
The term traces back to Robbins and Monro's 1951 stochastic approximation method, long before its role in modern deep learning [1].
In practice, "SGD" in frameworks like PyTorch, TensorFlow, and scikit-learn almost always means mini-batch SGD, not strict single-example updates [4][5][6].
The learning rate is usually the most influential hyperparameter, and no single schedule or value works best for every task.
Batch size and learning rate interact; changing one often means revisiting the other, especially at large scale [13].
Momentum, Nesterov momentum, and averaged SGD are established ways to improve on vanilla SGD without switching to a fully adaptive optimizer.
Adaptive optimizers like Adam and AdamW are not universally better than SGD; the right choice depends on the task, architecture, and available tuning time [8][12].
SGD is not guaranteed to find the global minimum on non-convex problems, and training loss improvements do not by themselves prove good generalization.
Actionable Next Steps
Pick one small dataset and implement the from-scratch NumPy example above to build direct intuition for how gradients and updates interact.
Run the same training task with torch.optim.SGD or keras.optimizers.SGD at three different learning rates and compare the loss curves.
Repeat the same experiment at two different batch sizes, keeping everything else fixed, and note how the training curve's smoothness changes.
Add momentum (e.g., 0.9) to your SGD optimizer and compare the number of epochs needed to reach a similar loss value.
If working with sparse, high-dimensional data such as text, try scikit-learn's SGDClassifier or SGDRegressor after standardizing your features.
Apply one learning-rate schedule (step decay or cosine decay) to your best-performing configuration and check whether final validation performance improves.
Log your experiments (learning rate, batch size, momentum, final validation score) so you can compare configurations systematically rather than by memory.
Glossary
Batch: A group of training examples processed together before a parameter update.
Batch gradient descent: Optimization using the gradient computed from the entire training dataset per update.
Batch size: The number of training examples used in one mini-batch.
Convergence: The point at which further training produces little or no further improvement in the loss.
Empirical risk minimization: The practice of minimizing average loss over a finite training sample as a proxy for minimizing true expected loss.
Epoch: One complete pass through the entire training dataset.
Generalization: How well a trained model performs on new, unseen data.
Gradient: A vector of partial derivatives showing how the loss changes with respect to each parameter.
Gradient clipping: Capping the magnitude of gradients to prevent unstable, oversized parameter updates.
Iteration: One single parameter update, typically corresponding to one mini-batch.
Learning rate: A scalar controlling the size of each parameter update step.
Loss function: A function measuring how far a prediction is from the true target value.
Mini-batch: A small, randomly sampled subset of the training data used for one update in mini-batch SGD.
Momentum: A technique that accumulates a running average of past gradients to smooth and speed up optimization.
Nesterov momentum: A momentum variant that computes the gradient at a look-ahead position based on the current velocity.
Objective function: The function being minimized or maximized during training, often the average loss over the training data.
Optimizer: The algorithm responsible for updating model parameters based on gradients.
Overfitting: When a model fits training data very well but performs poorly on new data.
Parameter: A learnable numeric value inside a model, such as a weight or bias.
Regularization: Techniques that discourage overly complex models to improve generalization.
SGD (stochastic gradient descent): An optimization algorithm that updates parameters using gradients estimated from a random example or mini-batch rather than the full dataset.
Stationary point: A point where the gradient is zero or near-zero, which may be a minimum, maximum, or saddle point.
Stochastic gradient: An estimate of the true gradient computed from a random sample of the data.
Training set: The portion of data used to fit a model's parameters.
Validation set: Data held out from training, used to tune hyperparameters and monitor generalization.
Vanishing/exploding gradients: Conditions where gradients become extremely small or extremely large during training, disrupting learning.
Weight decay: A regularization technique that shrinks parameter values toward zero at each update.
Sources & References
Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586
Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2), 223–311. https://doi.org/10.1137/16M1080173
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org/ (Chapter 8: Optimization for Training Deep Models)
PyTorch. torch.optim.SGD documentation. https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html (accessed July 24, 2026)
Keras. keras.optimizers.SGD documentation. https://keras.io/api/optimizers/sgd/ (accessed July 24, 2026)
scikit-learn. 1.5. Stochastic Gradient Descent. https://scikit-learn.org/stable/modules/sgd.html (accessed July 24, 2026)
Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013). On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, PMLR 28, 1139–1147. http://proceedings.mlr.press/v28/sutskever13.html
Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations. https://arxiv.org/abs/1412.6980
Polyak, B. T., & Juditsky, A. B. (1992). Acceleration of Stochastic Approximation by Averaging. SIAM Journal on Control and Optimization, 30(4), 838–855. https://doi.org/10.1137/0330046
Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12(61), 2121–2159. https://jmlr.org/papers/v12/duchi11a.html
Tieleman, T., & Hinton, G. (2012). Lecture 6.5-RMSProp: Divide the Gradient by a Running Average of Its Recent Magnitude. COURSERA: Neural Networks for Machine Learning.
Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations. https://arxiv.org/abs/1711.05101
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint. https://arxiv.org/abs/1706.02677
