What Is Momentum in Machine Learning?
- Jul 28
- 32 min read
Training a deep network with plain gradient descent can feel like pushing a shopping cart with one bad wheel across a parking lot: every small correction sends it veering sideways, and useful forward progress crawls. Momentum in machine learning fixes this by giving the optimizer a memory of where it has already been moving, so it keeps rolling in a consistent direction instead of re-deciding its path from scratch at every single step. That one idea, borrowed from decades-old optimization theory, is a big part of why modern deep learning models can train in hours instead of weeks.
TL;DR
Momentum in machine learning is an optimization technique that adds a fraction of the previous update to the current gradient step, building up velocity in directions where gradients keep agreeing.
It mainly speeds up training along consistent gradient directions and reduces zig-zagging on steep, narrow loss surfaces often called ravines.
The core mechanism is an exponentially weighted accumulation of past gradients, controlled by a momentum coefficient, usually written as μ or β.
Momentum interacts closely with the learning rate: raising momentum effectively raises the step size too, so the two must be tuned together, not separately.
Nesterov accelerated gradient evaluates the gradient after a look-ahead step, which can make it more responsive to changing curvature than classical, heavy-ball style momentum.
Momentum never guarantees a better-performing model. It changes how fast and how smoothly training reaches a minimum, and too much of it can cause overshoot or oscillation.
What is momentum in machine learning?
Momentum in machine learning is an optimization technique that speeds up gradient-based training by accumulating a moving average of past gradients into a velocity term. Instead of reacting only to the current gradient, the optimizer keeps moving in directions that have been consistently downhill, which speeds convergence and reduces oscillation on uneven loss surfaces.
Table of Contents
What Momentum Means in Machine Learning
Momentum in machine learning is a modification to gradient-based optimization that gives each parameter update a "memory" of previous updates. Rather than moving strictly in the direction of the current gradient, a momentum-based optimizer moves in a blend of the current gradient and an accumulated velocity built from earlier steps.
This idea did not start in machine learning. It comes from classical numerical optimization. Boris Polyak introduced the heavy-ball method in 1964 as a way to accelerate iterative methods for solving convex minimization problems (Polyak, 1964). Decades later, momentum became one of the default tools for training neural networks, largely because of work showing that a well-tuned momentum schedule combined with sensible weight initialization could train deep and recurrent networks that were previously very difficult to optimize (Sutskever, Martens, Dahl, and Hinton, 2013).
In practice, momentum shows up almost everywhere gradient descent is used to train a model, from simple linear regression solved with stochastic gradient descent to large-scale model training for deep neural networks. It is a setting on optimizers such as SGD, and it is also embedded, in modified form, inside more advanced optimizers such as Adam.
The core practical effect of momentum is this: it accelerates progress in directions where the gradient has been consistent across recent steps, and it dampens movement in directions where the gradient keeps flipping sign. That combination often means fewer training steps to reach a good loss value, though "fewer steps" is not the same thing as "a better final model" — a distinction this guide returns to throughout.
Why Ordinary Gradient Descent Can Struggle
To understand why momentum helps, it is worth being precise about what plain gradient descent actually does.
A machine learning model has a set of parameters, often called weights, that determine its predictions. Training tries to find parameter values that minimize a loss function, a number that measures how wrong the model's predictions are on the training data. The gradient of the loss with respect to the parameters is a vector that points in the direction of steepest increase of the loss. Gradient descent moves the parameters in the opposite direction, scaled by a step size called the learning rate.
In full-batch gradient descent, the gradient is computed using the entire training set at every step. In stochastic gradient descent, it is computed using one training example at a time, and in mini-batch gradient descent, using a small random subset, or mini-batch, of the data. Mini-batch training is by far the most common setup in modern deep learning, because it balances computational efficiency with a reasonably accurate estimate of the true gradient.
Plain gradient descent runs into two related problems.
First, noisy gradients. When gradients are estimated from a mini-batch rather than the full dataset, each estimate is a noisy approximation of the true gradient direction. Without any smoothing, the parameters can jitter step to step, following the noise almost as much as the true signal.
Second, poorly conditioned loss surfaces. Real loss surfaces are rarely bowl-shaped in every direction. Many are shaped like a long, narrow ravine: steep along some directions and nearly flat along others. A well-known description of this problem appears in the original momentum literature, where the loss surface's curvature in different directions is described using the ratio between the largest and smallest curvature, known as the condition number (Sutskever et al., 2013). When this ratio is large, plain gradient descent has to use a small learning rate to avoid diverging along the steep direction, which makes progress along the flat direction painfully slow. The optimizer bounces back and forth across the narrow direction while barely advancing along the long one.
Both problems get worse together. Noisy mini-batch gradients amplify the oscillation inside a ravine, because each noisy estimate can point slightly differently across the steep walls, adding wasted motion on top of the genuine zig-zag caused by curvature. This is the specific pair of problems that momentum is designed to address.
The Intuition Behind Momentum
Two complementary mental models help explain what momentum is doing, and it is worth understanding where each one breaks down.
Intuition one: a heavy ball rolling downhill. Imagine a heavy ball rolling down the loss surface instead of a weightless point that jumps wherever the local slope points. A heavy ball builds up speed as it rolls consistently in one direction, and it resists sudden changes in direction because of its own inertia. On a ravine-shaped surface, the ball's inertia along the long, flat direction keeps carrying it forward, while the oscillations across the narrow direction tend to cancel out over several bounces. This is why classical momentum is often called the heavy-ball method (Polyak, 1964).
This physical picture is useful, but it is only an analogy. A real loss landscape for a neural network is a mathematical surface in a very high-dimensional parameter space, not a physical hill with gravity and friction. Momentum's "velocity" is a bookkeeping term inside an update rule, not a literal physical quantity, and the optimizer does not experience momentum the way a rolling ball experiences inertia in three-dimensional space.
Intuition two: an exponentially weighted average of recent gradients. Mathematically, momentum keeps a running, decaying average of past gradients. Directions that show up consistently across many recent gradients get reinforced and add up. Directions that flip back and forth cancel out in the average. This framing explains the noise-smoothing effect of momentum directly: it behaves similarly to how a moving average smooths a noisy time series, without needing any physical metaphor.
Both intuitions describe the same update rule from different angles. The physical picture is easier to visualize; the moving-average picture is more accurate to what the algorithm is actually computing, especially when gradients are noisy estimates rather than exact values.
Momentum Mathematics, Step by Step
Before writing any equation, it helps to fix the symbols.
θ_t: the parameter vector at step t (the model's weights at that point in training).
g_t: the gradient, or stochastic-gradient estimate, of the loss with respect to θ at step t.
v_t: the velocity, or momentum buffer, an accumulated running direction built from past gradients.
η (eta): the learning rate, controlling the overall step size.
μ (mu): the momentum coefficient, a value usually between 0 and 1 that controls how much of the previous velocity is retained.
Classical momentum appears in two common formulations in the literature and in code, and they express the same underlying idea with different scaling conventions.
Formulation A (gradient-scaled velocity):
v_t = μ * v_(t-1) + g_t
θ_t = θ_(t-1) - η * v_t
Formulation B (learning-rate-scaled velocity):
v_t = μ * v_(t-1) - η * g_t
θ_t = θ_(t-1) + v_t
In Formulation A, the velocity accumulates raw gradients, and the learning rate is applied once, at the final update. In Formulation B, the learning rate is baked into the velocity itself. Under a fixed learning rate the two are mathematically equivalent up to a rescaling of v, but they are not always numerically interchangeable inside real frameworks, especially once the learning rate changes over training (via a schedule) or once weight decay and dampening enter the picture. This is precisely why PyTorch's own documentation flags that its SGD-with-momentum implementation "subtly differs" from the formulation used by Sutskever et al. (2013) and by some other frameworks, and recommends checking the exact update rule a given framework uses rather than assuming a single universal formula (PyTorch documentation).
The momentum buffer v_0 is typically initialized to zero, or, in some implementations, to the first gradient itself on the very first step (this is exactly how PyTorch initializes it, as shown later in the PyTorch section). Either choice affects only the first couple of updates; the buffer quickly reflects the accumulated recent gradient history once training is underway.
It is worth stating plainly what these formulas do not claim. They do not say momentum always reduces the number of steps needed. They do not say momentum is stable at any learning rate. And convergence-rate results proven for heavy-ball and Nesterov methods mostly rely on convexity or strong convexity assumptions (Polyak, 1964; Nesterov, 1983); deep neural network loss surfaces are non-convex, so those guarantees do not transfer automatically. Momentum is still widely used in non-convex deep learning because it tends to help empirically, not because the convex-case proofs cover this setting (Kidambi et al., 2018, cited in later non-convex momentum analyses).
A Worked Numerical Example
The clearest way to see momentum's effect is to trace a few update steps by hand, side by side with plain gradient descent, on a simple one-dimensional quadratic loss.
Suppose the loss is L(θ) = θ^2, so the gradient is g(θ) = 2θ. Start at θ_0 = 10. Use learning rate η = 0.1 for both methods, and momentum coefficient μ = 0.9 for the momentum run, using Formulation A (v_t = μv_(t-1) + g_t, θ_t = θ_(t-1) - ηv_t), with v_0 = 0.
Plain gradient descent:
| Step | θ (before) | g = 2θ | θ (after) = θ − ηg | |---|---|---|---| | 1 | 10.000 | 20.000 | 8.000 | | 2 | 8.000 | 16.000 | 6.400 | | 3 | 6.400 | 12.800 | 5.120 | | 4 | 5.120 | 10.240 | 4.096 |
Gradient descent with momentum (μ = 0.9):
| Step | θ (before) | g = 2θ | v = 0.9v_prev + g | θ (after) = θ − ηv | |---|---|---|---|---| | 1 | 10.000 | 20.000 | 20.000 | 8.000 | | 2 | 8.000 | 16.000 | 34.000 | 4.600 | | 3 | 4.600 | 9.200 | 39.800 | 0.620 | | 4 | 0.620 | 1.240 | 37.060 | -3.086 |
After the same four steps, plain gradient descent has reached θ = 4.096, while momentum has reached θ = -3.086, overshooting past zero. This single example demonstrates two real effects at once. First, momentum clearly moves faster: its velocity term keeps growing because every gradient so far has pointed the same way (θ has stayed positive and decreasing), so the accumulated push is larger than the current gradient alone. Second, that same acceleration causes overshoot: by step 4, momentum has enough accumulated velocity to carry θ past the minimum at zero and out the other side, something plain gradient descent, with no memory, cannot do on this simple curve.
This is the central trade-off in one small table: momentum trades some risk of overshoot for a real chance at faster progress, and the size of that trade-off depends heavily on the momentum coefficient and the learning rate chosen together, not on either one in isolation.
How Momentum Changes the Optimisation Path
The worked example above used a one-dimensional loss, which cannot show the zig-zag behavior that motivates most real uses of momentum. In two or more dimensions, particularly on a ravine-shaped loss surface, the story becomes clearer.
On a ravine, gradient descent without momentum tends to take a jagged, sawtooth path: large oscillations across the narrow, steep direction, combined with slow creeping progress along the long, shallow direction. Momentum changes this path in a specific way. Across the steep direction, successive gradients tend to flip sign as the optimizer overshoots back and forth, so their contributions to the velocity partly cancel out, damping the oscillation. Along the shallow direction, successive gradients tend to point the same way step after step, so their contributions to the velocity add up, accelerating progress precisely where plain gradient descent was slowest.
The result is an optimization path that looks smoother and more direct toward the minimum, with reduced side-to-side oscillation and faster movement along the dominant, persistent direction. This behavior is exactly what motivated Polyak's original heavy-ball analysis for ill-conditioned convex problems (Polyak, 1964), and it is the main reason momentum remains useful on the highly non-convex, often ravine-like loss surfaces found in deep learning, even though the formal convergence guarantees from convex analysis do not directly apply there.
It is important not to overstate this benefit. Momentum smooths and redirects an existing gradient signal; it cannot manufacture information the gradients do not contain, and on flat regions or saddle points where gradients are close to zero in every direction, momentum can help carry the optimizer through, but it does not always do so, and it can also carry the optimizer past a good minimum if it is too large, as the worked example already showed.
Classical Momentum vs Nesterov Accelerated Gradient
Classical, heavy-ball momentum computes the gradient at the current position and then applies the accumulated velocity. Nesterov accelerated gradient, introduced by Yurii Nesterov for convex optimization (Nesterov, 1983), changes the order of operations: it first takes a "look-ahead" step using the existing velocity, then evaluates the gradient at that look-ahead point, and only then updates the velocity and the parameters.
The intuition is that classical momentum can be thought of as a ball that measures the slope where it currently stands and then commits to its accumulated push, even if that push is about to carry it somewhere the slope suggests it should slow down. Nesterov's method instead "peeks ahead" to where the momentum is about to carry the parameters, checks the slope there, and corrects the update using that more current information. In regions where the loss surface's curvature is changing quickly, this look-ahead correction can make the optimizer more responsive and, in some settings, more stable than classical momentum, although Nesterov's original acceleration guarantees were proven for convex problems, not for the non-convex loss surfaces typical of deep neural networks (Nesterov, 1983).
Frameworks do not all express Nesterov's update the same way. The version widely used in deep learning is based on a reformulation described by Sutskever, Martens, Dahl, and Hinton (2013), which rewrites Nesterov's original two-step recursion into a single-step update that is easier to implement inside standard deep learning training loops. PyTorch's own SGD documentation explicitly notes that its implementation of Nesterov momentum is based on this Sutskever et al. formulation but still differs subtly from it in how the learning rate and velocity interact (PyTorch documentation). Because of this, it is important to consult the specific framework you are using rather than assuming that "Nesterov momentum" means one universal formula everywhere.
It would be inaccurate to claim Nesterov momentum is always faster or always more stable than classical momentum. Its benefit depends on the shape of the loss surface, the learning rate, and the momentum coefficient in use; in practice, many deep learning practitioners find the two perform similarly on a given task, and the choice often comes down to what a given framework makes easy to configure.
| Aspect | Classical (Heavy-Ball) Momentum | Nesterov Accelerated Gradient | |---|---|---| | Gradient evaluated at | Current parameter position | Look-ahead position after applying existing velocity | | Originator | Boris Polyak, 1964 | Yurii Nesterov, 1983 | | Typical framing | Ball rolling with inertia | Ball that "checks ahead" before committing | | Convergence theory | Proven acceleration for strongly convex problems | Proven O(1/k²) rate for convex, Lipschitz-gradient problems | | Deep learning usage | Common default in SGD-with-momentum | Common alternative flag (nesterov=True) in the same optimizers | | Practical difference | Can overshoot more before correcting | Can correct sooner because it uses look-ahead gradient information |
Momentum in Batch, Stochastic and Mini-Batch Gradient Descent
Momentum behaves somewhat differently depending on how the gradient itself is estimated.
In full-batch gradient descent, the gradient at every step is exact, computed over the whole training set, so momentum's main job is purely to accelerate progress across ill-conditioned directions; there is no gradient noise to smooth.
In pure stochastic gradient descent, where each step uses a single training example, gradients are highly noisy individual estimates. Momentum's exponentially weighted averaging effect becomes especially valuable here, since it smooths out much of the per-example noise, though it does not eliminate it, and very high momentum combined with very noisy per-example gradients can make training less stable rather than more.
In mini-batch gradient descent, the most common setting in practice, gradients are averaged over a small batch of examples, which already reduces noise somewhat compared to single-example updates. Momentum adds a further layer of smoothing on top of the mini-batch averaging. Batch size and momentum interact: smaller batches produce noisier per-step gradients, which momentum smooths more aggressively, while larger batches already produce fairly stable gradients, so the marginal benefit of a high momentum coefficient can be smaller, and a very high momentum coefficient with a large batch size can sometimes make the effective step size too large, since momentum increases how far a consistent gradient signal is allowed to carry the parameters.
None of this changes the core update rule from the Momentum Mathematics section above; what changes is simply how noisy g_t is at each step, and therefore how much of momentum's benefit comes from acceleration versus noise-smoothing in a given setting.
Choosing the Momentum Coefficient and Learning Rate
Momentum and the learning rate are not independent knobs. Because velocity accumulates gradients over many steps, raising the momentum coefficient effectively increases the size of the steps the optimizer takes once velocity has built up, even if the learning rate itself stays fixed. In rough terms, the "effective" step size scales with 1 / (1 - μ) under Formulation A once the velocity has stabilized, which is why a common practical guideline is that increasing momentum usually calls for decreasing the learning rate to keep the overall update size in a stable range.
A momentum coefficient around 0.9 is a frequently used starting point in deep learning practice, and it appears as a common example value in official framework documentation for SGD (PyTorch documentation; Keras documentation), but it is a starting point, not a universal law. Lower values, such as 0.5 to 0.8, retain less history and behave closer to plain gradient descent, which can be safer on noisy or unstable objectives. Higher values, such as 0.95 to 0.99, retain much more history and accelerate more aggressively, which can speed up training on well-behaved, smooth objectives but raises real risk of overshoot and oscillation, especially if the learning rate is not reduced to compensate.
Signs that momentum is set too high include a loss curve that oscillates or diverges after initially decreasing, parameter updates that visibly overshoot and then have to correct back, and training that becomes more unstable specifically after momentum or the learning rate is increased. Signs that momentum is too low include a loss curve that decreases smoothly but much more slowly than expected, and a training run that looks similar to plain gradient descent despite momentum being enabled.
Batch size, weight decay, and momentum also interact. Larger batch sizes are often paired with a somewhat larger learning rate and sometimes a reduced momentum coefficient, since larger batches already produce lower-variance gradients. Weight decay, a regularization technique that shrinks parameters toward zero, is applied differently depending on whether a framework mixes it directly into the gradient before the momentum step or applies it separately after the momentum step, a distinction popularized by the AdamW paper on decoupled weight decay (Loshchilov and Hutter, 2019). Combining weight decay with momentum requires knowing which convention a given framework uses, since the two orders of operations produce different effective updates over time. For further background on this kind of tuning, see this guide's related discussion of hyperparameter tuning and regularization.
Fine-tuning a pretrained model is a case where lower momentum, a lower learning rate, or both, are often preferred, since large accumulated updates risk erasing useful pretrained weights before the new task-specific signal has had a chance to guide training. For non-stationary training, such as continual or online learning where the data distribution itself shifts over time, a persistently high momentum coefficient can cause the optimizer to keep "coasting" in a direction that was correct for older data but is no longer correct for the current distribution, which is one reason some practitioners reduce momentum or reset the buffer when the training distribution changes materially.
Momentum Schedules, Warm-Up, Dampening and Restarts
Momentum does not have to stay fixed throughout training, and several practical adjustments are common in real training pipelines.
Momentum schedules. Rather than fixing μ at a single value, some training setups slowly increase momentum from a lower value toward a higher one over the first portion of training. Sutskever et al. (2013) specifically found that a well-designed, gradually increasing momentum schedule, paired with careful weight initialization, was important for successfully training deep and recurrent networks with SGD and momentum, and that poorly initialized networks could not be trained with momentum at all, while well-initialized networks performed noticeably worse without properly tuned momentum.
Warm-up. Many modern training recipes combine a learning-rate warm-up, where the learning rate starts small and increases over the first steps or epochs, with a stable momentum setting. Warm-up exists mainly to avoid large, unstable updates before the model's weights and the optimizer's internal statistics have settled, and it interacts with momentum because a high momentum coefficient combined with a still-ramping learning rate can produce a rapidly growing effective step size if not managed carefully.
Dampening. PyTorch's SGD optimizer exposes a dampening parameter that reduces the weight given to the current gradient when it is added into the velocity buffer (PyTorch documentation). A dampening value of 0 is the standard heavy-ball update; values closer to 1 reduce how much each new gradient contributes to velocity, softening the accumulation effect. PyTorch also requires dampening to be zero when Nesterov momentum is enabled (PyTorch documentation).
Restarts and buffer resets. Some training regimes intentionally reset the momentum buffer to zero at specific points, for example, right after a large change to the training setup, such as switching datasets, unfreezing new layers during fine-tuning, or restarting a learning-rate schedule. Resetting prevents the optimizer from continuing to apply accumulated velocity that reflects a training regime that no longer applies.
Gradient clipping. In recurrent networks and other settings prone to large gradient spikes, gradient clipping, which caps the norm or value of the gradient before it is used, is commonly combined with momentum to prevent a single large gradient from injecting an outsized, destabilizing push into the velocity buffer that then persists across several subsequent steps.
Sparse gradients. For models with sparse gradient updates, such as large embedding tables where only a few rows update per step, dense momentum buffers can behave unexpectedly, since a parameter that receives a gradient only occasionally can still carry velocity from steps where it was not directly updated. This is one reason some sparse-gradient scenarios use optimizers designed specifically for sparse updates rather than plain SGD with momentum.
Implementing Momentum From Scratch in Python
Seeing the update rule as code, without any framework in the way, makes the mechanics concrete. The example below minimizes the same simple quadratic used earlier, L(θ) = θ^2, using classical momentum in NumPy.
import numpy as np
def train_with_momentum(theta_init, lr=0.1, momentum=0.9, steps=10):
theta = theta_init
velocity = 0.0
history = []
for step in range(steps):
grad = 2 * theta # gradient of L(theta) = theta^2
velocity = momentum * velocity + grad
theta = theta - lr * velocity
history.append((step + 1, theta, grad, velocity))
return theta, history
final_theta, log = train_with_momentum(theta_init=10.0)
for step, theta, grad, velocity in log:
print(f"step={step} theta={theta:.4f} grad={grad:.4f} velocity={velocity:.4f}")
This implementation mirrors Formulation A from the mathematics section: the parameter theta starts at 10.0, the velocity buffer starts at zero, and each iteration computes the gradient, updates the velocity as a weighted combination of the old velocity and the new gradient, then updates the parameter using the learning rate and the new velocity. Running this with lr=0.1 and momentum=0.9 reproduces the same first four values shown in the worked numerical example earlier in this guide, and continuing past step 4 shows the parameter oscillating with decreasing amplitude before settling near zero, the minimum of this loss function.
Using Momentum in PyTorch
PyTorch exposes classical and Nesterov momentum directly as parameters of torch.optim.SGD (PyTorch documentation).
import torch
import torch.nn as nn
model = nn.Linear(10, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.01,
momentum=0.9,
dampening=0.0,
weight_decay=0.0,
nesterov=False,
)
inputs = torch.randn(32, 10)
targets = torch.randn(32, 1)
optimizer.zero_grad()
predictions = model(inputs)
loss = loss_fn(predictions, targets)
loss.backward()
optimizer.step()
lr sets the learning rate, momentum sets the momentum coefficient, dampening reduces the weight given to the newest gradient inside the velocity buffer, weight_decay adds an L2 penalty, and nesterov=True switches on Nesterov-style momentum, which PyTorch requires to be paired with dampening=0 (PyTorch documentation). The call order matters: optimizer.zero_grad() clears old gradients, loss.backward() computes new gradients through automatic differentiation, and optimizer.step() applies the momentum update using those gradients. For background on how the gradients behind this call are actually computed, see this guide's companion article on automatic differentiation and on the underlying computational graph.
PyTorch's documentation is explicit that its SGD-with-momentum implementation "subtly differs" from the formulation used by Sutskever et al. (2013) and by implementations in some other frameworks (PyTorch documentation). Concretely, PyTorch's classical-momentum update is:
v_(t+1) = momentum * v_t + g_(t+1)
p_(t+1) = p_t - lr * v_(t+1)
This matches Formulation A from the Momentum Mathematics section. Readers moving between PyTorch and other tools should treat this as a reminder to check each framework's documented convention before assuming behavior transfers directly, particularly when replicating results or ported hyperparameters from a paper or another codebase.
Using Momentum in TensorFlow and Keras
Keras and TensorFlow expose the same core idea through keras.optimizers.SGD (Keras documentation).
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Dense(1, input_shape=(10,))
])
optimizer = tf.keras.optimizers.SGD(
learning_rate=0.01,
momentum=0.9,
nesterov=False,
)
loss_fn = tf.keras.losses.MeanSquaredError()
inputs = tf.random.normal((32, 10))
targets = tf.random.normal((32, 1))
with tf.GradientTape() as tape:
predictions = model(inputs, training=True)
loss = loss_fn(targets, predictions)
gradients = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(gradients, model.trainable_variables))
learning_rate sets the step size, momentum sets the momentum coefficient, and nesterov=True enables Nesterov-style momentum. According to the official Keras documentation, the update rule when momentum is greater than zero is velocity = momentum * velocity - learning_rate * g, followed by w = w + velocity, and when nesterov=True, the rule becomes w = w + momentum * velocity - learning_rate * g (Keras documentation). This matches Formulation B from the Momentum Mathematics section, where the learning rate is baked into the velocity itself, in contrast to PyTorch's Formulation A convention. This is a concrete, documented example of the exact "different sign and scaling conventions" issue flagged earlier in this guide: the same conceptual algorithm, momentum-based SGD, is written differently by two major frameworks, and swapping hyperparameters between them without adjusting for this difference can produce different training behavior. Readers coming from model weights work in one framework and moving to another should re-verify momentum-related hyperparameters rather than porting them unchanged.
How to Diagnose Momentum During Training
A handful of observable signals help identify whether momentum is helping, hurting, or simply not doing very much in a given run.
A smoothly decreasing loss curve that reaches a good value in fewer steps than a no-momentum baseline is the clearest positive signal. A loss curve that decreases at first and then begins oscillating or increasing, especially right after momentum or the learning rate was raised, points toward momentum, the learning rate, or their combination being set too high. A loss curve that looks almost identical to a plain gradient descent run suggests momentum is currently contributing very little, which can happen when the momentum coefficient is set too low, or when gradients are already close to zero, such as late in training near convergence.
Tracking gradient norms alongside the loss can help distinguish these cases: a gradient norm that stays large while the loss oscillates suggests an unstable step size, whereas a gradient norm that shrinks steadily alongside a decreasing loss is a healthier pattern. Some practitioners also track the velocity buffer's norm directly; a velocity norm that keeps growing without bound over many steps, even as the loss stops improving, is a sign that momentum is accumulating faster than the gradient signal justifies.
| Training Symptom | Possible Momentum-Related Cause | Suggested Adjustment | |---|---|---| | Loss decreases, then oscillates or diverges | Momentum coefficient or learning rate too high for this loss surface | Lower momentum, lower learning rate, or both, and re-run a short comparison | | Loss decreases very slowly, similar to plain gradient descent | Momentum coefficient too low, or momentum not actually enabled | Increase momentum incrementally, for example from 0.5 toward 0.9, while watching stability | | Training was stable, then destabilized after unfreezing layers or switching data | Momentum buffer still reflects a training regime that no longer applies | Reset the momentum buffer, or reduce momentum temporarily after the change | | Occasional large loss spikes, especially in recurrent models | A single large gradient injected a big push into the velocity buffer | Add gradient clipping alongside momentum | | Training is stable at small batch size but destabilizes at larger batch size | Momentum plus a larger batch size is producing too large an effective step | Reduce momentum, reduce learning rate, or both, when scaling up batch size |
Common Momentum Mistakes and Failure Modes
Treating 0.9 as a fixed rule rather than a starting point. A momentum coefficient of 0.9 shows up often in documentation examples and published training recipes, but it is not a universal optimum. Using it without any adjustment for a specific loss surface, batch size, or learning rate can leave real performance on the table in either direction.
Raising momentum without lowering the learning rate. Because effective step size grows with momentum, increasing momentum while leaving the learning rate untouched is a common way to accidentally destabilize a previously stable training run.
Porting hyperparameters between frameworks unchanged. As shown in the PyTorch and Keras sections above, the same momentum coefficient can produce different effective updates depending on which formulation a framework uses internally. Copying a momentum value from a paper or another codebase without checking the target framework's convention is a frequent, avoidable source of confusing results.
Confusing optimizer momentum with unrelated "momentum" parameters. Some framework components use the word momentum for a completely different exponential moving average, most notably batch normalization's running statistics parameter. Conflating the two, and tuning one while intending to tune the other, is a common and easy-to-miss mistake covered in more detail in the next section.
Ignoring momentum when fine-tuning. Applying the same aggressive momentum settings used for training a model from scratch to a fine-tuning run on a pretrained model can cause large, destabilizing updates that overwrite useful pretrained weights faster than the new, task-specific gradient signal can guide the model toward a better solution.
Assuming momentum fixes a fundamentally broken setup. Momentum smooths and accelerates an existing gradient signal; it does not fix problems such as an incorrectly implemented loss function, badly scaled input features (see standardization), or a learning rate that is simply too large for the problem at any momentum setting.
Momentum vs AdaGrad, RMSprop, Adam and AdamW
Momentum-based SGD is one member of a broader family of gradient-based optimizers, several of which combine momentum-like ideas with adaptive, per-parameter learning rates.
AdaGrad adapts the learning rate for each parameter individually based on the historical sum of squared gradients for that parameter, which helps with sparse features but can cause the effective learning rate to shrink too aggressively over long training runs. RMSprop, proposed informally by Geoffrey Hinton in unpublished lecture notes and widely implemented in major frameworks, addresses that shrinking-learning-rate problem by using an exponentially decaying average of squared gradients instead of an ever-growing sum. The Adam optimizer combines the RMSprop-style adaptive, per-parameter learning rate with a momentum-like exponential moving average of the gradients themselves, tracked as Adam's first moment, alongside a second moment tracking squared gradients (Kingma and Ba, 2015). AdamW modifies Adam by decoupling weight decay from the adaptive gradient update, applying weight decay directly to the parameters rather than folding it into the gradient before the adaptive scaling is applied (Loshchilov and Hutter, 2019).
It is not accurate to describe Adam as simply "SGD with momentum." Adam adds a second, independent mechanism, the adaptive per-parameter scaling from its second-moment estimate, that plain SGD with momentum does not have at all. This means Adam and SGD-with-momentum can behave quite differently on the same problem, and neither one is universally better; each has settings where it tends to perform well and settings where the other is often preferred.
| Optimizer | Core Mechanism | Typical Strength | Typical Trade-off | |---|---|---|---| | SGD with Momentum | Exponentially weighted average of gradients (velocity) | Well-understood, memory-light, often generalizes well with careful tuning | More sensitive to learning rate and momentum tuning | | RMSprop | Per-parameter adaptive learning rate from decaying squared-gradient average | Handles varying gradient scales across parameters well | No explicit momentum term unless combined with one | | Adam | Combines momentum-like first moment with RMSprop-style second moment | Often converges quickly with less manual tuning, popular default for many deep learning tasks | Weight decay behavior is coupled with the adaptive update unless using AdamW; can generalize worse than tuned SGD on some tasks | | AdamW | Adam with decoupled weight decay applied directly to parameters | Cleaner separation of regularization from the adaptive gradient update | Still carries Adam's overall complexity and memory cost relative to plain SGD |
Choosing between these is task-dependent. Many practitioners default to Adam or AdamW for faster initial convergence with less manual learning-rate tuning, while some large-scale image classification and other well-studied benchmarks still favor carefully tuned SGD with momentum, sometimes reporting better final generalization after a longer, more carefully scheduled training run. Neither claim generalizes automatically to every task, dataset, or model architecture.
Momentum Beyond SGD: EMA and Terminology Traps
The word "momentum" and the underlying mechanism of an exponential moving average show up in several places in deep learning that are conceptually related but functionally distinct from optimizer momentum, and mixing them up is a common source of confusion.
Adam's first moment is itself an exponential moving average of gradients, mechanically similar to classical momentum, but it is packaged inside Adam's larger update alongside the second-moment, adaptive-learning-rate mechanism, so tuning Adam's beta1 parameter is not the same exercise as tuning SGD's momentum parameter, even though both control a similar kind of averaging.
Adam's second moment tracks an exponential moving average of squared gradients, used to scale the effective learning rate per parameter. It has no direct analogue in classical SGD-with-momentum at all.
Batch normalization's momentum parameter controls how quickly a layer's running statistics, the mean and variance used at inference time, are updated toward newly observed batch statistics during training. Several frameworks use the opposite convention here compared to optimizer momentum: a batch-normalization momentum close to 1 can mean the running statistic barely changes per batch in one framework's convention, while a value close to 0 means the same thing in another's. This is exactly the kind of "opposite coefficient convention" trap that makes it essential to check a specific framework's documentation rather than assuming familiarity with optimizer momentum transfers directly.
Exponential moving averages of model weights, sometimes called EMA weights or a teacher model in a student-teacher or self-distillation setup, maintain a smoothed copy of the model's own parameters across training steps, entirely separate from the optimizer's internal momentum buffer. A model can use SGD with momentum for the optimizer step and simultaneously maintain an EMA of its weights for a more stable evaluation checkpoint; these are two independent mechanisms operating on different quantities.
Target-network updates, common in reinforcement learning, similarly maintain a slowly updated copy of a network's weights using an exponential moving average, again a separate concept from optimizer momentum, despite sharing the same underlying mathematical pattern of exponential smoothing.
Gradient accumulation, which sums or averages gradients across several forward-backward passes before taking a single optimizer step, is sometimes confused with momentum because both involve combining information across steps, but gradient accumulation changes what a single optimizer step sees before any momentum is applied, rather than changing how the optimizer's internal velocity is computed.
| Term | What It Actually Tracks | Where It Lives | |---|---|---| | Optimizer momentum (classical/Nesterov) | Exponential moving average of gradients | Inside the SGD-style optimizer's velocity buffer | | Adam first moment | Exponential moving average of gradients | Inside the Adam optimizer's internal state | | Adam second moment | Exponential moving average of squared gradients | Inside the Adam optimizer's internal state | | Batch normalization momentum | Running mean and variance of activations | Inside the batch-normalization layer, not the optimizer | | EMA of model weights | The model's own parameters, smoothed over time | A separate copy of the weights, outside the optimizer | | Target-network update | A separate, slowly updated copy of a network's weights | Common in reinforcement-learning training loops |
When Should You Use Momentum?
Momentum tends to help when gradients across steps point in a broadly consistent direction for stretches of training, which is common in many standard supervised learning setups trained with mini-batch SGD, and when the loss surface has some degree of the ravine-like, ill-conditioned shape that motivated the original heavy-ball method (Polyak, 1964).
Momentum is less clearly beneficial, or requires more careful tuning, in a few situations: very small datasets where a handful of noisy gradient estimates can cause the velocity buffer to accumulate misleading direction; highly non-stationary training, such as certain online or continual learning setups, where the correct gradient direction itself keeps changing and a high momentum coefficient can cause the optimizer to keep pursuing an outdated direction; and settings already using an adaptive optimizer such as Adam or AdamW, where the momentum-like first-moment mechanism is already built in and a separate, additional momentum setting does not apply in the same way.
Momentum does not, on its own, change what a model has learned to represent, and a faster-converging training run is not automatically a better-generalizing one; the relationship between optimization speed and generalization depends on many other factors, including model architecture, regularization, and how well the train-test split reflects real deployment data.
Practical Momentum Tuning Checklist
Confirm the framework's exact momentum update convention (PyTorch's Formulation A style versus Keras/TensorFlow's Formulation B style) before porting hyperparameters between tools.
Start from a documented, commonly used momentum value such as 0.9 as a baseline, not as a final answer.
Adjust momentum and learning rate together; treat them as a single combined decision, not two independent ones.
Watch the loss curve and gradient norm together during early training to catch oscillation or divergence quickly.
Reduce momentum, the learning rate, or both, when increasing batch size substantially.
Consider lower momentum, a lower learning rate, or both, when fine-tuning a pretrained model.
Add gradient clipping alongside momentum for recurrent networks or other architectures prone to gradient spikes.
Reset the momentum buffer after major training-regime changes, such as unfreezing new layers or switching datasets.
Do not confuse optimizer momentum with Adam's internal moment estimates, batch-normalization momentum, or EMA-of-weights techniques; tune each independently.
Re-run a short, controlled comparison against a no-momentum or Adam baseline whenever momentum-related changes are made, rather than relying on assumptions from a different project or framework.
FAQ
What is momentum in machine learning, in one sentence?
Momentum in machine learning is an optimization technique that accelerates gradient-based training by accumulating a moving average of past gradients into a velocity term, instead of reacting only to the current gradient at each step.
Why does momentum speed up gradient descent?
Momentum speeds up gradient descent by adding up gradient contributions that point consistently in the same direction across steps, which accelerates progress along persistent directions, while gradient contributions that flip back and forth tend to cancel out in the velocity term.
Does momentum always improve a model?
No. Momentum can speed up and smooth the optimization process, but it does not guarantee a better-performing final model. Faster convergence and better generalization are related but separate outcomes, and excessive momentum can even destabilize training.
Is 0.9 always the best momentum value?
No. A momentum coefficient of 0.9 is a common starting point seen in official documentation and published recipes, but the ideal value depends on the loss surface, batch size, and learning rate, and should be tuned rather than assumed.
How are the learning rate and momentum related?
Raising the momentum coefficient effectively increases the size of the steps the optimizer takes once velocity has built up, even with a fixed learning rate, so increasing momentum usually calls for reducing the learning rate to keep training stable.
What is the difference between momentum and Nesterov momentum?
Classical momentum computes the gradient at the current position before applying accumulated velocity, while Nesterov accelerated gradient first takes a look-ahead step using existing velocity and computes the gradient there, which can make it more responsive to changing curvature (Nesterov, 1983).
What is the difference between momentum and Adam?
Momentum-based SGD only accumulates an exponentially weighted average of gradients. Adam adds a second, independent mechanism, an adaptive per-parameter learning rate driven by a moving average of squared gradients, on top of a momentum-like first moment (Kingma and Ba, 2015).
Does momentum affect only training speed, or also final accuracy?
Momentum primarily affects training dynamics, such as convergence speed and stability, rather than directly determining final accuracy. However, an unstable or poorly tuned momentum setting can prevent a model from reaching a good solution at all, which indirectly affects final accuracy.
When should the momentum buffer be reset?
The momentum buffer is commonly reset after major changes to the training setup, such as unfreezing new layers during fine-tuning, switching to a different dataset, or restarting a learning-rate schedule, so that accumulated velocity from the old regime does not carry over.
Is momentum useful for small datasets?
Momentum can still help on small datasets, but small datasets often produce noisier gradient estimates from batch to batch, so a lower momentum coefficient, or careful monitoring for oscillation, is generally safer than defaulting to a high value.
Can momentum cause training to diverge?
Yes. If the momentum coefficient, the learning rate, or their combination is too high for a given loss surface, the accumulated velocity can push parameter updates far enough to overshoot repeatedly, causing the loss to oscillate or diverge rather than converge.
Do PyTorch and TensorFlow implement momentum the same way?
No. PyTorch's SGD documentation explicitly states its implementation differs from the formulation used by Sutskever et al. (2013) and other frameworks (PyTorch documentation), while Keras documents a different convention where the learning rate is applied directly inside the velocity update (Keras documentation), so hyperparameters should be re-verified when moving between frameworks.
Key Takeaways
Momentum in machine learning accumulates an exponentially weighted average of past gradients into a velocity term, accelerating progress along consistent gradient directions.
The technique traces back to Boris Polyak's 1964 heavy-ball method and was shown to be practically important for training deep and recurrent networks by Sutskever, Martens, Dahl, and Hinton in 2013.
Momentum and the learning rate must be tuned together, since higher momentum effectively increases the optimizer's step size once velocity has accumulated.
Nesterov accelerated gradient evaluates the gradient after a look-ahead step, distinguishing it from classical, heavy-ball style momentum.
PyTorch and Keras/TensorFlow document different, non-identical momentum update conventions, so hyperparameters should not be ported unchanged between frameworks.
Momentum smooths some stochastic-gradient noise and reduces zig-zagging on ill-conditioned loss surfaces, but it does not remove noise entirely and cannot fix an otherwise broken training setup.
Adam is not simply SGD with momentum; it combines a momentum-like first moment with an entirely separate, adaptive per-parameter scaling mechanism.
Optimizer momentum must be distinguished from similarly named concepts, including Adam's moment estimates, batch-normalization momentum, and exponential moving averages of model weights.
Faster convergence from momentum is not the same thing as better generalization, and the best momentum setting depends on the specific task, architecture, and training configuration.
Actionable Next Steps
Establish a no-momentum baseline by training the same model with plain SGD first, so later comparisons have a clear reference point.
Select a starting momentum value, such as 0.9, and justify any deviation from it based on batch size, noise level, or loss-surface behavior observed during initial runs.
Coordinate momentum and learning-rate choices together as a single tuning decision rather than adjusting one while holding the other fixed.
Monitor the loss curve and gradient norm during early training to catch oscillation, divergence, or stagnation quickly.
Run a controlled comparison between classical momentum, Nesterov momentum, and an adaptive optimizer such as Adam or AdamW on a representative subset of the task before committing to one for full-scale training.
Document the exact framework, version, and momentum convention used for any experiment, since PyTorch and Keras/TensorFlow implement the update differently.
Revisit momentum settings whenever batch size, dataset, or model architecture changes materially, rather than assuming previously tuned values still apply.
Glossary
Gradient: A vector showing the direction and rate of steepest increase of a function, used in optimization to determine which way to move parameters to reduce loss.
Gradient descent: An optimization method that repeatedly moves parameters in the opposite direction of the gradient to reduce a loss function.
Stochastic gradient descent: A variant of gradient descent that estimates the gradient from a single training example, or a small batch, rather than the full dataset.
Mini-batch: A small, randomly sampled subset of the training data used to compute one gradient estimate during training.
Learning rate: A hyperparameter that scales how large each parameter update is during gradient-based optimization.
Momentum: A technique that accumulates an exponentially weighted average of past gradients into a velocity term, used to accelerate and smooth gradient-based optimization.
Momentum coefficient: The hyperparameter, often written μ or β, that controls how much of the previous velocity is retained at each step.
Velocity: The accumulated, exponentially weighted direction built from past gradients inside a momentum-based optimizer.
Momentum buffer: The stored velocity value maintained by an optimizer across training steps.
Heavy-ball method: Boris Polyak's 1964 formulation of classical momentum, named for its analogy to a heavy ball rolling with inertia.
Nesterov accelerated gradient: A momentum variant that evaluates the gradient after a look-ahead step using the existing velocity, rather than at the current position.
Oscillation: Repeated back-and-forth movement of parameter updates across a steep dimension of the loss surface, often visible as instability in the loss curve.
Loss surface: The high-dimensional surface formed by plotting a model's loss as a function of its parameters.
Conditioning: A description of how differently a loss surface curves in different directions; poor conditioning means some directions are far steeper than others.
Dampening: A parameter that reduces how much a new gradient contributes to the momentum buffer at each step.
Warm-up: A training technique where the learning rate starts small and gradually increases over the initial steps or epochs of training.
Optimiser: An algorithm that updates a model's parameters during training in order to reduce the loss function.
Exponential moving average: A running average that weights recent values more heavily than older ones, decaying older contributions over time.
First moment: In the Adam optimizer, an exponential moving average of gradients, mechanically similar to classical momentum.
Second moment: In the Adam optimizer, an exponential moving average of squared gradients, used to scale the effective learning rate per parameter.
Weight decay: A regularization technique that shrinks parameter values toward zero during training to reduce overfitting.
Gradient clipping: A technique that caps the size of a gradient before it is used in an update, preventing a single large gradient from destabilizing training.
Convergence: The point at which an optimization process stops making significant further progress toward reducing the loss.
Sources & References
Polyak, B.T. (1964). "Some methods of speeding up the convergence of iteration methods." USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17.
Nesterov, Y. (1983). "A method for solving a convex programming problem with convergence rate O(1/k²)." Soviet Mathematics Doklady, 27, 372–376.
Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). "On the importance of initialization and momentum in deep learning." Proceedings of the 30th International Conference on Machine Learning (ICML), PMLR, Volume 28, pp. 1139–1147. https://proceedings.mlr.press/v28/sutskever13.html
Kingma, D.P. and Ba, J. (2015). "Adam: A Method for Stochastic Optimization." International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1412.6980
Loshchilov, I. and Hutter, F. (2019). "Decoupled Weight Decay Regularization." International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1711.05101
PyTorch documentation. "SGD — torch.optim.SGD." PyTorch Foundation. https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html (Accessed 2026.)
Keras documentation. "SGD." Keras.io. https://keras.io/api/optimizers/sgd/ (Accessed 2026.)
TensorFlow documentation. "tf.keras.optimizers.SGD." TensorFlow. https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD (Accessed 2026.)
