What Is the Adam Optimizer?
- Jul 28
- 30 min read
Training a neural network almost always comes down to one repeated question: given the current error, how should each weight change? Plain gradient descent answers this with a single fixed step size for every parameter, which works poorly when some weights need large corrections and others need tiny ones, or when gradients are noisy from mini-batch sampling. The Adam optimizer was built to solve exactly this problem by giving every parameter its own adaptive step size, computed from a running memory of past gradients. This article explains what Adam is, the math behind it, how to implement and tune it in PyTorch and TensorFlow/Keras, and when a different optimizer might serve you better.
TL;DR
Adam (Adaptive Moment Estimation) is a first-order, gradient-based optimizer that keeps a moving average of past gradients (first moment) and past squared gradients (second moment) for every parameter.
Both moving averages start at zero, which biases early estimates toward zero; Adam corrects this with a bias-correction step before using them.
Because bias correction and per-parameter scaling work together, Adam often performs well with little manual tuning, which is its main practical appeal.
Adam still needs a sensible global learning rate; adaptive scaling changes update size per parameter, but it does not make the learning rate irrelevant.
Its main limitations are extra memory for optimizer state, a documented theoretical convergence counterexample, and cases where a well-tuned SGD recipe generalizes better.
AdamW is not "Adam with L2 regularization" — it decouples weight decay from the gradient-based update, and most modern training recipes now default to AdamW rather than classic Adam.
What Is the Adam Optimizer?
The Adam optimizer is a first-order, gradient-based optimization algorithm used to train machine learning models. It stands for Adaptive Moment Estimation and adjusts each parameter's update individually using moving averages of past gradients and squared gradients. Introduced by Kingma and Ba in 2014, it is widely used because it combines momentum-like behavior with adaptive, per-parameter learning rates.
Table of Contents
Adam Optimizer Definition in Plain English
Adam stands for Adaptive Moment Estimation. It is a first-order optimization method, meaning it uses only the gradient of the loss function — not second derivatives such as curvature — to decide how to update each parameter. "First order" simply means the algorithm looks at the slope of the error surface at the current point, the same information used by plain stochastic gradient descent, rather than relying on a Hessian or other curvature estimate that second-order methods require.
What makes Adam different is that it maintains two running, exponentially weighted averages for every trainable parameter: one that tracks the recent direction of the gradient (the first moment) and one that tracks the recent magnitude of the squared gradient (the second raw moment). Note the wording carefully: the second raw moment is not automatically the same thing as a centered statistical variance, because it is not computed relative to a mean gradient — it is simply an exponential average of squared gradient values, which is closer to an uncentered second moment in the statistical sense.
Using these two averages, Adam adapts the effective step size separately for every parameter. A parameter that has recently received large, consistent gradients gets a smaller effective step; a parameter with small or sparse gradients gets a relatively larger one. Think of it like adjusting your stride on uneven ground: on loose gravel you take smaller, careful steps, and on flat pavement you can stride out further, even though your overall walking effort per step stays similar. That analogy captures the intuition, but the real mechanism is the arithmetic of the moving averages described in later sections, not literal terrain-sensing.
Because this adaptive scaling operates independently for each parameter, Adam is described as invariant to diagonal rescaling of the gradients — a property the original authors highlighted as useful when different parameters naturally live on very different numeric scales, such as the weights of an early convolutional layer versus a final classification layer.
Why Adam Was Developed
The Role of an Optimizer
An optimizer's job is to update model parameters so that a loss function decreases over training. Every optimizer answers the same two questions differently: which direction should each parameter move, and how large should that move be. The choice of optimizer does not change what the model is capable of representing; it changes how efficiently and reliably training finds a good set of parameters.
Problems With a Single Fixed Learning Rate
Plain SGD applies one global learning rate to every parameter. In deep networks, gradient scales vary enormously between layers and even between individual weights, so one fixed rate is often too large for some parameters and too small for others, slowing convergence or causing instability in the parameters that receive unusually large or small gradients relative to the rest of the network.
Stochastic and Noisy Gradients
Mini-batch training produces noisy gradient estimates that change from step to step, because each batch is only a small random sample of the full dataset. A single noisy gradient can point the update in a direction that does not reflect the overall trend of the loss surface, and averaging recent gradients helps filter out this noise before it drives a large, unhelpful step.
Sparse Gradients
Some problems, such as embedding lookups or natural language tasks with large vocabularies, produce sparse gradients where most parameters get zero or near-zero gradient on any given step, because only a handful of tokens or features are active in a given batch. An optimizer needs to handle long stretches of near-zero gradient for a parameter without losing useful history about that parameter's occasional, informative updates.
Non-Stationary Objectives
Deep learning objectives are non-stationary: the loss landscape a parameter experiences early in training looks nothing like the landscape it experiences later, because every other parameter is also changing at the same time. The original Adam paper, Kingma and Ba (2014/2015), specifically designed Adam to remain effective under noisy, sparse, and non-stationary conditions, and reported that its hyperparameters have intuitive interpretations that typically require little tuning across a range of such problems.
Historical Relationship to Momentum, AdaGrad, and RMSProp
Adam builds on two separate lines of prior work. Momentum methods average recent gradients to smooth out noisy updates and carry useful "velocity" through flat or noisy regions of the loss surface. AdaGrad and RMSProp instead divide the learning rate by a running measure of gradient magnitude, so parameters with historically large gradients get smaller effective steps. Adam is often summarized as combining these two ideas, but it is not a naive merger — it applies bias correction to both moving averages, which AdaGrad and plain RMSProp do not, and this correction meaningfully changes early-training behavior, particularly during the first tens to hundreds of steps when the averages have not yet accumulated much history.
The Intuition Behind Adam
Every optimization update has two separable questions: direction and scale. Momentum-style averaging of the raw gradient answers the direction question — it smooths out noisy, conflicting gradient signals into a more consistent path toward lower loss. Averaging the squared gradient answers the scale question — parameters with a history of large gradient magnitude get their updates shrunk, while parameters with small or sparse gradients get theirs enlarged, relative to each other.
Consider two parameters in the same network receiving very different gradient scales: parameter A consistently receives gradients around 2.0 in magnitude, while parameter B receives gradients around 0.02, twenty times smaller. Under a single fixed learning rate, B would barely move while A moves quickly, and training would effectively be bottlenecked by whichever parameter needs the smallest safe step size. Adam's second-moment tracking scales A's updates down and B's updates up relative to each other, so both parameters can make meaningful progress on a comparable timescale, without requiring the practitioner to hand-tune a separate learning rate for every layer.
It is important not to overstate this: "adaptive" describes per-parameter relative scaling, not full self-tuning. Adam still needs a global learning rate, alpha, chosen sensibly for the problem — adaptive scaling changes how that global rate is distributed across parameters, but a poorly chosen alpha can still cause slow convergence if it is too small, or instability and divergence if it is too large, regardless of how well the per-parameter scaling behaves.
Why averaging recent gradients stabilizes direction, and why frequently large gradients lead to different scaling than consistently small ones, both come down to the same underlying arithmetic: an exponential moving average weights recent observations more heavily than old ones, so a parameter's recent gradient history — not its entire training history — determines its current effective step size at any moment.
How the Adam Optimizer Works Step by Step
One complete Adam update, repeated every training step, follows this sequence:
Compute the gradient of the loss with respect to the current parameters using backpropagation.
Update the exponential moving average of the gradient (the first moment), blending the new gradient into the running average.
Update the exponential moving average of the squared gradient (the second moment), blending the new squared gradient into its own running average.
Apply bias correction to both moving averages, since they were initialized at zero and would otherwise understate their true magnitude early in training.
Scale the bias-corrected first moment by the square root of the bias-corrected second moment, plus a small stability constant, to produce the final per-parameter step.
Update the parameter by subtracting the learning rate times that scaled value.
Repeat for the next mini-batch, incrementing the time-step counter by one.
For every trainable parameter, the optimizer stores two extra numbers of the same shape as the parameter itself: the running first moment and the running second moment. This means Adam's optimizer state occupies roughly twice the memory footprint of the model's own parameters — a real, practical memory cost worth remembering for very large models, where optimizer state can become a larger burden on accelerator memory than the model weights themselves.
# Pseudocode for one Adam step
for each parameter theta:
g = gradient(loss, theta) # step 1
m = beta1 * m + (1 - beta1) * g # step 2 (element-wise)
v = beta2 * v + (1 - beta2) * g * g # step 3 (element-wise)
m_hat = m / (1 - beta1 ** t) # step 4
v_hat = v / (1 - beta2 ** t) # step 4
theta = theta - lr * m_hat / (sqrt(v_hat) + eps) # steps 5-6
Adam's Mathematical Formula Explained
Adam's recurrence relations, following the original paper's notation, are:
g_t = gradient of the loss at step t
m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2
m_hat_t = m_t / (1 - beta1^t)
v_hat_t = v_t / (1 - beta2^t)
theta_t = theta_(t-1) - alpha * m_hat_t / (sqrt(v_hat_t) + epsilon)
Here g_t is the gradient at time step t, m_t and v_t are the first and second raw moment estimates, m_hat_t and v_hat_t are their bias-corrected versions, theta_t is the parameter vector, alpha is the learning rate, beta1 and beta2 are the exponential decay rates for the two moving averages, epsilon is a small numerical-stability constant, and t is the time step counter, starting at 1. All operations — squaring, division, and the square root — are element-wise across the parameter vector; there is no matrix inversion or cross-parameter interaction anywhere in the update.
Because m_0 and v_0 are initialized to zero, the raw averages m_t and v_t are biased toward zero during the first several steps, especially when beta1 and beta2 are close to 1. Bias correction divides each average by (1 - beta^t), a factor that starts small and approaches 1 as t grows, which inflates the early estimates back toward their true magnitude and shrinks in effect over time until it becomes negligible after enough steps have accumulated.
Epsilon exists purely to prevent division by zero when v_hat_t is extremely small; its exact placement inside or outside the square root differs slightly between the original paper's Algorithm 1 and some framework implementations, which is why documented epsilon values are not always numerically interchangeable across libraries — Keras and TensorFlow explicitly note that their epsilon corresponds to the "epsilon hat" formulation described just before Section 2.1 of the Kingma and Ba paper, rather than the epsilon written inside Algorithm 1 itself.
Neither m_t nor v_t are exact population moments in a statistical sense. They are exponentially weighted running averages over a finite, decaying window of recent gradients, and the effective size of that window is controlled by beta1 and beta2: a beta2 of 0.999 corresponds to an effective averaging window of roughly one thousand recent steps, while a lower beta2 would shorten that window and make the second moment react more quickly to recent changes in gradient magnitude.
A Worked Numerical Example
Consider a single parameter theta starting at 1.0, with alpha = 0.1, beta1 = 0.9, beta2 = 0.999, and epsilon = 1e-8. Suppose the first gradient computed is g_1 = 0.6.
First moment: m_1 = 0.9 * 0 + 0.1 * 0.6 = 0.06
Second moment: v_1 = 0.999 * 0 + 0.001 * 0.6^2 = 0.001 * 0.36 = 0.00036
Bias-corrected first moment: m_hat_1 = 0.06 / (1 - 0.9^1) = 0.06 / 0.1 = 0.6
Bias-corrected second moment: v_hat_1 = 0.00036 / (1 - 0.999^1) = 0.00036 / 0.001 = 0.36
Update: theta_1 = 1.0 - 0.1 * 0.6 / (sqrt(0.36) + 1e-8) = 1.0 - 0.1 * 0.6 / 0.6 = 1.0 - 0.1 = 0.9
Notice that bias correction recovered m_hat_1 = g_1 and v_hat_1 = g_1^2 exactly at t = 1, which is expected: with only one gradient observed, the corrected estimate should equal that single observation exactly, since there is no history yet to average over.
Adam Hyperparameters and Default Settings
Learning Rate (alpha)
Controls the overall step size. The original paper and most frameworks default to 0.001. Too high causes instability or divergence, visible as loss that oscillates or explodes; too low causes slow convergence, visible as a loss curve that barely moves over many steps. Because alpha interacts with every other hyperparameter, it is usually the first value to search when tuning.
Beta 1
Decay rate for the first-moment (gradient) average. Default 0.9. Higher values produce smoother, more momentum-like direction estimates that react slowly to sudden gradient changes; lower values make the direction estimate track the most recent gradient more closely, at the cost of more noise carrying through into the update.
Beta 2
Decay rate for the second-moment (squared gradient) average. Default 0.999. Higher values make the per-parameter scaling change more slowly over time, effectively averaging over a longer recent window of gradient magnitude; this default is deliberately closer to 1 than beta1 because the second moment is meant to capture a more stable, longer-term picture of each parameter's typical gradient scale.
Epsilon
Small constant preventing division by zero. PyTorch defaults to 1e-8; Keras and TensorFlow default to 1e-7, and Keras documentation notes this corresponds to the "epsilon hat" formulation just before Section 2.1 of the Kingma and Ba paper, not the epsilon inside Algorithm 1 itself. In mixed-precision training, a slightly larger epsilon can help avoid numerical instability when v_hat_t is very small.
Weight Decay
Plain PyTorch Adam defaults weight_decay to 0 (off), applying it in a coupled, L2-style fashion if enabled. AdamW defaults it to 0.01 in PyTorch and 0.004 in Keras, applying it in the decoupled fashion described later in this article. These defaults are not interchangeable, and switching between Adam and AdamW without adjusting weight decay can quietly change regularization strength.
AMSGrad Option
AMSGrad is an optional boolean, off by default in every major framework, that changes how the second moment is used by keeping a running maximum instead of a plain exponential average. It is covered in more detail later in this article.
Gradient Clipping
Not part of Adam itself, but frequently layered on top of it. Clipping caps the norm or value of gradients before the optimizer step, which helps prevent occasional large gradient spikes from producing a destructively large update.
Learning-Rate Schedules
Also not part of Adam itself, but very commonly combined with it. Warmup followed by decay is standard for transformer-style training; simpler step or cosine decay schedules are common elsewhere.
Framework-Specific Options
Modern framework implementations add extra options beyond the original paper, such as PyTorch's fused and foreach implementations for performance, or Keras's exponential moving average utilities, gradient accumulation steps, and loss-scale handling for mixed precision.
Learning rate — controls overall step size — default 0.001 across the original paper, PyTorch, and Keras/TensorFlow.
Beta 1 — first-moment decay — default 0.9 everywhere.
Beta 2 — second-moment decay — default 0.999 everywhere.
Epsilon — numerical stability constant — 1e-8 in the original paper and PyTorch; 1e-7 in Keras/TensorFlow.
Weight decay — L2-style penalty — 0 by default in plain PyTorch Adam; 0.01 by default in PyTorch AdamW; 0.004 by default in Keras AdamW.
AMSGrad — long-term-memory variant toggle — off by default in PyTorch, Keras, and TensorFlow.
These are not framework "errors" — they are legitimate, documented differences in how each library interprets the same underlying algorithm, and mixing framework defaults without checking current documentation is a common source of subtly different training runs between teams working in different frameworks on the same architecture.
Implementing Adam from Scratch
Framework-independent pseudocode was shown earlier. Below is a compact, educational NumPy implementation for a single parameter vector, intended to make the mechanics concrete rather than to replace a tested framework optimizer in production.
import numpy as np
def adam_step(theta, grad, m, v, t, lr=0.001,
beta1=0.9, beta2=0.999, eps=1e-8):
m = beta1 * m + (1 - beta1) * grad
v = beta2 * v + (1 - beta2) * (grad ** 2)
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
return theta, m, v
# usage
theta = np.zeros(5)
m = np.zeros(5)
v = np.zeros(5)
for t in range(1, 101):
grad = compute_gradient(theta) # user-supplied
theta, m, v = adam_step(theta, grad, m, v, t)
State initialization sets m and v to zero arrays matching the parameter shape, exactly mirroring the algorithm's initialize step. The time step t must start at 1, not 0, because the bias-correction formula divides by (1 - beta^t), and t = 0 would divide by zero. All arithmetic above is element-wise, which is why NumPy's vectorized operations work directly across the whole parameter array without an explicit loop over individual weights.
Production code should almost always use a tested framework implementation such as PyTorch or Keras rather than a hand-written version like this one, since those implementations handle edge cases such as sparse gradients, multiple parameter groups, mixed precision, distributed training, and performance optimizations like fused kernels that a small educational example intentionally leaves out for clarity.
Using Adam in PyTorch and TensorFlow/Keras
Using Adam in PyTorch
In PyTorch, the optimizer is created from the model's parameters, and a standard training step calls zero_grad, then backward, then step, in that order: zero_grad clears old gradients so they do not accumulate across batches, backward computes new gradients via autograd by walking the computational graph in reverse, and step applies the Adam update using those fresh gradients. Calling these out of order, or forgetting zero_grad, is one of the most common sources of subtly wrong training runs in PyTorch code.
import torch
model = MyModel()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3,
betas=(0.9, 0.999), eps=1e-8)
for inputs, targets in dataloader:
optimizer.zero_grad()
outputs = model(inputs)
loss = loss_fn(outputs, targets)
loss.backward()
optimizer.step()
Using Adam in TensorFlow/Keras
Keras exposes Adam directly through model.compile, and defaults are usually a reasonable starting point for a first training run:
import tensorflow as tf
model = build_model()
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
loss='categorical_crossentropy',
metrics=['accuracy']
)
model.fit(train_ds, epochs=10, validation_data=val_ds)
A concise GradientTape example for custom training loops, useful when the training step needs logic beyond what model.fit provides:
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
with tf.GradientTape() as tape:
predictions = model(x_batch, training=True)
loss = loss_fn(y_batch, predictions)
grads = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))
Framework Differences to Check
Default epsilon differs: 1e-8 in PyTorch, 1e-7 in Keras/TensorFlow, which can matter in mixed-precision or very-small-gradient settings.
PyTorch's weight_decay argument on the base Adam class behaves as coupled L2-style; use the separate AdamW class, or set decoupled_weight_decay=True on Adam in recent PyTorch versions, for decoupled behavior.
Confirm the current argument names and defaults against official documentation before upgrading framework versions, since optimizer APIs do evolve — for example, newer PyTorch releases added options such as fused and capturable that did not exist in earlier versions.
Advantages of the Adam Optimizer
Often delivers strong out-of-the-box performance across many task types without extensive tuning, which is especially valuable early in a project.
Parameter-wise adaptive scaling helps when different layers or weights need very different effective step sizes, without manual per-layer learning rates.
Handles noisy mini-batch gradients and sparse gradients — for example, in embedding layers or large-vocabulary NLP models — reasonably well.
Frequently needs less manual learning-rate-schedule tuning than plain SGD in many practical settings, though not in every setting.
Scales comfortably to large parameter spaces typical of modern deep networks, including very large transformer models.
Computationally efficient first-order updates with modest per-step overhead relative to second-order methods.
Simple, well-documented APIs in every mainstream framework, which lowers the barrier to correct usage and reduces implementation bugs.
None of these are universal guarantees. "Often," "many," and "frequently" are doing real work in the sentences above — Adam's advantages are empirical tendencies observed across many problems, not mathematical guarantees for every architecture, dataset, or objective, and any specific claim of superiority should be checked against your own validation results rather than assumed from general reputation.
Limitations and Convergence Caveats
Storing two moment estimates per parameter roughly doubles optimizer memory relative to the model's own parameter count, which matters at large scale — for a model with billions of parameters, Adam's optimizer state alone can require several times more accelerator memory than a memory-light optimizer would. Despite adaptive per-parameter scaling, Adam remains sensitive to the global learning rate alpha; a poor choice can still cause slow training or instability, regardless of how well beta1 and beta2 are set.
Reddi, Kale, and Kumar identified a theoretical counterexample — a simple convex optimization setting where Adam provably fails to converge to the optimal solution — and traced the cause to how the exponential moving average of squared gradients can shrink the effective learning rate too aggressively after encountering an informative large gradient, then fail to recover that sensitivity in time to correct course. They proposed AMSGrad, which keeps a running maximum of the second moment instead of a decaying average, as a fix that restores convergence guarantees in that specific setting.
It is important to separate this theoretical counterexample from everyday empirical failure: the counterexample describes a specific constructed scenario, not a claim that ordinary Adam training runs regularly fail to converge in practice. Many successful, widely deployed models were trained with plain Adam. Separately, several published comparisons have found that carefully tuned SGD with momentum can generalize better than Adam on some image-classification benchmarks, which is part of the motivation behind AdamW's decoupled weight decay.
Practical interactions worth watching include batch size, since larger batches change effective gradient noise and often tolerate a larger learning rate; learning-rate schedules, which interact with how quickly the moving averages adapt; regularization strength, which behaves differently under coupled versus decoupled weight decay; gradient clipping, which changes the raw gradient Adam actually sees; and mixed-precision training, where a too-small epsilon or overly aggressive loss scaling can introduce numerical instability that would not appear in full precision.
Adam vs SGD, Momentum, AdaGrad, RMSProp, and Other Optimizers
Plain SGD — updates each parameter by a fixed learning rate times the raw gradient. No per-parameter adaptation, no momentum. Strength: simple, well understood, low memory. Trade-off: sensitive to learning rate choice, slow on ill-conditioned surfaces where different parameters need very different step sizes.
SGD with momentum — adds an exponential moving average of the gradient to smooth the update direction. Strength: faster progress through noisy or shallow regions of the loss surface. Trade-off: still one global learning rate for all parameters, so it does not solve the per-parameter scaling problem.
AdaGrad — divides the learning rate by the accumulated sum of all past squared gradients. Strength: well suited to sparse features, since rarely updated parameters keep a relatively larger effective step. Trade-off: the accumulated denominator only grows, so the effective learning rate can shrink toward zero over long training runs.
RMSProp — replaces AdaGrad's ever-growing sum with an exponential moving average of squared gradients, so old gradients decay out of the average over time. Strength: works well on non-stationary objectives, including recurrent networks. Trade-off: no momentum term of its own, so direction is not smoothed the way Adam's does.
Adam — combines momentum-style gradient averaging with RMSProp-style squared-gradient averaging, plus bias correction for both. Strength: adaptive scaling with smoothed direction and corrected early-step bias. Trade-off: extra memory for two moment estimates per parameter; global learning rate still matters.
AdamW — Adam with weight decay decoupled from the gradient-based update. Strength: better-behaved, more predictable regularization for adaptive optimizers, often improving generalization over plain Adam. Trade-off: weight decay and learning rate both still require tuning, and they should be tuned together rather than in isolation.
AMSGrad — Adam variant using the running maximum of past second moments instead of the exponential average. Strength: restores theoretical convergence guarantees in the Reddi et al. counterexample. Trade-off: does not reliably outperform Adam empirically across the board, and adds a small amount of extra memory to store the running maximum.
Nadam — incorporates Nesterov-style momentum into Adam's update. Strength: can slightly speed convergence on some problems by using a look-ahead estimate of the gradient. Trade-off: an additional variant to validate rather than a guaranteed improvement over plain Adam.
There is no universally best optimizer, because "best" depends on architecture, data characteristics, batch size, and how much tuning budget is available. For rapid experimentation, Adam or AdamW are common defaults because they need less manual tuning to get moving on a new problem. Transformer-style workloads, including the Transformer architecture used across modern language and vision models, very commonly use AdamW with warmup and decay schedules as the near-universal default. Computer vision, especially large-scale image classification, sometimes still favors carefully tuned SGD with momentum for the best final generalization, particularly on well-studied benchmark datasets where such recipes have been refined over years.
Sparse features or embedding-heavy models often benefit from Adam's handling of sparse gradients, since RMSProp-style scaling prevents rarely updated parameters from being effectively frozen. Fine-tuning pretrained models, including vision transformers, typically uses AdamW with a small learning rate to avoid disturbing already-useful pretrained weights too aggressively. When reproducing a published result, matching the paper's exact optimizer, hyperparameters, and schedule matters more than any general optimizer preference, since small configuration differences can meaningfully change the final outcome.
AdamW and Decoupled Weight Decay
Weight decay shrinks parameter values slightly on every update to discourage overly large weights and reduce overfitting. L2 regularization adds a penalty term proportional to the squared weight magnitude directly into the loss function before the gradient is computed. For plain SGD, these two approaches are mathematically equivalent once rescaled by the learning rate — but Loshchilov and Hutter showed in "Decoupled Weight Decay Regularization" that this equivalence breaks down for adaptive optimizers like Adam, because the L2 penalty gets folded into the gradient before Adam's adaptive per-parameter scaling is applied, which distorts the intended regularization strength unevenly across parameters depending on each parameter's own second-moment estimate.
AdamW's core idea is to apply weight decay directly to the parameters, separately from the gradient-based Adam update, rather than adding it to the loss gradient beforehand. This decoupling means the weight decay strength no longer interacts unpredictably with each parameter's adaptive learning rate, so a single weight-decay setting behaves consistently across all parameters rather than being amplified or dampened by each parameter's own gradient history. Loshchilov and Hutter reported that this change substantially improved Adam's generalization performance, making it competitive with tuned SGD with momentum on image-classification benchmarks such as CIFAR-10 and ImageNet32x32, where plain Adam had previously underperformed.
PyTorch exposes this as a separate torch.optim.AdamW class, with a default weight_decay of 0.01, and also as a decoupled_weight_decay=True flag on the base Adam class in recent versions. Keras exposes it as keras.optimizers.AdamW, with a default weight_decay of 0.004. Because these framework defaults differ from each other, and from PyTorch's own AdamW default, weight decay strength should be checked and tuned per project rather than assumed to transfer cleanly between frameworks or between Adam and AdamW.
It is not accurate to call ordinary Adam with an L2 penalty "identical to AdamW" — they are related but mathematically distinct once the optimizer applies adaptive scaling, and the paper's own empirical results are precisely why this distinction is treated as meaningful rather than a minor implementation detail. In practice, most modern training recipes, especially for transformer-based models, now default to AdamW rather than plain Adam.
Adam Variants: AMSGrad, Nadam, AdaMax, and More
AMSGrad
Changes: keeps the maximum of all past second-moment estimates rather than an exponential average that can decay. Why proposed: to fix the theoretical non-convergence example identified by Reddi, Kale, and Kumar in "On the Convergence of Adam and Beyond." Practical reason to consider it: training that appears to stall or oscillate specifically after large, informative gradient spikes. Caveat: it does not reliably beat plain Adam on typical deep-learning benchmarks, so it is a tool to test on your own problem, not a default upgrade to apply blindly.
Nadam
Changes: substitutes Nesterov-style "look-ahead" momentum for Adam's standard momentum term, applying the momentum update slightly earlier in the calculation. Why proposed: Nesterov momentum has shown faster convergence in some momentum-based settings by anticipating where the parameter is heading before computing the gradient step. Practical reason to consider it: modest speed-ups have been reported on certain tasks. Caveat: gains are problem-dependent and should be validated on your own data, not assumed from general reputation.
AdaMax
Changes: replaces the L2-norm-based second moment with an infinity-norm-based estimate, discussed by Kingma and Ba in the original Adam paper itself as an extension. Why proposed: the infinity norm can be more numerically stable in certain embedding-heavy or sparse-gradient settings, since it tracks the maximum recent gradient magnitude rather than an average of squares. Practical reason to consider it: models with occasional very large gradient spikes that might otherwise distort a squared-gradient average. Caveat: less commonly used and less extensively benchmarked than Adam or AdamW across modern architectures.
AdamW
Already covered above: decouples weight decay from the gradient-based update. Why proposed: to fix Adam's regularization inconsistency identified by Loshchilov and Hutter. Practical reason to consider it: any model using explicit weight decay, which describes most modern deep networks, especially transformer-based ones. Caveat: still requires tuning both learning rate and weight decay together, since changing one can shift the effective strength of the other.
When to Use Adam—and When Not To
Reasons to start with Adam or AdamW: fast baselines when you need results quickly and cannot afford an extensive tuning sweep; noisy stochastic training with small batches, where averaged gradients help stabilize the direction of each update; sparse gradients in embedding or NLP-style models; large modern architectures where per-layer manual tuning would be impractical; fine-tuning pretrained checkpoints, where a small, adaptive learning rate helps avoid disturbing useful pretrained weights; and situations where an extensive optimizer-tuning budget is not yet justified because the project is still in an early, exploratory phase.
Reasons to compare against another optimizer: generalization quality is the primary concern and a well-established SGD-with-momentum recipe already exists for the architecture, particularly in well-studied computer-vision benchmarks; optimizer memory overhead is a real bottleneck at scale,; training is unstable even after careful Adam hyperparameter tuning; precise weight-decay behavior matters for the regularization strategy, favoring an explicit comparison between Adam and AdamW; or reproducibility requires matching a published training recipe exactly, in which case the paper's chosen optimizer should simply be replicated.
Cautious language matters here — Adam or AdamW "often" work well as a starting point, and switching "may" help depending on the task, but no blanket rule replaces empirical comparison on your own validation data, run under conditions that match your actual deployment goals.
How to Tune the Adam Optimizer
Establish a baseline using framework default hyperparameters and record the exact framework version and defaults used, so later comparisons have a fixed reference point.
Run a disciplined learning-rate search, typically sweeping across a log scale (for example 3e-5, 1e-4, 3e-4, 1e-3, 3e-3), keeping other hyperparameters fixed while alpha changes.
Add warmup for the first few hundred to few thousand steps if training with large batches or transformer-style architectures, since Adam's early-step updates can be less stable before the moving averages accumulate meaningful history.
Add a decay schedule, such as cosine or step decay, for longer training runs, so the effective learning rate reduces as training approaches convergence.
Adjust beta1 downward from 0.9 only if updates look too noisy to smooth; adjust beta2 downward from 0.999 only if the model needs to react faster to recent gradient changes, such as in highly non-stationary training regimes.
Check epsilon if training in mixed precision or observing NaNs, since a too-small epsilon can cause numerical instability in low-precision arithmetic where very small denominators are more likely to underflow.
Choose AdamW and tune weight decay separately from learning rate rather than assuming a shared default, since the two hyperparameters interact.
Account for batch-size interactions: larger batches generally tolerate larger learning rates, and gradient accumulation should be tuned alongside, not independently of, the learning rate and batch size together.
Apply gradient clipping only when there is a clear justification, such as observed gradient spikes or exploding gradients in recurrent architectures, rather than as a default safety net.
Change one major factor at a time, log every run's exact hyperparameters, and monitor both training and validation curves before drawing conclusions about what helped.
Symptom: loss barely moves → Likely cause: learning rate too low, or gradients vanishing → Adjustment to test: raise alpha on a log scale; check gradient norms directly.
Symptom: loss oscillates wildly → Likely cause: learning rate too high, or epsilon too small in mixed precision → Adjustment to test: lower alpha; increase epsilon.
Symptom: training diverges to NaN → Likely cause: exploding gradients or unstable mixed precision → Adjustment to test: add gradient clipping; verify loss scaling settings.
Symptom: validation lags training badly → Likely cause: overfitting or insufficient regularization → Adjustment to test: switch to AdamW; increase weight decay gradually.
Troubleshooting Adam: Common Problems and Fixes
Before changing any hyperparameter, confirm the basics: check that gradients are actually flowing (not all zero across a whole layer), confirm the loss function and labels are correctly paired and correctly shaped, and verify the learning rate actually reached the optimizer, since schedulers or configuration files are a common source of silent misconfiguration where the intended value never actually takes effect.
Loss is not decreasing — verify gradients are non-zero and the loss function matches the task; then test a higher learning rate on a small subset of data to confirm the model can learn at all.
Loss oscillates — lower the learning rate, or check for a data or normalization bug feeding inconsistent batches into the model.
Training diverges — apply gradient clipping and confirm the learning rate is not simply too large for the current batch size and architecture.
NaN or infinity values — check for division-by-zero-prone custom operations, verify epsilon is not effectively zero, and inspect mixed-precision loss scaling settings.
Validation quality lags training quality — this usually points to overfitting; consider AdamW with tuned weight decay, more data augmentation, or earlier stopping of training.
Updates appear too small — check that the second-moment estimate is not saturating at an unexpectedly large value; verify the learning rate and beta2 are appropriate for the observed gradient scale.
Updates are unstable early in training — add learning-rate warmup so the moving averages accumulate enough history before large steps are taken.
Mixed-precision instability — increase epsilon slightly, or use the framework's automatic mixed-precision utilities, which handle loss scaling more carefully than a manual implementation.
Unexpected behavior after resuming a checkpoint — Adam's internal moment estimates and time-step counter are part of optimizer state, and if they are not restored alongside model weights, training effectively restarts its adaptive scaling from zero, which can cause a visible loss spike right after resuming.
Results differ between frameworks — check for epsilon differences (1e-8 vs 1e-7), coupled versus decoupled weight decay, and any differences in how gradient clipping or mixed precision are applied by default.
Adam Best Practices for Real Projects
Start from an established training recipe for your architecture when one exists, rather than guessing hyperparameters from scratch.
Always save optimizer state alongside model weights in checkpoints, not just the weights themselves, so training can truly resume rather than partially restart.
Record every hyperparameter used for a run, including framework version, so results can be reproduced later by you or by teammates.
Track the exact learning-rate schedule, not just its initial value, since the schedule shape often matters as much as the starting point.
Monitor gradient norms during training when debugging instability, rather than guessing blindly at which hyperparameter to change.
Use gradient clipping only for a clear, observed reason, such as exploding gradients in a recurrent or very deep network.
Keep training loss and validation metrics visually and numerically separate to catch overfitting early, rather than relying on training loss alone.
Compare Adam against AdamW whenever regularization strength matters to the final result, since the two can produce meaningfully different generalization outcomes.
Never assume framework defaults are optimal for your specific architecture and dataset; treat them as a sensible starting point, not a final answer.
Re-run important comparisons across multiple random seeds before concluding one optimizer configuration is better than another, since single-run differences can be noise.
Document the framework and library versions used, since optimizer implementations do change between releases, sometimes in ways that affect numerical results.
Check whether bias terms and normalization parameters, such as LayerNorm or BatchNorm scale and shift, should receive weight decay according to the architecture's reference implementation — many established recipes deliberately exclude them.
Common Myths and Misconceptions About Adam
Myth: "Adam requires no learning-rate tuning." Reality: adaptive per-parameter scaling still operates under a global learning rate that needs a sensible value; a poorly chosen alpha still causes real problems.
Myth: "Adam always trains faster." Reality: faster early progress is common but not universal, and total wall-clock or sample efficiency depends heavily on the specific task and architecture.
Myth: "Adam always generalizes worse than SGD." Reality: some benchmarks show this gap; AdamW substantially narrows or removes it in many of the same settings, and results vary by architecture and dataset.
Myth: "The second moment is simply the gradient variance." Reality: it is an exponential average of squared gradients, not a centered statistical variance computed relative to a mean gradient.
Myth: "Adam and AdamW are the same." Reality: they differ specifically in whether weight decay is coupled into the gradient update or applied separately to the parameters.
Myth: "Bias correction is optional and unimportant." Reality: without it, early-training updates are systematically biased toward zero, which can meaningfully slow the very first steps of training.
Myth: "Adaptive learning rates make schedules unnecessary." Reality: warmup and decay schedules remain common and useful even with Adam, especially for transformer-style training at scale.
Myth: "Default beta values are optimal for every model." Reality: 0.9 and 0.999 are reasonable general-purpose defaults, not universal optima for every architecture and dataset.
Myth: "A theoretical convergence counterexample means Adam never converges in practice." Reality: the counterexample describes one specific constructed scenario; countless successful models have been trained with standard Adam.
FAQ
What does Adam stand for in machine learning?
Adam stands for Adaptive Moment Estimation. It refers to the algorithm's use of adaptive, per-parameter estimates of the first moment (mean) and second raw moment (uncentered squared magnitude) of the gradients to guide each parameter update during training.
Is Adam an optimizer or a loss function?
Adam is an optimizer, not a loss function. The loss function measures how wrong a model's predictions are; the optimizer, like Adam, decides how to change the model's parameters in order to reduce that loss over the course of training.
Why does Adam use first and second moments?
The first moment, a moving average of the gradient, smooths the update direction, similar to momentum. The second moment, a moving average of squared gradients, scales the step size per parameter, similar to RMSProp. Together they combine smoothed direction with adaptive, per-parameter scale.
What is a good learning rate for Adam?
0.001 is the standard default used in the original paper and in PyTorch and Keras, and it is a reasonable starting point for many problems. The right value still depends on architecture, batch size, and task, so a learning-rate search on a log scale is recommended rather than assuming one number fits every project.
What do beta 1 and beta 2 control?
Beta 1 controls the decay rate of the first-moment gradient moving average and defaults to 0.9. Beta 2 controls the decay rate of the second-moment squared-gradient moving average and defaults to 0.999. Higher values make each average change more slowly over time.
Why is bias correction necessary?
Because the moving averages start at zero, early estimates are biased toward zero, especially with beta values close to 1. Bias correction divides each average by one minus beta raised to the power t, which restores an accurate estimate during the first several steps of training.
What is the difference between Adam and AdamW?
Plain Adam folds an L2-style weight-decay penalty into the gradient before applying adaptive scaling, which distorts the effective regularization unevenly across parameters. AdamW decouples weight decay so it is applied directly to the parameters, separately from the adaptive gradient update, which tends to improve generalization.
Is Adam better than SGD?
Neither is universally better. Adam and AdamW often converge faster with less tuning, while carefully tuned SGD with momentum sometimes generalizes better on certain image-classification tasks. The right choice depends on the architecture, dataset, and the tuning budget available for the project.
Can Adam still suffer from vanishing or exploding gradients?
Yes. Adam adapts the step size per parameter, but it does not change how gradients are computed through the network. Extremely deep or poorly initialized networks can still produce vanishing or exploding gradients that Adam alone cannot fully correct.
Should optimizer state be saved in a checkpoint?
Yes, in most cases. Adam's moment estimates and time-step counter are part of its internal state. Restoring only the model weights without this state effectively restarts the adaptive scaling from zero, which can cause a noticeable disruption right after training resumes.
Key Takeaways
Adam adapts each parameter's update individually using moving averages of gradients and squared gradients, rather than applying one fixed step size to the whole model.
Its two moving averages — first moment and second raw moment — are initialized at zero and require bias correction to be accurate during early training.
Adam's adaptive scaling reduces the amount of manual tuning many projects need, but it does not remove the need to choose a sensible global learning rate.
AdamW's decoupled weight decay is now the more common default in modern training recipes, especially for transformer-based models.
A documented theoretical convergence counterexample motivated AMSGrad, but it describes one specific scenario rather than everyday empirical failure.
Careful tuning, monitoring, and saving of optimizer state, not just model weights, matter as much as the choice of optimizer itself.
No single optimizer is best for every architecture, dataset, and objective; comparison against alternatives like SGD with momentum remains worthwhile for many projects.
Actionable Next Steps
Establish an Adam or AdamW baseline using your framework's current default hyperparameters.
Record the exact framework version, defaults, and hardware used for that baseline run.
Run a disciplined, log-scale learning-rate search around the default before changing anything else.
Add a warmup period and a decay schedule if training a large or transformer-style model.
Monitor training loss, validation loss, and gradient norms together, rather than in isolation from each other.
Save both model weights and optimizer state in every checkpoint you intend to resume from later.
Compare your chosen optimizer against at least one credible alternative, such as SGD with momentum or the other of Adam and AdamW.
Document the final recipe, including every hyperparameter, so the result can be reproduced later by you or a teammate.
Glossary
Optimizer: The algorithm that updates a model's parameters during training in order to reduce the loss function step by step.
Objective function: The function an optimizer tries to minimize or maximize during training; in supervised learning this is usually the loss function itself.
Loss function: A function that measures how far a model's predictions are from the correct answers, giving the optimizer something concrete to reduce.
Gradient: The vector of partial derivatives of the loss with respect to each parameter, indicating the direction of steepest increase in the loss.
Stochastic gradient: A gradient estimated from a small random subset, or mini-batch, of the training data rather than the full dataset at once.
Learning rate: A hyperparameter controlling how large each parameter update step is, usually denoted alpha in Adam's equations.
Momentum: A technique that averages recent gradients to smooth the direction of parameter updates and carry velocity through noisy regions.
Exponential moving average: A running average that weights recent values more heavily than older ones, using a fixed decay rate close to but less than 1.
First moment: In Adam, the exponential moving average of the raw gradient, used to smooth the update direction over recent steps.
Second raw moment: In Adam, the exponential moving average of the squared gradient, used to scale the update size differently for each parameter.
Bias correction: An adjustment applied to Adam's moving averages to counteract their zero initialization during the earliest training steps.
Epsilon: A small constant added to Adam's denominator to prevent division by zero when the second moment estimate is very small.
Parameter-wise learning rate: An effective learning rate that differs for each individual parameter, based on that parameter's own recent gradient history.
Weight decay: A regularization technique that shrinks parameter values slightly on every update to discourage overly large weights.
L2 regularization: A penalty added to the loss function proportional to the squared magnitude of the model's weights, encouraging smaller values.
AdamW: A variant of Adam that applies weight decay directly to parameters, decoupled from the adaptive gradient-based update.
AMSGrad: A variant of Adam that uses the running maximum of past second-moment estimates instead of a decaying exponential average.
Epoch: One complete pass through the entire training dataset, often consisting of many individual mini-batch steps.
Mini-batch: A small subset of the training data used to compute one gradient estimate and one optimizer update at a time.
Gradient clipping: A technique that caps the magnitude of gradients before the optimizer update, used to prevent exploding gradients.
Learning-rate schedule: A rule that changes the learning rate over the course of training, such as warmup followed by gradual decay.
Optimizer state: The internal values an optimizer maintains between steps, such as Adam's first and second moment estimates and its time-step counter.
Sources & References
Kingma, D. P., & Ba, J.. Adam: A Method for Stochastic Optimization. arXiv preprint. First posted Dec 2014; latest revision Jan 2017.
Reddi, S. J., Kale, S., & Kumar, S.. On the Convergence of Adam and Beyond. International Conference on Learning Representations (ICLR) 2018. 2018 (arXiv posting 2019).
Loshchilov, I., & Hutter, F.. Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR) 2019. First posted Nov 2017; ICLR version Jan 2019.
PyTorch documentation. torch.optim.Adam. PyTorch (Linux Foundation). Accessed 2026.
PyTorch documentation. torch.optim.AdamW. PyTorch (Linux Foundation). Accessed 2026.
Keras documentation. Adam optimizer. Keras.io. Accessed 2026.
Keras documentation. AdamW optimizer. Keras.io. Accessed 2026.
