top of page
Search

What Is the AdamW Optimizer?

  • 1 hour ago
  • 26 min read
AdamW optimizer visualization with deep learning equations and training charts.

Pick the wrong weight-decay setting on Adam and a model can look fine in training loss while quietly overfitting or under-regularizing in ways that are hard to trace back to the optimizer. AdamW exists because the popular fix for that problem — adding an L2 penalty to the loss — behaves differently once Adam's adaptive scaling gets involved, and decoupling weight decay from that scaling turned out to matter for how well models trained with Adam actually generalize.


TL;DR


  • AdamW is Adam with weight decay applied directly to the weights, separately from the gradient-based adaptive update, instead of folded into the loss as an L2 penalty.

  • The "W" stands for weight decay — decoupled weight decay, to be precise.

  • Loshchilov and Hutter introduced AdamW in "Decoupled Weight Decay Regularization" (ICLR 2019) after showing that L2 regularization and weight decay, which are mathematically equivalent for plain SGD, are not equivalent once Adam's per-parameter adaptive scaling is involved.

  • PyTorch's torch.optim.AdamW and Keras's keras.optimizers.AdamW both implement decoupled weight decay natively, with different default weight-decay values (0.01 in PyTorch, 0.004 in Keras) — always check the current docs rather than assuming one number applies everywhere.

  • AdamW is a common default for training transformer-based models, but it is not a universal replacement for SGD or plain Adam; the right optimizer still depends on the architecture, the training budget, and the regularization the task needs.


What Is AdamW in One Paragraph?


AdamW is a variant of the Adam optimizer used to train neural networks. It keeps Adam's adaptive, per-parameter learning rates but applies weight decay directly to the model's weights instead of adding it to the loss gradient. This decoupling, introduced by Ilya Loshchilov and Frank Hutter in 2019, makes regularization behave more predictably and is now a standard choice for training transformer models.





Table of Contents



What Is the AdamW Optimizer?


AdamW is an optimization algorithm used to train neural networks. It is a modified version of Adam (Adaptive Moment Estimation), one of the most widely used optimizers in deep learning (Kingma and Ba, 2015).


The "W" in AdamW stands for weight decay. Specifically, it refers to decoupled weight decay: a way of shrinking model weights toward zero during training that is applied directly to the parameters, separately from the gradient-based update that Adam computes from the loss.


In plain Adam, if you want L2-style regularization, the common approach is to add a penalty term to the loss function. That penalty then flows through the same gradient computation as everything else, which means it gets scaled by Adam's adaptive, per-parameter learning rates before it reaches the weights. AdamW skips that step. It computes the regular Adam update from the loss gradient, and then, as a separate operation, shrinks each weight by a fraction of itself. The two operations no longer share the same adaptive scaling.


AdamW was introduced by Ilya Loshchilov and Frank Hutter in the paper "Decoupled Weight Decay Regularization", published at ICLR 2019. Since then it has become a standard optimizer for training transformer-based language and vision models, and it ships as a built-in optimizer in PyTorch, Keras, and most other major deep learning frameworks.


Why Was AdamW Created?


Adam became popular quickly after its 2015 introduction because it converges fast and needs relatively little tuning. But practitioners noticed something odd: models trained with plain stochastic gradient descent (SGD) plus a weight-decay term often generalized to new data at least as well as, and sometimes better than, models trained with Adam plus an equivalent-looking L2 penalty, even when Adam converged faster on the training set.


Loshchilov and Hutter's 2019 paper traced this gap to how L2 regularization interacts with Adam's per-parameter adaptive learning rates. For plain SGD, adding an L2 penalty to the loss and directly decaying the weights by a fixed fraction each step are mathematically the same operation, under a particular scaling. For Adam, they are not. Because Adam divides each parameter's update by a running estimate of that parameter's gradient magnitude, an L2 penalty folded into the gradient gets inflated for parameters with small typical gradients and shrunk for parameters with large ones. The regularization strength ends up depending on training dynamics in a way the practitioner did not choose and usually cannot predict in advance.


The paper's proposed fix, decoupled weight decay, separates the regularization step from Adam's adaptive gradient step so that the amount of decay applied to a weight depends only on the weight itself and the chosen weight-decay coefficient, not on the optimizer's internal gradient statistics for that parameter. The authors reported that this decoupling improved generalization and made weight decay and learning rate easier to tune independently, across several image-classification benchmarks — a result that should be read as evidence for their reported setups, not as a universal guarantee for every architecture and dataset.


A Quick Refresher on How Adam Works


Adam adapts the learning rate for every parameter individually by tracking two running statistics of the gradient: a first-moment estimate (roughly, the recent average direction of the gradient) and a second-moment estimate (roughly, the recent average magnitude of the gradient, squared). Below is the core idea, kept conceptual and mapped to code later in this article.


Gradients and the two moment estimates


At each training step t, Adam computes the gradient g_t of the loss with respect to each parameter, then updates two exponential moving averages:


  • First moment: m_t = β1 · m_(t-1) + (1 − β1) · g_t — a smoothed estimate of the gradient's direction, similar in spirit to momentum.

  • Second moment: v_t = β2 · v_(t-1) + (1 − β2) · g_t² — a smoothed estimate of the gradient's squared magnitude, used to scale down updates for parameters whose gradients are consistently large.


β1 and β2 are decay rates between 0 and 1 that control how much weight recent gradients get relative to older ones. Common values are β1 = 0.9 and β2 = 0.999, meaning the second-moment estimate changes slowly and mostly reflects long-run gradient behavior for that parameter.


Bias correction


Because m_t and v_t start at zero, they are biased toward zero in the earliest steps of training. Adam corrects for this with bias-corrected estimates, m̂_t = m_t / (1 − β1^t) and v̂_t = v_t / (1 − β2^t), which inflate the early estimates back toward their true magnitude and shrink toward the raw values as t grows.


The adaptive update


Adam then updates each parameter θ using θ ← θ − η · m̂_t / (√v̂_t + ε), where η is the learning rate and ε is a small constant, typically around 1e-8, added only to prevent division by zero. Because v̂_t is tracked separately for every parameter, parameters with historically large gradients get smaller effective steps, and parameters with small gradients get relatively larger ones — this is what "adaptive learning rate" means in Adam.


This refresher covers what is needed to understand AdamW; for the full derivation and convergence analysis, see the original Adam paper.


L2 Regularization vs. Weight Decay


This distinction is the entire reason AdamW exists, so it is worth being precise about it.


L2 regularization


L2 regularization adds a penalty term to the loss function proportional to the sum of squared weights: L_total = L_original + (λ/2) · ||θ||². When you differentiate this new loss to get the gradient used in training, the penalty contributes an extra term equal to λ·θ to the gradient of every parameter. That extra term then goes through whatever optimizer you are using, exactly like the rest of the gradient.


Weight decay


Weight decay, as originally used with plain SGD, is usually described as directly shrinking each weight by a small fraction every step: θ ← (1 − λ) · θ, applied alongside — but not mixed into — the ordinary gradient step. For plain SGD without momentum, adding an L2 penalty to the loss and applying this direct shrinkage produce the same parameter updates, provided the weight-decay coefficient is scaled correctly relative to the learning rate. That equivalence is where the common (and, for adaptive optimizers, misleading) idea that "L2 regularization and weight decay are the same thing" comes from.


Where the equivalence breaks for Adam


Adam does not apply gradients directly — it divides each parameter's gradient-based update by that parameter's own adaptive scale, √v̂_t. If you fold an L2 penalty into the gradient before this division happens, the effective amount of regularization each parameter receives gets divided by that parameter's adaptive scale too. Parameters with small, consistent gradients — which is common for large embedding matrices or normalization parameters — end up regularized more strongly than parameters with large, noisy gradients, and the practitioner has no direct, transparent way to correct for this because it depends on training dynamics.


AdamW's decoupled weight decay applies the θ ← (1 − η·λ) · θ shrinkage after Adam's adaptive update is computed, so the amount of decay a weight receives is governed by the weight-decay coefficient and the current learning rate — not by that weight's gradient history. This does not make weight decay and L2 regularization "equivalent for Adam"; it makes them two genuinely different mechanisms, and AdamW deliberately picks the one whose behavior is easier to reason about and tune.


Note: not every framework's weight_decay parameter behaves the same way. A parameter with that name can mean an L2 penalty on the loss (as in plain torch.optim.Adam) or true decoupled decay (as in torch.optim.AdamW). Always check the specific optimizer's documentation rather than assuming the name tells you the mechanism.

How AdamW Works Step by Step


At a high level, each AdamW training step does the following for every trainable parameter:


  1. Compute the gradient g_t of the loss with respect to the parameter, using the current mini-batch.

  2. Update the first-moment estimate m_t and second-moment estimate v_t using β1 and β2, as in standard Adam.

  3. Compute the bias-corrected estimates m̂_t and v̂_t.

  4. Compute the adaptive step: adaptive_update = m̂_t / (√v̂_t + ε).

  5. Apply the adaptive step to the parameter: θ ← θ − η · adaptive_update.

  6. Apply weight decay as a separate operation on the same parameter: θ ← θ − η · λ · θ (equivalently written θ ← (1 − η·λ) · θ).


Steps 5 and 6 are both scaled by the same learning rate η in the formulation above, which is how Loshchilov and Hutter originally proposed it and how PyTorch's implementation works — the decay update is deliberately computed outside the √v̂_t division that makes step 5 adaptive, but it still moves in step with the learning rate schedule unless the framework offers a way to decouple that too. This is why changing your learning rate schedule can change the effective strength of weight decay even when the weight_decay coefficient itself is untouched — something worth checking when comparing runs.


AdamW Equations Explained


The equations below collect the update rule described conceptually above, with every symbol defined. Notation follows the original Adam and AdamW papers closely; exact operator ordering can vary slightly between framework implementations, so treat this as the canonical form rather than byte-for-byte pseudocode for any one library.


m_t = beta1 * m_(t-1) + (1 - beta1) * g_tv_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2m_hat_t = m_t / (1 - beta1^t)v_hat_t = v_t / (1 - beta2^t)theta <- theta - eta * m_hat_t / (sqrt(v_hat_t) + epsilon)   # adaptive updatetheta <- theta - eta * lambda * theta                        # decoupled weight decay

Symbol

Meaning

θ (theta)

A model parameter (weight) being trained.

g_t

The gradient of the loss with respect to θ at step t.

m_t

First-moment estimate — a smoothed running average of g_t.

v_t

Second-moment estimate — a smoothed running average of g_t squared.

β1, β2 (beta1, beta2)

Decay rates controlling how quickly m_t and v_t forget old gradients.

η (eta)

The learning rate.

λ (lambda)

The weight-decay coefficient.

ε (epsilon)

A small constant added for numerical stability, preventing division by zero.


The key structural point is the last line: the weight-decay term θ ← θ − η·λ·θ never passes through the √v̂_t denominator that makes the update on the line above it adaptive. That is the entire mechanical difference between AdamW and "Adam plus L2 regularization," expressed in one line of algebra.


AdamW vs. Adam: What Is the Difference?


Adam and AdamW share the same first-moment and second-moment tracking and the same adaptive update rule. They differ only in how — and whether — weight decay is applied.


Aspect

Adam (with L2 penalty)

AdamW

Adaptive moments

Same m_t / v_t tracking

Same m_t / v_t tracking

Weight decay treatment

L2 penalty added to the loss gradient

Applied directly to weights, outside the gradient

Interaction with adaptive scaling

Regularization strength is divided by √v̂_t, so it varies per parameter

Regularization strength depends only on λ and η, not on gradient history

Hyperparameter meaning

weight_decay in plain Adam typically means an L2 coefficient

weight_decay in AdamW means a decoupled decay coefficient

Typical use

Still common; some codebases use plain Adam with weight_decay = 0 and no regularization

Common default for transformer training and general-purpose regularized training


AdamW is not universally better than Adam. When no weight decay is used at all (weight_decay = 0 in both), the two optimizers produce identical updates — the distinction only matters once regularization enters the picture. Whether decoupled decay helps a given model depends on the architecture, dataset size, and training budget; the original paper's results were strongest on the image-classification benchmarks it tested, and later results across NLP and vision tasks have been broadly, but not universally, favorable to decoupling.


AdamW vs. SGD With Momentum


SGD with momentum remains a strong, sometimes preferred, choice for many computer-vision architectures, particularly when training budgets are large enough to tune a learning-rate schedule carefully.


Aspect

AdamW

SGD with Momentum

Adaptivity

Per-parameter adaptive learning rates

Single global learning rate (shared momentum buffer, not per-parameter scaling)

Momentum

Built into the first-moment estimate

Explicit momentum term, typically 0.9

Learning-rate sensitivity

Often more forgiving of a suboptimal learning rate early on

Often needs a carefully tuned schedule to match Adam-family results

Weight decay

Decoupled, via λ

Direct parameter shrinkage; well understood and easy to reason about

Optimizer state

Two extra tensors per parameter (m and v)

One extra tensor per parameter (momentum buffer)

Common use cases

Transformers, fine-tuning, rapid prototyping

Some CNN training pipelines, settings with long, well-tuned schedules


AdamW's existence did not make SGD obsolete. Some well-known image-classification results still favor SGD with a carefully tuned step-decay or cosine schedule, particularly at large training budgets, and SGD's memory footprint is smaller since it tracks one running statistic per parameter instead of two. AdamW tends to win on ease of getting a reasonable result without extensive learning-rate-schedule tuning, which is part of why it is common in settings — like fine-tuning a pretrained transformer — where practitioners want fast, dependable convergence more than they want the last fraction of a percent of benchmark accuracy.


AdamW Hyperparameters Explained


  • Learning rate (η): Controls the overall step size. Too high causes divergence or instability; too low slows convergence. This is the hyperparameter most worth tuning first.

  • Weight decay (λ): Controls how strongly weights are pulled toward zero each step. Higher values regularize more strongly but can underfit if pushed too far.

  • β1 (beta1): Decay rate for the first-moment (gradient direction) estimate. A common default is 0.9; lower values make the optimizer react faster to recent gradients.

  • β2 (beta2): Decay rate for the second-moment (gradient magnitude) estimate. A common default is 0.999; some transformer training recipes lower this (for example to 0.95–0.98) for better stability with noisy gradients.

  • ε (epsilon): A small constant preventing division by zero. Rarely tuned, but very small ε values can occasionally cause instability in mixed-precision training, in which case a larger ε (e.g., 1e-6 instead of 1e-8) is a common fix.

  • AMSGrad: An optional variant, from Reddi, Kale, and Kumar's "On the Convergence of Adam and Beyond" (ICLR 2018), that keeps a running maximum of v_t instead of the plain exponential average, intended to address a theoretical non-convergence issue in some Adam settings. It is available as a flag in both PyTorch's and Keras's AdamW implementations but is not the default in either.


Framework defaults are convenient starting points, not universal recommendations. As of current documentation, PyTorch's torch.optim.AdamW defaults to lr = 0.001, betas = (0.9, 0.999), eps = 1e-8, and weight_decay = 0.01, while Keras's keras.optimizers.AdamW defaults to learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-7, and weight_decay = 0.004. Notice the weight-decay defaults differ by roughly 2.5x between the two libraries — a reminder to check the value your framework actually uses rather than assuming it matches what you last read elsewhere.


How to Tune Learning Rate and Weight Decay


There is no single correct learning rate or weight-decay value for AdamW — the right values depend on the model architecture, batch size, dataset size, training length, and whatever learning-rate schedule you pair the optimizer with. What follows is a process, not a lookup table.


  1. Start from a documented baseline close to your setup — a paper training a similar architecture on similar data, or your framework's example scripts — rather than a generic default.

  2. Tune the learning rate first, holding weight decay fixed. A short learning-rate range test (increasing the learning rate over a few hundred steps and watching where loss stops improving) is a fast way to bound a reasonable range.

  3. With a learning rate that trains stably, tune weight decay next. Increase it if you see a widening gap between training and validation loss; decrease it if training loss will not fall enough or validation loss tracks training loss closely but both are high (a sign of underfitting).

  4. Evaluate on a held-out validation set, not training loss alone — weight decay's whole purpose is to trade some training performance for better generalization.

  5. Add or adjust a learning-rate schedule (warmup plus decay) once the base learning rate and weight decay look reasonable; schedules interact with both.

  6. Watch for underfitting (both training and validation loss stay high) versus overfitting (training loss keeps falling while validation loss rises or plateaus), and adjust weight decay and model capacity accordingly.

  7. Change one major variable at a time. Adjusting the optimizer, the schedule, and the model architecture simultaneously makes it impossible to attribute a result to any one change.

  8. For conclusions that matter, repeat runs across a few random seeds — a single run's result can be noise, especially for smaller datasets or shorter training budgets.


For fine-tuning a pretrained model, both the learning rate and weight decay are typically set lower than for training from scratch, since large steps risk erasing what the pretrained weights already encode. Exact values are architecture- and dataset-specific enough that this article won't assert a single universal number.


Which Parameters Should You Exclude From Weight Decay?


Many training recipes exclude certain parameters from weight decay entirely, using separate parameter groups. This is a common convention, not a mathematical requirement of AdamW itself.


  • Bias terms: Bias parameters are low-dimensional and, unlike weight matrices, don't multiply the input — shrinking them toward zero is often judged to add little regularization benefit while potentially limiting the model's ability to fit an appropriate offset.

  • LayerNorm and BatchNorm scale (and bias) parameters: These normalization parameters directly rescale activations. Decaying them toward zero can conflict with the normalization layer's role, so many transformer training recipes exclude them.

  • Other small or special parameters: Embedding tables are sometimes treated differently across recipes; there is no single convention here, so check what a given codebase does rather than assuming.


This is a convention popularized by influential training recipes (including several transformer pretraining setups), not a universal rule every model must follow — some codebases apply weight decay uniformly and still train successfully. Below is a PyTorch pattern for setting this up with explicit parameter groups rather than fragile name-based string matching alone:


decay_params, no_decay_params = [], []for name, param in model.named_parameters():    if not param.requires_grad:        continue    # Exclude 1-D parameters: biases and norm scale/bias terms    if param.ndim <= 1 or name.endswith('.bias'):        no_decay_params.append(param)    else:        decay_params.append(param)optimizer = torch.optim.AdamW(    [        {'params': decay_params, 'weight_decay': 0.01},        {'params': no_decay_params, 'weight_decay': 0.0},    ],    lr=3e-4,)

Filtering on param.ndim (1-dimensional tensors are almost always biases or norm parameters) is more robust than matching only on parameter names, since naming conventions vary between model implementations. Still, always print out and sanity-check which parameters landed in which group before a long training run.


AdamW in PyTorch


PyTorch has shipped torch.optim.AdamW as a built-in optimizer for several years, implementing the decoupled weight decay from the original paper. As of current PyTorch documentation, its signature defaults to lr = 0.001, betas = (0.9, 0.999), eps = 1e-8, weight_decay = 0.01, and amsgrad = False.


Basic example


import torchmodel = MyModel()optimizer = torch.optim.AdamW(    model.parameters(),    lr=3e-4,    betas=(0.9, 0.999),    eps=1e-8,    weight_decay=0.01,)for batch in dataloader:    optimizer.zero_grad()    loss = compute_loss(model, batch)    loss.backward()    optimizer.step()

Parameter groups (selective weight decay)


See the code example in the previous section for a complete parameter-group pattern that excludes biases and normalization parameters from weight decay while applying it to everything else.


Using AdamW with a learning-rate scheduler


from torch.optim.lr_scheduler import LambdaLRwarmup_steps = 1000total_steps = 100000def lr_lambda(step):    if step < warmup_steps:        return step / max(1, warmup_steps)    progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)    return max(0.0, 1.0 - progress)  # linear decay after warmupscheduler = LambdaLR(optimizer, lr_lambda)for step, batch in enumerate(dataloader):    optimizer.zero_grad()    loss = compute_loss(model, batch)    loss.backward()    optimizer.step()    scheduler.step()

Note that recent PyTorch versions also added a decoupled_weight_decay flag directly to torch.optim.Adam, letting Adam behave like AdamW without switching classes. Whether that flag is present, and its exact default, depends on your installed PyTorch version — check torch.__version__ and the matching documentation before relying on it.


AdamW in Keras and TensorFlow


Keras provides keras.optimizers.AdamW as a first-class, built-in optimizer implementing decoupled weight decay. As of current Keras documentation, it defaults to learning_rate = 0.001, weight_decay = 0.004, beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-7, and amsgrad = False.


import kerasoptimizer = keras.optimizers.AdamW(    learning_rate=3e-4,    weight_decay=0.004,    beta_1=0.9,    beta_2=0.999,    epsilon=1e-7,)model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])model.fit(train_dataset, epochs=10, validation_data=val_dataset)

This differs from older tensorflow_addons.optimizers.AdamW patterns, which predate AdamW's inclusion as a built-in Keras optimizer; TensorFlow Addons is no longer the recommended path now that tf.keras.optimizers.AdamW ships natively. If a codebase you're referencing still imports from tensorflow_addons, treat that as a signal to check whether a native replacement is available before copying the pattern. Note also that plain keras.optimizers.Adam accepts an optional weight_decay argument (defaulting to None) that, when set, applies AdamW-style decoupled decay rather than an L2 penalty — so on current Keras, Adam and AdamW can be made to behave the same way; verify this against the version you have installed, since optimizer internals do change between releases.


Why Is AdamW Common in Transformer Training?


Transformer models became popular for sequence tasks starting with "Attention Is All You Need" (Vaswani et al., 2017), which itself used the original Adam optimizer, not AdamW — AdamW was published two years later. As decoupled weight decay gained traction after 2019, it was adopted widely across transformer pretraining and fine-tuning recipes for practical reasons rather than any formal requirement that transformers must use it.


  • Adaptive optimization suits transformers' large, heterogeneous parameter spaces — embedding tables, attention projections, and feed-forward layers can have very different gradient scales, and Adam's per-parameter scaling handles that without manual tuning per layer.

  • Regularization matters more as models scale. Large transformer models are prone to overfitting on limited fine-tuning data, and decoupled weight decay gives a more predictable regularization knob than an L2 penalty would under Adam's adaptive scaling.

  • Pretrained transformer fine-tuning workflows favor optimizers with dependable, fast convergence over a small number of epochs, which is where AdamW tends to be practical.

  • AdamW paired well with the warmup-plus-decay learning-rate schedules that became standard for transformer training, discussed in the next section.


It is not accurate to say every transformer model uses AdamW — some large-scale training runs use other optimizers (including Adafactor, LAMB, or more recent alternatives designed to reduce optimizer memory at scale), and the field continues to explore alternatives. AdamW is best described as a strong, widely adopted default rather than a technical requirement.


Learning-Rate Schedules, Warmup, and AdamW


The optimizer and the learning-rate schedule are separate concepts that are easy to conflate. AdamW defines how a given learning rate is turned into a parameter update; the schedule defines how that learning rate changes over the course of training. Swapping one without adjusting the other is a common source of confusing results.


  • Linear warmup: The learning rate ramps up from a small value (often zero) to its target value over the first few hundred or thousand steps. This is common with AdamW because early in training, before the second-moment estimates v_t have stabilized, large steps can be unstable.

  • Cosine decay: The learning rate follows a cosine curve down to a small value (or zero) over the remaining training steps after warmup. Common in transformer pretraining.

  • Linear decay: The learning rate decreases linearly to zero (or a floor value) after warmup — used in several well-known fine-tuning recipes.

  • Constant schedules: The learning rate stays fixed, sometimes used for short fine-tuning runs, though warmup is still frequently kept even here.


Because AdamW's decoupled weight-decay term is typically scaled by the same learning rate η as the adaptive update (in the formulation used by most current framework implementations), changing the learning-rate schedule changes the effective weight decay applied at each step, even if the weight_decay coefficient itself never changes. This is a genuinely important, easy-to-miss detail: two runs with identical weight_decay values but different schedules are not applying identical regularization over the course of training.


Common AdamW Mistakes and Debugging


  • Confusing L2 regularization with decoupled weight decay: These are different mechanisms once Adam's adaptive scaling is involved. Fix: check whether your optimizer's weight_decay parameter is documented as an L2 penalty or as decoupled decay before assuming a value carries over from one to the other.

  • Copying hyperparameters from an unrelated model: A weight-decay value tuned for a small CNN is not a safe default for a large transformer, or vice versa. Fix: start from a baseline that matches your architecture and data scale as closely as possible.

  • Applying weight decay blindly to every parameter: Decaying biases and normalization scales toward zero can hurt training in ways that are easy to miss. Fix: use parameter groups, as shown earlier, and inspect which parameters landed in which group.

  • Forgetting to tune the learning rate: Weight decay only matters once the learning rate is in a reasonable range; tuning decay before the learning rate wastes experiments. Fix: tune learning rate first, as described in the tuning section above.

  • Using excessive weight decay: Very large weight-decay values can prevent the model from fitting the training data at all. Fix: watch training loss — if it won't fall even with a small learning rate, try reducing weight decay before assuming a bug.

  • Assuming optimizer defaults are identical across frameworks: PyTorch and Keras ship different default weight-decay values for AdamW, as noted earlier. Fix: always check the specific version's documentation.

  • Changing the optimizer and the scheduler at the same time while debugging: This makes it impossible to tell which change caused a result. Fix: change one variable at a time.

  • Comparing Adam and AdamW without controlling other variables: A fair comparison holds the learning-rate schedule, weight-decay coefficient (adjusted for the mechanism), and everything else constant. Fix: design the comparison explicitly before running it.

  • Relying on outdated library examples: Old tutorials may reference deprecated APIs like tensorflow_addons.optimizers.AdamW. Fix: cross-check against the current official documentation for your installed version.

  • Assuming poor training is automatically an optimizer problem: Data quality, learning-rate scale, model capacity, and initialization all cause symptoms that can look like a bad optimizer choice. Fix: rule out the simpler explanations first.


When Should You Use AdamW?


AdamW is a reasonable default in several common situations:


  • Training or fine-tuning transformer-based models, where it is the most widely used starting point in current practice.

  • Transfer learning and fine-tuning generally, where fast, dependable convergence from a pretrained checkpoint matters more than squeezing out the last fraction of accuracy through extensive schedule tuning.

  • Situations where explicit, predictable regularization is desired and you want to tune weight decay independently of the learning rate.

  • Rapid prototyping, where Adam-family optimizers' relative forgiveness of an imperfect learning rate speeds up iteration.


Other optimizers may be reasonable too. Plain Adam without weight decay is fine when no regularization is needed at all. SGD with momentum remains competitive, and in some published results superior, for certain convolutional architectures given a well-tuned schedule and enough training budget. Newer optimizers designed for very large-scale training (targeting lower memory overhead or better scaling behavior) are active areas of research and may outperform AdamW in specific large-model settings. The right choice depends on your architecture, your compute and time budget for tuning, and what the literature reports for similar setups — not on a blanket rule.


Advantages, Limitations, and Trade-Offs


Advantages


  • Decoupled weight decay gives a regularization knob whose effect is easier to reason about and tune independently of the learning rate.

  • Adaptive per-parameter learning rates make AdamW relatively forgiving of an imperfectly chosen base learning rate, which speeds up early experimentation.

  • Broad framework support — AdamW ships natively in PyTorch, Keras, and most major libraries, so switching to it rarely requires custom code.

  • Strong track record as a practical default across a wide range of modern architectures, particularly transformers.


Limitations and trade-offs


  • Additional optimizer state: AdamW tracks two extra tensors per parameter (m and v), roughly doubling optimizer memory compared to SGD with momentum — a real constraint at very large model scales.

  • Still requires tuning: Decoupling weight decay from the adaptive update makes it easier to tune, not automatic; learning rate, weight decay, and schedule still need deliberate experimentation.

  • No universal superiority: AdamW is not guaranteed to outperform SGD or plain Adam on every architecture or dataset; published comparisons are mixed depending on the setup.

  • Behavior depends on the overall training recipe: The interaction between weight decay and the learning-rate schedule (discussed earlier) means AdamW's effective regularization is a property of the whole training setup, not the optimizer in isolation.


A Practical AdamW Tuning Recipe


A concrete starting workflow for training or fine-tuning a model with AdamW, assuming you already have a working training loop and a validation set:


  1. Establish a baseline: pick a learning rate, weight decay, and schedule from a paper or framework example that trains a similar architecture on similar data. Run it once, end to end, as your reference point.

  2. Run a short learning-rate range test to bound a reasonable learning-rate interval before committing to a long run.

  3. Add warmup (typically a few hundred to a few thousand steps, scaled to your total training length) if you don't already have it.

  4. Hold weight decay fixed at your baseline value and tune the learning rate within the bounded range from step 2, watching validation loss, not just training loss.

  5. With a stable learning rate, sweep weight decay across an order-of-magnitude range around your baseline (for example, 5x smaller and 5x larger) and compare validation performance.

  6. Once learning rate and weight decay both look reasonable, decide whether to exclude biases and normalization parameters from decay, and re-check validation performance with that change.

  7. Lock in a schedule (warmup plus your chosen decay shape) and run a final, longer training run at the selected hyperparameters.

  8. If the result matters for a decision, repeat the final run across two or three random seeds and report the spread, not just a single number.


No step in this recipe assumes a specific numeric hyperparameter is universally optimal — the point of the process is to find good values for your specific model and data, using the framework defaults discussed earlier only as a starting point.


FAQ


What does the W in AdamW mean?


The W stands for weight decay — specifically decoupled weight decay. AdamW applies this weight decay directly to the model's parameters as a separate step from the gradient-based adaptive update that plain Adam computes, rather than folding it into the loss as an L2 penalty.


Is AdamW better than Adam?


Not universally. When no weight decay is used, AdamW and Adam produce identical updates. When regularization is needed, AdamW's decoupled decay is generally easier to tune and, per the original paper and much follow-up practice, often improves generalization — but results vary by architecture and dataset, so it isn't a guaranteed win in every case.


What is the main difference between Adam and AdamW?


Adam applies weight decay, if used, as an L2 penalty added to the loss gradient, so it gets scaled by Adam's adaptive per-parameter learning rates. AdamW applies weight decay as a separate operation on the weights themselves, outside that adaptive scaling.


Is AdamW just Adam with L2 regularization?


No. That's a common misconception this article addresses directly. L2 regularization added to the loss and AdamW's decoupled weight decay produce different updates once Adam's adaptive per-parameter scaling is involved, even though they can look similar for plain SGD.


What weight decay should I use with AdamW?


There is no single correct value — it depends on your model, dataset, and training length. PyTorch's AdamW defaults to 0.01 and Keras's to 0.004; treat both as starting points to tune, not as universal recommendations, using a validation set to guide the search.


What learning rate should I use with AdamW?


It depends on the model and batch size, but a short learning-rate range test — increasing the learning rate over a few hundred steps and watching where loss stops improving — is a fast, practical way to find a reasonable starting range for your specific setup.


Why is AdamW used for transformers?


Transformers have large, heterogeneous parameter spaces where Adam's adaptive per-parameter scaling helps, and their fine-tuning and pretraining recipes benefit from AdamW's more predictable regularization. It became a practical default after 2019, but it is not a strict requirement — some transformer training runs use other optimizers.


Should biases receive weight decay?


Many training recipes exclude bias terms from weight decay, since biases are low-dimensional and don't multiply the input, so decaying them is judged to add little regularization value. This is a common convention rather than a mathematical necessity, and some codebases decay all parameters uniformly.


Should LayerNorm parameters receive weight decay?


Commonly, no — many transformer training recipes exclude LayerNorm (and BatchNorm) scale and bias parameters from weight decay, since these parameters directly rescale activations and decaying them toward zero can work against the normalization layer's purpose. Check your specific codebase's convention.


Does AdamW need a learning-rate scheduler?


It isn't required, but most current training recipes pair AdamW with a schedule — commonly warmup followed by cosine or linear decay — because AdamW's decoupled weight-decay term is typically scaled by the same learning rate as the adaptive update, so the schedule affects both.


Does AdamW use more memory than SGD?


Yes. AdamW tracks two extra tensors per parameter (the first and second moment estimates), roughly doubling optimizer memory compared to SGD with momentum, which tracks only one extra tensor per parameter. This matters more as model size grows.


Can AdamW be used for CNNs?


Yes, AdamW works for convolutional architectures. Some well-known CNN results still favor SGD with momentum and a carefully tuned schedule, particularly at large training budgets, so it's worth comparing both for a specific CNN task rather than assuming AdamW is automatically better.


Is AdamW suitable for fine-tuning pretrained models?


Yes — AdamW is widely used for fine-tuning, typically with a lower learning rate and often lower weight decay than training from scratch, since large updates risk erasing useful information already encoded in the pretrained weights.


What are the default AdamW parameters in PyTorch or Keras?


As of current documentation, PyTorch's torch.optim.AdamW defaults to lr=0.001, betas=(0.9, 0.999), eps=1e-8, and weight_decay=0.01. Keras's keras.optimizers.AdamW defaults to learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-7, and weight_decay=0.004. Always confirm against your installed version, since defaults can change between releases.


When should I not use AdamW?


Consider alternatives when a well-tuned SGD-with-momentum recipe is documented to outperform Adam-family optimizers for your specific architecture (common in some CNN benchmarks), when optimizer memory is a hard constraint at very large model scale, or when a specialized optimizer designed for your exact training regime has demonstrated a clear advantage in the literature.


Key Takeaways


  • AdamW separates weight decay from Adam's adaptive gradient update, applying it directly to the weights instead of folding it into the loss as an L2 penalty.

  • The distinction matters because Adam's per-parameter adaptive scaling distorts an L2 penalty's effective regularization strength in ways that are hard to predict or control.

  • AdamW was introduced by Loshchilov and Hutter in "Decoupled Weight Decay Regularization" (ICLR 2019), building directly on Kingma and Ba's original 2015 Adam paper.

  • PyTorch's and Keras's built-in AdamW implementations use different default weight-decay values (0.01 and 0.004, respectively) — always confirm current defaults rather than assuming.

  • Many training recipes exclude biases and normalization parameters from weight decay using parameter groups, a common convention rather than a strict rule.

  • The learning-rate schedule and weight decay interact, because AdamW's decoupled decay term is typically scaled by the same learning rate as the adaptive update.

  • AdamW is a strong, widely used default for transformer training and fine-tuning, but it is not universally superior to Adam or SGD — the right optimizer still depends on the task.


Actionable Next Steps


  1. Check which optimizer and weight-decay mechanism your current training script actually uses — read the class name and its documentation, don't assume from the parameter name alone.

  2. If you're using plain Adam with an L2 penalty added to your loss, try switching to torch.optim.AdamW or keras.optimizers.AdamW with an equivalent starting weight-decay value, and compare validation performance.

  3. Set up parameter groups that exclude biases and normalization parameters from weight decay, following the code pattern shown earlier, and confirm the grouping is correct before a long run.

  4. Run a short learning-rate range test on your specific model and dataset rather than reusing a value from an unrelated project.

  5. Add a warmup-plus-decay learning-rate schedule if you don't already have one, and re-check whether your weight-decay value still makes sense under the new schedule.

  6. Track validation loss alongside training loss for every experiment, so you can tell whether a weight-decay change is actually improving generalization.


Glossary


  1. Adam: An optimization algorithm that adapts the learning rate for each parameter using running estimates of the gradient's mean and variance.

  2. AdamW: A variant of Adam that applies weight decay directly to the weights, decoupled from the adaptive gradient update.

  3. Optimizer: The algorithm that updates a model's parameters during training to minimize a loss function.

  4. Gradient: The direction and rate of change of the loss function with respect to a parameter; used to decide how to update that parameter.

  5. Learning rate: A scalar that controls how large each parameter update step is.

  6. Weight decay: A regularization technique that shrinks parameter values toward zero during training, applied directly to the weights.

  7. L2 regularization: A penalty added to the loss function proportional to the sum of squared parameter values, discouraging large weights.

  8. Momentum: A technique that smooths parameter updates by incorporating a running average of past gradients.

  9. First moment: A running average of the gradient itself, used by Adam to estimate the gradient's typical direction.

  10. Second moment: A running average of the squared gradient, used by Adam to estimate the gradient's typical magnitude.

  11. Beta1 (β1): The decay rate controlling how quickly Adam's first-moment estimate forgets older gradients.

  12. Beta2 (β2): The decay rate controlling how quickly Adam's second-moment estimate forgets older gradients.

  13. Epsilon (ε): A small constant added to Adam's denominator to prevent division by zero.

  14. Bias correction: An adjustment Adam applies to its moment estimates early in training to counteract their initialization at zero.

  15. Adaptive learning rate: A per-parameter learning rate that changes based on that parameter's gradient history, rather than a single global rate.

  16. Parameter group: A subset of a model's parameters given its own optimizer settings, such as a different weight-decay value.

  17. Scheduler: A component that changes the learning rate over the course of training according to a defined rule.

  18. Warmup: A schedule phase at the start of training where the learning rate increases gradually from a small value to its target.

  19. Normalization (LayerNorm/BatchNorm): A technique that rescales activations within a network, using its own learned scale and bias parameters.

  20. Generalization: How well a trained model performs on new, unseen data rather than just the data it was trained on.

  21. AMSGrad: A variant of Adam-family optimizers that uses a running maximum of the second-moment estimate instead of a plain exponential average.


Sources & References





bottom of page