What Is a Learning Rate Schedule?
- 2 days ago
- 24 min read
A learning rate that gets a model moving fast in the first few epochs is often the exact same value that keeps it from settling into a good minimum later on. Drop it too soon and training crawls before the model has learned anything useful. Leave it high too long and validation loss oscillates instead of settling down. This tension between speed and stability is not a bug in gradient descent. It is a basic feature of optimizing a messy, non-convex loss surface with a single step-size knob. A learning rate schedule is the standard answer: a rule for changing that step size as training moves forward, instead of freezing it at one value for the whole run.
TL;DR
A learning rate schedule is a rule for changing the optimizer's step size during training, not a single fixed value.
Schedules can be predetermined, such as step decay or cosine annealing, or metric-responsive, such as ReduceLROnPlateau.
Warmup followed by cosine or linear decay, and the one-cycle policy, are common defaults for Transformers and CNNs trained from scratch.
Adaptive optimizers like Adam and AdamW still often benefit from an external schedule on their base learning rate.
The right schedule depends on total training steps, the optimizer, batch size, and how stable your validation metric is. No single schedule wins everywhere.
What Is a Learning Rate Schedule?
A learning rate schedule is a rule that changes an optimizer's learning rate as training progresses, instead of keeping it fixed. The change can follow a predetermined curve, such as step decay or cosine annealing, or respond to a monitored metric like validation loss. Its goal is to balance fast early progress with stable, precise refinement later in training.
Table of Contents
What Is a Learning Rate Schedule?
In gradient-based training, the learning rate, usually written as η, α, or simply lr, is the multiplier applied to the gradient before it updates each parameter. A learning rate schedule is a rule that changes this multiplier as training proceeds, rather than holding it fixed for the entire run.
More formally, a schedule is a function lr(t) that maps the current training step, epoch, or evaluation event t to a learning rate value. That mapping can come from a closed-form formula, such as step decay or cosine annealing, or from a stateful rule that reacts to a monitored quantity, such as ReduceLROnPlateau watching validation loss.
A schedule is different from a single manual change. Dropping the learning rate once by hand partway through a run is an ad hoc adjustment, not a schedule. A schedule is a reusable, defined policy, ideally logged and reproducible, that specifies exactly how and when the rate changes for the whole run, not just at one point someone happened to intervene.
Schedules split into two broad families. Predetermined schedules follow a fixed curve set in advance, based on elapsed time, independent of how training is actually going. Metric-responsive schedules instead watch a signal, typically a validation metric, and change the rate only when that signal stops improving. Both are hyperparameter strategies: they configure how the optimizer behaves, and they do not change the model's architecture, its parameters, or its weights directly.
A useful analogy: think of parking a car. Full throttle gets you across the lot quickly, but you ease off the accelerator for the last few metres so you do not hit anything. A learning rate schedule is that same easing-off, applied to an optimizer instead of a pedal.
theta(t+1) = theta(t) - lr(t) * grad_L(theta(t))
Here, theta(t) is the set of model parameters at step t, lr(t) is the scheduled learning rate at step t, and grad_L(theta(t)) is the gradient of the loss function with respect to those parameters, typically computed via automatic differentiation over the model's computational graph. Everything else about the update rule stays the same; only lr(t) changes across the run instead of remaining a constant lr.
Why Change the Learning Rate During Training?
A large learning rate can produce rapid early progress, since big steps move parameters across the loss surface quickly when they start far from any good solution. But a rate that is too large for the local curvature can overshoot a minimum, causing the loss to oscillate or, in extreme cases, diverge outright.
A small learning rate supports careful refinement once the model is already in a reasonable region of parameter space, allowing it to settle into a lower-loss point without overshooting. But a rate that is too small for too long can make optimization painfully slow, or can stall useful progress well before the training budget is used up.
The core reason a single fixed value rarely suits the whole run is that the loss landscape a model is walking through is not simple or smooth. Curvature changes across training, mini-batch sampling introduces noise into every gradient estimate (see mini-batch gradient descent), and the relationship between training loss and validation loss can shift as the model starts to overfit or as generalization behavior changes. A step size well suited to the start of training is often poorly suited to fine convergence near the end, which is the practical justification for scheduling rather than fixing the rate.
How a Learning Rate Schedule Works
Several units of "progress" matter here and are easy to conflate. An epoch is one full pass over the training dataset. A mini-batch is one subset of examples used for a single gradient estimate. An optimizer step is one parameter update, normally triggered once per mini-batch (or less often, when using gradient accumulation). A global training step is a running counter of optimizer steps across the whole run. A scheduler update is the moment the schedule recomputes lr(t), which may or may not happen at the same frequency as an optimizer step.
Schedules are commonly written using notation like lr(t) or the Greek eta with a subscript, eta_t, where t indexes whichever unit the schedule is actually defined over: epoch-based schedules index by epoch number, while step-based schedules index by optimizer step.
Training phase | Example learning rate | Intended effect |
|---|---|---|
Warmup (first ~500 steps) | Rises from near 0 to 1e-3 | Avoid early instability from large, poorly calibrated initial updates |
Main training (steps 500-8,000) | Cosine decay from 1e-3 toward 1e-5 | Steady refinement as the model approaches a good region |
Final steps (8,000-10,000) | Roughly 1e-5 | Fine convergence without destabilizing what has already been learned |
(These figures are illustrative, not universal defaults; actual values depend heavily on the model, optimizer, and dataset.)
Beyond frequency, schedules also differ in mechanism. Epoch-based updates recompute the rate once per epoch. Batch-based updates recompute it once per optimizer step. Step-based schedules are typically a closed-form function of the global step count. Metric-triggered updates, like ReduceLROnPlateau, only change the rate after an evaluation shows no improvement. Closed-form schedules can be computed from t alone with no memory of past behavior, while stateful schedulers, such as ReduceLROnPlateau, must track things like how many evaluations have passed since the last improvement. Composite schedules chain several of these together, for example a short warmup phase followed by a longer decay phase, with the boundary between them handled explicitly.
Learning Rate Schedules vs Adaptive Optimizers
Stochastic gradient descent (SGD) applies the same single learning rate to every parameter at every step. SGD with momentum adds a running average of past gradients to smooth the update direction, but still uses one global learning rate. Adam and AdamW go further: they maintain per-parameter running estimates of the gradient's first and second moments, so each parameter effectively gets its own adapted step size, scaled by these statistics.
That per-parameter adaptation is not the same thing as a learning rate schedule. Adam's adaptivity adjusts relative step sizes across parameters based on their own gradient history; a schedule adjusts the single global multiplier that sits in front of all of those per-parameter updates, changing over the course of training. The two mechanisms operate independently and are frequently combined: Adam or AdamW still have a global learning rate hyperparameter, and that value can still be scheduled to decay, warm up, or cycle exactly as it would with SGD.
This is why AdamW training recipes for Transformer models routinely pair AdamW with warmup followed by linear or cosine decay applied to AdamW's base learning rate. "Adaptive optimizer" and "adaptive learning-rate schedule" describe two different things: one is a property of the optimizer's per-parameter update rule, the other is a property of how the optimizer's global rate changes over time.
Optimizer | Adapts per parameter? | Commonly scheduled too? | Typical schedule pairing |
|---|---|---|---|
SGD | No | Yes, often necessary | Step decay or cosine annealing |
SGD with momentum | No (only smooths direction) | Yes | Step decay or one-cycle |
Adam | Yes | Often, though less strictly required | Warmup plus decay |
AdamW | Yes, with decoupled weight decay | Often, especially for Transformers | Linear or cosine decay with warmup |
Main Types of Learning Rate Schedules
This section works through the schedules most commonly used in practice today, from the simplest baseline to metric-responsive approaches. For each one: the intuition, a readable formula where useful, the parameters involved, advantages, limitations, and typical use cases.
Constant Learning Rate
A constant rate is the baseline, not really a "schedule" at all, since lr(t) never changes. It remains a perfectly reasonable choice for small models, quick experiments, or cases where a well-tuned single value already trains reliably and there is little appetite for extra tuning overhead.
Step Decay
lr(t) = lr0 * gamma ^ floor(t / step_size)
Here lr0 is the initial rate, gamma is a decay factor (commonly around 0.1 to 0.5), and step_size is the interval, usually in epochs, between drops. Step decay is simple and easy to reason about, but the abrupt drops can momentarily destabilize training and require the boundaries to be chosen well, often from experience with similar runs.
Multi-Step Decay
Multi-step decay works the same way as step decay, but the drop points are specific, named milestones (for example, particular epoch numbers) rather than a uniform interval. This suits cases where prior runs, or domain knowledge, suggest training tends to plateau at particular, uneven points rather than at regular intervals.
Exponential Decay
lr(t) = lr0 * gamma ^ t
Exponential decay reduces the rate continuously by a constant multiplicative factor at every step. Frameworks also implement a "staircase" variant, which applies the same formula but only recomputes the value once per epoch rather than continuously, giving a smoother overall curve than step decay but coarser than continuous exponential decay.
Linear Decay
lr(t) = lr0 + (lr_end - lr0) * (t / T)
Linear decay moves the rate in a straight line from a start value lr0 to an end value lr_end over a total duration T (usually total steps). It is easy to reason about and is a common partner for warmup in fine-tuning recipes.
Polynomial Decay
lr(t) = (lr0 - lr_end) * (1 - t / T) ^ power + lr_end
Polynomial decay generalizes linear decay: setting power to 1 reproduces linear decay exactly, while higher powers concentrate most of the decline near the end of training and keep the rate relatively high for longer beforehand.
Inverse-Time Decay
lr(t) = lr0 / (1 + decay_rate * t / decay_steps)
Inverse-time decay reduces the rate more sharply in early steps and then levels off, so later steps see progressively smaller reductions rather than a constant proportional drop.
Inverse-Square-Root Decay
lr(t) ~ 1 / sqrt(t)
Inverse-square-root decay is most associated with the original Transformer training recipe, where it is applied after a warmup phase (Vaswani et al., 2017). This does not mean every Transformer since then uses this exact schedule; many current recipes instead pair warmup with cosine or linear decay.
Cosine Annealing
lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T_max))
Cosine annealing reduces the rate along one half-period of a cosine curve, from lr_max down to lr_min, over T_max total steps or epochs. It decays slowly at first, more steeply through the middle, then slowly again near the end. It was introduced alongside warm restarts by Loshchilov and Hutter (SGDR, ICLR 2017) and is now a common default across many CNN and Transformer training setups, though it does not guarantee convergence to a better minimum than any other schedule.
Cosine Annealing with Warm Restarts
This extends cosine annealing by periodically resetting the rate back to lr_max at defined cycle boundaries, with each subsequent cycle sometimes made longer via a multiplier (often called T_mult). The word "restart" refers only to the learning rate curve jumping back up; it does not reset the model's weights or the optimizer's own internal state, both of which continue accumulating from where they were.
Cyclical Learning Rates
Proposed by Smith (2017), cyclical learning rates move back and forth in a triangular wave between a lower bound and an upper bound over a defined half-cycle length. The reasoning is that periodically increasing the rate can help the optimizer escape sharp or unhelpful regions of the loss surface that a purely decaying schedule might get stuck near.
The One-Cycle Policy
Introduced by Smith and Topin under the banner of super-convergence, the one-cycle policy rises from an initial learning rate up to a maximum over roughly the first 10 to 40 percent of training, then anneals down to a value far lower than the initial rate for the remainder of the run. Momentum is often cycled inversely to the rate when the optimizer supports it. Because the shape depends on total step count, correctly estimating total_steps (accounting for epochs, steps per epoch, and any gradient accumulation) is essential, and it is normally stepped once per batch, not once per epoch. One-cycle is a specific, asymmetric shape; it should not be treated as simply another generic cyclical schedule.
ReduceLROnPlateau
Unlike every schedule above, ReduceLROnPlateau is metric-responsive rather than time-based. It watches a monitored metric, commonly validation loss, in a given mode (min for loss, max for accuracy-like metrics), and multiplies the rate by a factor when that metric fails to improve by more than a set threshold for a number of evaluations equal to patience. A cooldown period can delay re-triggering right after a drop, and min_lr sets a floor below which the rate will not fall. Because it depends on an evaluated metric rather than elapsed steps, it needs a reasonably stable validation signal to behave sensibly.
The table below compares all of these at a glance.
Schedule | Update trigger | Shape | Key parameters | Good starting use case |
|---|---|---|---|---|
Constant | Never changes | Flat | lr | Small models, quick experiments |
Step decay | Fixed interval | Stepwise drops | gamma, step_size | Classic CNN training with a known epoch budget |
Multi-step decay | Named milestones | Stepwise drops at custom points | milestones, gamma | Runs where plateaus are known from past experience |
Exponential decay | Every step or epoch | Smooth or staircase decline | gamma | Continuous, predictable decay without hard boundaries |
Linear decay | Every step | Straight-line decline | lr0, lr_end, T | Fine-tuning pretrained models with a fixed step budget |
Polynomial decay | Every step | Curved decline, tunable steepness | power, T | When decay should stay gentle then sharpen near the end |
Inverse-time decay | Every step | Fast-then-flat decline | decay_rate, decay_steps | Long runs needing an early aggressive drop |
Inverse-square-root decay | Every step, after warmup | Smooth, slowing decline | warmup_steps | Transformer recipes following the original warmup setup |
Cosine annealing | Every step | Smooth cosine curve down | T_max, lr_min | General-purpose default for CNNs and Transformers |
Cosine annealing with warm restarts | Every step, cycle resets | Repeating cosine cycles | T_max, T_mult | Long training runs that benefit from periodic re-exploration |
Cyclical learning rates | Every step | Triangular oscillation | base_lr, max_lr, step_size | Escaping sharp minima or saddle regions |
One-cycle policy | Every step | Rise then long anneal | max_lr, pct_start, total_steps | Fast, high-performance training with a known step budget |
ReduceLROnPlateau | Every evaluation | Stepwise drops on stagnation | factor, patience, mode | Unknown training duration, stable validation metric |
Warmup, Cooldown, Restarts, and Composite Policies
Warmup is a short initial phase in which the rate rises from a low (sometimes zero) value up to its intended starting or peak value, instead of starting at that peak immediately. Linear warmup increases the rate in a straight line; gradual or exponential warmup increases it more slowly at first.
Warmup tends to help in settings prone to early instability: large-batch training, where Goyal et al. (2017) paired a linear batch-size-to-learning-rate scaling rule with a warmup phase specifically to offset that instability, and Transformer training, where Vaswani et al. (2017) used several thousand warmup steps before inverse-square-root decay. Warmup is a phase, not a guarantee of improved results; smaller models on modest datasets with conservative batch sizes often train perfectly well without it.
Composite policies chain phases together explicitly: warmup followed by cosine decay, warmup followed by linear decay, or a final cooldown segment at a very low, steady rate to squeeze out the last bit of refinement. Restarts periodically reset part of the curve, as described above for cosine annealing. When chaining schedules, boundary continuity matters: an abrupt jump in rate at a phase boundary can destabilize training briefly, while a smooth handoff, where the end of one phase matches the start of the next, usually transitions without disruption.
As an illustrative example: a hypothetical 10,000-step run might use a 500-step linear warmup from 0 up to 3e-4, followed by cosine decay from 3e-4 down to 3e-6 across the remaining 9,500 steps. These specific numbers are illustrative starting points, not universal defaults.
Choosing the Right Schedule for Your Model
The right choice is conditional on context. The list below groups common situations with schedules that often fit them reasonably well, not a claim that any of them is universally correct.
Small tabular models: a constant rate or simple step decay is usually enough.
Conventional CNN training from scratch: step decay, multi-step decay, or cosine annealing are all well-established choices.
Transfer learning: a small constant rate or short decay on the unfrozen layers, often with discriminative rates across layers.
Fine-tuning pretrained language models: linear decay with a short warmup is a widely used default.
Training Transformers from scratch: warmup followed by cosine or inverse-square-root decay.
Short experimental runs: constant or simple decay, to minimize schedule-tuning overhead relative to the run's value.
Long production training runs: cosine annealing or one-cycle, provided total steps can be estimated well.
Noisy validation metrics: favor time-based schedules over ReduceLROnPlateau, since a noisy metric can trigger premature drops.
Unknown training duration: ReduceLROnPlateau, or a schedule that does not require a total-step estimate up front.
Known, fixed compute budget: cosine annealing, one-cycle, and linear decay all use a known budget well.
Compute-limited projects: fewer knobs to tune, such as step decay or a constant rate, to limit tuning cost.
The right pick also depends on the optimizer, model architecture, dataset size, number of training steps, batch size, how stable validation behaves, the checkpointing strategy in use, tolerance for additional tuning, and whether the total training duration is even known in advance.
Learning Rate Schedules with SGD, Adam, and AdamW
Momentum and the learning rate interact: high momentum combined with a high rate compounds the risk of oscillation, since momentum already smooths and amplifies the effective step. Weight decay behaves differently across optimizers too. AdamW decouples weight decay from the gradient-adaptive update (Loshchilov and Hutter, 2019), so its regularization strength does not scale with the schedule the same way an L2 penalty folded directly into the gradient would in plain SGD or Adam.
Parameter groups let different parts of a model use different rates, and different schedule multipliers, at once, commonly used to give a pretrained backbone a smaller rate than a newly initialized prediction head, an approach often called discriminative learning rates. Switching optimizers usually means retuning both the base rate and the schedule's parameters, since Adam-family optimizers typically operate well at rates an order of magnitude or more smaller than typical SGD rates.
Epoch-Based vs Step-Based Scheduling
Classic step decay is normally stepped once per epoch. Most modern schedules, including cosine annealing and one-cycle, are stepped once per optimizer step (that is, per mini-batch). ReduceLROnPlateau is stepped once per evaluation, typically once per epoch, immediately after computing the monitored metric.
Several things can quietly break the intended timing. Gradient accumulation changes the relationship between "batches seen" and "optimizer steps taken"; total_steps used by a step-based schedule needs to divide by the accumulation factor, not just the raw batch count. A dataloader whose length varies across epochs (for example, when not dropping a final partial batch) changes steps_per_epoch, which matters for any schedule computed from that value. In distributed training, a "step" usually refers to one synchronized update across all workers, not one worker's local batch. Under mixed-precision training, a gradient-scaler overflow can cause the optimizer to skip an update while the scheduler is still stepped, silently shifting the schedule relative to what was intended, a documented behavior in PyTorch's automatic mixed precision utilities.
Resuming interrupted training also needs care: the scheduler's own internal state, not just an epoch counter, has to be restored, or the learning-rate curve restarts from the wrong point entirely. Calling a scheduler at the wrong time, or at the wrong frequency, produces a schedule that runs faster or slower than the one that was actually intended.
PyTorch Learning Rate Scheduler Examples
The examples below use current torch.optim.lr_scheduler APIs. Always confirm exact behavior against the documentation for your installed PyTorch version, since scheduler defaults and edge cases have shifted across releases.
Epoch-Based StepLR
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
for epoch in range(num_epochs):
for batch in train_loader:
loss = compute_loss(model, batch)
optimizer.zero_grad()
loss.backward()
optimizer.step() # updates parameters every batch
scheduler.step() # StepLR updates once per epoch, after the epoch's batches
Batch-Based OneCycleLR
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4, momentum=0.9)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
optimizer, max_lr=0.01, steps_per_epoch=len(train_loader), epochs=num_epochs
)
for epoch in range(num_epochs):
for batch in train_loader:
loss = compute_loss(model, batch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
scheduler.step() # OneCycleLR is stepped once per batch, not per epoch
ReduceLROnPlateau on Validation Loss
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
optimizer, mode="min", factor=0.5, patience=3
)
for epoch in range(num_epochs):
train_one_epoch(model, optimizer, train_loader)
val_loss = evaluate(model, val_loader)
scheduler.step(val_loss) # the monitored metric is passed directly, once per evaluation
Warmup Plus Cosine Decay (Chained)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
warmup = torch.optim.lr_scheduler.LinearLR(
optimizer, start_factor=0.01, total_iters=500
)
decay = torch.optim.lr_scheduler.CosineAnnealingLR(
optimizer, T_max=total_steps - 500
)
scheduler = torch.optim.lr_scheduler.SequentialLR(
optimizer, schedulers=[warmup, decay], milestones=[500]
)
for step in range(total_steps):
train_one_step(model, optimizer)
scheduler.step() # SequentialLR hands off from warmup to decay at the milestone
(This simplified example omits data loading, checkpointing, and logging so the scheduler placement stays visible; a production loop would include those around the same core calls.)
TensorFlow and Keras Learning Rate Schedule Examples
The examples below use current Keras optimizers.schedules and callbacks APIs. As with PyTorch, confirm exact argument names against the documentation for your installed TensorFlow or Keras version.
A Built-In CosineDecay Schedule
decay_steps = 8000
lr_schedule = keras.optimizers.schedules.CosineDecay(
initial_learning_rate=1e-3, decay_steps=decay_steps
)
optimizer = keras.optimizers.AdamW(learning_rate=lr_schedule)
# Passing the schedule object directly to the optimizer means Keras
# recomputes the rate internally at every optimizer step automatically.
CosineDecay With Built-In Warmup
lr_schedule = keras.optimizers.schedules.CosineDecay(
initial_learning_rate=0.0,
decay_steps=8000,
warmup_target=1e-3,
warmup_steps=500
)
optimizer = keras.optimizers.AdamW(learning_rate=lr_schedule)
# The rate rises linearly to warmup_target over warmup_steps, then
# follows a cosine decay down toward initial_learning_rate * alpha.
ReduceLROnPlateau as a Callback
reduce_lr = keras.callbacks.ReduceLROnPlateau(
monitor="val_loss", mode="min", factor=0.5, patience=3
)
model.fit(
train_ds, validation_data=val_ds, epochs=num_epochs,
callbacks=[reduce_lr]
)
# ReduceLROnPlateau needs a plain, mutable learning rate on the optimizer,
# not a LearningRateSchedule object, since it adjusts the value directly.
Three distinct mechanisms are worth telling apart here: passing a schedule object directly as the optimizer's learning_rate, which Keras evaluates internally every step; the LearningRateScheduler callback, which calls a Python function of the epoch index to set the rate at each epoch boundary; and ReduceLROnPlateau, which is metric-based and only works against a directly mutable learning rate rather than a schedule object.
How to Tune a Learning Rate Schedule
A practical process, roughly in order:
Establish a constant-rate baseline first, so later comparisons have something concrete to beat.
Run a learning-rate range test, as described by Smith (2017): sweep the rate upward over a short run and watch where loss stops improving, to bound a sensible max_lr.
Move coarse-to-fine: start with wide steps across orders of magnitude, then narrow in once a promising region appears.
Keep total training steps comparable across compared schedules, so differences reflect the schedule and not an uneven compute budget.
Repeat runs where variance matters, especially for shorter runs where a single seed can be noisy.
Log the actual learning rate used at every step, not just the nominal schedule, to confirm it behaved as intended.
Select the checkpoint using validation performance, not training loss alone, since the two can diverge as generalization shifts.
Change one major schedule decision at a time so its effect can be separated from other hyperparameter changes.
Parameters worth tuning specifically include the initial rate, maximum rate, minimum or final rate, warmup duration, decay duration, step boundaries, decay factor, plateau patience, cycle length, and number of restarts. Any numeric values offered as starting points here or elsewhere in this article are illustrative, not universal defaults; for instance, a warmup fraction of roughly 1 to 6 percent of total steps shows up often in Transformer training recipes, but that is a common illustrative range, not a rule.
How to Monitor Whether the Schedule Is Working
Useful signals include training loss, validation loss, task-specific validation metrics, gradient norms, the actual learning-rate history, parameter update norms where available, signs of instability or divergence, plateaus, oscillation, overfitting, how well the model recovers after a decay step, and whether meaningful improvement only shows up once the rate has become very small. No single curve reliably identifies the cause on its own; the table below pairs common symptoms with candidate causes and a next test.
Symptom | Possible LR-related cause | Other possible cause | Sensible next test |
|---|---|---|---|
Loss diverges or turns to NaN early | Rate too high, or warmup too short | Bad data batch, exploding-gradient bug | Lower the rate or add warmup; check gradient norms directly |
Validation loss plateaus early while training loss keeps falling | Decayed too early or too aggressively | Overfitting, or too little regularization | Extend the high-rate phase; compare the train/validation gap |
Improvement only appears in the final low-rate steps | Total steps mis-estimated for the schedule's shape | Model needed more capacity or data | Re-run a range test; adjust decay duration and re-check |
Validation metric oscillates without settling | Rate still too high for this phase | Small or noisy validation set | Reduce the rate, or widen patience on a plateau scheduler |
Common Mistakes and Troubleshooting
Starting with an excessively high rate: back it off and consider adding warmup.
Starting with an unnecessarily low rate: run a range test to find a more productive starting point.
Applying decay too early: extend the high-rate phase and re-check validation behavior.
Applying decay too aggressively: use a gentler factor or a smoother curve, such as cosine instead of a hard step.
Ending training before the low-rate phase becomes useful: extend the budget or shorten the decay duration.
Calling the scheduler before or after the wrong operation: step it after optimizer.step(), in the order your framework's documentation specifies.
Updating an epoch-based scheduler every batch, or a batch-based scheduler only once per epoch: match the call frequency to the scheduler type.
Passing training loss into a scheduler meant to monitor validation loss: pass the intended metric explicitly.
Using the wrong mode for a monitored metric (min versus max): match mode to whether the metric should rise or fall.
Forgetting gradient accumulation when computing total steps: divide by the accumulation factor before setting schedule boundaries.
Failing to save and restore scheduler state on resume: checkpoint the scheduler alongside the optimizer and model.
Changing batch size without reconsidering the rate: revisit both the base rate and any schedule boundaries tied to step counts.
Combining several schedule techniques without understanding how they interact: verify the combined curve by logging it directly.
Comparing schedules across unequal compute budgets: hold total steps constant across the comparison.
Treating the final checkpoint as automatically better than the best validation checkpoint: it usually is not.
Confusing warm restarts with resetting model weights: restarts only affect the learning-rate curve, not the parameters.
Assuming a schedule can fix broken data pipelines, incorrect labels, or a buggy loss function: it cannot; fix the underlying issue first.
A Practical Decision Framework
Before training, work through these questions:
Is the total number of optimizer steps known in advance?
Is validation stable enough to support metric-based scheduling?
Is the optimizer SGD-like or adaptive (Adam-family)?
Is this training from scratch or fine-tuning a pretrained model?
Is warmup justified given the batch size, model type, and known instability risks?
Will the scheduler update per batch or per epoch?
How will scheduler state be checkpointed alongside the model and optimizer?
What metric determines the best checkpoint to keep?
Situation | Total steps known? | Validation stable? | Reasonable starting schedule |
|---|---|---|---|
Fine-tuning a pretrained model | Yes | Usually | Linear decay with short warmup |
Training a Transformer from scratch | Yes | Usually | Warmup with cosine or inverse-square-root decay |
Long-running production job, duration uncertain | No | Depends | ReduceLROnPlateau |
Fast experimental iteration | Yes, short | Often noisy | Constant rate or simple decay |
Illustrative worked example: fine-tuning a pretrained image classifier on 20,000 labeled photos for 15 epochs, batch size 32, using AdamW. Total optimizer steps are known (20,000/32 times 15, rounded), so a linear decay with a short warmup of a few hundred steps is a reasonable starting point. The practitioner logs the actual rate every step, checkpoints on validation accuracy rather than training loss, and re-runs a shorter range test only if early validation behavior looks unstable. This sequence, not any single numeric choice, is the transferable part of the example.
Advanced Considerations
Large-batch training often needs a larger base rate, commonly guided by the linear scaling rule as a heuristic rather than a universal law, and typically paired with warmup.
Distributed data-parallel training usually treats a synchronized update across all workers as one step for scheduling purposes.
Gradient accumulation changes the true number of optimizer steps relative to raw batches processed; schedule boundaries need to account for that.
Mixed precision can cause a skipped optimizer step during a gradient-scaler overflow while the scheduler is still advanced, quietly shifting the intended curve.
Separate parameter groups let frozen and unfrozen layers, or a backbone and a new head, use different rates and different schedule multipliers during transfer learning.
Very short training runs generally benefit less from elaborate schedules, since there is little time for a multi-phase curve to matter.
Long-horizon and continual learning setups may need schedule resets or new warmup phases each time new data arrives.
Checkpoint resumption requires restoring the scheduler's own serialized state, not only the epoch or step counter.
Hyperparameter-search leakage can occur if a schedule is tuned by watching the same validation data used for final model selection; keep a held-out set separate where possible.
Random-seed variance matters most for shorter runs; repeat comparisons where the outcome is close.
Early stopping and learning-rate decay interact: an aggressive decay near a patience-based stopping point can either help by stabilizing metrics or mask a plateau that would otherwise trigger stopping sooner.
Weight decay and regularization strength can behave differently under a changing rate depending on whether decay is coupled to the gradient step (as in plain SGD or Adam) or decoupled (as in AdamW).
FAQ
Is a learning rate schedule always necessary?
No. A well-tuned constant rate is a reasonable choice for small models, short runs, or quick experiments. Schedules tend to matter more as training runs get longer, or as batch sizes and model sizes grow, since more is at stake in getting the fine-convergence phase right.
What is the difference between a learning rate and a learning rate schedule?
A learning rate is a single numeric value, the step-size multiplier used in a parameter update. A learning rate schedule is the rule that determines how that value changes across the whole training run, whether through a fixed formula or a response to a monitored metric.
Does Adam need a learning rate scheduler?
Not strictly. Adam's per-parameter adaptation already adjusts relative step sizes based on gradient history. But its single global rate can still benefit from an external schedule, and many Transformer and fine-tuning recipes pair Adam or AdamW with warmup and decay regardless.
Which learning rate schedule is best?
There is no universal answer. Cosine annealing and one-cycle are strong general-purpose defaults with a known step budget, warmup plus linear or cosine decay is common for fine-tuning, and ReduceLROnPlateau suits cases where the total duration is not known in advance. The right choice depends on the model, optimizer, and data.
When should the learning rate decrease?
Typically once early, rapid progress has slowed and the model is refining rather than exploring broadly. Time-based schedules decrease it on a fixed curve regardless of behavior; ReduceLROnPlateau decreases it specifically when a monitored metric stops improving.
What is learning rate warmup?
Warmup is a short initial phase in which the rate rises from a low value up to its intended starting point, rather than beginning at that peak immediately. It is commonly used in large-batch training and Transformer pretraining to reduce early instability, though it is not required for every model.
Is cosine annealing better than step decay?
Neither is universally better. Cosine annealing gives a smoother, continuous decline, which some training runs handle more gracefully than step decay's abrupt drops, but step decay remains simple, well understood, and effective in plenty of settings. The comparison depends on the specific model and data.
How often should a scheduler update?
It depends on the scheduler type. Classic step decay is typically stepped once per epoch. Most modern schedules, including cosine annealing and one-cycle, are stepped once per optimizer step. ReduceLROnPlateau is stepped once per evaluation, using the relevant monitored metric.
What happens if the learning rate becomes too small?
Parameter updates shrink to the point where they barely move the model, so the remaining training steps make little further progress, effectively wasting whatever compute budget is left in that phase of the run.
Can a scheduler be changed during training?
Yes, though it needs to be done carefully. Chaining or sequential schedulers let a run move from one policy to another at a defined boundary, and resuming from a checkpoint mid-schedule requires restoring the scheduler's own saved state, not just the step or epoch counter.
Key Takeaways
A learning rate schedule changes the optimizer's step size over training instead of leaving it fixed, balancing fast early progress against stable later refinement.
Predetermined schedules follow a fixed curve; metric-responsive schedules like ReduceLROnPlateau react to validation behavior instead.
Cosine annealing, one-cycle, and warmup-plus-decay are strong general-purpose starting points, but none of them wins in every setting.
Adaptive optimizers such as Adam and AdamW still commonly pair with an external schedule on their global rate.
Getting the update frequency right, per step versus per epoch versus per evaluation, matters as much as picking the schedule's shape.
Gradient accumulation, mixed precision, and distributed training can all quietly shift a schedule's true timing if step counts are not handled carefully.
Scheduler state must be checkpointed and restored alongside the optimizer and model when resuming interrupted training.
A schedule cannot fix broken data, incorrect labels, or a buggy training loop; those need to be fixed directly.
Actionable Next Steps
Establish a constant-learning-rate baseline before trying any schedule.
Log the actual learning rate used at every step throughout training.
Choose a schedule based on whether total training duration is known and how stable validation is.
Run a controlled comparison against the baseline, holding total steps equal.
Inspect both training and validation behavior, not training loss alone.
Save optimizer and scheduler state together at every checkpoint.
Revise only one major schedule decision at a time before comparing results again.
Glossary
Annealing: gradually reducing the learning rate over training, especially along a smooth curve like cosine annealing.
Batch: a subset of training examples used to compute one gradient estimate.
Convergence: the point where further training produces little further improvement in the loss.
Cooldown: a final phase of training at a very low, steady learning rate, or a delay period after a plateau-triggered drop before another can trigger.
Cyclical learning rate: a schedule that moves the rate back and forth between a lower and upper bound instead of only decreasing.
Decay: a reduction in the learning rate over time or steps.
Epoch: one full pass through the training dataset.
Gradient: the direction and rate of steepest increase of the loss with respect to the model's parameters.
Gradient accumulation: summing gradients across several mini-batches before performing one optimizer update, used to simulate a larger effective batch size.
Hyperparameter: a setting, like the learning rate or schedule type, chosen before training rather than learned from data.
Learning rate: the multiplier applied to the gradient when updating parameters.
Learning rate schedule: a rule for changing the learning rate as training proceeds.
Momentum: an optimizer technique that smooths updates using a running average of past gradients.
Optimizer: the algorithm, such as SGD or Adam, that uses gradients to update model parameters.
Optimizer step: one parameter update, usually triggered once per mini-batch.
Parameter group: a subset of a model's parameters assigned its own learning rate or schedule multiplier.
Plateau: a period where a monitored metric stops improving.
Scheduler: the component that computes the learning rate at each point in training according to a schedule.
Warm restart: a point where a cyclical schedule resets its rate back up to a higher value without resetting the model's weights.
Warmup: an initial phase where the learning rate rises from a low value up to its intended level.
Weight decay: a regularization technique that shrinks parameter values during training, applied either coupled with or decoupled from the gradient update depending on the optimizer.
Sources & References
PyTorch Foundation. "torch.optim.lr_scheduler." PyTorch documentation. docs.pytorch.org/docs/stable/optim.html
PyTorch Foundation. "OneCycleLR." PyTorch documentation. docs.pytorch.org (OneCycleLR)
PyTorch Foundation. "SequentialLR." PyTorch documentation. docs.pytorch.org (SequentialLR)
Keras Team. "Learning rate schedules API." Keras documentation. keras.io/api/optimizers/learning_rate_schedules
Keras Team. "CosineDecay." Keras documentation. keras.io (CosineDecay)
Keras Team. "CosineDecayRestarts." Keras 2 API documentation. keras.io (CosineDecayRestarts)
Loshchilov, I. and Hutter, F. "SGDR: Stochastic Gradient Descent with Warm Restarts." ICLR 2017. arXiv:1608.03983
Loshchilov, I. and Hutter, F. "Decoupled Weight Decay Regularization." ICLR 2019. arXiv:1711.05101
Smith, L. N. "Cyclical Learning Rates for Training Neural Networks." WACV 2017. arXiv:1506.01186
Smith, L. N. and Topin, N. "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates." 2019. arXiv:1708.07120
Goyal, P. et al. "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." 2017. arXiv:1706.02677
Vaswani, A. et al. "Attention Is All You Need." NeurIPS 2017. arXiv:1706.03762
Kingma, D. P. and Ba, J. "Adam: A Method for Stochastic Optimization." ICLR 2015. arXiv:1412.6980
PyTorch Foundation. "Automatic Mixed Precision package - torch.amp." PyTorch documentation. docs.pytorch.org/docs/stable/amp.html
