What Is RMSProp (Root Mean Square Propagation)? Complete 2026 Guide

11 minutes ago
24 min read

RMSProp optimizer visualizing adaptive learning rates and loss convergence.

Pick the wrong learning rate for a neural network and training either crawls for hours or blows up into NaNs within a few steps — and the frustrating part is that the "wrong" rate for one weight is often the "right" rate for another. RMSProp was one of the first widely used fixes: instead of one learning rate for the whole model, it gives every parameter its own, continuously recalculated from that parameter's own recent gradients.

TL;DR

RMSProp keeps a running, exponentially decaying average of each parameter's squared gradients, then divides the gradient by the square root of that average before applying the learning rate.
It was introduced by Geoffrey Hinton in Lecture 6e of his 2012 Coursera course, Neural Networks for Machine Learning — there is no separate peer-reviewed RMSProp paper (Tieleman & Hinton, 2012).
RMSProp fixes AdaGrad's main weakness: instead of accumulating all past squared gradients forever, which shrinks the learning rate toward zero, it lets old gradients gradually lose influence.
PyTorch's torch.optim.RMSprop defaults to lr=0.01, alpha=0.99, eps=1e-8; Keras's RMSprop defaults to learning_rate=0.001, rho=0.9, epsilon=1e-7 — the two frameworks also place epsilon differently in the formula.
RMSProp can be combined with momentum and used in a "centered" form that normalizes by an estimated variance instead of the raw squared-gradient average.
Adam adds first-moment tracking and bias correction on top of the same squared-gradient idea, which is why Adam-family optimizers are the more common default in 2026 — though RMSProp is still a legitimate, lighter-weight choice.

What Is RMSProp? (Quick Answer)

RMSProp (Root Mean Square Propagation) is an adaptive-learning-rate optimization algorithm for training neural networks. It divides each parameter's gradient by the square root of an exponentially decaying average of that parameter's recent squared gradients, which stabilizes step sizes and lets each parameter learn at its own effective rate.

Bonus: AI in Business: Applications, Benefits & Implementation Guide

Bonus Plus: The Complete Guide to Physical AI: What It Is and Why It Matters

Bonus Plus Pro: AI Humanoid Robots: How They Work, Who's Building Them, and What's Next

What Is RMSProp?

RMSProp, short for Root Mean Square Propagation, is an adaptive-learning-rate optimization algorithm used to train neural networks and other models fit with gradient descent. Instead of applying one fixed learning rate to every parameter, RMSProp keeps a separate, per-parameter running estimate of how large that parameter's recent gradients have been, and uses that estimate to rescale each update. Concretely, RMSProp tracks an exponentially decaying moving average of the squared gradient for every parameter. On each step it divides the current gradient by the square root of that moving average, plus a small constant for numerical stability, and only then multiplies by the learning rate. A parameter whose recent gradients have been large gets a smaller effective step. A parameter whose recent gradients have been small gets a comparatively larger effective step. RMSProp sits in the same family of adaptive-gradient methods as AdaGrad (Duchi, Hazan & Singer, 2011) and Adam (Kingma & Ba, 2015), and it ships as a built-in optimizer in both PyTorch (torch.optim.RMSprop) and TensorFlow/Keras (tf.keras.optimizers.RMSprop and keras.optimizers.RMSprop).

Why RMSProp Was Created: The Problem with AdaGrad

AdaGrad, introduced by Duchi, Hazan and Singer in 2011, was one of the first widely used adaptive-learning-rate methods. It divides each parameter's gradient by the square root of the sum of every squared gradient that parameter has ever produced during training. That accumulate-forever design is useful early on, but it has a structural weakness: because the sum of squared gradients can only grow, the effective learning rate for every parameter shrinks monotonically over the course of training. In long training runs, especially in the non-convex, many-iteration setting typical of deep learning, this can push the effective step size toward zero well before the model has converged. Hinton addressed this directly in Lecture 6e of his 2012 Coursera course, Neural Networks for Machine Learning. His fix was to replace AdaGrad's ever-growing sum with an exponentially decaying moving average of squared gradients. Older squared gradients gradually lose influence instead of being remembered forever, so the effective learning rate can recover if recent gradients shrink, rather than being locked into permanent decay.

The Core Intuition Behind RMSProp

Picture two weights in the same network. One has a gradient that is consistently large and noisy from step to step. The other has a gradient that is small and steady. A single global learning rate forces a compromise: large enough to make real progress on the steady weight, small enough that it does not destabilize the noisy one. RMSProp removes that compromise. For the noisy weight, the moving average of squared gradients stays high, so its update is divided by a large number and shrinks. For the steady weight, the moving average stays low, so its update is barely scaled down at all. Each parameter effectively gets its own learning rate, continuously recalculated from its own recent gradient history. The exponential decay matters just as much as the squaring. RMSProp does not weight a gradient from step one the same as a gradient from step ten thousand — recent squared gradients count for more than old ones, so the per-parameter scaling keeps adapting as training dynamics change, instead of freezing in place.

How RMSProp Works Step by Step

Compute the gradient of the loss with respect to every parameter, exactly as in ordinary gradient descent.
Square that gradient element-wise, so every value becomes non-negative and sign information is discarded.
Update a per-parameter moving average of squared gradients by blending in the new squared gradient and decaying the previous average.
Take the square root of that moving average and add a small constant, epsilon, for numerical stability.
Divide the raw (unsquared, signed) gradient by this value to produce a rescaled gradient.
Multiply the rescaled gradient by the learning rate and subtract it from the parameter to get the new value.
Repeat every training step, letting the moving average keep adapting to recent gradient behavior.

RMSProp Equation: The Mathematics Explained

The canonical RMSProp update, consistent with Hinton's original formulation, is usually written as:

v_t = rho * v_(t-1) + (1 - rho) * g_t^2 theta_t = theta_(t-1) - eta * g_t / (sqrt(v_t) + epsilon)

Here theta is the parameter being optimized, g_t is the gradient of the loss with respect to theta at step t, v_t is the per-parameter exponentially decaying moving average of the squared gradient, eta is the base learning rate, rho is the decay factor (sometimes called alpha), epsilon is a small constant added for numerical stability, and t indexes the training step. The squaring (g_t^2) and the division (g_t / (sqrt(v_t) + epsilon)) are both applied element-wise, independently for every parameter — v_t is not a single global number, it is a full accumulator with one entry per parameter.

Squaring the gradient throws away its sign and keeps only its magnitude, which is exactly what you want when the goal is to measure how large recent gradients have been, not which direction they pointed. Taking the square root afterward brings that magnitude back to the same units as the original gradient, so the denominator can sensibly rescale it. Because v_t blends in only a (1 - rho) fraction of the newest squared gradient and multiplies everything already stored by rho, older values are discounted by a compounding factor of rho every step: a squared gradient from many steps ago has been multiplied by rho raised to a large power and has shrunk toward zero. As an informal rule of thumb, a larger rho (closer to 1) corresponds to a longer effective memory of past gradients, though RMSProp does not define this as an exact, fixed-length window.

Effective Learning Rate: Why the Denominator Matters

The quantity eta / (sqrt(v_t) + epsilon) behaves as an effective, per-parameter learning rate. When a parameter's recent gradients have been persistently large, v_t is large, the denominator is large, and the effective step shrinks — this damps oscillation on steep or noisy directions. When recent gradients have been small, v_t is small, the denominator is small, and the effective step grows relatively larger — this speeds up progress on flat or quiet directions. This is a genuinely useful self-adjusting mechanism, but it is not magic. RMSProp does not discover an optimal learning rate on its own; eta is still a hyperparameter you choose, and a poor choice of eta, rho, or epsilon can still produce slow or unstable training. RMSProp reshapes the geometry of the optimization problem per parameter; it does not remove the need for sensible hyperparameters.

A Worked RMSProp Numerical Example

To make the update rule concrete, here is one parameter, theta, optimized for three steps with a learning rate of eta = 0.1, decay rho = 0.9, epsilon = 1e-8, and starting value theta_0 = 5.0. Suppose the observed gradients at each step are g_1 = 2, g_2 = -3, and g_3 = 1.

Step 1 — gradient g_1 = 2. Previous average v_0 = 0. New average: v_1 = 0.9(0) + 0.1(2^2) = 0.4. Denominator: sqrt(0.4) + epsilon ≈ 0.632456. Scaled update: 0.1 × 2 / 0.632456 ≈ 0.316228. New parameter: theta_1 = 5.0 − 0.316228 = 4.683772.

Step 2 — gradient g_2 = −3. Previous average v_1 = 0.4. New average: v_2 = 0.9(0.4) + 0.1(9) = 1.26. Denominator: sqrt(1.26) + epsilon ≈ 1.122497. Scaled update: 0.1 × (−3) / 1.122497 ≈ −0.267261. New parameter: theta_2 = 4.683772 − (−0.267261) = 4.951033.

Step 3 — gradient g_3 = 1. Previous average v_2 = 1.26. New average: v_3 = 0.9(1.26) + 0.1(1) = 1.234. Denominator: sqrt(1.234) + epsilon ≈ 1.110856. Scaled update: 0.1 × 1 / 1.110856 ≈ 0.090025. New parameter: theta_3 = 4.951033 − 0.090025 = 4.861008.

Notice what happened at step 2: the gradient magnitude tripled compared to step 1 (from 2 to 3), but the scaled update only grew from about 0.316 to about 0.267 in magnitude — it actually shrank slightly, because the moving average v had already risen from the large step-1 gradient and partly absorbed the shock. This is RMSProp's self-normalizing behavior in miniature: large recent gradients build up v and progressively damp the update, rather than letting one big gradient produce one enormous, destabilizing step.

RMSProp From Scratch in NumPy

A minimal, from-scratch implementation makes the update rule easy to see in code. This teaching version is not a byte-for-byte reproduction of PyTorch's or Keras's internal kernels — those add extra options like momentum, centering, weight decay, and framework-specific epsilon placement — but it implements the same core mechanism and reproduces the worked example above exactly.

import numpy as np

def rmsprop_update(params, grads, cache, lr=0.01, rho=0.9, eps=1e-8):
    """One RMSProp step for a dict of parameters (matches the worked example)."""
    for key in params:
        cache[key] = rho * cache[key] + (1 - rho) * (grads[key] ** 2)
        params[key] -= lr * grads[key] / (np.sqrt(cache[key]) + eps)
    return params, cache

theta = {"w": np.array([5.0])}
cache = {"w": np.zeros_like(theta["w"])}

for g in [2.0, -3.0, 1.0]:
    grads = {"w": np.array([g])}
    theta, cache = rmsprop_update(theta, grads, cache, lr=0.1, rho=0.9, eps=1e-8)
    print(theta["w"])
# -> [4.68377223]
# -> [4.95103262]
# -> [4.86100799]

How to Use RMSProp in PyTorch

PyTorch exposes RMSProp as torch.optim.RMSprop. Its documented defaults are lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, and centered=False (PyTorch, current documentation). PyTorch's implementation takes the square root of the moving average before adding epsilon — that is, sqrt(v) + eps — and describes the effective learning rate as the scheduled learning rate divided by (sqrt(v) + eps).

import torch

optimizer = torch.optim.RMSprop(
    model.parameters(),
    lr=0.01,          # learning rate, default 1e-2
    alpha=0.99,        # smoothing/decay constant, default 0.99
    eps=1e-08,         # numerical-stability term, default 1e-8
    weight_decay=0,    # L2 penalty, default 0
    momentum=0,        # momentum factor, default 0 (off)
    centered=False,    # centered variant, default False
)

optimizer.zero_grad()
loss.backward()
optimizer.step()

PyTorch Argument Notes

alpha is PyTorch's name for the decay factor referred to as rho elsewhere in this article — it is not a learning rate. Setting momentum > 0 adds a separate velocity buffer on top of the adaptive scaling. Setting centered=True adds a running mean-of-gradients buffer and normalizes by an estimated variance instead of the raw uncentered second moment, at extra memory and compute cost (PyTorch, current documentation).

How to Use RMSProp in TensorFlow and Keras

Keras exposes RMSProp as keras.optimizers.RMSprop (and, for TensorFlow's integrated Keras, tf.keras.optimizers.RMSprop). Its documented defaults are learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, and centered=False (Keras documentation, current). This implementation uses plain momentum, not Nesterov momentum, and — this is a real, documented difference from PyTorch — it adds epsilon inside the square root, computing sqrt(v + eps) rather than sqrt(v) + eps. PyTorch's own documentation explicitly notes that "TensorFlow interchanges these two operations."

import tensorflow as tf

optimizer = tf.keras.optimizers.RMSprop(
    learning_rate=0.001,  # default 0.001
    rho=0.9,               # decay factor, default 0.9
    momentum=0.0,           # default 0.0 (off)
    epsilon=1e-07,          # default 1e-7
    centered=False,         # default False
)

model.compile(optimizer=optimizer, loss="categorical_crossentropy")
model.fit(x_train, y_train, epochs=10)

Why the Epsilon Placement Difference Matters

In practice, with the default epsilon values, PyTorch's sqrt(v) + eps and Keras's sqrt(v + eps) behave almost identically once v is not tiny, because epsilon is only there to prevent division by exactly zero. The difference becomes more noticeable only when v itself is extremely small (very early in training, or for parameters with near-zero gradients), where the two formulas can diverge slightly. This is a genuine implementation difference worth knowing about, not an error in either framework.

RMSProp Hyperparameters and How to Tune Them

RMSProp has a small number of hyperparameters, but each one changes training behavior in a distinct way. There is no single universally best configuration — the right values depend on the model, data, and task — but the sections below explain what each one controls and what to watch for.

Learning Rate (eta)

The learning rate is the base step-size multiplier applied after RMSProp's per-parameter rescaling. RMSProp's adaptive scaling changes how a given gradient is rescaled, but it does not remove the need to choose eta — a poor base learning rate still causes slow training (too low) or divergence and oscillation (too high). A common starting point is the framework default (0.01 in PyTorch, 0.001 in Keras) with a small grid or logarithmic search around it. If the loss oscillates or produces NaNs, the learning rate is a first suspect; if progress is extremely slow despite falling loss on early steps, it may be too small.

Rho / Alpha / Decay

This decay factor (called rho in Keras, alpha in PyTorch) controls how quickly old squared gradients lose influence in the moving average. Values close to 1 (like the common defaults of 0.9 or 0.99) retain a longer effective memory of recent gradient magnitude; smaller values forget faster and make v_t react more quickly to sudden changes in gradient scale, at the cost of noisier scaling. Note that rho is not a learning rate and should not be tuned as if it were one — it governs memory length, not step size.

Epsilon

Epsilon exists purely for numerical stability, to prevent division by a value that is exactly (or very close to) zero when v_t is tiny, typically at the very start of training or for parameters that receive almost no gradient. As covered above, some frameworks add epsilon before taking the square root and some add it after, which is a real, documented implementation difference rather than an error. Epsilon is rarely worth tuning aggressively; the framework defaults (1e-8 in PyTorch, 1e-7 in Keras) work for the great majority of use cases.

Momentum

RMSProp's adaptive per-parameter scaling and momentum are two different mechanisms that solve different problems, and RMSProp on its own has no momentum. Adaptive scaling adjusts step size based on recent gradient magnitude; momentum accumulates a velocity from the direction of recent gradients to smooth out the trajectory and push through small local irregularities in the loss surface. Both PyTorch and Keras let you add plain momentum to RMSProp via the momentum argument (default 0, i.e. off), which layers a velocity buffer on top of the adaptive scaling rather than replacing it.

Centered

The centered variant maintains an additional running mean of the raw (unsquared) gradient and uses it to estimate the gradient's variance, normalizing by that variance estimate instead of by the raw uncentered second moment. Both PyTorch and Keras document that this centered version "first appears" in the paper Generating Sequences With Recurrent Neural Networks (Graves, 2013). Centering can help stabilize training in some settings, but it requires an extra per-parameter buffer, adding memory and a small amount of extra computation compared to standard, uncentered RMSProp.

Weight Decay

Where a framework exposes it (weight_decay in PyTorch, an optional weight_decay in current Keras), weight decay applies an L2-style penalty that shrinks parameters toward zero, added into the effective gradient before the RMSProp rescaling is applied. This is a separate mechanism from RMSProp's core adaptive-scaling logic — it's a regularization technique layered on top, distinct from the decoupled weight decay used in AdamW, which is applied directly to the parameter rather than folded into the gradient before adaptive scaling.

Centered RMSProp and RMSProp With Momentum

Two optional extensions are commonly bundled with framework RMSProp implementations, and it is worth being precise about what each one actually changes, since it is easy to conflate them with the core algorithm.

Centered RMSProp keeps a second moving average — this time of the raw gradient itself, not its square — and uses it to compute an estimated variance: roughly, the mean of the squared gradient minus the square of the mean gradient. Dividing by the square root of this variance estimate, rather than by the raw second moment, is meant to give a normalization that better reflects how much the gradient is actually fluctuating around its recent trend. Both PyTorch's and Keras's documentation trace this specific variant to Alex Graves's 2013 paper on generating sequences with recurrent neural networks, rather than to Hinton's original 2012 course material. Centering adds one extra per-parameter buffer, so it costs more memory and a little more computation than plain RMSProp.

RMSProp with momentum adds a completely separate mechanism: a velocity buffer that accumulates a fraction of the previous velocity plus the current (already adaptively rescaled) update, smoothing the optimizer's trajectory across steps the way classical momentum smooths SGD. This is not the same thing as Adam's first-moment tracking, even though both involve a moving average of gradient-derived quantities — RMSProp's momentum, when enabled, is applied after the adaptive rescaling step, whereas Adam tracks first and second moments of the raw gradient in parallel and combines them with bias correction.

RMSProp vs SGD, Momentum, AdaGrad, Adam, and AdamW

SGD | No | No | No | No | None beyond parameters | Simple, well understood, often generalizes well with a good schedule. | Needs careful learning-rate tuning and scheduling; slow on ill-conditioned problems.

The table above compares six optimizers along the dimensions that matter most in practice: whether they scale each parameter's step individually, whether they track a momentum-like first moment, whether they track a second-moment (squared-gradient) estimate, whether they apply bias correction, and roughly how much extra optimizer state they require relative to the parameters themselves.

RMSProp and Adam are related but not interchangeable. Both maintain a decaying moving average of squared gradients and use it to rescale updates — that part of Adam is directly descended from the same idea RMSProp popularized. Adam adds two things RMSProp's canonical form does not have: an exponentially decaying moving average of the raw gradient itself (a first-moment estimate, functioning like built-in momentum) and bias correction terms that counteract the fact that both moving averages start at zero and are therefore biased low in early steps. RMSProp, as originally described, has neither of those by default, though momentum can be added on top in most implementations.

AdamW is not simply "Adam with weight decay" — ordinary Adam with an L2 penalty folds that penalty into the gradient before the adaptive rescaling is applied, which interacts with the per-parameter scaling in a way that can weaken the intended regularization. AdamW instead applies weight decay directly to the parameters, decoupled from the gradient-based update, which has become a common default especially for training transformer-based models.

None of this makes any one optimizer universally "better." Adam-family methods are common defaults because they tend to need less manual tuning across a wide range of architectures, but SGD with momentum, properly tuned, still produces excellent or superior generalization on some vision benchmarks, and RMSProp remains a legitimate, lower-memory choice, particularly for recurrent architectures and reinforcement learning settings where it has a long track record.

Advantages of RMSProp

Fixes AdaGrad's core weakness: the effective learning rate no longer shrinks monotonically toward zero over long training runs.
Well suited to non-stationary objectives, including recurrent neural networks and reinforcement learning, where the right step size for a parameter can change as training progresses.
Conceptually simple, with few hyperparameters and no bias-correction machinery to reason about.
Adapts automatically to parameters with very different gradient scales, without hand-tuning a separate learning rate per layer.
Modest memory overhead in its standard form — one extra buffer per parameter, the same order of magnitude as AdaGrad or plain momentum.

Limitations and Failure Modes of RMSProp

No bias correction: because v_t starts at zero, early-training estimates of the squared-gradient average are biased low, which can make the very first steps behave differently than later, well-warmed-up steps.
Still requires a tuned base learning rate — the adaptive scaling changes how a gradient is rescaled, it does not choose eta for you.
Can generalize worse than a well-tuned SGD-with-momentum baseline on some tasks, a tension documented across adaptive-gradient research broadly, not unique to RMSProp.
Poorly chosen rho or epsilon can cause oscillation (rho too small, reacting too fast to noisy gradients) or sluggish adaptation (rho too close to 1).
Like other first-order stochastic optimizers, RMSProp offers no general convergence or generalization guarantee on arbitrary non-convex objectives.

When Should You Use RMSProp?

RMSProp tends to be a good fit when training recurrent neural networks or other models with non-stationary, sequentially dependent objectives, where Hinton originally popularized it. It is also a reasonable choice in reinforcement learning settings with noisy, changing reward signals, and in general as a lightweight adaptive optimizer when you want less optimizer state than Adam keeps, or want to closely follow a course, paper, or codebase that specifies it.

Adam or AdamW is usually the more practical default when you want a low-maintenance optimizer for a new architecture, especially large transformer-based models, where AdamW's decoupled weight decay is now a common standard choice. SGD with momentum, and a carefully tuned learning-rate schedule, is often worth the extra tuning effort when training convolutional vision models where flat-minima generalization is a priority and compute budget allows for schedule tuning.

In all cases, the only reliable way to know which optimizer is best for a specific model and dataset is to run a controlled comparison, since published results on other tasks do not guarantee the same ranking on yours.

Common RMSProp Mistakes and Troubleshooting

Symptom: Loss diverges or becomes NaN. Likely cause: Learning rate too high, or epsilon too small allowing a near-zero denominator. Fix: Lower the learning rate; confirm epsilon matches the framework's documented default; add gradient clipping if spikes are the cause

Symptom: Training oscillates without settling. Likely cause: Learning rate too high relative to rho, or rho too low (short memory reacting to every noisy gradient). Fix: Reduce the learning rate, or raise rho toward the framework default (0.9–0.99) for a longer effective memory

Symptom: Training progresses extremely slowly. Likely cause: Learning rate too low, or rho too close to 1 with a cold-start v_t suppressing early updates. Fix: Increase the learning rate modestly; verify v_t is actually growing early in training by logging it

Symptom: Results differ between PyTorch and TensorFlow with 'the same' settings. Likely cause: Different default epsilon (1e-8 vs 1e-7) and different epsilon placement (outside vs inside the square root). Fix: Match epsilon values explicitly and treat small early-training differences as expected, not a bug

Symptom: Changing rho without understanding its effect. Likely cause: Treating rho like a learning rate and tuning it as the primary knob. Fix: Tune eta first for step size; treat rho as a secondary knob that controls memory length, not step size

Symptom: Copying hyperparameters from one framework to another. Likely cause: PyTorch and Keras have different default learning rates (0.01 vs 0.001) and epsilons. Fix: Re-tune, or explicitly set every argument to match the source framework's exact defaults

Symptom: Confusing the squared-gradient accumulator with momentum. Likely cause: Assuming v_t is a velocity term the way SGD-momentum's buffer is. Fix: Remember v_t rescales step size based on magnitude; momentum, if enabled, is a separate buffer that smooths direction

Symptom: Forgetting optimizer state when resuming training. Likely cause: Reinitializing the optimizer instead of loading its saved state dict. Fix: Save and restore the optimizer's state (including v_t and any momentum/centered buffers) alongside model weights

Symptom: Exploding gradients from an unrelated cause. Likely cause: A separate architectural or data issue (e.g. unstable recurrent unrolling) unrelated to RMSProp itself. Fix: Add gradient clipping (e.g. clip-by-norm) independent of the optimizer choice

Computational and Memory Complexity

RMSProp's computational cost per step is O(P), where P is the number of parameters — one multiply-add-style operation per parameter to update v_t, plus one division to rescale the gradient. This is the same asymptotic cost as plain SGD, and in practice it is negligible compared to the cost of the forward and backward passes through the network.

Memory overhead comes from the extra state RMSProp must store. In its standard (uncentered, no-momentum) form, RMSProp keeps one additional buffer, v_t, the same shape as the parameters — roughly doubling the memory devoted to parameters plus optimizer state compared to plain SGD, which needs no persistent per-parameter state at all. Enabling momentum adds a second buffer (a velocity term), and enabling the centered variant adds a third (a running mean of the raw gradient), so a fully centered, momentum-enabled RMSProp configuration can require roughly three extra parameter-sized buffers in total. For context, Adam and AdamW require two extra buffers (first and second moment estimates) by default, so plain RMSProp typically has a smaller memory footprint than Adam, while centered RMSProp with momentum can approach or exceed it.

RMSProp in Modern Deep Learning

Adam-family optimizers, and AdamW in particular, are the most common default choice for training large modern architectures, especially transformer-based language and vision models, largely because they tend to need less per-task tuning out of the box and AdamW's decoupled weight decay has become a standard regularization choice for that model family. That popularity does not make RMSProp obsolete.

RMSProp has a long, well-documented history in recurrent neural network training, which is the setting Hinton originally introduced it for, and it remains a reasonable choice for sequence models and other non-stationary objectives. It has also seen substantial historical use in reinforcement learning research, where noisy, non-stationary reward signals suit an optimizer that adapts its scaling continuously rather than assuming a stationary gradient distribution. Its smaller optimizer-state footprint compared with Adam or AdamW can matter in memory-constrained settings.

The practical takeaway is not that one optimizer is universally superior in 2026. It is that Adam and AdamW are sensible starting points for most new projects because they are well-tested defaults across a wide range of tasks, while RMSProp remains a legitimate, specifically useful tool — not a historical curiosity — particularly for recurrent architectures, reinforcement learning, and situations where a simpler, lighter-weight adaptive method is preferable. The right choice ultimately depends on the model, the data, and empirical results from your own experiments, not on which optimizer happens to be most fashionable.

FAQ

What does RMSProp stand for?

RMSProp stands for Root Mean Square Propagation. The name refers to how the algorithm rescales each gradient using the square root of a running average of that gradient's squared values.

Who invented or introduced RMSProp?

RMSProp is credited to Geoffrey Hinton, who introduced it in Lecture 6e of his 2012 Coursera course, Neural Networks for Machine Learning, with the lecture material co-credited to Tijmen Tieleman.

Is there an original RMSProp paper?

No. Unlike AdaGrad or Adam, RMSProp was never published as a standalone, peer-reviewed paper. It is standardly cited as Tieleman, T. and Hinton, G. (2012), Lecture 6.5-rmsprop, COURSERA: Neural Networks for Machine Learning, 4(2), 26–31 — a course lecture, not a journal or conference paper.

How does RMSProp work?

RMSProp keeps an exponentially decaying moving average of each parameter's squared gradients, then divides that parameter's current gradient by the square root of the average (plus a small stability constant) before applying the learning rate. This gives every parameter its own effective, continuously updated step size.

Why does RMSProp square the gradient?

Squaring discards the gradient's sign and keeps only its magnitude, so the moving average measures how large recent gradients have been regardless of direction. The square root is taken later to bring that magnitude back to the gradient's original scale.

What does rho do in RMSProp?

Rho (called alpha in PyTorch) is the decay factor controlling how quickly old squared gradients lose influence in the moving average. Values close to 1 give a longer effective memory; smaller values make the average react faster to recent changes. Rho is not a learning rate.

What is epsilon in RMSProp?

Epsilon is a small constant added to the denominator to prevent division by zero when the moving average is very small, most often near the start of training. Frameworks differ on whether epsilon is added before or after the square root; both are documented, valid choices.

Is RMSProp better than Adam?

Neither is universally better. Adam adds momentum-like first-moment tracking and bias correction on top of the same squared-gradient idea RMSProp uses, and tends to need less tuning as a general-purpose default. RMSProp remains a lighter-weight, still-effective choice, particularly for recurrent networks and reinforcement learning.

What is the difference between RMSProp and AdaGrad?

AdaGrad accumulates the sum of all squared gradients ever observed, which causes its effective learning rate to shrink continuously and can stall training over long runs. RMSProp replaces that permanent accumulation with an exponentially decaying moving average, so old gradients lose influence instead of being remembered forever.

What is centered RMSProp?

Centered RMSProp keeps an additional moving average of the raw (unsquared) gradient and uses it to estimate the gradient's variance, normalizing by that variance estimate rather than by the raw squared-gradient average. Both PyTorch and Keras document this variant as first appearing in Alex Graves's 2013 paper on generating sequences with recurrent neural networks.

Can RMSProp use momentum?

Yes. Both PyTorch's and Keras's implementations expose a momentum argument (off by default) that adds a separate velocity buffer on top of RMSProp's adaptive per-parameter scaling, distinct from Adam's built-in first-moment tracking.

What learning rate should I start with?

A sensible starting point is the framework's documented default — 0.01 in PyTorch's torch.optim.RMSprop, 0.001 in Keras's RMSprop — followed by a small search around that value for your specific model and data, since the right value is task-dependent.

Is RMSProp still used?

Yes. While Adam and AdamW are more common defaults for many modern architectures, RMSProp continues to be used for recurrent networks, reinforcement learning, and situations calling for a simpler, lower-memory adaptive optimizer.

When should I use RMSProp?

It is a reasonable choice for recurrent or sequential models, non-stationary objectives, and reinforcement learning, or whenever you want an adaptive optimizer with a smaller memory footprint than Adam. For many new transformer-style architectures, AdamW is a more common starting point.

What are RMSProp's disadvantages?

RMSProp has no built-in bias correction, still requires a tuned base learning rate, offers no convergence or generalization guarantees, and can be outperformed by a well-tuned SGD-with-momentum baseline on some vision tasks.

Key Takeaways

RMSProp rescales each parameter's gradient by the square root of an exponentially decaying moving average of that parameter's squared gradients.
It was introduced by Geoffrey Hinton in Lecture 6e of his 2012 Coursera course; there is no separate, peer-reviewed original RMSProp paper.
RMSProp fixes AdaGrad's main flaw — a monotonically shrinking effective learning rate — by letting old squared gradients decay instead of accumulating forever.
PyTorch (lr=0.01, alpha=0.99, eps=1e-8) and Keras (learning_rate=0.001, rho=0.9, epsilon=1e-7) use different defaults and different epsilon placement (outside vs inside the square root).
RMSProp can be extended with momentum and with a centered variant that normalizes by an estimated variance instead of the raw second moment.
Adam builds on the same squared-gradient idea but adds first-moment tracking and bias correction, which is why Adam-family optimizers are more common defaults today.
RMSProp remains a legitimate choice for recurrent networks, reinforcement learning, and lower-memory adaptive optimization — not an obsolete method.

Actionable Next Steps

Read Hinton and Tieleman's original Lecture 6e slides to see the reasoning that motivated RMSProp firsthand.
Implement the from-scratch NumPy version above and reproduce the worked numerical example to confirm you understand the update rule.
Try torch.optim.RMSprop or keras.optimizers.RMSprop on a small model, logging the loss with a few different learning rates and rho values.
Compare RMSProp against Adam and SGD-with-momentum on the same task and dataset to see how they actually differ for your specific problem.
If training an RNN or working in reinforcement learning, experiment with RMSProp as a first-choice optimizer given its documented history in those areas.
Read the AdaGrad and Adam papers directly (both linked in Sources & References) to understand exactly what each one adds relative to RMSProp.

Glossary

Gradient: The vector of partial derivatives of a loss function with respect to a model's parameters; it points in the direction of steepest increase of the loss.

Optimizer: An algorithm that updates a model's parameters, using gradients, to reduce a loss function during training.

Learning rate: A scalar hyperparameter that scales how large a step an optimizer takes in the direction indicated by the gradient.

Adaptive learning rate: A learning-rate scheme that adjusts automatically, often per parameter, based on the optimizer's own tracked statistics rather than staying fixed.

Exponential moving average: A running average where more recent values are weighted more heavily than older ones, which are discounted by a compounding decay factor.

Squared gradient: A gradient value multiplied by itself, used to measure the magnitude of recent gradients while discarding their sign.

RMS: Root mean square: the square root of the mean of a set of squared values, used here to convert an averaged squared gradient back into the gradient's original scale.

Decay factor: The hyperparameter (rho or alpha) controlling how quickly older values lose influence in an exponential moving average.

Epsilon: A small constant added to a denominator to prevent division by zero or near-zero values.

Parameter: A learnable value in a model, such as a weight or bias, adjusted during training to minimize the loss.

Momentum: An optimization technique that accumulates a velocity from recent gradients to smooth an optimizer's trajectory across steps.

AdaGrad: An adaptive-gradient optimizer that divides each gradient by the square root of the sum of all squared gradients observed so far for that parameter.

RMSProp: An adaptive-gradient optimizer that divides each gradient by the square root of an exponentially decaying moving average of that parameter's squared gradients.

Adam: An optimizer that combines an exponentially decaying moving average of the gradient (a first moment) with one of the squared gradient (a second moment), plus bias correction.

Centered RMSProp: A variant of RMSProp that normalizes by an estimated variance of the gradient, computed from separate running averages of the gradient and its square, rather than by the raw squared-gradient average.

Effective learning rate: The learning rate actually applied to a parameter after any adaptive per-parameter rescaling, as opposed to the base learning rate hyperparameter.