Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview - a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. Most people learn neural networks by staring at the model. Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows. But when you actually train an LLM, there is another piece of machinery making billions of decisions every second: the optimizer. A 70-billion-parameter model does not "learn" because gradient descent tells it which direction is better. It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates. For the last decade, the dominant answer has largely been some form of Adam, and increasingly AdamW. The interesting part is that Adam is not some mysterious LLM-specific invention. The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era. Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto. Three years later, the Transformer paper used Adam directly in its training recipe. Then came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization. By 2025, Adam was sufficiently influential to receive an ICLR Test of Time award. So what exactly is Adam doing? And why is AdamW usually what you actually want when training a Transformer? 1. First, forget Adam: what problem is the optimizer solving? Suppose your neural network has parameters theta = [theta_1, theta_2, ..., theta_N] and your training batch produces a loss L . Backpropagation gives you g = dL/dtheta The simplest possible optimizer is gradient descent: theta m -> v -> normalization So the shrinkage of a parameter becomes entangled with Adam's gradient statistics. That produces a surprising effect. Two parameters with the same weight magnitude can receive different effective regularization depending on their gradient history. Suppose: theta_1 = 1 theta_2 = 1 and the only difference is that: sqrt(v_1) = 0.1 sqrt(v_2) = 10 The same regularization contribution gets normalized very differently. So the thing you thought was: "shrink every weight by some amount" has turned into something closer to: "shrink weights according to how the optimizer's adaptive statistics happen to scale their gradients." That is not the same operation. Enter AdamW In 2017, Ilya Loshchilov and Frank Hutter proposed a simple fix. Don't put weight decay inside the gradient. Do it separately. Instead of conceptually doing: g Adam exponential moving averages | | | v | m_t, v_t | | | v | adaptive update | +----> AdamW weight decay | v parameters There are several consequences worth keeping in your head. Learning rate is still incredibly important Adam does not eliminate the need to tune the learning rate. The optimizer normalizes gradients, but alpha still determines the global scale of movement. A useful mental model is: Adam decides: "How large should this parameter's step be relative to its gradient history?" Learning rate decides: "How aggressive should the entire optimizer be?" That is why learning-rate schedules remain central in LLM training. The Transformer paper, for example, used a warmup followed by inverse-square-root decay rather than holding the learning rate constant. beta1 controls gradient-memory timescale The moving average m_t = beta1*m_(t-1) + (1-beta1)*g_t has an effective memory on the order of roughly: 1 / (1 - beta1) steps. So: beta1 = 0.9 means roughly a ten-step memory scale. That is not an exact cutoff; it is an intuition for the EMA timescale. Likewise: beta2 = 0.999 corresponds to a much longer memory: ~1000 steps for the second-moment estimate. This is why changing beta values is not just changing some arbitrary constants. You're changing the temporal horizon over which the optimizer interprets gradient behavior. epsilon is mostly a numerical stabilizer The denominator is: sqrt(v_hat) + epsilon The epsilon prevents division by something vanishingly small. In many practical regimes, it is not the dominant behavioral hyperparameter. But in low-gradient or low-precision regimes, its interaction with numerical scale can matter. Adam is not Newton's method A common misunderstanding is: "Adam uses second-order information." Not really. It tracks a second moment of gradients: E[g^2] but it does not construct the Hessian: H = d^2L/dtheta^2 and does not estimate the full curvature matrix. Adam is still a first-order optimizer. Its sophistication comes from using historical statistics of first-order information. 8. Why developers should care beyond knowing the equations If you are debugging LLM training, AdamW is not an implementation detail. It can directly influence: training stability loss curves sample efficiency generalization memory footprint distributed-training architecture hyperparameter sensitivity A few practical examples: Training is unstable You might immediately suspect: bad initialization bad normalization bad data exploding gradients But the optimizer configuration is also part of the system. A learning rate that is perfectly reasonable under one optimizer can behave differently under another. Loss decreases but validation quality stagnates Weight decay becomes interesting. Because AdamW separates optimization from regularization, you can reason about: learning rate and weight decay as two separate control knobs. That conceptual separation is much cleaner than treating "L2 regularization" as something buried inside the gradient. GPU memory is unexpectedly full Check the optimizer state. For a 7B model: Adam moments ≈ 56 GB in FP32 That number alone can explain a lot of apparently mysterious infrastructure decisions. You scale the model and everything changes This is an increasingly interesting research question. The optimal AdamW weight decay is not necessarily a universal constant that you can blindly copy from a smaller model. Recent work has explicitly studied how the optimal weight decay changes with model size, dataset size, and training dynamics. In other words, once you're operating at serious scale, "just set AdamW to 0.1" is more cargo cult than theory. 9. The big picture: Adam is really a control system for noisy learning The cleanest mental model I know is this: A neural network is trying to optimize an absurdly high-dimensional function using noisy measurements. The raw gradient says: "Here is what today's minibatch thinks you should do." Adam says: "Fine. But I also remember what the gradients have been doing lately." It keeps track of: direction -> m scale -> v and uses those statistics to construct an adaptive update. AdamW then says: "And separately, I want the parameters to decay." That separation turns out to matter. So the evolution is roughly: SGD | +-- momentum | +-- adaptive scaling | v Adam | +-- decoupled weight decay | v AdamW And the reason this matters for LLMs is not that Adam is mathematically glamorous. It is that training billion-parameter models is fundamentally an optimization-and-systems problem. The model might contain 70 billion parameters, but every one of those parameters is being updated by a tiny piece of state maintained over the entire training trajectory. That makes the optimizer part of the model's computational machinery. The next time you see: optimizer = AdamW(...) you are not looking at five lines of boilerplate. You are looking at a compact algorithm that is simultaneously doing: momentum adaptive normalization bias correction parameter updates regularization for billions of variables, potentially millions of times. That is a rather extraordinary amount of machinery hiding behind one constructor. One question for you When you train or fine-tune an LLM, how much attention do you actually pay to the optimizer compared with the model architecture and data? Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius - how far its impact reaches through your call graph - so you can focus attention where it actually matters. Spend code review effort where business risk is highest - not spread evenly across every diff. Try LiveReview on your codebase: Top comments (0)