Lesson 4 of 5 · How Learning WorksRead
Optimization: What Actually Changes the Weights
How the optimizer turns the notes from backpropagation into real steps: with momentum, with a step size of its own for every fader, with a gentle pull toward zero and on a schedule for the learning rate.
At the end of the last lesson, each of the roughly eight billion faders on Llama 3.1 8B’s mixing desk had a note on it. Backpropagation had worked it out for one batch, a stack of text that the loss turns into a single error number. Each fader stands for a parameter, its setting for that parameter’s number. Its note says, for this batch only: up or down, and how strongly. All the notes together are called the gradient. Not a single fader has moved yet.
With the next batch, many notes point a little differently, some the opposite way. What do you think: where does it lead to follow the notes every time?
Who turns: the optimizer, with momentum
From the lesson on training you know the simplest answer: step equals slope times learning rate. A note is this slope, for each fader separately: which way is downhill, and how steeply. With the made-up apple model and a learning rate of one twenty-fourth, every step halved the rest of the way to the valley. With one sixth, the fader jumped back and forth between 0 and 6. A fixed calculation like this, which turns notes into steps, is called an optimizer. In training, the optimizer is what turns the faders, step by step.
Following the simple rule to the letter causes a problem, as a made-up model with two faders shows. Its error forms a foggy hillside, as in the lesson on training, now with two directions: one fader moves you across, the other along. The hillside is a narrow ravine: across, the ground falls steeply toward its middle; along, it only slopes gently downhill. With large steps you land on the opposite side each time, like the apple model at 6, and your path zigzags. Smaller steps stop the jumping, but then you barely progress along the ravine. And the hillside wobbles: every batch delivers slightly different notes.
Smooth made-up numbers show what helps. Across, you jump from side to side, so the notes there alternate between +8 and −8. Along, they say 1 every time. The notes only apply to their batch and are gone afterwards. What remains of them from batch to batch is the momentum: not physics, but a stored number per fader that takes in a bit of each new note.
The optimizer follows the momentum instead of every note. The new momentum is old momentum and a tenth new note. The step is learning rate times momentum, not times note. What points back and forth across almost cancels out; what always points the same way along accumulates. After twenty notes, the momentum across barely swings around zero, while along it has grown to almost the full note.
Its technical term is Momentum. A textbook describes it as an average of earlier slopes in which older ones count less and less, and the optimizer keeps moving in their direction. This form of momentum averages: the new note enters with only 10 percent. GPT-3, a forerunner of the models behind ChatGPT, kept these 90 percent in training. The share is chosen, so it is a hyperparameter.
With this averaging, the path along the ravine stays below the full note. It becomes calmer, so it can take larger steps without building up across.
One level deeper: two ways of writing momentum
The textbook’s formula, however, adds up the earlier notes without dividing, and at the same learning rate its steps become up to ten times as large. The main text’s form works like this sum with a learning rate ten times smaller. A fader that keeps getting the same note gets a momentum ten times that note in the sum: each new note comes in fully, and the old momentum keeps only 90 percent. The main text’s form stays at that note itself, because it takes only 10 percent of the new note. For a comparison, the summing form needs a learning rate ten times smaller. The summing form goes back to Boris Polyak (1964).
Momentum smooths out the back and forth. But it does nothing about some faders always getting large notes and others tiny ones.
Every fader its own step size
Earlier lessons said only this: large language models set each fader’s step by how large its slopes have been so far, and keep two helper values per fader for that. What are they for?
Two made-up faders, each with four notes from four batches. Instead of the mixing above, the table simplifies and takes a plain average of the four notes twice: with signs, where plus and minus cancel out, and without signs. The second is called usual size here: how strong the notes are on average, whichever way they point. For P the notes cancel out with signs; without signs each lies between 8 and 11.
| Fader | Notes | Average with signs | Usual size |
|---|---|---|---|
| P | 10, −8, 9, −11 | 0 | about 9.5 |
| Q | 0.1, 0.1, 0.1, 0.1 | 0.1 | 0.1 |
With the plain rule, P’s last note gives a step of 11 times the learning rate, Q’s a step of 0.1 times, a hundred and ten times less. Yet P’s notes contradict each other, while Q always gets the same message, quietly. What do you think: which of the two should take the larger step, though the plain rule has it the other way round?

A good hiker in the ravine would choose a step length for each direction, not for each spot: short steps across, where the slope keeps changing, and long strides along, where it is flat but steadily downhill. Where the slope kept flipping during their last steps, they grow more careful. The optimizer has as many directions as there are faders and knows only the notes, so it works out a step length for each of billions of them on its own.
That is what the method Adam does, introduced by two researchers in 2014. For each fader it keeps two helper values: the momentum and the usual size of its notes. Its step is learning rate times momentum, divided by the usual size. For P the momentum is almost zero, so there is almost no step. For Q, momentum and usual size are equal, so the step is one full learning rate. Adam divides out the size of the notes: the usual size only measures how strong they are, while the momentum decides whether they point steadily one way.
Still, Adam is no model and learns nothing: it is a fixed calculation with chosen settings. Momentum and usual size are running values of training, not parameters, so the finished model does not need them. GPT-3 was trained with Adam.
One level deeper: How Adam measures the usual size
Adam measures the usual size differently from the table, which only shows the order of magnitude. It multiplies each note by itself, so every value is positive, and keeps a running average of these, which is extended each batch, like the momentum. From this average it takes the square root and gets back to the order of magnitude of the notes. With the simple average from the table, this works out for P like this: 10 times 10 is 100, 8 times 8 is 64, 9 times 9 is 81 and 11 times 11 is 121. Together that is 366; divided by 4, that is 91.5; and its square root is about 9.6, almost the same usual size as in the table.
Which fader moves further, P or Q?
Q. Its steps point the same way each batch and add up: after a hundred such batches, about a hundred learning rates far. P’s steps are tiny and point this way one time and that way the next. They largely cancel out, and P stays roughly where it was.
Live demo · try it yourself
Three rules in a ravine
Invented simulationA made-up mini model with two faders. Its error forms a long, narrow ravine, seen here from above: steep across, flat along, the valley on the left. Every batch brings noisy notes, and three rules turn the faders with the same notes: the plain rule, momentum and Adam.
- Plain rule
- Momentum
- Adam
Solution
About twice as fast: in this run Adam reaches the valley after 44 notes, the plain rule after 76, momentum after 74. How big the lead is depends on the notes, as “New notes” shows; the three levels of the answer list are a setting of the demo. Adam divides each fader’s momentum by the usual size of its notes. Across, where the notes are large and point back and forth, that makes the steps small; along, where they are small but steady, the step becomes about one full learning rate. Momentum alone makes the path calmer than the plain rule, but not faster along the ravine. At a learning rate of 0.22 the plain rule flies out of the ravine while momentum stays inside. With 99 percent momentum it turns sluggish and no longer arrives.
A made-up toy, not a trained model: two faders, much steeper across than along; the numbers come from the demo itself. The noise on the notes is invented and the same for all three rules. “Valley reached” means an error below 0.3 here. Real models have billions of faders and a slope with just as many directions; which rule suits them best cannot be read off this picture.
Adam is almost complete. But Meta’s report on the Llama 3 series names something with one more letter: AdamW.
The W: a gentle pull toward zero
In its research paper, Meta trained the largest Llama 3 model, Llama 3.1 with 405 billion faders, with AdamW. What does the W mean?

Picture a small spring on every fader of the mixing desk that pulls it toward zero. At every step, the optimizer first moves each fader a tiny bit toward zero, in proportion to its value, and then takes the step from momentum and usual size. The spring has no resting point other than zero, and it always pulls, whatever the note says. This pull is called weight decay. No fader is removed in the process, and none jumps to zero.
But what happens to a fader whose note says zero at every batch?
It shrinks, step by step, by the same small share of its value. After many steps it is close to zero. Faders the model needs get notes that keep pushing back against the pull. So faders that contribute nothing stay small, and no fader grows without reason.
Why a letter of its own? Adding the pull to the note would not work: Adam would divide it, together with the note, by the note’s usual size, so for a fader with large notes the pull would become tiny. So in 2017 two researchers proposed applying the pull separately, “decoupled”. The W in AdamW stands for this separate weight decay. The spring’s strength is a chosen number; according to the research paper, GPT-3 used 0.1. Common implementations multiply the pull by the learning rate, so at each step every fader shrinks by learning rate times 0.1 times its value. Large models have tiny learning rates, so that is a small fraction of the value.
That leaves the learning rate itself, fixed so far.
Up first, then down for a long time: the learning-rate schedule
If so much depends on the learning rate, why not pick a good one and keep it?
Live demo · try it yourself
Constant or on a schedule?
Invented simulationA made-up mini model with one fader, the valley at zero. Every batch brings a noisy note, even right at the bottom of the valley. Three learning rates run 100 steps with the same notes: constantly high (0.8), constantly low (0.01), and a schedule that changes the learning rate during the run.
- Constant high
- Constant low
- Schedule
Solution
The schedule. Constant high is almost in the valley after a few steps but keeps jittering, because every noisy note triggers a big step. Constant low barely jitters but is still far from the valley after a hundred steps. The schedule first ramps the learning rate up over ten steps, then takes big steps and tiny ones at the end: the noise hardly pushes it any more, and it comes to rest close to the bottom.
A made-up toy, not a trained model: one fader, the error is the fader setting times itself, halved. The noise on the notes is invented. The warm-up has little to do here: the toy has no random starting values and no per-fader step sizes like Adam, which it protects in real runs. New notes change the curves; the schedule almost always ends lowest, but not for every set of notes.
Because training needs different steps at start and end. Toward the end, the model is close to the valley, but the notes keep wobbling, because every batch contains different text. A textbook gives this wobbling as the reason to lower the learning rate with the simple rule: it does not vanish at the lowest point. Adam wobbles for the same reason. With large steps the model would keep being pushed out of the valley, like the apple model that jumped between 0 and 6. Small steps let it come to rest.
At the start, the faders are set at random and the notes are large and wild. From so few notes, Adam estimates the usual size only roughly, so its step sizes swing widely, and some become much larger than needed. With a high learning rate, such steps easily go wrong. A research team proposed that a slow start cushions this uncertainty, and backed it with experiments and theory. So the learning rate starts near zero and is ramped up quickly. This is called warm-up. After that, it slowly falls.
This pre-set course is called a learning-rate schedule. Like the learning rate itself, it is a hyperparameter. For the largest model in this series, Meta reports that it rose for 8,000 steps and then slowly fell over more than a million steps. The chart shows the values. At the end of training, the learning rate fell to zero.
What would happen if, toward the end, you suddenly jumped back to the peak learning rate?
The steps would become large again. Near the valley, the notes still wobble, and every large step pushes the model out, as the apple model did at one sixth. The loss would probably rise again. That is why a schedule ends with small steps and does not jump back.
One level deeper: What the optimizer stores per fader
The lesson on model size gave you the 16 bytes per parameter of a common setup. They come from the ZeRO research paper: 2 bytes for the 16-bit number, 2 bytes for its note and 12 bytes for the optimizer. The 12 bytes are the more precise 32-bit copy of the number plus Adam’s momentum and usual size, 4 bytes each.
So three quarters of what is stored per parameter belongs to the optimizer. Worked out for Llama 3.1 8B, that is about 96 of the 128 gigabytes. That is where ZeRO comes in. When many chips train together, each holds only its share of the optimizer values and turns only those faders. The paper works through an example with 7.5 billion parameters on 64 chips: 120 gigabytes per chip without splitting, 31.4 gigabytes when only the optimizer state is spread out.
At the start, Llama 3.1 8B’s eight billion faders had notes, and none had moved. Now you know who does the turning. How much text a model sees during training, and how you can tell that it is making progress, is shown in the next lesson.
Explain it more simply
- Explain more simplyCore idea and an everyday comparison
- With an exampleOne concrete example, step by step
- In more detailEvery step on its own, in plain words
Recall moment
What stayed with you?
Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.
Preparing questions …
Questions from the community
Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.