Lesson 6 of 6 · Inside the ModelRead
Output Head: From the Last State to a Prediction
How a language model computes a score for every token from its last state and why only the last position counts.
The previous lesson ended with a state at every position: a list of numbers that the blocks had filled with context. In the small language model Qwen3-0.6B, each state contains 1,024 numbers. A chat window, though, shows words, not the numbers behind them.
After “Die Hauptstadt von Frankreich ist” (German for “The capital of France is”), this same model predicts “Paris” at 47.5 percent. On another run with the same opening, it wrote “Zürich”. What calculation produced the 47.5 percent?
What comes out is a scoreboard, not a word
At the end, the model produces the score list mentioned earlier. It resembles the quiz-night scoreboard with three teams from the lesson on probability and softmax. This board has far more rows: for Qwen3-0.6B, there are 151,936. All but 267 rows belong to a token, one of the pieces known to the tokenizer. These 267 rows are spare: no piece carries their number. The fixed list of these pieces is the vocabulary. Every row gets a score, even the one for a comma or a Chinese character.

For this topic area, the free base version of Qwen3-0.6B, without chat training, was run on an ordinary computer. After “Die Hauptstadt von Frankreich ist”, “Paris” has the highest score at 20.5. A blank line, “____”, comes second at 19.1, as in worksheets and quizzes. Blank lines of other lengths take third and fourth place because each length is a separate token. The model apparently saw such texts often in training. “Bern” comes fifth. Softmax turns the scores into percentages.
By the rule of thumb from the softmax lesson, one point ahead means about 2.7 times as much. “Paris” is almost one and a half points ahead of the blank line and gets about four times as much: 47.5 against 12.0 percent.
“Bern” is three and a half points behind and only reaches 1.4 percent. All rows together make 100 percent. The top five have about two thirds, and the remaining rows share the last third in tiny pieces. Only as a share of all rows does “Paris” come to exactly 47.5 percent, the number from the start of this lesson.
You know from the lesson on input and output that the model rates every vocabulary entry and a separate step picks one: after “The cat sat”, “on” was ahead. Often, the chosen piece is not even a whole word. After “Der Hund jagt die” (“The dog chases the”), Qwen3 put word beginnings in the lead, such as “T” (as in “Taube”, pigeon), “F” (as in “Fliege”, fly) and “Kat” (as in “Katze”, cat). Which word these pieces become is decided only in the following rounds. What gets chosen is a token.
Scores can all be negative. In the older GPT-2, part of the experiment, “dog” led after “The dog chased the” with −86.4 and still got almost 20 percent. Only the gaps between the scores count: the highest gets the largest share, even if it is negative.
Where the points come from: one row per token
The part of the model that fills the board is called the output head. It is not a block but a simple step: only a comparison, no mixing. For every vocabulary entry, it has a row with as many numbers as the state. A token’s score measures how well the state fits its row.
Suppose states had only three numbers and the vocabulary four tokens: “cat”, “pigeon”, “duck” and “cloud”. For illustration, the three positions are called “animal”, “quick” and “object”. After “The dog chases the”, the last position holds the state (1.0 | 0.5 | −1.0): an animal fits, preferably a quick one, an object rather not.
The row for “cat” reads (2.0 | 1.0 | −1.5): clearly an animal, quick, not an object. The numbers say how well a word fits, not what it is.
The row for “duck” reads (0.5 | 0 | −1.0); its score is below.
The output head multiplies position by position and adds the results. You know this calculation from the attention lesson. For “cat”, the first position is 1.0 times 2.0, so 2.0.
The second position gives 0.5 times 1.0, so 0.5.
The third position: −1.0 times −1.5 gives plus 1.5, because minus times minus is plus. Together that is 4.0, the score of “cat”.
“cloud” has the row (−1.0 | 0 | 0). What score does it get? Work it out before reading on.
The score is −1.0. The sentence calls for an animal, and a cloud is not one. At the “animal” position, the state is positive but the row is negative, so the score goes down. The same rule applies throughout: matching signs add points, opposite signs subtract them. The farther a number is from zero, the more it matters.
So a high score does not mean that state and row are equal. If the state were exactly the row of “pigeon” (1.0 | 1.0 | −1.0), “cat” would still be ahead.
The score of “cat” would then be 4.5, that of “pigeon” 3.0.
“Cat” has pigeon’s signs everywhere and lies farther from zero in two positions.
Real models do the same, just bigger. In Qwen3-0.6B, the output head compares the state with 151,936 rows of 1,024 numbers each, about 156 million parameters set during training. This happens for every token a chatbot writes. There, the positions have no names.
Live demo · try it yourself
Which row fits the state?
The toy example from the text, now to move around. The three sliders are the state after “The dog chases the.” For every word, the output head multiplies position by position and adds up. The word with the highest score is at the top.
Try it: first guess which slider you have to move, and which way, to get “cloud” to the top. Then check. And: set the state to exactly the row of “pigeon” (1.0 | 1.0 | −1.0). Is “pigeon” in front then?
Scores, best on top
- catanimal × 2.0quick × 1.0object × −1.5
- pigeonanimal × 1.0quick × 1.0object × −1.0
- duckanimal × 0.5quick × 0object × −1.0
- cloudanimal × −1.0quick × 0object × 0
All numbers are made up and come from the example in the text, not from a real model. The names of the three positions (“animal,” “quick,” “object”) are invented for this demo only: real states have over a thousand positions without names. Only the scores are shown, not the percentages softmax makes of them.
No one wrote the output head’s rows; training set their numbers.
One level deeper: the output head as a single matrix calculation
Stacked, the rows form a matrix, in GPT-2 of shape 50,257 × 768: one row per token, 768 numbers per row, as many as the state h. All comparisons together are one multiplication:
Scores = h · Wᵀ
W is the matrix of all rows; the superscript T means it is flipped so its rows meet the state’s positions. Out comes a vector with 50,257 scores. Recalculated for GPT-2, the result differs from the model’s own scores by less than 0.0002. Right before the head runs a normalization; “last state” means the state after it. Experts call the scores logits.
The output head’s rows recall something from the start of the path.
The same table at the input and the output
At the input, every token ID fetches its embedding from a large table; in the lesson on embeddings, “apple” and “peach” lay close together. This table has one row per vocabulary entry, each as long as a state. The output head’s table looks like that. Could it be the same one?
A second table can sit at the output. Many models share the same table with the input, though. Experts call this weight tying. “Weights” here means parameters, the numbers in the table. The table is used twice: at the front to fetch an ID’s numbers, at the back to compare the last state with every row. A high score for “Paris” then means: the state at the last position points in a similar direction to the embedding of “Paris”.
A model’s blueprint says whether the table is shared. This is the small file shipped with the parameters (see the lesson on parameters, training and use). For Qwen3-0.6B, it says yes; for the larger Qwen3-8B, no: it has a second table of its own at the output. Sharing can make language models slightly better, as researchers showed in 2016, and above all it saves space.
The Qwen3-0.6B table has about 156 million numbers. Each takes two bytes here, as with Llama 3.1 8B in the lesson on model size.
Sharing saves 156 million times two bytes, about 0.3 gigabytes. Without sharing, a second table of the same size would be added at the output: about a quarter more for a 0.6-billion model.
Small models often run on laptops or phones, where every gigabyte counts.
One level deeper: how much memory the shared table saves
The whole model has about 0.6 billion parameters. The table, with about 156 million numbers, makes up a good quarter of that; the rest, about 0.44 billion, lies outside.
The config file holds the switch tie_word_embeddings. For Qwen3-0.6B it is true, for Qwen3-8B false. With shared weights, the numbers exist only once in memory, and the program uses them in both places: at the front to look up, at the back to compare. The config file names the number format bfloat16, so two bytes per number. For large models this hardly matters; on a phone it does. This holds only for the table.
So far, it was about a single state. But the model has one at every position.
Which position counts
For Qwen3-0.6B, “Die Hauptstadt von Frankreich ist” consists of seven tokens: “Die”, “Haupt”, “stadt”, “von”, “Frank”, “reich” and “ist”. Each has its state after the last block. Which goes into the output head?
During generation, only the state of the last position, here “ist” (“is”). The other six would predict tokens that are already in the text.
Does the model only look at the last word? No: as the previous lesson described, the blocks mixed information from all earlier positions into this state. The word “ist” alone points to no capital. Only the mixed-in context makes its state fit the row of “Paris” well.
In training, every position of a text delivers its own board. How many predictions are in a single pass with “The dog chases the cat”, if every word is one token? Think it over.
Five positions deliver a board each, four can be checked directly: “The” should predict “dog”, “dog” the word “chases”, “chases” the word “the”, and “the” the word “cat”. For “cat”, the next token is not in the text. No position can copy, although the whole sentence is in the model: the causal mask from the previous lesson lets every position see only what comes before it. Each gets the real text before it, not the model’s own guess (technical term: teacher forcing). So no prediction waits for another; all run at once.
That is why answers appear piece by piece, although training processes whole texts at once: during generation, the real next text does not exist yet. The next token can only be computed after the previous one has been chosen.
One full round, from text to the next token
Now the whole path in one go. The text is split into tokens, each with its ID. Every ID fetches its embedding from the table. The blocks mix context into the states, one after another. The state of the last position goes into the output head, and out comes the board with 151,936 scores. Softmax turns them into percentages. A selection step picks a token, and it is appended. Then the next round starts with one more token.
The selection step decides what the board turns into. If the model always takes the most likely token (greedy decoding), “Paris” follows every time. If instead the prize wheel from the softmax lesson is spun (sampling), the result is open: every token’s segment is as large as its percentage. In the experiment, “Zürich” came out first; then a colon followed by the sentence again; the third time, “Paris”.
How does “Zürich” get onto the wheel when it is not even in the top five? For the model, “Zürich” is not one token but three: “Z”, “ür” and “ich”. The word beginning “Z” is 17th on the board, at about 0.6 percent. The wheel lands on this narrow segment roughly once every 170 spins, so hitting it on the first try was luck. Once “Z” is appended, “ür” and “ich” follow in the experiment. The output head ranked “Paris” first; the selection step made the draw.
A token that has been appended is never taken back; each further round builds on it. Tapping “Regenerate” in a chatbot keeps the first board: only the draw is new, and from the first different token on, the later ones change.
Live demo · try it yourself
Scores, percentages, draw
Real numbers from the small model Qwen3-0.6B: the output head delivers a score for every token, softmax turns them into percentages, and only then is a token drawn. Pick an opening, set the temperature and draw.
Try it: for the opening “Die Hauptstadt von Frankreich ist” (“The capital of France is”), “Paris” has 47.5 percent at temperature 1. Roughly how often does it come up in 20 draws? And what changes at temperature 0?
- The output head delivers scores
- Softmax turns them into percentages
- Only then is a token drawn
Loading the board …
The demo’s data could not be loaded. The text explains the principle completely without it.
Scores of Qwen3-0.6B-Base (the base version without chat training), computed once in advance on an ordinary computer, with the same setup as the experiment in the first lesson; percentages over all 151,936 rows of the board. Shown are the twelve highest scores, rounded to two decimals (the text uses one). The demo only draws the next token and writes no further; top-k and top-p are not built in. “< 0.1%” means less than a tenth of a percent. Such leftovers belong to the 100%; their value is not shown.
Always taking the most likely token has a weakness. In the experiment, GPT-2 thus continued “Der Hund jagt die” with “Welt des Welt des Welt des Welt des” (“world of the world of the …”). That is the loop of repeated phrases from the softmax lesson: once a phrase is in the text twice, repeating it often becomes the most likely continuation.
One level deeper: does everything have to be recomputed every round?
As described, the whole text runs through the model every round, although only one token was added. The solution is called the KV cache. You know keys (labels) and values (contents) from the previous lesson. Those of earlier tokens are kept, because later tokens do not change them. Each round, only the new token goes through the blocks.
Measured with GPT-2 (only as an order of magnitude), 200 tokens took about 3.6 seconds with the cache and about 14 without, roughly four times as long. The cache is not free: it grows with every token and can fill much of the memory for long texts.
When the loop ends and what you can adjust
Remember the end token from the lesson on tokenization in the model, the one that closes a turn in Llama? In the basics, it was called the stop marker. It has its own row in the output head and gets a score every round; choosing it signals the end; the chat app then stops. The length limit is a setting outside the model. If a long answer stops mid-sentence and the app offers to continue, the limit was hit.
Whether you can influence the selection step depends on the program you use; the calculation before it does not. You know temperature from the softmax lesson: a low value enlarges the prize wheel’s large segments, a high value evens out the shares.
One level deeper: what providers report and allow
Programs reach a model through the provider’s interface, an API. It reports why an answer ended: at Anthropic “end_turn” if the model finished by itself, otherwise for example the maximum number of tokens or a stop sequence set by the program.
Besides temperature, there are often top-k and top-p: before the spin, only the k most likely tokens remain, or only as many as together reach the share p. Some providers restrict them (as of October 2026). For OpenAI’s GPT-6 models, temperature and top-p are dropped depending on the model: for some always, for others once the model thinks before answering. For models after Claude Opus 4.6, Anthropic accepts only temperature 1.0, no top-k, and top-p only from 0.99. A top-p of 0.99 keeps 99 percent of the wheel and cuts only hair-thin segments.
That completes the calculation behind the 47.5 percent: the last state times the row of “Paris”, then softmax over all rows. But where do its numbers come from? The last state is computed anew for every text, from the tokens and the model’s parameters. The embeddings, the parameters in the blocks and the rows of the output head, by contrast, were set by training. Training uses this lesson’s prediction: at every position, the next token is predicted and compared with the real one. How this makes a better model is shown in the next topic area, “How Learning Works”.
Explain it more simply
- Explain more simplyCore idea and an everyday comparison
- With an exampleOne concrete example, step by step
- In more detailEvery step on its own, in plain words
Recall moment
What stayed with you?
Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.
Preparing questions …
Questions from the community
Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.