Lesson 4 of 9 · FoundationsRead
Token IDs: What a Number Means
Shows what a token ID is: a number that only has meaning within its tokenizer’s vocabulary. It also explains why each model must use the tokenizer it was trained with, and why the number itself reveals nothing about the piece.
A language model calculates only with numbers, not text. That means every piece gets a number, such as the piece “Frank” in “Frankreich” (France).
Every piece gets a number
Every piece in the vocabulary has a fixed number, its token ID. The tokenizer sets these numbers once, when it learns its vocabulary. After that, they never change. The number only shows where a piece sits in the vocabulary.

The vocabulary is the box from the previous lesson. Every piece in it has a number. For every piece in your text, exactly that piece is looked up, and its number is copied down. The tokenizer does not choose anything; it follows its rules. In the end, there is a row of numbers in the same order as the pieces. A fresh row is created for every text, whatever its length. Each piece contains only a sequence of characters, with no explanation of what it means. Looking through the box alone tells you nothing about a piece’s meaning.
Take the sentence “The cats sit.” The symbol ␣ represents a space stored as part of the piece. The tokenizer has split the sentence into “The”, “␣cat”, “s”, “␣sit”, and “.”. Because the vocabulary has no piece “␣cats”, “cats” becomes two pieces.
How many numbers does the sentence get? Guess before you read on.
Five, one for each piece. The table shows which number is on which card:
| Token ID | Text piece |
|---|---|
| 417 | “The” |
| 82 | “␣cat” |
| 903 | “s” |
| 771 | “␣sit” |
| 13 | “.” |
The pieces and ID assignments in this example are made up rather than taken from a real tokenizer. The five cards represent just five entries from a much larger box, so their numbers do not run from 1 to 5. The real vocabulary of GPT-2, an older language model from OpenAI, has 50,257 entries, and ID 417 stands for a different piece there.
The sentence therefore becomes a row of five numbers, in the same order as its pieces. Converting text into numbers in this way is called encoding. The model then receives these five numbers as its input. Your messages to a chatbot are converted into numbers in this way, too, before the model sees them.
The way back: numbers become text
A chatbot’s answer also starts as a sequence of numbers. The model outputs a score for every piece in the vocabulary. Each score says how well that piece would fit next. The selection step uses these values to pick a piece of text and appends it to the text. The list is ordered by the vocabulary’s numbers: entry 417 belongs to the piece “The”. If the selection step picks entry 82, “␣cat” is chosen.
The tokenizer turns the numbers back into readable text by looking up each one and joining the stored pieces in order. This reverse step is called decoding.
So the five numbers in the table become “The cats sit.” again. The spaces are not lost, because they are already stored in “␣cat” and “␣sit”. Order matters, too: 82, 903, 417 gives “catsThe”, with a space at the start.
Before you read on: what comes out when 417 and 82 swap places?
Type your own sentence into the tool and swap the numbers. Then switch to “Characters”. Which number does “T” get now?
Live demo · try it yourself
Your sentence, three tokenizers
Invented simulationWhich number does each token of your sentence get? Guess first; the demo then shows the numbers. After the first swap, you can switch between three invented tokenizers.
The way back: IDs become text again
Pick two IDs to swap them. The tokenizer looks up every ID and joins the pieces in this order.
Solution
The word-piece tokenizer turns “The cats sit.” into five tokens with the IDs 417, 82, 903, 771 and 13, as in the text. Swap 417 and 82 and you get “ catThes sit.”: each number only brings back its own piece, space included. In “Characters” mode the same sentence gets entirely different numbers: the capital “T” of “The” gets the number 84 there, the number under which this character is listed in the international character table. That is why a model needs exactly the tokenizer it was trained with.
All vocabularies and IDs are invented and tiny. Only in “Characters” mode are the IDs real: they are the characters’ numbers in the international character table. In “Word pieces” mode, characters that no piece of the vocabulary covers get invented numbers from 5000 on. A real tokenizer knows tens or hundreds of thousands of pieces and assigns its own numbers, which only hold within its vocabulary.
If you swap them, you get “catThes sit.”, with a space at the start. Each number only brings back its own piece, space included.
An ID is a label
Does the number 417 tell the model that “The” is an article? No. In the invented mini vocabulary, 417 stands for “The”. In the real GPT-2 vocabulary, it stands for “el”, a word part. A number is only valid within its vocabulary. The number itself carries no meaning for the model. It makes no difference whether the number is large or small; the model uses it only to look something up.
Where does the model keep its numbers for “The”? The spam filter from the first lesson stored a single number, called a weight, for each word. A language model keeps a whole group of learned numbers for each token ID. In the made-up example, that group contains some numbers that training has set. These numbers are also parameters, set during training. The ID tells the model which group to fetch, and that group becomes the starting point for the calculation.
Why the tokenizer and the model form a fixed pair
Can a finished model use a different tokenizer? Imagine two tokenizers with the same number of cards but different ID assignments: in the first, 417 stands for “The”; in the second, for “el”, as in GPT-2. The model was trained with the first but now receives numbers from the second. If “el” occurs in the text, the other tokenizer returns 417. What does the model do with that number? Take a moment to think before you read on.
It fetches the group it learned for ID 417, which belongs to “The” in its own vocabulary. No error appears because 417 is a valid ID for this model. The model simply keeps calculating with the group for “The”, even though the other tokenizer meant “el”.
That is why every model needs exactly the tokenizer it was trained with, including its rules and vocabulary. The model learned its parameters for exactly one assignment of numbers to pieces. With a different assignment, the numbers no longer match the learned groups. If you switch to a model with a different tokenizer in a chat app, your message is split and numbered differently there. It then often consists of a different number of tokens, too. Several models can share one tokenizer; they then share the numbers, but not the learned numbers.
In the tool above, the same sentence got different numbers from each of the three tokenizers.
The numbers are only the beginning
The next lesson shows how these groups of numbers are arranged in a table: Scalar, Vector, Matrix, Tensor.
Explain it more simply
- Explain more simplyCore idea and an everyday comparison
- With an exampleOne concrete example, step by step
- In more detailEvery step on its own, in plain words
Recall moment
What stayed with you?
Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.
Preparing questions …
Questions from the community
Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.