Lesson 4 of 9 · Foundations

Token IDs: What a Number Means

Shows what a token ID is: a number that only has meaning within its tokenizer’s vocabulary. It also explains why each model must use the tokenizer it was trained with, and why the number itself reveals nothing about the piece.

A language model calculates only with numbers, not text. That means every piece gets a number, such as the piece “Frank” in “Frankreich” (France).

Every piece gets a number

Every piece in the vocabulary has a fixed number, its token ID. The tokenizer sets these numbers once, when it learns its vocabulary. After that, they never change. The number only shows where a piece sits in the vocabulary.

An open wooden box of index cards; in front of it, five empty cards taken out and lined up in a row
The vocabulary as a box: for every piece of your sentence, exactly that piece is looked up, and its number is copied down.
#

The vocabulary is the box from the previous lesson. Every piece in it has a number. For every piece in your text, exactly that piece is looked up, and its number is copied down. The tokenizer does not choose anything; it follows its rules. In the end, there is a row of numbers in the same order as the pieces. A fresh row is created for every text, whatever its length. Each piece contains only a sequence of characters, with no explanation of what it means. Looking through the box alone tells you nothing about a piece’s meaning.

Take the sentence “The cats sit.” The symbol ␣ represents a space stored as part of the piece. The tokenizer has split the sentence into “The”, “␣cat”, “s”, “␣sit”, and “.”. Because the vocabulary has no piece “␣cats”, “cats” becomes two pieces.

How many numbers does the sentence get? Guess before you read on.

Five, one for each piece. The table shows which number is on which card:

Token IDText piece
417“The”
82“␣cat”
903“s”
771“␣sit”
13“.”

The pieces and ID assignments in this example are made up rather than taken from a real tokenizer. The five cards represent just five entries from a much larger box, so their numbers do not run from 1 to 5. The real vocabulary of GPT-2, an older language model from OpenAI, has 50,257 entries, and ID 417 stands for a different piece there.

The sentence therefore becomes a row of five numbers, in the same order as its pieces. Converting text into numbers in this way is called encoding. The model then receives these five numbers as its input. Your messages to a chatbot are converted into numbers in this way, too, before the model sees them.

The sentence “The cats sit.” is split into five text pieces (The, ␣cat, s, ␣sit, period), next to them their IDs 417, 82, 903, 771, and 13
A sentence is first split into text pieces, then each piece is translated into its number using the invented mini vocabulary. The symbol ␣ marks a space that belongs to the piece.
#

The way back: numbers become text

A chatbot’s answer also starts as a sequence of numbers. The model outputs a score for every piece in the vocabulary. Each score says how well that piece would fit next. The selection step uses these values to pick a piece of text and appends it to the text. The list is ordered by the vocabulary’s numbers: entry 417 belongs to the piece “The”. If the selection step picks entry 82, “␣cat” is chosen.

The tokenizer turns the numbers back into readable text by looking up each one and joining the stored pieces in order. This reverse step is called decoding.

So the five numbers in the table become “The cats sit.” again. The spaces are not lost, because they are already stored in “␣cat” and “␣sit”. Order matters, too: 82, 903, 417 gives “catsThe”, with a space at the start.

Before you read on: what comes out when 417 and 82 swap places?

Type your own sentence into the tool and swap the numbers. Then switch to “Characters”. Which number does “T” get now?

Live demo · try it yourself

Your sentence, three tokenizers

Invented simulation

Which number does each token of your sentence get? Guess first; the demo then shows the numbers. After the first swap, you can switch between three invented tokenizers.

Solution

The word-piece tokenizer turns “The cats sit.” into five tokens with the IDs 417, 82, 903, 771 and 13, as in the text. Swap 417 and 82 and you get “ catThes sit.”: each number only brings back its own piece, space included. In “Characters” mode the same sentence gets entirely different numbers: the capital “T” of “The” gets the number 84 there, the number under which this character is listed in the international character table. That is why a model needs exactly the tokenizer it was trained with.

All vocabularies and IDs are invented and tiny. Only in “Characters” mode are the IDs real: they are the characters’ numbers in the international character table. In “Word pieces” mode, characters that no piece of the vocabulary covers get invented numbers from 5000 on. A real tokenizer knows tens or hundreds of thousands of pieces and assigns its own numbers, which only hold within its vocabulary.

#

If you swap them, you get “catThes sit.”, with a space at the start. Each number only brings back its own piece, space included.

An ID is a label

Does the number 417 tell the model that “The” is an article? No. In the invented mini vocabulary, 417 stands for “The”. In the real GPT-2 vocabulary, it stands for “el”, a word part. A number is only valid within its vocabulary. The number itself carries no meaning for the model. It makes no difference whether the number is large or small; the model uses it only to look something up.

A label tag
An ID is only a label: a number that says nothing about the content.
#

Where does the model keep its numbers for “The”? The spam filter from the first lesson stored a single number, called a weight, for each word. A language model keeps a whole group of learned numbers for each token ID. In the made-up example, that group contains some numbers that training has set. These numbers are also parameters, set during training. The ID tells the model which group to fetch, and that group becomes the starting point for the calculation.

Token ID 417 leads to two separate places: in the tokenizer to the text piece The, in the model to a group of learned numbers
The ID 417 leads to two separate places: in the tokenizer to the text piece “The” from the mini vocabulary, and in the model to a group of learned numbers. The model itself sees only the number.
#

Why the tokenizer and the model form a fixed pair

Can a finished model use a different tokenizer? Imagine two tokenizers with the same number of cards but different ID assignments: in the first, 417 stands for “The”; in the second, for “el”, as in GPT-2. The model was trained with the first but now receives numbers from the second. If “el” occurs in the text, the other tokenizer returns 417. What does the model do with that number? Take a moment to think before you read on.

It fetches the group it learned for ID 417, which belongs to “The” in its own vocabulary. No error appears because 417 is a valid ID for this model. The model simply keeps calculating with the group for “The”, even though the other tokenizer meant “el”.

Two index cards with the same number 417: the model’s tokenizer holds “The”, the foreign tokenizer holds “el”
A different tokenizer has its own vocabulary, where ID 417 refers to a different piece. “The” is invented for this example; “el” comes from GPT-2.
#

That is why every model needs exactly the tokenizer it was trained with, including its rules and vocabulary. The model learned its parameters for exactly one assignment of numbers to pieces. With a different assignment, the numbers no longer match the learned groups. If you switch to a model with a different tokenizer in a chat app, your message is split and numbered differently there. It then often consists of a different number of tokens, too. Several models can share one tokenizer; they then share the numbers, but not the learned numbers.

In the tool above, the same sentence got different numbers from each of the three tokenizers.

The numbers are only the beginning

The next lesson shows how these groups of numbers are arranged in a table: Scalar, Vector, Matrix, Tensor.

Recall moment

What stayed with you?

Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.

Preparing questions …

Questions from the community

Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.

Ask a question about this lesson

Last changed on · commit 4d7449b