Lesson 4 of 6 · Inside the Model

Embeddings: How a Number Becomes a Learned Vector

Why a token ID tells the model nothing about similarity, how training turns random numbers into similar profiles for tokens used in similar ways, and what a large vocabulary costs.

Barcode symbol
A barcode tells the checkout which item it is, but nothing about what the item has in common with others.
#

At the supermarket checkout, the barcode reveals exactly one thing: which item it is. No barcode says that apples and peaches are both fruit. The same goes for a language model and your chat message. From the previous lesson, you know that your message is a sequence of token IDs, each saying only which piece of text is meant.

The smallest version of the freely available model GPT-2 has an English vocabulary (the fixed list of all pieces a tokenizer knows). Whenever GPT-2 is mentioned here, it means this version. Mid-sentence, with a space in front, “apple” has the number 17180, “laptop” 13224, and “peach” 47565. By number, the laptop is much closer to the apple than the peach is. Does the model therefore treat apple and laptop as related?

What a number lacks

A token ID is the number on an index card, as in the lesson on token IDs. It says which piece of text is meant and means nothing in itself. The tokenizer assigned the numbers while building its vocabulary, without regard to meaning. So neighboring numbers do not stand for similar pieces.

The tokenizer’s card holds only the piece of text. The model uses the same number as a page number in its own thick reference book. You know this from the lesson on scalar, vector, matrix, and tensor. In GPT-2, every page holds 768 numbers. In the computer, that is one row of a large table; the token ID is its row number. So the model looks up the token’s numbers, not a fact like “France | Paris.”

Here, this row is called the token’s profile: a fixed sequence of numbers belonging to this token alone.

Unlike numbers, profiles can be compared. In a made-up example, the profile has only two places, labeled “grows on trees” and “has a battery.” The apple scores high on the first and low on the second, the peach almost the same, and the laptop the other way round. The sketch below shows the numbers. Real profiles have 768 places, and none has a name.

Pair of axes: the arrows of apple (0.9 | 0.1) and peach (0.8 | 0.2) point flat to the right, the laptop’s (0.1 | 0.9) steeply upward
A made-up example: each profile of two numbers becomes an arrow from the origin. Which arrows point in the same direction?
#

Plot the first number to the right and the second upward, and each profile becomes an arrow from the origin. The arrows are compared by direction, even with 768 places that nobody can draw. If two profiles point in a similar direction, they are called neighbors.

In the sketch, the arrows of apple and peach point almost the same way, the laptop’s in a quite different one.

How random numbers become profiles

Who entered these numbers? Nobody. Before training, the table holds random numbers, usually small values scattered around zero, and every token starts with a different profile. At this point, “apple” shares as little with “peach” as with “laptop”.

Training then works as in the lesson on parameters, training, and use: the model predicts the next token. Then its parameters, the faders on the mixing desk, are adjusted a little so the prediction fits better. The numbers in the table are such faders too. So whenever a token occurs in a training sentence, its row is adjusted as well.

Suppose a small model has one row each for “apple” and “peach.” Its training texts contain sentences like “The apple is ripe.” and “The peach is ripe.” After “apple,” the model should predict “is,” so the row “apple” is shifted until “is” fits better. The same follows “peach.” “Laptop,” by contrast, appears in sentences like “The laptop has a battery.”

Why do the two rows become similar? After the table, every token goes through the same computation with the same faders. In a made-up toy model, each profile has a single number. The computation after it is “times 2,” giving the score (an assessment of how well a piece of text fits) for “is” as the next token. For this example, the score should be exactly 10; too much is as wrong as too little. Real scores only count relative to the others.

The row “apple” starts with a small number, so its score is too low. Because five times two makes ten, the number is nudged step by step toward five. The row “peach” holds a number that is too large; it also moves toward five. Both end up at five, because the same computation should deliver the same result.

After “laptop,” by contrast, comes “has.” There the score for “is” should be low, so the number in the row “laptop” moves down, away from five. In real models, the computation after the table is much longer, and its faders are adjusted in training too. In those models too, tokens followed by similar things are adjusted alike. With many numbers per row, their arrows point in similar directions.

A made-up example: because the same thing follows “apple” and “peach,” both rows are adjusted alike during training.
#

Linguists had this idea as early as the 1950s: words that occur in similar contexts tend to have similar meanings. It is called the distributional hypothesis. The linguist J. R. Firth summed it up in 1957: “You shall know a word by the company it keeps.”

Live demo · try it yourself

From random numbers to profiles

Invented simulation

Every word of 42 short sentences has a profile of just two numbers, drawn here as a point. At the start, the numbers are random. Each training step takes one sentence, lets every word predict the two words after it, and adjusts the rows used a little. Every point is the tip of an arrow from zero, as in the sketch; to keep things simple, the demo measures the distance between the points, not their direction.

The made-up word “glorp,” the star, occurs in only two sentences: “The glorp is ripe.” and “The glorp hangs on the tree.” Guess which word its point ends up next to. Then start the training.

applepeachdogcatlaptopphoneeatpeelpetfeedchargebarksglorp
  • Fruit
  • Animals
  • Devices
  • Actions
  • Puzzle word
The numbers of “glorp” and the two points nearest to it: number 1 sets how far right the point lies, number 2 how far up. The “Last sentence” column shows how much these numbers have shifted through the sentence learned last; a dash means that sentence did not change this row.
WordNumber 1Number 2Last sentence
apple
peach
dog
cat
laptop
phone
eat
peel
pet
feed
charge
barks
glorp

0

Solution

“glorp” ends up with “apple” and “peach.” In its two sentences, the same follows it as follows them: “is ripe,” “hangs on the tree.” So every training step adjusts its row the way it adjusts the rows of the fruit words, and the points move together. What a glorp is was written nowhere; the groups and colors come from the demo only. Real models learn their table the same way, only with billions of sentences and hundreds of numbers per row.

A real calculation in miniature: 42 made-up sentences instead of billions of texts, 2 numbers per token instead of GPT-2’s 768, and the starting values are spread wider than in real models so that you can see them. The random start is fixed, so it runs the same for everyone. 13 of the 45 words are drawn; “the,” “is,” and the rest learn along but stay hidden. The colors come from the demo only; the training knows no groups. Real language models predict only the next token; two words help the mini training get by with few sentences. The sentences are made up, the calculation is not: what a large model has learned cannot be read off here.

#

Today’s chatbot models learn their table the same way, only with billions of sentences instead of a handful.

Similar use, similar profile

You can check this on GPT-2, whose table is freely available. You can work out for yourself how similar the directions of two profiles are. The same direction gives 1, a right angle 0. In the sketch, apple and peach point in almost the same direction, apple and laptop much less.

Live demo · try it yourself

Neighbors in the real GPT-2

Real output, computed in advance

Pick a word and see which profiles in GPT-2’s table are closest to its own, and which tokens sit right next to its ID in the vocabulary.

“peach” is already set under “Compare with.” In the sketch, apple and laptop came to about 0.2, arrows at right angles to 0. Guess what “laptop” reaches with “apple” in GPT-2: about 0, about 0.2, or more? Then pick “laptop” and watch the line for two random tokens.

Loading the GPT-2 values …

ID versus profile:

Distance between the IDs

Similarity of the directions (1 = same, 0 = at right angles)

Closest neighbors and the IDs next to its own

The 8 closest profiles

    Right next to it in the vocabulary

      Solution

      “laptop” reaches 0.36, more than in the sketch, but clearly less than “peach” with 0.53, although the ID of “laptop” is much closer to that of “apple.” Even two random tokens reach 0.27 on average, because all of GPT-2’s profiles point a little in one shared direction. What counts, then, is how far a value lies above chance. The tokenizer assigns IDs without regard to meaning, and the tokens right next to “apple” in the vocabulary have nothing to do with fruit. The numbers in the rows were set by training: in texts, similar things often follow “apple” and “peach,” so their rows were adjusted similarly. This holds for every language model that learns its profiles in training.

      Real values from the table of profiles of the smallest GPT-2 (openai-community/gpt2, revision 607a30d), computed in advance. What is measured is how similar the directions of the profiles are, where 1 would mean pointing the same way; only whole words with a leading space count as neighbors, no word pieces, digits, or punctuation. The vertical line shows what 20,000 random token pairs reach on average (0.27). This is similarity at the input, before the model takes the sentence into account.

      #

      This solves the opening puzzle: in GPT-2, “apple” and “peach” reach 0.53, “apple” and “laptop” only 0.36. Two random tokens average 0.27 in GPT-2, although they have nothing to do with each other. So the laptop lies a little above chance, the peach far above it, although its number is much farther away. Among whole words, the nearest neighbors of “apple” are “apples” and “Apple,” followed by “cider,” “peach,” “lemon,” and “fruit.” Nobody told the model they belong together; they appeared in similar sentences.

      GPT-2 bar lists: neighbors of apple mostly fruit (plus Apple), of Apple mostly tech (plus apple), all 0.51 to 0.70, random 0.27
      The nearest neighbors of “apple” and “Apple” in GPT-2’s table (a selection, recomputed for this lesson). The bars show how similar the directions of the profiles are; 1 would mean pointing the same way. The vertical line marks 0.27, what two random tokens reach on average.
      #

      For GPT-2, the capitalized “Apple” is a separate token with its own profile. Its neighbors are “iPhone,” “apple,” “iOS,” “Microsoft,” “iPad,” and “Macintosh.” Microsoft is neither an apple nor a phone; the name just occurs in similar texts. So similar profiles stand for similar use in text. No profile contains a definition.

      One level deeper: how do you measure whether two profiles are similar?

      Usually with cosine similarity: 1 means the same direction, 0 a right angle, −1 opposite. Multiply the numbers in the same place, add up the products, and divide by the lengths of the two arrows (the square root of the sum of the squares). As a formula: cos(v, w) = (v · w) / (|v| · |w|). With the sketch: apple and peach give 0.9 · 0.8 + 0.1 · 0.2 = 0.74, divided by √0.82 · √0.68 ≈ 0.75, so cos ≈ 0.99.

      In GPT-2, apple and peach reach 0.533, apple and laptop 0.357; 20,000 random pairs average 0.27, and 95 out of 100 were below 0.35. Why is chance not at 0? All profiles point a little in one shared direction: every row has a positive cosine with the average of all rows. Subtract this average, and random pairs average 0. What matters, then, is the distance from the random value.

      Embedding and embedding matrix

      Three blank cards with line patterns: the two left ones lie close together and look almost alike, the right one lies apart and differs
      Profiles without labels: tokens used in similar ways carry similar patterns of numbers; a token used differently carries a different one.
      #

      In GPT-2, “apple” has a whole row of numbers. The first three are 0.119, −0.175, and 0.129. What does the second one mean? Nobody can say, and the same goes for the other numbers in the row. The profile analogy has a limit here: a profile on paper has fields like height or eye color that someone filled in. In GPT-2, no field has a name, and textbooks note that the individual numbers have no clear meaning. What a profile expresses lies in all its numbers together.

      One level deeper: what lies in the other numbers?

      The other numbers in the row are just as nameless. A single place usually does not stand for a named feature such as “fruit,” and its sign alone reveals nothing tangible. What matters is the pattern of all the numbers together. The ID-card comparison breaks down here: a card has fields that each hold something unambiguous, whereas each number in a profile contributes only a part of a pattern, much as a single stroke in a face says little. Two profiles count as similar because their numbers together point in the same direction, not because one particular place matches. So nobody compares just a single number. Anyone who wants to know whether two tokens are related computes with all the numbers together.

      In technical terms, a token’s profile is called its embedding. Because it is an ordered list of numbers, it is a vector. The whole table with one row per token of the vocabulary is called the embedding matrix.

      In GPT-2, the embedding matrix has one row for each of its just over 50,000 vocabulary entries. At 768 numbers per row, that makes almost 39 million numbers, about a third of the more than one hundred million parameters of this smallest GPT-2 version.

      One row per token, not per word

      Does “Apfel” (German for apple) get a row of its own? That depends on how the vocabulary is cut. In GPT-2, “apple” is a whole token, because its vocabulary comes from English texts. German words often split into pieces in GPT-2. “Katze” (cat), with a space in front, becomes “␣Kat” and “ze”; the symbol ␣ stands for the space that belongs to the piece. “Apfel” (apple) becomes “␣Ap,” “f,” and “el.”

      So GPT-2’s embedding matrix has no row for “Apfel,” only rows for the three pieces. The row “␣Ap” is also used for words such as “Apotheke” (pharmacy) that begin with the same piece. That the three pieces together mean the fruit only emerges in the computing stages after the table.

      Why not simply take a larger vocabulary? In the lesson on token IDs, a made-up mini vocabulary had five entries: “The,” “␣cat,” “s,” “␣sit,” and “.”. With it, “The cats sit.” becomes five tokens; with “␣cats” as a sixth entry, four. In return, the embedding matrix gets a sixth row.

      Two mini vocabularies: with the entry ␣cats, “The cats sit.” goes from 5 to 4 tokens, and the embedding matrix grows from 5 to 6 rows
      A made-up mini vocabulary: with the sixth entry “␣cats,” the sentence gets one token shorter, but the embedding matrix gets one more row.
      #

      The new row always costs space in the table. It saves a token only when “cats” occurs in the text: one fewer position in the context window (the maximum number of tokens the model processes at once) and one fewer round when answering. In a sentence about dogs, it saves nothing.

      How big should the vocabulary be?

      Three tokenizers from OpenAI have different vocabulary sizes. GPT-2’s has more than 50,000 entries, GPT-4’s (a model behind ChatGPT) about 100,000, and its successor GPT-4o’s about 200,000. The largest has about four times as many entries as the smallest. Does a sentence then get four times shorter?

      Live demo · try it yourself

      How big should the vocabulary be?

      Real tokenizer, no model

      Three real tokenizers from OpenAI with about 50,000 to 200,000 entries split your sentence. Each shows how large an embedding matrix with one row per entry would be.

      The largest vocabulary has about four times as many entries as GPT-2’s. Does the cat sentence then need only a quarter of the tokens? Guess first, then split it, the German sentence too.

      Compare:

      Loading the tokenizers …

      GPT-2

      50,257 entries
      tokens…
      Table39 million

      50,257 × 768

      GPT-4

      100,277 entries
      tokens…
      Table77 million

      100,277 × 768

      GPT-4o · gpt-oss

      200,019 entries
      tokens…
      Table154 million

      200,019 × 768

      Solution

      The English cat sentence needs 9 tokens with all three, not a quarter as many. The German sentence goes from 14 to 12 to 9. A new entry only saves where its piece of text occurs: for the words of the English sentence, the larger vocabularies bring no new entries, while GPT-2 had to split the German “ Katze” and the largest vocabulary holds it whole. The table, by contrast, grows with every entry, whether it occurs in the text or not. That is why vocabulary size is a trade-off for every language model.

      Live in your browser: real tokenizers from the gpt-tokenizer library, GPT-2 (r50k_base), GPT-4 (cl100k_base), and GPT-4o (o200k_base; gpt-oss uses the same pieces). Entries counted with special tokens. No model computes here. The table size is an example calculation with 768 numbers per row, as in GPT-2; the real models behind the two larger tokenizers have longer rows: unpublished for GPT-4 and GPT-4o, 2,880 numbers for gpt-oss-20b.

      #

      “The cat is sitting on the windowsill.” needs 9 tokens with all three: for its words, the larger vocabularies bring no new entries. The German sentence “Die Katze sitzt auf dem Fensterbrett.” becomes 14 tokens in GPT-2, 12 in GPT-4, and 9 in GPT-4o. The largest vocabulary holds “␣Katze” whole; GPT-2 has to split it. An entry only saves where its piece of text occurs.

      The costs grow with every entry. Suppose GPT-2 had GPT-4o’s vocabulary, still with 768 numbers per row. Its embedding matrix would be about four times as large. It would have more numbers than the rest of the model, the computing stages after it. A second cost comes from training. During training, a row is adjusted mainly when its token occurs in the text. The larger the vocabulary, the more entries are so rare that their rows get little practice. Researchers found such undertrained entries in widely used models, where they can trigger strange behavior.

      The best size depends on the training effort, according to a 2024 study: larger models can handle larger vocabularies. A large model, trained on a great deal of text, sees even rare entries often enough; for a small one, even a medium-sized vocabulary can be too large.

      A bank is a bank, for now

      Symbol of a park bench next to a tree
      A bank to sit on or a bank for your money: at the model’s input, it is the same profile.
      #

      “I sit on the bank in the park.” “I pay money into the bank.” At the model’s input, it is the same token: same number, the same row in the embedding matrix. The row does not fix which meaning is meant here.

      Even so, a chatbot usually understands “bank” correctly. So the profile from the table is only the starting point.

      The profile does not reveal where “bank” stands in the sentence either: for the same token ID, the model looks up the same row, whether the token comes at the start of the sentence or at the end. How the computing stages after it mix in the rest of the sentence, and how they keep track of the order, is the topic of the next lesson.

      Recall moment

      What stayed with you?

      Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.

      Preparing questions …

      Questions from the community

      Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.

      Ask a question about this lesson

      Last changed on · commit 3dd66ba