Lesson 1 of 5 · How Learning Works

Text Becomes Many Practice Problems

Why nobody writes down answers to train a language model, how many texts become sequences of a fixed length, and which text gets in at all.

The path through the model ended with one sentence about training: at every position, the model predicts the next token, and the prediction is compared with the real one. Meta trained Llama 3.1, its freely available language model, on about 15 trillion tokens. Almost every one is a small practice problem: here is the text so far, what comes next?

Without an answer there is nothing to compare. So who wrote down answers to trillions of problems? A reading exercise with a card shows it.

The answer is already in the text

Put a card on a line in a book so that only the first three words show. Guess the next word, slide the card one word along and check, then go on. Nobody had to write the answer down. It was already there, only covered.

A hand holds a card over a line in a book; only the first three words are uncovered
The cover card: the beginning of the line is uncovered. The word right under the edge of the card is the answer to this one problem, and nobody had to write it specially.
#

Here the analogy stops: you guess word by word and only then look, but the model slides no card. It computes all the problems of a whole text in a single pass. When a text is being written, only the last position yields a list with one score for each possible token, as in the lesson on the output head. In training, by contrast, every position yields its own list: “The dog chases the cat” lets four predictions be checked at once. Each position saw only what comes before it. A built-in filter hides everything later, the rule “No looking ahead” from the lesson on transformer blocks and attention. So no position can copy its answer, even though that answer is in the same sequence.

This is how a language model’s practice problems come about. Recall “The cat sat on the sofa.” from the lesson on input and output: the beginning is the input. The piece that follows is its label, the correct answer. For the spam filter from the first lesson, people had to label every email by hand. With text, the card does that: every point in the text yields a problem with an answer.

Try an invented sentence: “The bus leaves again on time at eight in the morning.” A model computes with tokens, pieces of text from its tokenizer. To keep it simple, every word and the full stop count as one token here; a real tokenizer often splits more finely, but the principle stays.

Live demo · try it yourself

How many problems are in one sentence?

Simplified split; GPT-2 option: real tokenizer, no model

First estimate how many practice problems the sentence yields. Then look at each one: everything so far as the question, the next token as the answer.

Your own sentence and a real tokenizer
Split

Solution

A text of n tokens yields n − 1 problems, because the last token has no answer left. Word by word, the bus sentence has 12 tokens and so 11 problems. GPT-2’s real tokenizer happens to split this English sentence into 12 tokens as well; German words are often cut into more pieces.

The word-by-word split is simplified for the overview. The “GPT-2” split uses that model’s real tokenizer; no model computes anything here. Stepping through is only a reading aid: a model computes all problems of a text at the same time.

#

Count along:

Question: everything so farAnswer
Thebus
The busleaves
The bus leavesagain
The bus leaves againon
The bus leaves again ontime
The bus leaves again on timeat
The bus leaves again on time ateight
The bus leaves again on time at eightin
The bus leaves again on time at eight inthe
The bus leaves again on time at eight in themorning
The bus leaves again on time at eight in the morning.

That is 11 problems, one for every token followed by another. The final full stop has no answer left in this text. This holds for every text: it yields one problem fewer than it has tokens.

How many problems are in a whole paragraph?

Take “It stays dry today. Tomorrow it should rain.” With every word and full stop as one token, that is 10 tokens and 9 problems. A paragraph of 100 words yields about 100 problems, and more with a real tokenizer that splits more finely. This is how trillions of tokens become trillions of problems.

The model keeps no list of answers. Training only changes numbers, as with the spam filter: the problems nudge the parameters, the faders on the model’s mixing desk, bit by bit, so that the correct answer becomes more likely next time.

In pretraining, nobody wrote the answers specially. A chatbot seems like someone giving learned answers, and for a later step people really do write answers. Not for the trillions of problems before that: because the answer is in the text itself, every text becomes practice problems. This is called self-supervised learning. The text supplies problem and answer at once; it supervises itself, so to speak.

Almost all language models behind chatbots started out this way. This first phase is called pretraining. Its result is the base model you know from the lesson on training, for example Llama 3.1. Only then come examples that people have written or rated, turning the base model into a helpful chatbot. A later topic area covers that.

A sentence is short. Real texts, though, run from three words to a hundred pages, and the model practices on pieces of a fixed length. How does that fit?

Many texts, one fixed length

Remember the freight train from the lesson on tokenization inside the model? Every token was a wagon. Special tokens, markers that stand for no piece of text, were differently colored marker wagons. The buffer stop at the end of the track stood for the context window, the most tokens the model can process at once.

Training uses this length in full. The paper on GPT-3, a predecessor of the models behind ChatGPT, is precise: every sequence it practiced on had 2048 tokens, as many as fit its context window. Most texts are shorter, so they are joined like wagons into a long train. Between two texts there is always an end token, the marker that a text has ended; the lesson on input and output called it the stop marker. What is new is the cut, and here the analogy stops: nobody cuts up a real train. This one is cut into pieces of exactly 2048 tokens, each a training sequence. They are the same length because the model computes several at once on the graphics chips. For that, the pieces must fit into the same rectangular table of numbers. Shorter pieces would otherwise have to be filled up with padding tokens, placeholders that only take up space.

Two texts with end tokens, cut into sequences of 8 tokens; the second sequence begins mid-way through the cake sentence
Two invented texts, joined with end tokens and cut into sequences of 8 tokens each. Every box is a token. The first sequence ends with “The cake”; the second begins mid-sentence and has only begun, and in training the next text fills it up. The small context window is only for the overview; GPT-3 cut sequences of 2048 tokens.
#

Live demo · try it yourself

Several texts, one fixed length

Invented simulation

Two invented texts, each followed by an end token, are joined and cut into sequences. Set the window length and see where the cut separates the texts.

8

Solution

With their end tokens, the two texts have 13 tokens together. With window length 8, the first sequence holds the weather sentence, its end token and “The cake”, and the second starts in the middle of the cake sentence. Every sequence yields one problem fewer than it has tokens, so 7 for 8 tokens.

Every word and every full stop counts as one token here, simplified for the overview. The small window lengths are only for the overview; GPT-3 cut sequences of 2048 tokens.

#

Take two invented texts: “It stays dry today.” and “The cake is baking now.” With an end token after each, that is 13 tokens. Suppose the context window were only 8 tokens long. The first sequence then holds the weather sentence, its end token and “The cake”. The second begins mid-sentence with “is baking now”, and the next text fills the rest. So a text can be split across two sequences.

A sequence of 8 tokens yields 7 problems, as with the bus sentence. For GPT-3 that is 2047 problems per sequence, about 2000. The model does not learn the context window itself. It is a hyperparameter, set before training, like the number of layers, the transformer blocks one after another.

Two special problems sit around the end token. In the first, the question is “It stays dry today.” and the answer is the end token itself, so the model learns when a text is finished. What does it learn from the next one, where “The” follows the end token?

“The cake” has nothing to do with the weather. Any text can begin after the end token. From many such problems, the model learns that after this marker comes a new text, unrelated to the one before. The weather sentence is still in the sequence; the marker only shows that it does not belong. That is why the GPT-3 paper puts it between texts. Without it, the model would practice continuing two unrelated texts as one.

The same kind of marker ends an answer when the model is used. In GPT-2, whose tokenizer you know from the lesson on tokenizers, it is the very same marker that shows the end of a text when writing. Chat models like Llama 3.1 end a turn with a special token of their own. That they should place it at the end of an answer, they learn only afterwards, in a further training on conversations, as in the lesson on tokenization inside the model.

So every text becomes problems, about 2000 per sequence. Then why not simply take the whole web?

From the web to training text

The raw material often comes from Common Crawl, an organization that regularly collects web pages and offers the archive for free. It is a wild mix, like these five invented pages. One is a guide on looking after a bike, another a list of search terms such as “buy cheap bike”, one per line with bullet points. Then come a menu with “Home, About us, Contact”, the same guide under another address and a page in German. Which would you keep?

Three stacked sieves; sheets of paper fall in at the top, and only a small pile reaches the bottom
Several sieves stacked: each layer stands for a rule, each sheet for a page, and whatever stays on top is dropped. The real sieves are programs whose rules people have set.
#

The pages pass several stages, like a sieve with several layers. Unlike a real sieve, they sort not by size but by rules people have chosen. First the text is taken from the page’s source code, the instructions the browser builds the page from. Then the language is identified; the German page goes to the German part. Then quality rules apply. For Gopher, a language model from the AI company DeepMind, every page with fewer than 50 words was dropped. That hits the menu. Pages where more than 90 percent of lines start with a bullet point failed too, like the search-term list.

Next comes deduplication. The same page often appears on the web several times, copied or under a new address; programs keep one of the identical or near-identical pages. Otherwise the model would practice the same page again and again, and what appears very often in its text, a model can partly reproduce word for word. Developers also hold back text to test the model on text it never practiced on. If the same page sat in both the practice text and the held-back test text, the test would no longer measure cleanly. The GPT-3 paper gives both reasons.

Of the five pages, one is left for the English part: the guide, once.

The demo below adds three more pages, eight in all. Which pages stay, only the demo shows.
Why remove duplicate pages if they would mean more practice?

A copy teaches nothing new. Each repetition pushes the numbers further the same way, until exactly this continuation comes out on top every time. The text is still not stored as a sentence, but the numbers are set so that the model can reproduce it almost word for word. In a 2021 experiment, models trained on deduplicated text emitted memorized passages ten times less often.

Six stages from collecting web pages to mixing the sources; for GPT-3, 45 terabytes became 570 gigabytes
The stages from the web archive to training text. The amounts at the start and at the end come from the GPT-3 paper, with the raw text measured in compressed form.
#

According to the GPT-3 paper, filtering left only 570 gigabytes of the roughly 45 terabytes of raw text, measured in compressed form. More text sounds like more practice, but a search-term list only practices stringing search terms together.

Live demo · try it yourself

The data sieve, layer by layer

Real thresholds, invented pages

8 invented web pages run through three layers of a sieve. First estimate how many stay, then sieve layer by layer.

  1. Detect language
  2. Quality rules
  3. Remove duplicates

  • How to look after your bike

    bike-guide.example/care

    Show the full text
    A bike keeps running well if you look after it regularly. After a ride in the rain, wipe the frame and the chain dry. Once a month, give the chain a few drops of oil and wipe off what is left over with a cloth. Before every longer ride, check the tyre pressure and the brakes. If a brake sounds scratchy or feels weak, have a workshop adjust it. That keeps the bike safe and quiet.

  • Buy cheap bike

    bike-deals.example/offers

    Show the full text
    • buy cheap bike
    • buy bike online
    • cheap e-bike order
    • bike deal today
    • bike sale now
    • cheap road bike
    • buy bike London
    • buy bike Leeds
    • buy bike Bristol
    • cheap kids bike
    • used bike buy
    • bike bargain online
    • bike discount today
    • compare bike deals
    • buy bike fast
    • cheap mountain bike
    • order bike cheap
    • bike free delivery
    • buy bike now
    • best bike price

  • Menu

    cycling-club.example/

    Show the full text
    Home
    About us
    Contact
    Imprint
    Privacy

  • How to look after your bike

    bike-tips.example/care-copy

    Show the full text
    A bike keeps running well if you look after it regularly. After a ride in the rain, wipe the frame and the chain dry. Once a month, give the chain a few drops of oil and wipe off what is left over with a cloth. Before every longer ride, check the tyre pressure and the brakes. If a brake sounds scratchy or feels weak, have a workshop adjust it. That keeps the bike safe and quiet.

  • So pflegst du dein Fahrrad

    radratgeber.example/pflege

    Show the full text
    Ein Fahrrad fährt länger gut, wenn du es regelmäßig pflegst. Wisch nach einer Fahrt im Regen den Rahmen und die Kette trocken. Einmal im Monat bekommt die Kette ein paar Tropfen Öl, überschüssiges Öl nimmst du mit einem Lappen ab. Prüf vor jeder längeren Tour den Luftdruck der Reifen und die Bremsen. Klingt eine Bremse schleifend oder greift sie schwach, lass sie in einer Werkstatt einstellen. So bleibt das Rad sicher und leise.

  • You have to read this

    bike-news.example/

    Show the full text
    Ten tips for your next cycling holiday …
    Hardly anyone knows this lakeside route …
    What professionals do differently when packing …
    The one mistake that almost every cyclist makes …
    How to find the right saddle for you …
    Why your tyre keeps losing air again and again …
    The most beautiful coastal cycle paths at a glance …
    With this trick your chain stays clean for longer …
    What a good lock should really cost you …

  • Simple apple cake

    bakery.example/apple-cake

    Show the full text
    For a simple apple cake, mix 200 grams of flour, 100 grams of sugar, 125 grams of soft butter and two eggs into a smooth dough. Press the dough into a greased tin. Cut three apples into thin slices and lay them closely on the dough. Sprinkle a little cinnamon over them. Bake the cake at 180 degrees for about 40 minutes until it is golden brown. Let it cool a little before you cut it.

  • Order numbers

    parts-store.example/list

    Show the full text
    # 4711-0815
    # 4748-0868
    # 4785-0921
    # 4822-0974
    # 4859-1027
    # 4896-1080
    # 4933-1133
    # 4970-1186
    # 5007-1239
    # 5044-1292
    # 5081-1345
    # 5118-1398
    # 5155-1451
    # 5192-1504
    # 5229-1557
    # 5266-1610
    # 5303-1663
    # 5340-1716
    # 5377-1769
    # 5414-1822
    # 5451-1875
    # 5488-1928
    # 5525-1981
    # 5562-2034
    # 5599-2087

Solution

Of the 8 pages, 2 stay: the care guide and the recipe. The German page ends up in the German part, four pages fail the quality rules, and the copy drops out with the duplicates. At scale, it looks similar. For GPT-3, 45 terabytes of compressed raw text became 570 gigabytes.

All pages and addresses are invented. The quality rules are the published thresholds DeepMind used to filter the text for Gopher. Language detection is simplified; real filtering pipelines use a separate program for it. Duplicates only count here when the text is exactly the same.

#

Finally the sources are mixed, because only now is it clear how much usable text each one supplies. How much of each goes in is again a decision, the data mix. Meta reports about half general knowledge for Llama 3.1, a quarter mathematics and reasoning, plus code. That is one reason chatbots can write programs: they practiced on code like any other text. You can browse such a collection yourself in FineWeb, a freely available, filtered dataset from Common Crawl.

Whether all these texts may be collected, personal data included, is the subject of its own lesson: Training Data: Copyright and Privacy. What gets through the sieve shapes the model, most visibly when text is missing.

The model learns what is in the text

Meta’s figures put only 8 percent of the Llama 3.1 text in languages other than English, all of them together, and German is just part of that. English text, by contrast, is plentiful. So why not translate English text into German and practice on that?

Two bars: for Llama 3.1, 8 percent is in languages other than English; for Kolibri, over 20 percent is German
Data mix according to the developers’ reports: for the Llama 3.1 models by kind of text, for Kolibri only the share of German. For Llama 3.1, that is about 50 percent general knowledge, 25 percent mathematics and reasoning, 17 percent code and 8 percent languages other than English. The two bars divide the text differently and can only be compared on the language share.
#

Aleph Alpha’s report on its model Kolibri gives an example. “Drive safe!” is what you say in English when someone sets off. Translated, it easily becomes “Fahre sicher!” (roughly “drive safely!”), although, according to the report, “Komm gut an!” (roughly “arrive well!”) would be more natural. How would a model say goodbye in German if it knew German farewells mainly from translated text?

The answer to these problems is then “Fahre sicher!”, because that is what the text says, and each one pushes the numbers a little that way. The model takes on the patterns of its text, and the more often they appear, the more strongly. You know this from the lesson on probability and softmax: if many texts follow “The capital of Australia is” with “Sydney”, then “Sydney” gets a large segment, even though Canberra is the capital.

A single phrase is still the smaller problem. According to the report, translated text also keeps the source language’s view of the world. Such a body of text is in many respects an English view of the world, only in German.

So translated text does not replace missing German text; the model learns the traces of translation along with it. For Kolibri, Aleph Alpha therefore added German texts from the web or rephrased by a language model; according to the report, more than a fifth of the mix is German. Translated text went in only in a carefully selected amount, as a supplement.

Try it yourself. Ask a chatbot, in German, for a short farewell to someone with a long drive ahead. Does it sound like a word-for-word translation from English, such as “Fahre sicher!”, or like a German formula, such as “Gute Fahrt!” (roughly “have a good trip!”)? One answer proves nothing, because chance often plays a part in picking the next token. But over several tries, watch whether its German sometimes sounds translated.

So nobody specially wrote the answers down: they stand in the text, and from it come the trillions of problems, roughly one per token, in sequences of a fixed length. What is still open is how a prediction and the real token become a number that tells the training how far off it was. The next lesson shows that.

Recall moment

What stayed with you?

Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.

Preparing questions …

Questions from the community

Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.

Ask a question about this lesson

Last changed on · commit 3dd66ba