Lesson 1 of 6 · Inside the ModelRead
What an AI Model Actually Is
What is an AI model? Not a database of facts, but billions of numbers in which knowledge is spread out. That is where answers and made-up facts come from.
The foundations left a question unanswered: a model’s billions of numbers spell out no facts. So how does a chatbot know that Paris is the capital of France?
In an experiment, a small, openly available language model called Qwen3-0.6B-Base was given “Die Hauptstadt von Frankreich ist” (“The capital of France is”). “0.6B” stands for 0.6 billion parameters (stored numbers that training has set), “Base” for the basic version, not yet post-trained into a chatbot. For this exact wording, the leading next token (a piece of text from the model’s fixed list) was “Paris”, at just under 50 percent. The percentages say how well a piece fits, not whether it is true.
In other cases, the same mechanism that delivers “Paris” can invent a court ruling that never existed.
Where does it say that Paris is the capital?
A correct answer feels like a database lookup, as if the model held a row with this fact.
A real model file looks different. You know it from the foundations: a small blueprint, the architecture, and many numbers, the parameters. A larger Qwen model stores them in a few hundred number tables, as in the lesson on vectors and matrices. Each is named after a computing step that later lessons explain. None is called “countries” or “capitals.”

You know the mixing desk from the foundations: each fader stands for a parameter, its position for its value. Training set the faders; while answering, they stay put. So settings are stored. A fact like “Paris is the capital” does not appear in it as a line. On the desk, a signal enters on the left and leaves on the right. The meters show what passes through. In the model, this signal is your text, turned into numbers. The spam filter from the foundations added weights 2 and 3 for “Free: your prize is waiting” and got 5.
That 5 wasn’t stored anywhere; it was calculated for that email.
Many steps are sums with fixed weights (part of its parameters): the model multiplies incoming numbers by them, adds up the products and passes on the sum. Other steps differ; that depends on the model.
The numbers passed on are called intermediate values (the numbers between input and output). They arise anew for each text, like the desk’s meters.
The last intermediate values become the score list from the foundations, one score per text piece the model knows. Softmax (a calculation that turns scores into percentages) turns them into shares that add up to 100 percent.
That is as far as the picture goes: on a real desk, each meter shows the level of one channel, such as the vocals. An intermediate value usually stands for no particular content and belongs to no single fader.
No parameter is called “Paris”
What the model learned about Paris lives in its parameter settings, but in no single one. A made-up mini model shows how that works. A person set its ten faders, nothing was trained; real models have billions. It only shows how the same faders give different results for two sentence starts, not what a real model answers.
Sentence A, “The capital of France is,” comes in as three made-up numbers. Each meter shows a sum of products. A second stage works on the meters the same way and produces two scores: Paris 9, France 6. Sentence B, “Paris is the capital of,” comes in as three other numbers. The same faders give the scores Paris 6, France 9.
How does the mini model compute the meters?
Sentence A comes in as 2, 1 and 1. For meter 1, faders 1 to 3 sit at 2, −1 and 1. Multiply each fader setting by its incoming number, then add the results: 2·2 − 1·1 + 1·1 = 4. Meter 2 uses faders 4 to 6 (−1, 2 and 1) and comes to 1. The second stage uses the meters: Paris 2·4 + 1·1 = 9, France 1·4 + 2·1 = 6. For sentence B, 1, 2 and 1 come in; the same computation gives meter values 1 and 4.
A fader in the second stage feeds meter 1 into the France score. Think briefly: if it jumps from one to three, does only sentence A’s answer tip?
Sentence A gives the meters four and one, sentence B one and four.
For sentence A, meter 1 shows a 4. Turn the fader up, and France overtakes Paris.
One level deeper: what a fader jump does
The fader that feeds meter 1 into the France score starts at 1; at 3, it counts meter 1 three times instead of once. For sentence A, meter 1 is 4, so 2 · 4 = 8 is added: France rises from 6 to 14 and is then ahead of Paris, which stays at 9. For sentence B, the same meter shows only a 1. There, 2 · 1 = 2 is added: France goes from 9 to 11, Paris stays at 6. The lead grows; the answer stays.
The difference thus comes from the meter the fader is attached to. A fader only amplifies what is currently arriving at its meter. The Paris score is not affected, because this fader feeds only into the France score.
In a real model, the same parameters serve many questions. A change therefore shifts many things a little, but not in a way that makes every related answer follow along. Facts can still be changed on purpose. In 2022, researchers rewrote individual facts in GPT models (the family behind ChatGPT) by recomputing certain parameters in middle stages. But later tests showed that related facts often don’t follow. Give a person a new father in the model, and it often still names the old siblings, those who share the same father. Unlike a table row, a fact can’t be cleanly corrected in one place.
What researchers find in the intermediate values
No one has yet mapped all the faders involved in “Paris”. But researchers can study the intermediate values.
A team at Anthropic, the company behind Claude, did so with a small language model. A single intermediate value, like one meter in the picture, fired for academic citations, English dialogue and Korean text. So this intermediate value does not stand for a single concept.
In a version of Claude, the team used a helper program to extract millions of recurring combinations across many intermediate values. The texts that made a combination fire revealed what it represented, such as famous people, countries or cities. Such combinations are called features. A feature is like a chord: a single note occurs in many chords and alone doesn’t reveal which is sounding. Only the notes together make C major. Unlike chords, nobody defined the features; they arose during training. Which ones you find also depends on the helper program.
Parameters, intermediate values and features are three different quantities. Parameters are stored and fixed, like the faders. Features are combinations of intermediate values, like a pattern across many meters at once. A feature for Paris is not yet the fact that Paris is France’s capital. That fact only emerges when the calculation for a question puts “Paris” first.
One level deeper: how features share intermediate values
Many features share the same intermediate values, each as its own combination; this is called superposition. For 82 percent of the features examined in Claude 3 Sonnet, no single neuron, no intermediate value like the one with Korean text, was strongly linked to it. So more features fit in than there are intermediate values: in a small model with one computing stage, tens of thousands could be extracted from 512 of them. This works because only a few are usually active at once; when several are, they interfere with each other.
Features influence the answer. In an experiment, the team held the Golden Gate Bridge feature at ten times its maximum while computing, without changing parameters. The model then described itself as the Golden Gate Bridge. The experiment doesn’t show where a fact sits.
Three questions where a table behaves differently
A country | capital table reads each row from both sides, finds every entry equally well and reports “no match” for missing entries. The small model and a larger one of its kind with 4 billion parameters received three questions. They got the reversed start “Paris is the capital of” (in German: “Paris ist die Hauptstadt von”) and questions about the birth dates of Albert Einstein and the nonexistent physicist Bernhard Quelling. Here the model answers alone; some chatbots add web search hits as extra input. Pause to think: what does the small model write next in each case?
Live demo · try it yourself
A model keeps writing
Real output, computed in advancePick a sentence start from the experiment and let a real language model continue it, token by token. At every step you see the five text pieces that fit best, with their percentages.
Try it: for the reversed start “Paris ist die Hauptstadt von” (Paris is the capital of), does the right answer lead again? And does the model stop for the invented Bernhard Quelling, or does it write a date anyway? Guess first, then check.
Solution
For the reversed start, the small model puts “Deutschland” (Germany) ahead, the larger one “Frank…”, the beginning of “Frankreich” (France). For the invented Quelling, both write a date without hesitating, even though the top digit there often leads with only 10 to 20 percent; for Einstein’s year digits it is mostly over 95 percent. The reason is the same: the model computes which text piece fits after this start; there is no row that could be missing and report “no match”. Carried over: the rarer a person is in texts, the less a fluently stated date tells you about whether it is true.
Real values from Qwen3-0.6B-Base and Qwen3-4B-Base, two openly available base versions without post-training into a chatbot, computed once in advance as in the experiment for this lesson. The sentence starts are German, as in the experiment, so the model continues in German. One run, one wording, no hit rate. You can only pick another candidate where that branch was computed in advance; there is no randomness here.
The first case, the reversed start: the small model put “Deutschland” (Germany) first, at just under 30 percent. The larger one put “Frank…”, the start of “Frankreich” (France), first at just over 60 percent. For Paris, the larger model manages the reversal, probably because texts name Paris and France in both orders. This one run doesn’t reveal why the small model was wrong.
Still, direction matters. In training, the start of a text is the input and the next piece is the answer the model should learn, as with “The cat sat” and “on.” By contrast, texts usually name a celebrity before their mother. In a 2023 study, GPT-4, a model behind ChatGPT, answered questions like “Who is Tom Cruise’s mother?” (Mary Lee Pfeiffer) correctly almost four times in five. It answered reversed questions like “Who is Mary Lee Pfeiffer’s son?” correctly only once in three. So the reversal often fails when learned from training alone. The son question contained only the mother’s name, not the whole statement. If the statement is in the text, say “Tom Cruise is the son of Mary Lee Pfeiffer”, the model can deduce the reverse.
The second case: some facts stick better than others. A study shows that a model answers correctly more often the more matching texts it saw in training, and that larger models retain more. Einstein’s birth date is a very common fact; this run shows only the size difference. Given “The physicist Albert Einstein was born on” (in German: “Der Physiker Albert Einstein wurde geboren am”), the small model wrote “August 14, 1879, in Zurich”; the larger one correctly wrote “March 14, 1879, in Ulm”.
The third case, the made-up Bernhard Quelling: both models promptly named a birth date, the small one “August 13, 1920, in the city of Berlin”. The calculation always yields a score list, with some text piece on top. “I don’t know” would be just one possible continuation; after this start, a date fits better. Neither model put the invisible stop marker that ends an answer first. Post-trained chatbots (further tuned with example conversations after pretraining) more often say they don’t know; that too is learned behavior, not a search result, and it sometimes fails.
Answers that were never written down
Computing lets a model combine what it learned. A version of Claude was asked for the capital of the US state where Dallas is. That is Texas, whose capital is Austin. The Anthropic team saw a feature for Texas fire first, triggered by “Dallas”. Together with the question about a capital, it led to Austin.
Think first: is the model really combining here, or retrieving a memorized answer? During the calculation, the researchers replaced the Texas feature with one for California by changing intermediate values, not parameters. The model then answered “Sacramento,” California’s capital. In this example, the second step depended on the first; this one experiment doesn’t show whether models always work this way.
Still, a model can sometimes reproduce training text almost word for word: hundreds of passages could be extracted from GPT-2. The fader settings hold such passages spread out; no single line stores them.
Why the model would rather guess than stay silent
In 2023, two New York lawyers filed a federal court brief that cited six nonexistent court decisions. ChatGPT had generated them, with docket numbers and quotations resembling real rulings. The model had learned what rulings and docket numbers look like from many texts. It produced the form even without the facts.
Fluent, plausible-sounding statements that aren’t true are called hallucinations.

The first cause lies in pretraining. A language model must produce the next piece at every position in a text, even where its training data offers little. The second lies in evaluation. In large sets of test questions, a correct answer often earns one point, a wrong one zero, and “I don’t know” zero too. It is like a multiple-choice exam without penalties: if you don’t know, you answer anyway. If such tests evaluate chatbots during post-training, guessing still pays. This isn’t a conscious choice.
Third, the model misapplies what it learned. In a version of Claude, the Anthropic team saw that “I can’t answer that” is initially the default answer to every question. A feature for “I know this,” triggered by, say, a well-known person, switches it off. If a name only seems familiar, the feature sometimes fires anyway, and the model writes whatever sounds plausible.
Facts you can’t derive from a rule are vulnerable, such as a person’s birth date when only a few texts about that person exist. A fact that appeared only once in training is usually not retained reliably, even with error-free data. The rarer a fact, such as a ruling, a quotation or a date of birth, the less a fluent answer reveals about its truth. A later lesson shows how to check them.
Not every AI model is a language model
You know the spam filter and image recognition, technically an image classifier, from the foundations: an email or a photo goes in, a verdict comes out, such as “spam” or “cat.” These are AI models too, but not language models.
Many image generators, like Stable Diffusion (a freely available image generator), work differently. They are diffusion models: they start from pure noise, like snow on an old TV, and make the whole image clearer over many steps, steered by your text. A language model instead appends piece after piece. Multimodal models process several kinds of input, such as a chatbot you send a photo and a question. Many answer with text; some also make images. All consist of a blueprint and parameters set by training.
This topic area follows the path through a language model, because most large language models today share one basic pattern: they predict the next token, seeing only the text before it. Your text becomes tokens. Each token gets a list of numbers. Many computing stages, called blocks, then mix in the sentence’s context. Finally, the exit, the output head, turns this into the score list.
So a chatbot knows that Paris is France’s capital because training set its parameters so that this question’s calculation puts “Paris” first. The model looks up no fact; it computes a continuation from your input that sometimes repeats training text almost word for word. That is why the model can recombine what it learned, and why it fluently fills gaps with inventions.
The path has four stations. The first turns your chat message into a long sequence of tokens that records who said what. Between the stations, layers of neurons compute; the lesson Layers shows how they are built.
Explain it more simply
- Explain more simplyCore idea and an everyday comparison
- With an exampleOne concrete example, step by step
- In more detailEvery step on its own, in plain words
Recall moment
What stayed with you?
Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.
Preparing questions …
Questions from the community
Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.