Lesson 2 of 6 · Inside the ModelRead
Layers: How Many Neurons Become a Network
Many neurons calculate using the same numbers, and their outputs become the inputs of the next layer. Why the lower limit makes a peak possible that two layers without it cannot form, and what a figure like “96 layers” means.
The basic neuron calculates a sum, adds a bias and sets negative values to zero. That last step is the lower limit. A network is made up of many such neurons, and their arrangement determines how they work together.
Many neurons on the same numbers
Picture a mixing-desk channel that receives numbers. Now place a second channel next to it, and a third. All three see the same inputs, but each has its own sliders. This group is called a layer.
A layer is a group of neurons that receive the same inputs. Each neuron performs its own calculation, using its own weights and bias.
The counting box shows this for two inputs and three neurons. Each row is one neuron. Each row contains two empty boxes for the weights, one per input. The amber box at the end contains the bias. Read from left to right: the weight for the first input, the weight for the second, then the bias.

The picture shows the layout of this example: the two switches feed a layer with the neurons A and B, whose outputs run into the output neuron C. Read the picture from left to right to follow the path of the numbers. Each arrow stands for a value that travels from one place to the next. The inputs do not calculate; they only pass on what you enter. Only the neurons calculate, and the output neuron turns the outputs of A and B into a single number at the end. The two-switch example in the section “Intermediate values: the output keeps going” uses this layout.
How many numbers are in a layer?
First, guess how many numbers are in the layer shown in the counting box. Then count them step by step. Each neuron has one weight per input, so with two inputs each neuron has two weights, and the three neurons together have six.
Add the biases. Each neuron has exactly one, so that is three more. Together that makes nine numbers.
The same rule applies to every other layer: one weight for each input and each neuron, plus one bias per neuron.
A different layer with three inputs and two neurons has three times two, so six weights.
Add two biases, one per neuron: eight numbers.
Intermediate values: the output keeps going
The output of a layer is not the end of the calculation. It becomes the input of the next layer. A number that travels from one layer to the next is called an intermediate value.
Consider an example with two switches, one at the top and one at the bottom, like a stair light. The numbers in this example were chosen by hand, not learned through training. The stair light only shows how this calculation works; it says nothing about what a real model with billions of trained numbers does. A pressed switch counts as 1, a switch that is not pressed as 0. The light should come on when exactly one of the two switches is pressed, that is, when C shows 1 at the end.
A and B both receive the two switches with a weight of 1. A’s bias is 0; for B, the bias adds −1.
Output neuron C receives only the outputs of A and B. It gives A a weight of 1 and B a weight of −2; its bias is 0.
What does C show when both switches are pressed? Guess first.
The table shows all three cases. The column for B already reflects the lower limit, which sets negative values to zero, as the neuron from the basics does.
| Switches pressed | A | B (after the lower limit) | C |
|---|---|---|---|
| none | 0 | 0 | 0 |
| one | 1 | 0 | 1 |
| both | 2 | 1 | 0 |
With no switch pressed, B is really −1. The lower limit turns that into 0, so C shows 0 there.
Without a lower limit, the peak is missing
The calculation above has one fixed step: the lower limit at neurons A and B. Now leave it out. Guess first: when no switch is pressed, does C show 0 or 2?
Live demo · try it yourself
The stair light without a lower limit
Invented simulationEnter the guess you just made. Afterwards you can switch the lower limit off and look.
Calculation and all positions
Set weights and biases
Solution
With the starting weights, without the lower limit, neuron B is −1 instead of 0 when no switch is pressed. The output neuron uses weight −2 for B: 0 plus (−2 · −1), which is 2. Why: without the lower limit every computation is a sum, and a sum of sums stays a sum. The 2 comes only from B's bias −1, multiplied by the weight −2. The lower limit was the only place that brought the result down to 0.
All numbers are invented and set by hand; they do not come from training a model. The demo only shows how the sum, the bias and the lower limit compute. Nothing about a real model with billions of such numbers can be read from this small network.
Now work through the four cases. If no switch is pressed, B stays at −1 without the lower limit. C then calculates −2 times −1 and shows 2.
With one or two pressed switches, nothing changes, because B was never negative there: C shows the same as above. Group the cases by the number of pressed switches, and the value falls by one each time: from 2 to 1 and then to 0. A peak would require the values to rise first and then fall again.
The reason holds for every network whose layers only calculate weighted sums plus a bias, with no lower limit or other step between them. Each output is a sum of the inputs, each multiplied by its weight, plus a bias. Feeding it into the next layer produces a sum of sums. That sum can be folded back into a single sum: each input receives a new weight: along each path from the input to the output, the weights are multiplied, and the paths are added together. All the biases together make up one fixed number. This is true for two layers as for ten.
In this example, both switches have the same weights. So C depends only on how many switches are pressed, and each further pressed switch changes C by the same amount, here −1. A series that always falls by the same amount cannot rise and then fall.
With the lower limit, by contrast, the value rises as soon as one switch is pressed, and falls again when both are pressed. That is the peak. It sits at exactly one pressed switch, and that spot is what makes the pattern “exactly one” possible. Without the lower limit, the peak is missing. It takes a step that does more than add up with weights, such as the lower limit does.
One level deeper: why does this example have no peak without a lower limit?
Without a lower limit, A is the number of pressed switches, and B is that same number minus 1. Each pressed switch raises A and B by 1. C calculates A minus twice B, so C falls by 1 for each pressed switch (1 minus 2). With no switch pressed, C stands at 2, because B is −1 and gets multiplied by −2. With zero, one or two pressed switches, this gives the values 2, 1, 0. The distance between two neighboring values is always the same, namely one. Because the calculation falls at a steady rate, it never turns around, and a peak requires such a turning point. Without a step between them, such layers stay a single calculation, however many there are, so in this example no peak appears.
What a figure like “96 layers” means
Descriptions of language models often include their number of layers. GPT-3, a language model from 2020, has 96 layers. Here, “layer” does not mean the kind shown in the counting box. It refers to the transformer block, one of a sequence of building blocks inside the model. The number 96 counts blocks, not neurons.
Each block contains a feed-forward network, a two-layer network in which information moves in only one direction, from input to output. Another step sits between the two layers, resembling the lower limit of the neuron; its exact form depends on the model.
A block has two main parts. In one, each word looks back at the words that come before it. The other is the two-layer feed-forward network described above. The numbers a block puts out are the inputs of the next block, just as the intermediate values A and B were the inputs of C.
Where the numbers for the first layer come from
Layers work with numbers, but the question you type is text. Before the first layer can process it, the text has to be translated into numbers.
The next lesson shows how text breaks into small pieces, how each piece gets a number, and how the layers then work with these numbers.
The weights are already set at this point. How they come about is explained in later lessons.
Explain it more simply
- Explain more simplyCore idea and an everyday comparison
- With an exampleOne concrete example, step by step
- In more detailEvery step on its own, in plain words
Recall moment
What stayed with you?
Answer without scrolling back. This is not a grade; the point is to retrieve what you learned from memory once. As soon as you pick an answer, you see whether it is right.
Preparing questions …
Questions from the community
Something still unclear? Ask your question about this lesson on the community board. Other learners and the project answer there, and you can see what others have already asked.