For looking things up
Glossary
107 terms, explained briefly and precisely — every lesson links right here whenever a technical word first appears.
A
- Activation FunctionThe step after a neuron’s sum that reshapes the number; the lesson The Artificial Neuron calls it the lower limit, which sets negative values to zero.
- AgentA combination of a model and a program that uses tools in a loop: the model writes calls, the program carries them out, and the results come back as text.
- AI (Artificial Intelligence)Umbrella term for systems that solve tasks usually associated with perceiving, language, planning, or decision-making.
- AlgorithmA general solution procedure made of a limited sequence of clear steps — the idea behind a program, not the concrete code itself.
- ArchitectureThe blueprint of a model: what is computed with the numbers and how many parameters there are for it, regardless of their values.
- AttentionThe method a language model uses to mix weighted information from earlier tokens into the state of a token.
B
- BackpropagationThe method that computes the gradient: a single backward pass from the error to the input, which gives every fader its slope but turns none.
- Base ModelA model after pretraining that has only learned to continue text and is not yet post-trained for conversations.
- BatchA batch is a set of examples in training, averaged into one error number; in inference, many requests computed together.
- BenchmarkA standardised test in which different AI models get the same tasks and are scored by a fixed rule.
- BiasA number a neuron adds to its weighted sum before the activation function; it stays fixed while the model answers, and some texts call it bias term.
- Byte Pair Encoding (BPE)A method that progressively merges frequent neighboring characters or previously formed text pieces into larger units.
C
- Causal maskThe mask in a language model’s attention that lets every token see only itself and earlier tokens.
- Chain of thoughtThe method in which a model writes its intermediate steps as text before it gives the answer.
- Chat TemplateThe fixed pattern of a chat model that joins messages with roles and special tokens into one single token sequence.
- Context WindowThe limited number of tokens a model can consider in one processing operation.
D
E
- EmbeddingThe learned list of numbers a language model looks up for every token; tokens used in similar ways get similar embeddings.
- Embedding matrixThe large table at the input of a language model, with one row of learned numbers for every token in the vocabulary.
- EpochOne complete pass through a training dataset: every example came up exactly once, so each was seen one time.
- Expert systemsEarly AI systems that captured the knowledge of specialists in many hand-written if-then rules, rather than learning from examples.
F
- Feed-Forward NetworkA network of layers whose numbers flow only forward, without feedback; often also called an MLP.
- Forward PassThe path the numbers take through a model from the input to the prediction: the same when answering as in training, where it covers all positions at once.
- FunctionA computational step that takes an input and computes an output from it — the same input always produces the same output.
G
H
I
- Image classifierA trained model that assigns images to one of several predefined categories, such as "cat" or "dog".
- In-Context LearningA model picks up a pattern from examples in its input, without its weights changing.
- InferenceUsing a fully trained model: with its fixed parameters, it computes an output from an input.
- InputThe data fed into a computational step — a number, an image, a text, or many values at once.
- Instruction ModelA model that results from post-training and is meant to follow instructions and react to questions with answers.
- Intermediate ValueThe number a neuron produces while calculating an input and passes on to the next layer; it is created anew for every text.
K
L
- Label (Target)The expected correct output that a model's prediction is compared against during training, also called the target.
- Language modelA model trained on large amounts of text that scores how well each text piece it knows fits next. This is how text emerges, one piece at a time.
- Lateral readingYou check a claim by leaving the page and looking outside it: who makes the claim, and what do other sources say?
- LayerA group of neurons that receive the same input and calculate in parallel; in model descriptions, “layer” often means a whole transformer block.
- Learning rateA hyperparameter that sets how strongly the training algorithm adjusts the parameters at each step.
- Learning rate scheduleThe course of the learning rate over a training run, fixed in advance: usually a short ramp up, the warm-up, then a long decline.
- Loss (Cross-Entropy)The one number that measures how far a model’s predictions are off during training; for language models it is the average of the penalty points.
- Lost in the MiddleFinding that information in the middle of long inputs is often used less well than information at the beginning or the end.
M
- Machine learningThe umbrella term for methods where a training algorithm sets a model's parameters from examples, instead of a human writing decision rules by hand.
- MatrixA matrix is a table of numbers with two axes, rows and columns, in which every number has a fixed address.
- MemorizationWhen a model has learned a training text so well that it can later reproduce it word for word.
- ModelA calculation whose behavior additionally depends on stored, trained numerical values (parameters).
- MomentumA moving average of earlier slopes that the optimizer follows instead of each single gradient, so back-and-forth cancels out.
- Multimodal modelAn AI model that processes several kinds of input, such as text and images together, and computes an answer from them.
N
O
- OptimizerThe fixed calculation rule in training that turns the gradient into a step for every parameter and actually changes the parameters.
- OutputThe result of a computational step — for a language model, not a finished answer but a list of scores for the next piece of text.
- Output HeadThe last computing layer of a language model: it turns the last state into a score for every entry in the vocabulary.
- OverfittingA model adapts to individual training examples instead of what they have in common, and gets worse on data it does not know.
P
- ParametersThe stored, adjustable numerical values a trained model consists of — also called weights.
- PermissionThe rule of a program that decides which tool calls run without a question, which ask first, and which do not run at all.
- Personal dataAll information that relates to a specific or identifiable person, such as names, phone numbers or email addresses.
- Position embeddingA vector for a place in the text that tells the model the order, usually added to the embedding and either learned or fixed by formula.
- Post-TrainingThe second training after pretraining, which uses examples written or rated by people to teach a model answers that fit a conversation.
- ProbabilityA number between 0 and 100 percent that states how often something happens if the same situation repeats very many times.
- ProbeA small additional reading device that checks the numbers in a model for whether they contain a particular piece of information.
- ProgramA sequence of instructions written by humans that a computer carries out step by step; every rule is fixed before it starts.
- PromptThe text input you give an AI model — it only works while it is sent along and doesn't change the model's stored parameters.
- Prompt injectionHidden instructions in text that a model processes, for example on a web page or in a tool result, that are meant to steer the model.
Q
R
- RAGRetrieval-augmented generation: suitable passages are searched for and placed in the context before the answer.
- Recommendation systemA system that selects possible content and ranks it by how relevant it might be to a person.
- Reversal curseThe finding that a model often does not answer in reverse a relationship it learned during training.
- Reward ModelA model that learns to predict which of several answers people prefer, and gives each answer a number as its rating.
- Rights reservationA rights holder’s declaration that their work may not be used for text and data mining, and why it must be machine-readable.
- robots.txtA text file on a website in which the operator tells crawlers which areas they may retrieve.
S
- Sample (Example)A matched pair of input and expected output that a model is trained on, also called a training example.
- SamplingChoosing the next text piece by weighted chance: each piece comes up roughly as often as its probability says.
- Saved memoryA piece of information that is stored outside the model and only enters a conversation when it is fetched.
- ScalarA scalar is a single number with no place in a list or table, the simplest block of numbers, with no axis at all.
- ScoreA numeric value a model uses to rate a possible answer, such as a class or the next piece of text, before it becomes a probability.
- Self-supervised learningTraining in which the answers are already in the data: for a language model, the next token in the text is the answer to every problem.
- SentencePieceA language-independent system for learning and applying subword tokenizers directly from raw text.
- SoftmaxThe calculation a model uses to turn a list of scores into probabilities that add up to 100 percent.
- Spam filterA trained model that classifies an email as spam or not spam, often expressed as a probability such as "92% spam".
- Special TokenA vocabulary entry that marks structure rather than text: the start, roles, message boundaries or the end of an answer.
- StepOne round of training: a batch of examples, a comparison with the right answers, every parameter adjusted once.
- Stochastic parrotA warning from the debate: a language model assembles word sequences without any link to meaning.
- Subword TokenA reusable text piece that can be smaller than a word, so that rare words can be built from known parts.
- SycophancyThe tendency of a chatbot to agree with the person it is talking to, even when its correct answer becomes wrong as a result.
- System PromptInstructions at the start of a conversation that reach the model as a message with the system role, before the user’s question.
T
- TemperatureA setting that divides the scores before softmax: low values make the selection more predictable, high values more varied.
- TensorA block of numbers with any number of axes; scalar, vector and matrix are special cases of it, and AI models compute in such blocks.
- Text and data miningThe automated analysis of large amounts of digital works to find patterns, trends and connections, and the legal exception for it.
- Thinking budgetA guideline for how much a model should think before it answers, which used to be set as a fixed number.
- Thinking effortThe computing work a model spends before an answer, measured by its thinking tokens and its passes through the network.
- Thinking tokensThe intermediate steps a model writes as text before its answer.
- TokenOne unit in a tokenizer’s fixed list of pieces (its vocabulary), for example a whole word, a word part, a single character, a punctuation mark, or a marker for structure.
- Token budgetThe total number of tokens a training run processes, repetitions included: the figure that makes training runs easiest to compare.
- Token IDThe fixed number assigned to a token in a particular tokenizer’s vocabulary. The model uses this number instead of the text itself.
- TokenizerA fixed process that turns text into tokens and token IDs, then turns those IDs back into text.
- ToolA function that a program carries out so that a chatbot can fetch information or complete tasks.
- Tool callThe request with which a model wants to use a tool: a piece of text in a fixed format that a program recognizes and carries out.
- Training algorithmThe fixed procedure applied to training examples that adjusts a model's parameters step by step.
- Training dataThe examples a training algorithm is applied to in order to set a model's parameters, such as labeled emails or large amounts of text.
- TransformerThe architecture presented in 2017, made of many identical blocks with attention, on which today’s language models are based.
- Transformer blockThe repeated building block of a transformer: attention mixes between tokens, then further processing works on each token on its own.
V
- Validation lossThe loss on held-out text that is never trained on, showing whether a model learns patterns rather than particulars.
- VectorAn ordered list of numbers in which every position is part of the information; a language model fetches one per token from a learned table.
- VocabularyThe fixed list of entries a tokenizer recognizes and a model uses to build its answers.
W
No term matches your search.