Glossary
Benchmark
A benchmark is a fixed set of test questions with which you compare AI models. All models get the same questions or tasks, and their answers are scored by a fixed rule. Only when all models get the same test with the same scoring are their results directly comparable.
An example: A fact test consists of short questions with known answers. Each right answer brings one point; each wrong answer and each skipped answer brings zero points. A model that guesses loses nothing and can win a point, so this scoring rewards guessing. The scoring decides whether a model is rewarded for guessing or for staying silent.
Not the same as general ability: A benchmark measures only what it asks. A model that does well is therefore not good at every task.
Where you’ll come across it: In reports on new models, in comparison tables, and in notes on which tests a provider used. A table without the name of the test and the scoring says little. So check whether the test is named and how the points were awarded. Without that information, the comparison cannot be checked.
Introduced in Why AI Is Convincingly Wrong: When Guessing Pays Off and a Confident Tone Proves Nothing.