Glossary

Self-supervised learning

Self-supervised learning is a way of training in which nobody has to write down answers, because they are already in the data. A language model learns this way in its pretraining: every beginning of a text is a question; the token that follows is the answer. Think of a cover card laid over a line of text, showing only the start. Unlike a reader, the model does not guess one word after another; it works through all positions in one pass and keeps no answer, it nudges its numbers. A text of 12 tokens therefore yields 11 practice problems, one fewer than it has tokens. Because this holds for every text, trillions of tokens become trillions of problems.

An example: In the sentence “The bus leaves again on time”, one problem is “The bus leaves again on”, and its answer is “time”. For this position the model names a guess, which is compared with “time”, and training adjusts the parameters a little so that “time” becomes more likely next time.

Not to be confused with learning without answers: Here, too, every problem has a correct answer, the label. It just does not come from people but from the text itself. With a spam filter, by contrast, people mark every email by hand. Fine-tuning a chatbot on answers people wrote or rated is not self-supervised learning either.

Where you’ll come across it: In reports on the pretraining of large language models, and in teaching material on machine learning, often next to supervised and unsupervised learning.

Introduced in Text Becomes Many Practice Problems.

Last changed on · commit 7edbb52