Pretraining — Pretraining is the first and largest stage of building a language model, in which it learns from vast amounts of general text. During pretraining, the model learns language, facts, and patterns of reasoning. It is extremely expensive and can take months. Later stages shape the pretrained model into a helpful assistant.

Added October 21, 2026