Tokenizer — A tokenizer is the tool that splits text into tokens and turns them into numbers a model can process. Different models use different tokenizers. A tokenizer explains why models sometimes struggle with spelling or counting letters, since they see chunks instead of characters. It also affects how much a given text costs to process.

Added October 21, 2026