Available inEnglishFrenchGermanHindiPortugueseRussianSpanish
How LLMs Work: Tokens to Transformers
Explain why the model you build against behaves the way it does, from the token it reads to the agent loop around it, without opening a slide deck. For developers who use LLM APIs daily and never took the mechanism apart; no calculus, no code. Sixty-six cards in ten chapters cover tokens, embeddings, attention, sampling, RLHF, hallucination, retrieval and agents, and nothing about which model leads this month.
You watch a reply appear one piece at a time. What is the unit the model actually reads and writes?
A token: a chunk of text from a fixed vocabulary
— Not the letter. Not always the whole word either.
Source
A tokenizer turns your text into numbers before the model sees anything, using a vocabulary learned from a corpus: common strings become one token, rare ones are split into several. Everything downstream is denominated in these units: the price you pay, the size of the context window, the model's blindness to spelling. A rough English rule of thumb is four characters per token, so 1,000 tokens is about 750 words. Commonly confused with: the word. Word and token coincide often enough in ordinary English prose to hide the difference, and then a name, a rare term or a non-English script costs several tokens each.