Tokens, windows and budgets

Start from bytes, merge what repeats

Byte-pair encoding begins with the smallest possible alphabet — bytes — and repeatedly merges the pair of symbols that occurs most often in the training text into a new single symbol. Run that loop a few tens of thousands of times and the vocabulary fills up with the units that actually recur: whole common words, frequent fragments like ##ing, ation or pre, and the space that begins a word as its own marker. A token is one of those learned units. Nothing in the definition says "word"; the algorithm optimises frequency, not linguistics.

The consequence is that a word boundary is invisible to the model. Common words cost one token each; a rare word shatters into a stem plus continuation pieces; a string the tokenizer never saw during training shatters all the way down to bytes. Below is a toy model of that behaviour — it is not any vendor's tokenizer, but it produces the same shapes: frequent short words stay whole, long or rare words split, and single non-alphanumeric characters become tokens of their own.

Three different units, three different counts

It is tempting to treat "token" as a synonym for "word" and multiply by a price. The three units move independently, and their ratio depends entirely on what the text is made of. English prose is the best case: it lands near four characters per token, or roughly 1.3 tokens per word, because its frequent words and morphemes were merged. Source code is worse, because identifiers, punctuation and indentation are not natural-language statistics. Non-Latin scripts are worse still: a script whose characters are individually rare in the training mix gets tokenized nearly character by character, so a single Chinese or Japanese word can cost several tokens where its English translation costs one.

Numbers are the quiet trap. Whether 128000, 3.14159 or 2026-09-15 is one token or four depends on what the merge list happened to capture, and the answer differs between one model family and the next. This is why arithmetic is a poor first task for a language model, and why "the model got the digit wrong" is often a tokenization fact before it is a reasoning fact.

Counts and ids are not portable

Each model family trains its own tokenizer over its own data mix, so it has a different vocabulary size, a different merge list, and therefore a different number of tokens for the same sentence. The integer id that stands for a token is even less meaningful: it is a serial number into that family's embedding table, assigned by construction order, not by any property of the text. Token #18447 in one family and #18447 in another have nothing whatsoever to do with each other.

The demo below sends the current passage through two mock tokenizers with different hash functions. The token strings are identical; the ids are entirely different. That is the whole point — a count from a tokenizer you are not actually going to call is a guess, and the estimate is only trustworthy when it comes from the same model version you will run.

Line items, and a reserve for the answer

The context window is the maximum number of tokens the model can attend to in a single call — the prompt plus the completion. Everything the model knows about this request must fit inside it: the system instructions, the few-shot examples, the retrieved passages, the conversation history, and then the answer it is about to write. Treat it as a budget with named line items, and reserve room for the output first, because a truncated answer is usually worse than a shorter prompt.

Drag the line items below. The stacked column is one prompt's composition against its window, the dashed line is the ceiling, and the readout warns before the input plus the reserve overruns it.

Nominal window sizes are a marketing number, not a working capacity. The middle of a long context is where information gets lost: accuracy is highest for material at the very start and the very end and sags in between, so a fact parked at position 40,000 of a 128,000-token window can be effectively absent. . Read the window as an upper bound you should rarely approach.

Two more line items deserve names. Prefix caching lets a provider reuse the computation for a byte-identical prompt prefix, so keeping stable content first and volatile content last is a budget decision as well as a latency one.

Turn the budget into a count

Most context decisions are really a packing problem: given a window, a reserve for the answer, some fixed overhead, and a per-item token cost, how many items fit? Each retrieved chunk, few-shot example or tool result has a price in tokens, and the number that fits falls as that price rises. The demo sweeps the per-item cost and plots the count that fits; the marker is the cost you currently have selected.

The design lesson is to measure items in tokens before you decide how many to buy. Ten chunks of 800 tokens and twenty chunks of 400 tokens occupy the same budget but behave very differently in retrieval quality — and neither is free.

Cheat sheet

Further reading

• Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016 (arXiv:1508.07909, 2015) — the paper that brought byte-pair encoding to neural models.
• Kudo & Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer", EMNLP 2018 (arXiv:1808.06226) — the unigram/LM subword view and the family of vocabularies in wide use.
• OpenAI, "How to count tokens with tiktoken", 2023 — the practical reason token counts are model-specific and must be measured, not estimated.
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL 2024 (arXiv:2307.03172) — the U-shaped accuracy curve behind the context-rot argument.
• Chroma, "Context Rot", July 2025 — degradation that begins well below the advertised window on many models.
• Anthropic, Prompt caching and OpenAI, Prompt caching guide — cache-read multipliers and the minimum cacheable length that shape prompt layout.