What is a token?
AI models do not read words. They read tokens. Three ideas, three minutes.
Speeds: machines.json, 2026-10-26. Context sizes: models.json, 2026-10-26.
A token is a piece of a word.
Models read and write in tokens, one at a time.
A piece of a word. 100 tokens is about 75 English words.
A short word is often one token. A long or rare word is split into a few. Each model splits text its own way, so counts differ a little from model to model.
When the model answers, it writes one token, then the next, then the next. That is why the answer appears bit by bit.
Speed = tokens per second.
Written tok/s. People read at about 5 tok/s.
- Slow: llama3.2 3B on a Raspberry Pi 5 (CPU only): 5.6 tok/s. About as fast as you read.
- Fast: gpt-oss-20b on an M5 Pro Mac: 76.6 tok/s. Much faster than you read.
These are published tests on real computers (Machines lists the source of each). A bigger model is slower on the same computer. More on speed: The basics.
Context = what it keeps in mind.
How many tokens the model can read at once: your chat, files and instructions.
Example: at 16 GB of memory, our recipes install Gemma 4 12B. Its maker says it can hold up to 262,144 tokens of context (HF config.json).
But more context needs more memory. So Ollama, the free engine that runs the model, starts smaller. It picks the size from your graphics memory: 4k (about 4,000) tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k from 48 GiB up.
Our memory numbers leave room for a chat of about 8K to 16K tokens. Ask for much more, and the model may not fit any more.
Source: Ollama docs: Context length, read 2026-10-22
Want a longer memory? You can change it.
Only if your memory has room to spare.
In the Ollama app: open Settings and move the context length slider. From the terminal, set it when you start Ollama:
OLLAMA_CONTEXT_LENGTH=64000 ollama serve64000 tokens is the example in Ollama's docs. They suggest at least 64,000 for agents, web search and coding tools.
Then check it still runs on the graphics chip: ollama ps shows a Processor column. 100% GPU is good. A CPU share means it did not fit, and it will be slower.
Sources: Ollama docs: Context length, read 2026-10-22; Ollama FAQ, read 2026-10-22
Where this comes from.
Each page with the day we read it.
- Ollama docs: Context length read 2026-10-22
Default context by graphics memory (under 24 GiB: 4k; 24–48 GiB: 32k; 48 GiB and up: 256k); a larger context needs more memory; how to change it.
- Ollama FAQ read 2026-10-22
Models stay in memory 5 minutes by default;
ollama stopunloads one now;ollama psshows what is loaded and whether it runs on the GPU.