Token (LLM)

A token is the unit of text a large language model actually processes. Before a model reads anything, a tokeniser splits the text into tokens, which can be whole words, pieces of words, punctuation or spaces. Everything practical about using an LLM is measured in tokens: how much text fits in the context window, how long an answer can be and how much an API call costs.

How it works

Tokenization for LLMs usually uses subword methods, so common words become a single token and rarer words are split into several. OpenAI’s rule of thumb for English is that one token is roughly four characters, or about three quarters of a word. Other languages, code and unusual strings often need more tokens for the same meaning.

Tokens are counted in both directions. Input tokens are your prompt plus anything you send with it (instructions, documents, chat history). Output tokens are what the model writes back. Providers price these separately, and output tokens usually cost more. Images and audio are converted into tokens too: Google’s Gemini documentation counts a small image as 258 tokens and tiles larger images into several 258-token blocks.

How it’s used in SEO and marketing

• Cost control: when you run an LLM over thousands of pages (meta descriptions, anchor text, classification), the bill is input tokens times rows plus output tokens times rows. MLforSEO’s internal linking script prints the tokens used and an estimated cost after every run.
• Agent fetch limits: AI agents cap how much of a fetched page they will process. The AI Search and Agentic SEO course describes a page coming back at 193,217 tokens against a 200,000-token window; anything past the limit is simply cut off.
• Chunking for retrieval: retrieval systems split pages into segments of roughly a few hundred tokens before creating embeddings, so a definition separated from its term at a chunk boundary can be lost. See chunking.
• Prompt design: shorter, structured prompts reduce cost and leave more room for the content you want analysed.

Example

You want to classify 5,000 product pages. Each prompt is about 600 tokens (instructions plus the page excerpt) and each answer about 20 tokens. That is roughly 3 million input tokens and 100,000 output tokens. Multiply by your provider’s current per-million-token prices to estimate the job before you start, and test on 50 rows first.

Limitations and things to watch

• Token counts depend on the tokeniser, so the same text costs a different number of tokens on different models. Use the provider’s own counter for estimates.
• More tokens in the context window does not guarantee the model uses all of it well; long inputs can still lose detail.
• Heavy pages (navigation, scripts, boilerplate) waste an agent’s token budget before it reaches your main content.

Learn it on MLforSEO

• Your robots.txt might be the silent killer of your AI traffic
• The chunk, not the page: content an AI can actually extract
• 10 Ways to Automate Internal Linking with AI
• AI Search Optimisation and Agentic SEO