How Many Tokens Is a Word? A Practical Token Budgeting Guide

Every LLM cost estimate starts with a token count, and most token counts start with a guess. The guess is usually low, and the error propagates through every projection built on it.

#The baseline for English

For ordinary English prose:

• 1 token ≈ 0.75 words
• 1,000 tokens ≈ 750 words ≈ 4,000 characters
• 1 page of text ≈ 500 tokens
• 1 typical email ≈ 100–300 tokens
• A 20-page PDF ≈ 12,000–15,000 tokens

Common words are single tokens. Longer or rarer words split into sub-word pieces — "unbelievable" might become three tokens. Punctuation and whitespace consume tokens too.

#Where the estimate breaks

Code: add 30% to 50%. Source code tokenises poorly because indentation, brackets, operators and camelCase identifiers all fragment. A 500-line Python file that is 15,000 characters may be 5,000 tokens rather than the 3,750 the character ratio suggests.

JSON and XML: add 30% to 60%. Structural characters — braces, quotes, colons, commas — are individually tokenised, and repeated key names are paid for on every object. A JSON array of 200 records with eight fields each pays for those eight key names 200 times.

Non-Latin scripts: 2× to 3×. Chinese, Japanese, Korean, Arabic, Hindi, Thai and Cyrillic all tokenise less efficiently than English in most tokenisers, because the training data is English-dominant. A Chinese document conveying the same meaning as an English one commonly uses two to three times the tokens.

If you serve non-English markets, this materially changes your per-user cost and should be modelled per language, not on a blended average.

Numbers and IDs. Long numeric strings, UUIDs and hashes tokenise into many small pieces. A table of 500 UUIDs is more expensive than it looks.

#Budgeting a real request

Do not estimate the whole prompt as one number. Break it down:

System prompt              ~800 tokens
Tool definitions         ~1,200 tokens
Retrieved context (4×500) ~2,000 tokens
Conversation history      ~1,500 tokens
Current user message        ~150 tokens
---------------------------------------
Total input              ~5,650 tokens

Expected output          ~600 tokens

Itemising like this does two useful things: it makes the estimate more accurate, and it immediately shows where the cost is. In this example, tool definitions and retrieved context are 57% of input — both cacheable, which is the obvious optimisation.

Feed the totals into the LLM API cost calculator with your request volume.

#Output tokens are the expensive ones

Output typically costs three to five times input per token. Two implications:

A verbose model is expensive. The same task done in 200 tokens instead of 800 saves more than trimming 600 input tokens would.

Estimate output generously. Output length is far more variable than input. A model that occasionally decides to be thorough can produce a long tail that dominates your bill. Set max_tokens and estimate at the 75th percentile of observed lengths, not the median.

#Measuring rather than estimating

Estimates are fine for planning. For anything you are committing budget to, measure:

Use the provider's tokeniser. Every major provider publishes one, and they differ from each other — a prompt that is 5,000 tokens for one model may be 5,400 for another.

Log actual usage. API responses return input and output token counts. Log them per request with the feature name, then aggregate. Within a week you will have real distributions rather than assumptions.

Watch the percentiles. Mean token counts hide the tail. If p50 is 4,000 and p99 is 40,000, a small share of requests is driving a large share of cost, and finding out why is usually productive.

#Context windows and effective limits

Large context windows create a temptation to fill them. Three reasons not to:

Cost scales linearly. A 200,000-token context costs 200,000 tokens' worth on every request.

Attention degrades. Retrieval accuracy over very long contexts is measurably worse than over well-selected short ones. More context frequently means worse answers.

Latency scales too. Time to first token rises with input length, which users feel.

Retrieving four relevant chunks beats stuffing forty, on both cost and quality.

#The estimate to carry around

For quick mental arithmetic on English text: characters ÷ 4. For code or JSON: characters ÷ 3. For non-Latin scripts: characters ÷ 1.5 to 2.

Then add 20% before you commit to a budget, because everyone's first estimate is low.

Frequently asked questions