How to Cut LLM API Costs by 70% Without Changing Models

Most teams' first response to a large AI bill is to switch to a cheaper model, which trades quality for cost. There is usually a great deal of savings available before that trade becomes necessary.

Here are six optimisations, ordered roughly by typical impact.

#1. Prompt caching — commonly 40% to 70%

If a substantial part of your input repeats across requests — a system prompt, tool schema, document, style guide, or product catalogue — caching it typically reduces the cost of that portion by around 90% on each read.

The requirement is that cached content must be a stable prefix. Structure prompts so unchanging material comes first and variable content last:

[ system prompt        ]  <- cached
[ tool definitions     ]  <- cached
[ retrieved documents  ]  <- cached if stable
[ conversation history ]  <- partially cached
[ current user message ]  <- never cached

Reordering a prompt so the variable part sits at the end is frequently a one-line change worth thousands of dollars a month. Set the cacheable share in the LLM API cost calculator to see the effect on your workload.

For RAG systems where 80% of input tokens are a stable context block, total spend commonly falls by more than half from this alone.

#2. Batch processing — 50% on eligible traffic

Batch APIs accept a set of requests and return results asynchronously, usually within 24 hours, at roughly half the standard price.

Audit what is genuinely synchronous. Most teams find 60% to 80% of their token volume is not user-facing: classification, enrichment, embedding generation, evaluation runs, nightly summarisation, dataset labelling, moderation backfills.

A hybrid architecture — synchronous for interactive requests, batch for the pipeline — captures the discount without affecting anyone's experience.

#3. Cap output length — 15% to 30%

Output tokens typically cost three to five times input tokens. Two changes:

Set max_tokens to a realistic ceiling. This bounds the worst case, which matters because a model that decides to be thorough on 2% of requests can dominate your bill.

Instruct for brevity in the prompt. "Respond in at most three sentences" or "return only the JSON object, no explanation" reliably reduces output length, and usually improves the product too.

For structured extraction, request only the fields you need. A schema with twelve fields where you use four is paying for eight.

#4. Trim retrieved context — 20% to 40% on RAG

Retrieval systems commonly send ten or twelve chunks when three or four would answer the question. Each unnecessary chunk is pure cost.

Rerank before sending. A cross-encoder reranker over twenty retrieved candidates, sending only the top three, usually improves answer quality and cuts input tokens by two thirds.

Tune chunk size. Smaller chunks retrieved more precisely beat large chunks retrieved loosely.

Deduplicate. Overlapping chunks from adjacent passages waste tokens on repeated text.

Better retrieval is almost always cheaper than more context.

#5. Routing — 30% to 60% where quality allows

Send the easy majority of requests to a smaller model and escalate only where needed. Escalation triggers that work in practice:

• A confidence or self-consistency check on the small model's output
• Input length or complexity above a threshold
• A schema validation failure on structured output
• Explicit user request for a more thorough answer

Model this by running the cost calculator twice at your estimated split. A 70/30 routing between a small and a frontier model often lands within a few percent of frontier-only quality at a fraction of the cost — but only if you actually measure the quality gap on your own data.

#What to do first

Measure before optimising. Log input tokens, output tokens, model and latency per request, and aggregate by feature. The distribution is almost always concentrated — one endpoint typically accounts for the majority of spend, and that is where the effort belongs.

Then, in order: implement caching, move eligible traffic to batch, cap outputs, trim retrieval, and cache identical requests at the application layer. Those four require no quality trade-off at all. Routing and model changes come afterwards, once the free savings are exhausted.

#The metric that matters

Optimise cost per successfully completed task, not cost per token. A cheaper model that needs three attempts, longer prompts or human correction can cost more in total than one capable call.

If AI inference is becoming a material part of your cost of goods sold, feed the resulting gross margin into the SaaS metrics calculator — it changes LTV and CAC payback more than most teams expect.

Frequently asked questions