Batch API vs Real-Time: When the 50% Discount Is Free Money

Batch APIs offer roughly a 50% discount in exchange for asynchronous delivery, usually within 24 hours. For a large share of production AI traffic, that trade costs nothing at all — because nobody is waiting.

Most teams underuse it because the eligible traffic is not obvious until you go looking.

#What batch processing is

You submit a file containing many requests, the provider processes them when capacity allows, and you collect results when the job completes. Typical turnaround is well under the stated 24-hour ceiling, often within an hour, though there is no guarantee.

The discount reflects the value of schedulable load to the provider — they can run your work during troughs rather than reserving capacity for it.

#Identifying eligible traffic

The test is simple: is a human waiting for this specific response right now? If not, it is a batch candidate.

Clearly eligible:

• Classification and tagging of incoming records
• Data enrichment and normalisation
• Embedding generation for a corpus
• Bulk summarisation — documents, transcripts, tickets
• Evaluation runs against a test set
• Content moderation backfills
• Nightly report and digest generation
• Dataset labelling and synthetic data creation
• Reprocessing after a prompt change

Not eligible:

• Chat responses
• Autocomplete and inline suggestions
• Anything behind a loading spinner
• Interactive search and retrieval

Depends on product design:

• "Analyse my document" — if you can email results, batch works
• Onboarding enrichment — if it completes before first login, batch works
• Weekly insights — always batchable

That third category is where the opportunity usually is. A feature designed as synchronous because that was the default can often be redesigned as asynchronous with a notification, halving its cost and improving perceived reliability.

#The audit

Log every LLM call for a week with feature, token counts and whether a user was blocked on the response. Then aggregate.

Teams commonly find 60% to 80% of token volume is not user-facing, and that a single background pipeline — enrichment, indexing or evaluation — accounts for most of it. Moving that one pipeline to batch is frequently a same-day change with an immediate 50% reduction in the largest line of the bill.

Model the effect in the LLM API cost calculator by toggling the batch option and adjusting your volume split.

#Architecting a hybrid pipeline

A structure that works well:

A single request abstraction with a priority flag. Application code calls one interface; the layer beneath routes to the synchronous or batch endpoint.

A queue for batch work. Accumulate requests and submit when either a size threshold or a time window is reached — for example, 500 requests or every fifteen minutes, whichever comes first.

Idempotency keys on every request, so a resubmission after a partial failure does not duplicate work or cost.

A fallback path. If a batch job fails or a deadline approaches, reroute to synchronous. Rare, but you want it defined before it happens.

Result reconciliation. Batch responses come back keyed by your request IDs. Store the mapping so results land in the right place.

#Combining batch with caching

These stack, and the combination is where the largest savings appear.

If your batch requests share a stable prefix — the same classification instructions and taxonomy across thousands of records — caching applies within the batch too. A workload with 80% cacheable input running through batch can see total costs fall by 80% or more against a naive synchronous implementation.

Order the optimisations: structure prompts for caching first, then move eligible traffic to batch. Doing it the other way round means restructuring the batch payloads afterwards.

#The operational caveats

No latency guarantee. The stated window is a ceiling, not a target. Do not build anything with a hard deadline inside it.

Rate and size limits. Providers cap requests per batch and concurrent jobs. Chunk large workloads accordingly.

Partial failures. Individual requests within a batch can fail while others succeed. Handle per-request errors rather than assuming a job is all-or-nothing.

Harder to debug. You lose the immediate feedback loop. Keep a synchronous path available for development and testing against a small sample.

Model availability. Not every model is offered on every batch endpoint. Check before designing around it.

#When it is not worth it

If your total AI spend is small, the engineering effort may exceed the savings. A useful threshold: if batchable traffic is under a few hundred dollars a month, the queueing and reconciliation infrastructure is probably not worth building yet.

Above that, the payback is usually measured in days.

Frequently asked questions