Cutting LLM costs by 10× without cutting quality

The first month of an AI feature in production is usually a pleasant surprise. The sixth is usually not, because token spend scales with usage and nobody optimised anything while it was cheap.

The good news: most production systems are wildly inefficient in four specific, fixable ways. Fixing them typically takes an order of magnitude off the bill without touching output quality. Here they are in order of leverage.

1. Prompt caching — the biggest lever, and the most commonly broken

Every request in a typical system re-sends the same enormous prefix: system prompt, tool definitions, few-shot examples, retrieved policy documents, conversation history. You pay full price for all of it, every single time.

Prompt caching lets the provider keep that prefix warm. Reads cost roughly a tenth of standard input pricing. Writes cost a modest premium — around 1.25× for the short-lived cache, around 2× for the long one — which means the short cache breaks even at two requests and the long one at three. For anything with a stable prefix and repeat traffic, that's an enormous saving.

And it silently doesn't work most of the time. Here's the mechanism, because understanding it is the whole job:

Caching is a prefix match. Any byte that changes anywhere in the prefix invalidates everything after it.

The prompt renders in a fixed order — tools, then system, then messages — and the cache key derives from the exact bytes up to each breakpoint. One different character at position 400 destroys the cache for everything from position 400 onward.

The things that quietly do this:

Pattern	Why it breaks
datetime.now() in the system prompt	Prefix differs on every request
A request ID or UUID early in the content	Same
json.dumps(d) without sorted keys, or iterating a set	Non-deterministic serialisation
User or session ID interpolated into the system prompt	Per-user prefix; nothing shared
Conditional system sections (if flag: system += ...)	Every flag combination is a distinct prefix
A tool list that varies per user	Tools render first — nothing after them caches

The architectural rules that follow:

• Freeze the system prompt. No dates, no names, no modes. Inject dynamic context later in the message list, where it invalidates only what comes after it.
• Serialise tools deterministically and don't change the set mid-conversation. Tool definitions render at position zero; changing them costs you the entire cache.
• Don't switch models mid-conversation. Caches are model-scoped. If a sub-task needs a cheaper model, run it as a separate call rather than swapping the main loop's model.
• Put volatile content last. Stable content first, per-session content next, per-turn content at the end.

Verify it. The response usage object reports cache_read_input_tokens. If it's zero across repeated requests that should share a prefix, you have a silent invalidator — diff the rendered bytes of two consecutive requests and you'll find it in minutes.

One more thing that catches people: the minimum cacheable prefix varies by model, and it isn't monotonic across generations. Depending on which model you're on it can be anywhere from ~512 to ~4096 tokens. A 3,000-token prompt caches on some models and silently doesn't on others — no error, just a zero in the usage field. Check the minimum for the model you're actually running.

2. Route per task, not per system

The second-biggest lever, and the one most teams skip because it feels like a downgrade.

It isn't. A single system contains steps with wildly different difficulty:

• Classifying ticket intent into one of twelve categories
• Extracting an order number from free text
• Deciding whether a message is angry
• Writing the customer-facing reply
• Making a refund judgement on an ambiguous case

The first three are narrow, high-volume, and eminently measurable. The last two need real judgement. Running everything on your most capable model means paying frontier prices for string classification.

Route by step. Small fast models for classification, extraction, routing, and formatting. Mid-tier for most generation. Frontier for anything customer-facing, judgement-heavy, or long-horizon and agentic.

The cost differential between tiers is meaningful — the smallest capable models run at a fraction of frontier per-token rates — and in the narrow tasks it's often more accurate, because you can build a real eval set for a task with a fixed label space and actually tune it.

Second dial: reasoning effort. Current frontier models expose an effort control that trades deliberation against tokens and latency. Most teams leave it at the default, which is usually tuned high. Sweep it on your own eval set — on routine operational work, a notch or two down frequently holds quality while cutting spend substantially. On long-horizon agentic runs, higher effort often reduces total cost, because better planning means fewer turns. Don't assume the direction; measure it.

3. Batch anything that isn't interactive

If a result isn't needed in this HTTP request, it shouldn't be paying interactive prices.

The Batch API processes asynchronously at 50% of standard pricing, with most batches completing within an hour and a 24-hour ceiling. It supports the full feature set — tools, caching, vision, structured output.

Everything on this list belongs in a batch:

• Nightly classification, enrichment or scoring runs
• Bulk content generation (product descriptions, translations, summaries)
• Backfilling embeddings or metadata over a corpus
• Eval runs
• Report generation
• Anything triggered by a cron rather than a user

Half your bill, for work nobody is waiting on. It's the easiest 50% you'll ever find, and it's usually left on the table because the code was written for the interactive path and never revisited.

4. Spend context discipline

Tokens you send are tokens you pay for, and long context has a second cost: models attend unevenly across a large window, so padding the prompt often makes output worse as well as more expensive.

Retrieve less, better. Five well-ranked passages beat twenty mediocre ones on both cost and accuracy. This is the practical payoff of reranking — it lets you send fewer chunks with more confidence.

Prune conversation history. In a long agentic run, tool results from twenty turns ago are almost never load-bearing. Clear stale tool outputs, or summarise the older history into a compact form. Most providers now offer this server-side.

Cap output. max_tokens is a real cost control, not just a safety limit. Set it to what the task actually needs. But set it high enough — truncated output means a retry, which costs more than the headroom would have.

Watch for the reasoning-token surprise. On models where reasoning is enabled by default, max_tokens caps thinking plus the visible response together. A limit sized tightly around the expected answer on a non-reasoning model will truncate mid-answer on a reasoning one. This catches people during migrations.

Putting numbers on it

A representative support system before optimisation:

• Every request sends an 8,000-token prefix uncached
• Everything runs on the frontier model
• Nightly enrichment runs synchronously through the same path
• Retrieval sends 20 chunks

After:

• Prefix cached; the stable portion bills at roughly a tenth
• Classification and routing on a small model; generation on mid-tier; frontier only for escalation-adjacent judgement calls
• Nightly enrichment moved to batch, at half price
• Reranking cuts retrieval to 6 chunks

None of that is a quality trade-off. Reranking improves accuracy. Routing improves the narrow tasks. Caching changes nothing about the output at all. The compounding of the four is where the order of magnitude comes from.

The order to do it in

1. Measure first. Log token counts per call path, tagged by feature. You cannot optimise a bill you can't attribute, and the distribution is almost never what people guess.
2. Fix caching. Highest leverage, no quality risk, and usually already half-configured and silently broken.
3. Move batchable work to batch. Mechanical, halves that portion of the bill.
4. Route by task. Requires evals per step — which you want anyway.
5. Tune context and effort. Ongoing, incremental, measured.

Step one is the one people skip, and it's the one that tells you whether the other four are worth doing at all.

---

We instrument spend before we optimise it, and hand the dashboard over with the code. Book an audit call.