How to choose a model — and why the answer is usually 'more than one'

"Which model should we use?" is the most common question we get and the one with the least useful general answer, because it's the wrong shape. It assumes a system uses one model.

Almost no well-built production system does.

The framing that works: route per step

A support system contains steps with radically different difficulty:

Step	Difficulty	Volume	Model tier
Detect language	Trivial	Every message	Smallest
Classify intent into 12 categories	Easy, narrow	Every message	Small
Extract order number from free text	Easy	Most messages	Small
Detect sentiment / escalation risk	Moderate, high stakes	Every message	Small–mid, heavily evaluated
Retrieve and rank relevant policy	Moderate	Most messages	Mid
Compose the customer reply	Needs judgement	Most messages	Mid–frontier
Decide an ambiguous refund	High stakes, judgement	~8%	Frontier
Investigate a multi-system failure	Long-horizon agentic	Rare	Frontier, high effort

Running all of that on your most capable model means paying frontier prices to detect that a message is in Dutch.

Routing per step isn't a compromise. On the narrow tasks it's frequently more accurate, because a task with a fixed label space and thousands of daily examples is one you can actually build a proper eval set for and tune.

The three tiers, functionally

Setting aside vendor names and version numbers, current models sort into three practical bands.

Small / fast. Classification, extraction, routing, formatting, simple transforms. Sub-second latency, roughly an order of magnitude cheaper per token than frontier. Use it wherever the task is narrow and measurable. If you can define the output space precisely, start here and only escalate if evals say you must.

Mid. The workhorse. Most generation, most retrieval-grounded answering, most tool use. Recent mid-tier models have closed much of the gap to frontier on coding and agentic work specifically. For a large share of production traffic this is the right default, and starting here rather than at the top is usually the better instinct.

Frontier. Deep reasoning, long-horizon autonomous work, complex multi-file engineering, anything where being wrong is expensive. Also the right choice for genuinely hard one-off tasks where the cost of the call is irrelevant next to the cost of a bad result.

The practical rule: start mid, measure, move up only where evals demand it and down wherever they permit it.

The second dial: reasoning effort

This is newer, less understood, and often a bigger lever than the model choice itself.

Current frontier models expose a control over how much deliberation goes into a response — how deeply they think, how many tool calls they make, how much they verify. Higher settings mean better results on hard problems and substantially more tokens and latency. Lower settings mean faster, terser, more literal execution scoped tightly to what was asked.

Three things worth knowing:

The relationship to cost isn't monotonic. On long-horizon agentic work, higher effort often reduces total cost — better planning means fewer turns, fewer wrong paths, less rework. On short interactive tasks, lower effort is straightforwardly cheaper. You have to measure per workload.

Low settings are stronger than people expect on current models. A frontier model at low effort frequently outperforms a previous generation at maximum effort. If you migrated a workload and kept the old effort setting, you're probably overpaying.

It's not a verbosity control. Teams reach for lower effort to get shorter answers and are surprised when it doesn't work. Response length is a prompting concern; effort governs deliberation. Use the right lever.

Sweep the range on your own eval set. Defaults are tuned for a general population, not your workload.

Selection criteria that actually matter

Task-specific accuracy, measured on your data. Public benchmarks tell you about public benchmarks. A model that leads a leaderboard may be worse than a small model at your specific classification task with your specific label taxonomy. Twenty of your own examples beat any leaderboard.

Latency budget. Interactive chat needs sub-two-second first tokens. A nightly batch job does not. Don't pay for speed in a cron job, and don't build an interactive feature on a model whose thinking phase takes minutes.

Context window vs. what you actually need. Enormous windows are useful and often a crutch. If you need to send a million tokens to answer a question, your retrieval is broken and the model is doing search on your behalf, expensively and less accurately.

Structured output support. If you need guaranteed schema conformance, verify the model enforces it rather than merely usually complying. This matters more for pipeline reliability than a few points of benchmark score.

Tool use quality. For agents, this is the differentiator — not raw reasoning but whether the model calls the right tool with correct arguments and knows when not to call one. Models differ substantially here and the difference doesn't show up in general benchmarks. Note also that models vary in how eagerly they reach for tools; a system tuned around one model's reluctance will over-trigger on a model that's keener, and vice versa.

Deployment constraints. Data residency, on-premise requirements, procurement approval, existing cloud commitments. These frequently decide the question before any technical criterion gets a vote. Find out early.

The costs of switching that nobody prices in

Model migration is rarely a string change, and teams consistently under-budget it.

Prompts don't transfer cleanly. A prompt tuned around one model's tendencies becomes a mis-tune on another. Aggressive tool-calling language written to overcome a reluctant model produces over-triggering on an eager one. Verbosity instructions written to counter a terse model overcorrect on a warmer one. Verification scaffolding that helped a weaker model produces redundant loops on one that self-verifies.

Token counts change. Different models tokenise differently — the same text can be 30% more tokens on one than another. That shifts your context budgets, your max_tokens limits, and your cost baseline even when per-token pricing is identical. Re-measure with the target model's own token counter rather than reusing old numbers.

Defaults shift silently. Whether reasoning is on by default, whether reasoning text is returned, what the default effort level is — these have changed between generations, without erroring. A workload that ran without reasoning by omission can start reasoning after a version bump, spending tokens and potentially truncating output against a max_tokens sized for the old behaviour.

Caches are model-scoped. Switching invalidates every warm prefix. First requests on the new model pay full price.

API surfaces evolve. Parameters get removed between generations. Sampling controls, fixed thinking budgets, response prefilling — all have been deprecated or removed on recent frontier models, returning errors rather than warnings.

The practical defence: an eval set that runs against any model. With one, migration is an afternoon: run the suite on the candidate, diff per case, tune what regressed. Without one, it's a month of vibes.

A default architecture

If you want somewhere to start rather than a framework to reason with:

1. Small fast model for classification, routing, extraction, language detection, and any narrow task with a fixed output space.
2. Mid-tier model as the default for generation and tool use, at moderate reasoning effort.
3. Frontier model for the escalation path, ambiguous judgement calls, and long-horizon agentic runs, at higher effort.
4. An abstraction thin enough to swap — one interface, per-step model configuration, no provider SDK calls scattered through business logic.
5. An eval set per step, so you know which direction to move each one.

Then measure, and let the numbers move each step up or down. That's the whole method — and it's the part that survives the next model release, which is more than can be said for any specific recommendation.

---

We benchmark per step on client data and re-benchmark quarterly — you're never locked to one provider. Book an audit call.