The Cheapest Model That Passes: Eval-Driven Model Selection

You are buying capability you do not use

The default model-selection process is aspirational: pick the most capable model available, on the theory that the best model gives the best results. It is also, for most workloads, a quiet waste of money. Your traffic is not uniformly hard. A large share of requests, classification, extraction, simple Q&A, formatting, routing, are easily within reach of a model that costs a small fraction of the frontier flagship. You are paying frontier prices to answer questions a much cheaper model would get right.

The price differential is stark enough to change architecture decisions. Per 2026 pricing trackers, the cheapest capable APIs sit around $0.10 per million input tokens (budget tiers like the smallest Gemini Flash, GPT nano, and Mistral Small), while flagship and reasoning models commonly run $2.50-$30 per million input tokens and $15-$168 on output. That is a 25-to-300x spread depending on the pair. At any real volume, that ratio is the difference between a feature that is profitable and one that is not. (Prices have also been falling fast, roughly 80% across 2025-2026, which is itself a reason to re-evaluate selection regularly.)

Routing: the highest-ROI architecture you are not using

The insight that powers cost-efficient LLM systems is that most queries do not need the most capable model. The RouteLLM research made this concrete: an intelligent router that sends easy queries to a cheap model and escalates only hard ones to an expensive model can achieve large cost savings, on the order of 50-85% in reported settings, while retaining most of the frontier model’s quality. Industry practice bears this out: routing roughly 70-90% of requests to a budget tier and reserving a flagship for the demanding remainder commonly knocks 60-80% off a bill without users noticing. The router is a small, fast classifier deciding, per request, how much horsepower the question actually warrants.

The related pattern is the cascade: try the cheap model first, and only escalate to the expensive one when a confidence or verification check says the cheap answer is not good enough. Cascades pay the expensive model’s price only on the requests that genuinely need it, but they add latency on escalated requests and require a reliable “is this answer good enough?” signal.

# Cascade: cheap model first, escalate only when verification fails
def cascade_answer(query):
    cheap = CHEAP_MODEL(query)
    if verify(cheap, query):              # confidence / self-check / grounding pass
        return cheap, {'tier': 'cheap', 'cost': cheap.cost}
    strong = FRONTIER_MODEL(query)        # pay the premium only when warranted
    return strong, {'tier': 'frontier', 'cost': cheap.cost + strong.cost}

# Router: a fast classifier decides the tier up front (no double-spend)
def routed_answer(query):
    difficulty = router_score(query)      # small/cheap difficulty classifier
    model = CHEAP_MODEL if difficulty < THRESHOLD else FRONTIER_MODEL
    return model(query), {'tier': 'cheap' if difficulty < THRESHOLD else 'frontier'}

The eval is what makes routing safe

Routing and cascading sound like obvious wins until you realize the risk: route a hard query to the cheap model and you ship a worse answer to a user. The thing that turns “risky cost optimization” into “measured engineering decision” is an eval that tells you, per task type, exactly what each model can and cannot handle. Without that, you are guessing where the difficulty threshold goes, and guessing wrong means silent quality regressions.

The disciplined process is to run your golden set against every candidate model and build a per-task quality-vs-cost table. Then your routing thresholds are not intuition; they are derived from measured pass rates.

# Build the quality-vs-cost table that drives model selection
def benchmark_models(models, golden_set_by_task):
    table = {}
    for model in models:
        for task, cases in golden_set_by_task.items():
            scores = [grade(model(c.input), c.gold) for c in cases]
            table[(model.name, task)] = {
                'pass_rate':  round(mean(s >= c.threshold for s, c in zip(scores, cases)), 4),
                'cost_per_1k': model.cost_per_1k_tokens,
            }
    return table

def cheapest_passing(table, task, bar=0.95):
    candidates = [(m, v) for (m, t), v in table.items()
                  if t == task and v['pass_rate'] >= bar]
    # The whole thesis in one line: cheapest model that clears the bar wins
    return min(candidates, key=lambda x: x[1]['cost_per_1k'], default=None)

This table is the deliverable. It answers the real question for each task, what is the cheapest model that clears my quality bar?, with a number instead of a hunch. It also tells you which tasks genuinely require the frontier model, so you stop apologizing for paying for those.

Watch the second-order costs

Sticker price per token is not the whole bill. A few traps that turn an apparent saving into a loss:

• Verbosity. A “cheaper” model that emits twice the tokens to say the same thing can cost more per useful answer. Measure cost per resolved request, not per token.
• Retry and escalation overhead. A cheap model that fails 20% of the time and triggers a frontier retry pays both prices. Factor escalation rate into the cascade math.
• Latency cost. Cascades add a round-trip on escalation; if that pushes you past your TTFT budget, the saving comes out of abandonment instead of margin.
• Prompt-caching eligibility. Pricing for cached input tokens varies by provider and model; a model with aggressive caching can beat a nominally cheaper one on a cache-friendly workload.

All of these are measurable, and all of them belong in the cost column of your quality-vs-cost table. The point of the table is to make the true, fully-loaded cost visible so the “cheap” choice is actually cheap.

Re-run it on a schedule

Model selection is not a one-time decision. Providers ship new tiers, drop prices, and silently update existing models constantly, the 2025-2026 pace of new cheap-but-capable models has been relentless, with industry-wide price cuts around 80%. A model that failed your bar six months ago may pass today at a tenth of the cost; the frontier model you depend on may have quietly drifted. Re-run the benchmark on a schedule, and treat “is there now a cheaper model that passes?” as a recurring optimization rather than a settled question.

Stacking the other cost levers on top

Model selection is the biggest lever, but it is not the only one, and they compound. Live pricing trackers like independent model-analysis dashboards are useful for keeping the cost column of your table current, but the structural savings come from combining routing with the techniques that reduce how many tokens you pay for in the first place:

• Prompt caching. If a large system prompt or retrieved context repeats across requests, cached input tokens are billed at a steep discount, often around a tenth of the normal rate, which can dwarf the savings from a model downgrade on cache-friendly workloads.
• Context compression. Trimming or summarizing retrieved context before it hits the model cuts input tokens directly, and on long-context RAG workloads the input side is usually where the bill lives.
• Output discipline. Asking for terse, structured output (and capping max tokens) controls the more expensive output side, where per-token prices are typically several times the input rate.

Each of these is independently measurable and, critically, each carries its own quality risk, over-compress the context and groundedness drops; cap output too hard and answers get truncated. That is the recurring theme: every cost optimization is a potential quality regression, and the only thing that lets you take the saving safely is an eval that catches the regression before users do.

The bottom line

Defaulting to the frontier model overpays for capability most requests never use, and the price gap to smaller models is 25-to-300x. Reframe selection from “which model is best?” to “what is the cheapest model that passes my eval, per task?” Build a quality-vs-cost table by running your golden set against every candidate, use it to set routing or cascade thresholds, account for verbosity, retries, latency, and caching in the true cost, and re-run it as the market shifts. Routing easy queries to cheap models can cut cost 60-85%, but only an eval makes that saving safe instead of a silent regression.