You Can't Eval Your Way to Quality Without Production Observability

The fundamental limit of offline eval

A golden dataset is one of the highest-leverage tools in AI quality engineering, and it has a hard ceiling. By construction, it contains the cases you anticipated. It is your hypothesis about what users will send and how the model should respond. Real traffic is the experiment that tests that hypothesis, and the experiment always finds inputs the hypothesis missed: the phrasing nobody imagined, the language you do not support, the adversarial probe, the edge case that only exists at 2am under load.

This is not a flaw in offline eval; it is its nature. Offline eval answers “does the model handle the cases I thought of?” That is necessary and valuable. But it structurally cannot answer “does the model handle the cases I didn’t think of?”, and those are exactly the ones that generate incidents, because nobody was watching for them. The teams that get blindsided are the ones who treated a green offline suite as proof of production quality.

Why “it works on my machine” is uniquely false for LLMs

Traditional software has a comforting locality: if it passes in staging, it usually behaves the same in production, because the code is identical and the inputs are bounded. LLM systems break both assumptions. The inputs are an open set, users will phrase things in ways your staging data never contained, and the “code” (the hosted model) can change underneath you with no deploy on your side. The result is that the distance between a green staging run and real production behavior is far larger than engineers’ intuition, trained on deterministic systems, leads them to expect.

That intuition gap is why teams chronically under-invest in production observability for AI features specifically. They ship the same monitoring they would for a CRUD service, uptime, error rate, latency, and those dashboards stay reassuringly green while the actual quality of the answers degrades. Status codes cannot see a hallucination; a 200 OK is returned for a confidently wrong answer just as readily as a correct one. The monitoring that matters for an LLM is monitoring of the content, and that is a different discipline with different tooling.

The two layers: tracing and online evaluation

Production eval observability has two distinct jobs. The first is tracing: capturing what happened on every request, the prompt, the retrieval context, the model and version, token counts, latency, and the final output, stitched into a span tree you can inspect. The second is online evaluation: scoring a sample of that live traffic for quality so you have metrics, not just logs. Tracing tells you what the system did; online eval tells you how well it did it.

The industry now has a standard for the first layer. OpenTelemetry’s GenAI semantic conventions define standard attribute names for LLM operations, gen_ai.request.model, input and output token counts, finish reasons, and more, so observability is consistent across providers and frameworks instead of a bespoke logging mess. The conventions are still maturing (most remain experimental as of 2026), but as the OpenTelemetry project’s own introduction to LLM observability argues, building on the open standard means your traces are portable and your tooling is not locked to one vendor, and major observability vendors now support them natively.

# Trace every LLM call with OpenTelemetry GenAI semantic conventions
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

def traced_generate(request):
    with tracer.start_as_current_span("chat") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", MODEL)
        span.set_attribute("gen_ai.request.temperature", TEMP)
        out = model(request)
        span.set_attribute("gen_ai.usage.input_tokens",  out.usage.input)
        span.set_attribute("gen_ai.usage.output_tokens", out.usage.output)
        span.set_attribute("gen_ai.response.model", out.model_version)  # catch silent swaps
        return out

Online evaluation: scoring traffic you can’t hand-review

You cannot manually read production output at scale, so online eval automates the scoring. Sample a representative slice of live traffic and run the same kinds of graders you use offline, groundedness checks, schema validation, safety classifiers, and LLM-as-judge for nuance, against real outputs. The result is a live time series of quality metrics that turns “it feels worse lately” into a dated, attributable signal.

# Online eval: sample live traffic, score it, alert on regressions
def online_eval(trace, sample_rate=0.05):
    if random() > sample_rate:
        return                                   # score a representative sample
    out = trace.output
    scores = {
        'grounded':  is_grounded(out, trace.retrieved_sources),
        'format_ok': validates_schema(out, trace.expected_schema),
        'safe':      safety_score(out) > 0.9,
        'relevant':  relevance(out, trace.input) > 0.8,
    }
    for metric, ok in scores.items():
        emit_metric(f"genai.quality.{metric}", 1.0 if ok else 0.0,
                    attrs={"model": trace.model_version})
        if not ok:
            log_failure(trace, metric)           # a real example to add to the golden set

# Alert when a live quality metric drops past threshold (drift, model swap, bad deploy)
def watch(metric, window="1h", floor=0.95):
    if rolling_mean(metric, window) < floor:
        alert(f"{metric} dropped below {floor:.0%} on live traffic")

Two refinements make this work in practice. First, sampling: you do not need to score every request, a representative sample gives you the trend at a fraction of the grading cost (OpenTelemetry’s ratio-based samplers handle this cleanly). Second, reference-free graders: production has no gold label, so your online graders must judge quality without a known-correct answer (groundedness against retrieved context, schema validity, safety scoring) rather than exact-match against a reference.

The flywheel: production failures become golden cases

The highest-value output of production observability is not the dashboard, it is the stream of real failures it surfaces. Every ungrounded answer, every parser break, every safety miss caught in production is a case your offline golden set was missing. Feed it back in. Over time this closes the loop offline eval cannot close on its own: production discovers the inputs you never imagined, and your golden set absorbs them so the next regression is caught before launch. Tooling increasingly supports this directly, promoting production traces into curated eval datasets.

What to watch, and the silent-swap tripwire

Beyond per-request quality scores, the metrics worth standing up: groundedness rate, format-adherence rate, safety-violation rate, refusal rate, p95/p99 latency, cost per request, and, critically, the response model version returned by the provider. Logging the actual model version on every response is your tripwire for silent provider updates: when your quality metrics shift and the response-model-version field shows a change you did not make, you have caught a silent model swap red-handed, with a date attached.

None of these require reading individual outputs by hand. They turn the unbounded, non-deterministic behavior of a production LLM into a small set of watchable numbers, which is the only way to manage something you cannot exhaustively test.

Trust the judge before you trust its scores

Online evaluation at scale usually leans on automated graders, classifiers and LLM-as-judge, because no human can read the volume. But an unvalidated judge is just another unverified model in your stack, and a miscalibrated one can mask a real regression or invent a phantom one. Before you alert on a judge’s scores, validate the judge: assemble a few hundred human-labeled production samples and confirm the grader agrees with human judgment at an acceptable rate. Re-validate periodically, because a judge built on a hosted model is subject to the same silent drift as the system it grades.

The practical pattern is a two-tier loop: cheap automated graders score the full sample and raise candidate alerts; a human reviews a small confirmation set when an alert fires. This keeps the cost of online eval low while ensuring a page at 3am reflects a real quality drop rather than a judge having a bad day. Treat grader agreement as a metric you monitor, exactly like the quality metrics themselves.

The bottom line

Offline eval proves the model handles the cases you anticipated; it is structurally blind to the long tail of real inputs you did not. Closing that gap requires production observability: trace every call using the OpenTelemetry GenAI conventions so your data is standard and portable, sample and score live traffic with reference-free graders for groundedness, format, and safety, and alert when any quality metric drops. Log the response model version to catch silent provider swaps, and feed every production failure back into your golden set. You cannot eval your way to quality without watching production, the wild is where your real test cases live.