TL;DR
Request ID (for connecting traces across distributed systems) Timestamp (precise; use UTC) Model name and version Temperature, top_p, max_tokens, and other parameters Full prompt text (not summarized) Full completion text Token count (input and output) Latency (time to first token, total time) Cost (per-token pricing × tokens used) User or session ID (for behavior correlation) Upstream context (what data was fed to the prompt) Dedicated LLM observability platform: Langsmith, Humanloop, Arthur (captures and indexes traces, makes them searchable) Time-series database: ClickHouse, TimescaleDB (cheap storage, fast queries on large volumes) Data warehouse: Snowflake, BigQuery (overkill for real-time alerting, but good for batch analysis) Certain user segments get worse recommendations Queries with specific keywords trigger lower-quality responses Long prompts degrade reasoning (or improve it) Model A outperforms Model B on your specific use case (contrary to benchmarks) For this user, what prompts led to good outcomes vs. bad outcomes? For this use case, which model configuration performs best? When did quality degrade, and what changed in that timeframe? Threshold alerts: "If accuracy drops below 85% for 10 minutes, page the on-call engineer" Trend alerts: "If accuracy has dropped 10% over the last 24 hours, send a warning" Anomaly alerts: "If today's response latency is 3 standard deviations above the weekly average, investigate" Comparison alerts: "If Model A's quality is more than 3% worse than Model B on the same queries, flag it" Saturation alerts: "If we're approaching our API token budget, warn before spending doubles" An alert fires: "Accuracy on financial advice queries dropped from 91% to 78%" Engineer investigates the last 50 queries that triggered the alert Engineer identifies the pattern: recent system prompt change removed a critical constraint about not making specific recommendations Engineer writes a test case: "On financial queries with uncertain inputs, AI should decline to recommend and suggest consulting a professional" Test is added to the permanent test suite and runs on every new system prompt or model change The change that caused the degradation is reverted; all is well Maintain a "test case generation queue." When quality drops, automatically extract the failing queries and mark them as candidates for new test cases. Have an engineer spend one hour per week reviewing this queue, writing permanent tests for any patterns. Run these tests in CI/CD before deploying new models, prompts, or configurations. Day 1: Choose an LLM observability platform (Langsmith is easiest to start) and set up basic tracing Day 2: Instrument prompt and completion logging with business outcome fields Day 3: Write aggregation scripts to compute hourly metrics from logs Day 4: Build a dashboard and set up baseline alerts (accuracy, cost, latency) Day 5: Document the process for converting observed issues into test cases; add the first 10 critical tests to CI/CD You ship a prompt change that hurts quality by 8%. It goes unnoticed for three days because you don't have quality metrics. By the time customers complain, you've served bad results to 50,000 people. You switch to a cheaper model and assume quality degrades 2%. Actually, it degrades 12% on a specific category of queries, but you won't know for a month when business metrics show lower conversion. You increase temperature from 0.7 to 0.8 to make outputs more creative. What you don't see: factual accuracy drops 11%, bad actors figure out how to trick the system, and security issues emerge. Level 1: You have logs. You can grep them when something breaks. Level 2: You have a dashboard. You can see what's happening in real time. Level 3: You have alerts. Problems wake you up, don't surprise you later. Level 4: You convert observations into tests. Regressions are caught in CI/CD, not production. Level 5: You have automated response. Detected issues trigger automated rollbacks or recovery. Tracing: Langsmith (free tier handles most projects) Logging: JSON files to S3, queryable via Athena or similar Metrics: Hourly aggregations written to a Postgres table Dashboard: Metabase or Grafana pointed at your metrics table Alerting: Threshold checks that post to Slack Testing: A test file that imports the last 30 days of edge cases and runs them against your current model