Token Economics: Why Your AI Feature Costs 10x What You Think

Your new AI feature launched three weeks ago. Engineering said it would cost $1,200 a month. You're on track to spend $12,000. No one can explain why. The obvious culprit is traffic, but you checked, usage is exactly as predicted. The real problem is something smaller and worse: the prompt is inefficient, the system prompts have bloat, the retrieval is pulling too much context, and the temperature settings are generating too many retries.

Token economics is the difference between AI being a reasonable cost center and it bankrupting your company. You need to treat token usage like you treat database queries: measure it, profile it, optimize it, and alert when it drifts. This is primarily a testing and monitoring problem, not an engineering problem.

The Hidden Cost Multipliers

Token usage compounds in ways that aren't obvious. A prompt that's 10% longer doesn't cost 10% more, it costs 10% more for every single query, forever. Run 1 million queries a month and that 10% becomes massive.

Common cost multipliers:

Verbose system prompts: Your system prompt is 2000 tokens. It gets repeated in every call. Cut it to 500 tokens and you save 1.5 million tokens a month if you have 1000 queries/day. That's $15/month in savings on a single prompt edit.

Unnecessary context in RAG: You retrieve 10 documents when 3 would suffice. Each extra document is tokens. Cut retrieval from 10 to 5 documents and watch token usage drop 40%.

Temperature and retry patterns: Set temperature too high and the model generates nonsense. You retry. Every retry costs tokens. Lower temperature, fewer retries, lower total cost.

API choice: Using GPT-4 for classification? Switch to GPT-3.5 and save 70% on that workload. Different models for different tasks: expensive model for reasoning, cheap model for formatting.

Caching misses: If you're not caching context between similar queries, you're reprocessing the same tokens. Implement prompt caching and watch costs drop 40-60%.

One bad prompt decision can waste more money than entire engineering projects. This needs to be tested.

Measuring Token Usage per Query

Start by understanding exactly what costs money. Instrument every LLM call to track tokens in and out.

interface TokenMetrics {
 inputTokens: number;
 outputTokens: number;
 totalTokens: number;
 costUsd: number;
 timestamp: Date;
 prompt: string; // version identifier
 model: string;
 feature: string; // which feature/prompt
 isRetry: boolean;
}

// Token tracking wrapper
class TokenTracker {
 private metrics: TokenMetrics[] = [];

 async trackCall(
 feature: string,
 prompt: string,
 callFn: () => Promise<{ usage: OpenAI.Usage; content: string }>
 ): Promise {
 const startTime = Date.now();
 const response = await callFn();

 const costUsd = this.calculateCost(
 response.usage.prompt_tokens,
 response.usage.completion_tokens
 );

 this.metrics.push({
 inputTokens: response.usage.prompt_tokens,
 outputTokens: response.usage.completion_tokens,
 totalTokens: response.usage.total_tokens,
 costUsd,
 timestamp: new Date(),
 prompt: hashPrompt(prompt),
 model: 'gpt-4',
 feature,
 isRetry: false
 });

 return response.content;
 }

 private calculateCost(inputTokens: number, outputTokens: number): number {
 // GPT-4 pricing: $0.03 per 1K input, $0.06 per 1K output
 const inputCost = (inputTokens / 1000) * 0.03;
 const outputCost = (outputTokens / 1000) * 0.06;
 return inputCost + outputCost;
 }

 getMetrics(): TokenMetrics[] {
 return this.metrics;
 }

 summarizeByFeature() {
 const byFeature: { [key: string]: any } = {};

 for (const metric of this.metrics) {
 byFeature[metric.feature] ??= {
 queries: 0,
 totalTokens: 0,
 totalCost: 0,
 avgTokensPerQuery: 0,
 avgCostPerQuery: 0
 };

 byFeature[metric.feature].queries++;
 byFeature[metric.feature].totalTokens += metric.totalTokens;
 byFeature[metric.feature].totalCost += metric.costUsd;
 }

 // Calculate averages
 for (const feature in byFeature) {
 const stats = byFeature[feature];
 stats.avgTokensPerQuery = stats.totalTokens / stats.queries;
 stats.avgCostPerQuery = stats.totalCost / stats.queries;
 }

 return byFeature;
 }
}

// Usage
const tracker = new TokenTracker();

const result = await tracker.trackCall(
 'chat-summarization',
 systemPrompt,
 async () => {
 return await openai.createChatCompletion({
 model: 'gpt-4',
 messages: [{ role: 'user', content: userPrompt }]
 });
 }
);

// After running queries
console.table(tracker.summarizeByFeature());

This tracking gives you visibility into cost by feature. Now you can identify which features are expensive and optimize them systematically.

Cost per Query Regression Testing

Treat cost per query like a performance metric. When someone updates a prompt, test whether it increases or decreases token usage.

interface CostBudget {
 feature: string;
 maxCostPerQuery: number; // in cents
 maxTokensPerQuery: number;
 warningThreshold: number; // 80% of max
}

// Define cost budgets for each feature
const costBudgets: CostBudget[] = [
 {
 feature: 'quick-summary',
 maxCostPerQuery: 5, // 5 cents
 maxTokensPerQuery: 500,
 warningThreshold: 0.8
 },
 {
 feature: 'detailed-analysis',
 maxCostPerQuery: 25, // 25 cents
 maxTokensPerQuery: 2500,
 warningThreshold: 0.8
 },
 {
 feature: 'code-review',
 maxCostPerQuery: 15, // 15 cents
 maxTokensPerQuery: 1500,
 warningThreshold: 0.8
 }
];

// Test cost before deploy
const validateCostBudget = async (
 feature: string,
 promptVersion: string,
 testQueries: string[] = [],
 iterations = 10
) => {
 const budget = costBudgets.find(b => b.feature === feature);
 if (!budget) throw new Error(`No cost budget defined for ${feature}`);

 const costs: number[] = [];
 const tokens: number[] = [];

 for (let i = 0; i < iterations; i++) {
 const query = testQueries[i % testQueries.length];

 const response = await tracker.trackCall(
 feature,
 promptVersion,
 async () => {
 return await openai.createChatCompletion({
 model: 'gpt-4',
 messages: [{ role: 'user', content: query }]
 });
 }
 );

 const metrics = tracker.getMetrics().slice(-1)[0];
 costs.push(metrics.costUsd * 100); // convert to cents
 tokens.push(metrics.totalTokens);
 }

 const avgCost = costs.reduce((a, b) => a + b) / costs.length;
 const avgTokens = tokens.reduce((a, b) => a + b) / tokens.length;
 const maxCost = Math.max(...costs);

 const costStatus = avgCost > budget.maxCostPerQuery ? 'FAIL' : 'PASS';
 const tokenStatus = avgTokens > budget.maxTokensPerQuery ? 'FAIL' : 'PASS';
 const warningCost = avgCost > (budget.maxCostPerQuery * budget.warningThreshold) ? 'WARNING' : 'OK';

 console.log(`Cost Budget Test: ${feature}`);
 console.log(`Avg cost per query: ${avgCost.toFixed(2)}¢ (budget: ${budget.maxCostPerQuery}¢) [${costStatus}]`);
 console.log(`Avg tokens per query: ${avgTokens.toFixed(0)} (budget: ${budget.maxTokensPerQuery}) [${tokenStatus}]`);
 console.log(`Max cost: ${maxCost.toFixed(2)}¢ [${warningCost}]`);

 if (costStatus === 'FAIL' || tokenStatus === 'FAIL') {
 throw new Error(`Cost regression: ${feature} exceeded budget`);
 }

 return { avgCost, avgTokens, status: costStatus };
};

// Run before merge
await validateCostBudget(
 'quick-summary',
 newPrompt,
 ['Summarize this document', 'What are the key points?', 'Brief summary please'],
 15
);

Cost regression tests catch expensive mistakes before they go to production. Someone rewrites a prompt in a verbose way? Your test catches it and blocks the PR.

Detecting Cost Anomalies in Production

Even with good controls, costs can drift. Set up anomaly detection to catch it before your bill shocks you.

interface CostAnomaly {
 feature: string;
 timestamp: Date;
 expectedCost: number;
 actualCost: number;
 percentageIncrease: number;
 possibleCause?: string;
}

class CostAnomalyDetector {
 private baseline: Map = new Map();

 // Build baseline from historical data
 buildBaseline(historicalMetrics: TokenMetrics[]) {
 const byFeature: { [key: string]: number[] } = {};

 for (const metric of historicalMetrics) {
 byFeature[metric.feature] ??= [];
 byFeature[metric.feature].push(metric.costUsd);
 }

 // Calculate mean and standard deviation for each feature
 for (const [feature, costs] of Object.entries(byFeature)) {
 const mean = costs.reduce((a, b) => a + b) / costs.length;
 const variance = costs.reduce((sq, x) => sq + Math.pow(x - mean, 2)) / costs.length;
 const stdDev = Math.sqrt(variance);

 this.baseline.set(feature, { mean, stdDev });
 }
 }

 // Detect anomalies in new metrics
 detectAnomalies(newMetrics: TokenMetrics[]): CostAnomaly[] {
 const anomalies: CostAnomaly[] = [];

 for (const metric of newMetrics) {
 const baseline = this.baseline.get(metric.feature);
 if (!baseline) continue;

 // Flag if cost is more than 2 standard deviations from mean
 const zScore = (metric.costUsd - baseline.mean) / baseline.stdDev;

 if (zScore > 2) {
 anomalies.push({
 feature: metric.feature,
 timestamp: metric.timestamp,
 expectedCost: baseline.mean,
 actualCost: metric.costUsd,
 percentageIncrease: ((metric.costUsd - baseline.mean) / baseline.mean * 100)
 });
 }
 }

 return anomalies;
 }
}

// Monitor in production
const detector = new CostAnomalyDetector();
detector.buildBaseline(lastMonthsMetrics);

// Check every hour
setInterval(async () => {
 const recentMetrics = await getRecentMetrics(lastHour);
 const anomalies = detector.detectAnomalies(recentMetrics);

 for (const anomaly of anomalies) {
 console.warn(
 `COST ANOMALY: ${anomaly.feature} ` +
 `${anomaly.percentageIncrease.toFixed(0)}% higher than baseline`
 );

 // Alert operations
 await sendSlackAlert({
 feature: anomaly.feature,
 increase: anomaly.percentageIncrease,
 actual: anomaly.actualCost,
 expected: anomaly.expectedCost
 });
 }
}, 60 * 60 * 1000); // check hourly

Prompt Optimization as Testing

When you optimize a prompt, you want to measure whether it's actually better. Better usually means "cheaper while maintaining quality." Test this systematically.

interface PromptComparison {
 originalPrompt: string;
 optimizedPrompt: string;
 originalMetrics: { avgCost: number; avgQuality: number; tokenCount: number };
 optimizedMetrics: { avgCost: number; avgQuality: number; tokenCount: number };
 improvement: {
 costReduction: number;
 qualityChange: number;
 recommendation: 'deploy' | 'investigate' | 'reject';
 };
}

// A/B test prompts
const comparePrompts = async (
 feature: string,
 originalPrompt: string,
 optimizedPrompt: string,
 testQueries: string[],
 iterations = 20
): Promise => {
 const originalCosts: number[] = [];
 const originalQuality: number[] = [];
 const originalTokens: number[] = [];

 const optimizedCosts: number[] = [];
 const optimizedQuality: number[] = [];
 const optimizedTokens: number[] = [];

 // Test original
 for (let i = 0; i < iterations; i++) {
 const query = testQueries[i % testQueries.length];
 const response = await openai.createChatCompletion({
 model: 'gpt-4',
 messages: [
 { role: 'system', content: originalPrompt },
 { role: 'user', content: query }
 ]
 });

 originalCosts.push((response.usage.total_tokens / 1000) * 0.06);
 originalTokens.push(response.usage.total_tokens);

 // Score quality (using evaluator)
 const quality = await evaluator.score(response.choices[0].message.content);
 originalQuality.push(quality);
 }

 // Test optimized
 for (let i = 0; i < iterations; i++) {
 const query = testQueries[i % testQueries.length];
 const response = await openai.createChatCompletion({
 model: 'gpt-4',
 messages: [
 { role: 'system', content: optimizedPrompt },
 { role: 'user', content: query }
 ]
 });

 optimizedCosts.push((response.usage.total_tokens / 1000) * 0.06);
 optimizedTokens.push(response.usage.total_tokens);

 const quality = await evaluator.score(response.choices[0].message.content);
 optimizedQuality.push(quality);
 }

 // Calculate averages
 const avgOriginalCost = originalCosts.reduce((a, b) => a + b) / originalCosts.length;
 const avgOriginalQuality = originalQuality.reduce((a, b) => a + b) / originalQuality.length;

 const avgOptimizedCost = optimizedCosts.reduce((a, b) => a + b) / optimizedCosts.length;
 const avgOptimizedQuality = optimizedQuality.reduce((a, b) => a + b) / optimizedQuality.length;

 const costReduction = ((avgOriginalCost - avgOptimizedCost) / avgOriginalCost * 100);
 const qualityChange = avgOptimizedQuality - avgOriginalQuality;

 // Decide whether to deploy
 let recommendation: 'deploy' | 'investigate' | 'reject' = 'reject';
 if (costReduction > 10 && qualityChange >= -0.05) {
 recommendation = 'deploy'; // Cost improved, quality didn't hurt
 } else if (costReduction > 0 && qualityChange > 0) {
 recommendation = 'deploy'; // Both improved
 } else if (costReduction < 0 && qualityChange > 0.1) {
 recommendation = 'investigate'; // Quality improvement might justify cost
 }

 return {
 originalPrompt,
 optimizedPrompt,
 originalMetrics: {
 avgCost: avgOriginalCost,
 avgQuality: avgOriginalQuality,
 tokenCount: originalTokens[0]
 },
 optimizedMetrics: {
 avgCost: avgOptimizedCost,
 avgQuality: avgOptimizedQuality,
 tokenCount: optimizedTokens[0]
 },
 improvement: {
 costReduction,
 qualityChange,
 recommendation
 }
 };
};

// Example: Test a shorter prompt
const comparison = await comparePrompts(
 'code-review',
 longSystemPrompt,
 shortSystemPrompt,
 codeExamples,
 20
);

console.log(`Cost reduction: ${comparison.improvement.costReduction.toFixed(1)}%`);
console.log(`Quality change: ${comparison.improvement.qualityChange.toFixed(2)}`);
console.log(`Recommendation: ${comparison.improvement.recommendation}`);