OpenAI Just Deprecated Your Model. Here's Your Testing Playbook.

It's Tuesday morning. You check your email and OpenAI announced GPT-3.5 Turbo is being sunset in 3 months.

You've built your entire product on that model.

This scenario isn't hypothetical. OpenAI deprecates models regularly. Anthropic will too. Azure rotates models. Every major provider has a sunset timeline.

The teams that handle this smoothly planned for it. The ones that panic didn't.

Why Model Deprecations Break Things

Model deprecations aren't just "swap old model for new model." They break in subtle ways:

• Behavioral changes: GPT-4 reasons differently than GPT-3.5. Outputs might be wordier, have different biases, or interpret ambiguous prompts differently
• Performance cliffs: The new model might be 30% faster but hallucinates more in specific domains
• API changes: Token costs change. Response formats shift. New models have different context windows
• Integration breaks: Tools, function calling, structured outputs might work differently
• Silent failures: No errors, but results silently degrade. A chatbot still responds, but with lower quality

The worst part: you don't know these break until production lights up with user complaints.

The teams that survive model deprecations are the ones that test exhaustively before cutoff.

The Timeline: When to Start

Model deprecation announcements usually give 3-6 months' notice. Here's when to do what:

Month 1: Baseline & Planning (Right Now)

Week 1: The moment you see a deprecation notice, do this:

• Document current model behavior extensively
• Run your full eval suite against current model, capture all metrics
• Identify which use cases depend on that model
• Get the replacement model (if available)

// Establish baseline before any changes
async function captureModelBaseline(
 deprecatedModel: string,
 replacementModel: string
): Promise {
 const testSuite = await loadComprehensiveTestSuite();
 const timestamp = new Date();

 const deprecatedResults = await evaluateModel(
 deprecatedModel,
 testSuite
 );

 const replacementResults = await evaluateModel(
 replacementModel,
 testSuite
 );

 const baseline: ModelBaseline = {
 timestamp,
 deprecatedModel,
 replacementModel,
 metrics: {
 deprecated: deprecatedResults,
 replacement: replacementResults,
 comparison: compareResults(deprecatedResults, replacementResults),
 },
 knownIssues: [],
 regressions: [],
 improvements: [],
 };

 await persistBaseline(baseline);
 return baseline;
}

interface ModelBaseline {
 timestamp: Date;
 deprecatedModel: string;
 replacementModel: string;
 metrics: {
 deprecated: EvalResults;
 replacement: EvalResults;
 comparison: ComparisonMetrics;
 };
 knownIssues: Issue[];
 regressions: Regression[];
 improvements: Improvement[];
}

Week 2-4: Planning sprint:

• Identify all code paths using the old model
• Plan fallback strategies (if replacement fails, what's plan B?)
• Estimate effort: migration code, testing, rollout
• Schedule: when can teams dedicate focus?
• Communication: set up weekly syncs between eng, product, and QA

Month 2: Evaluation & Testing (Deep Work)

Run Practical evaluation of the replacement model:

// Evaluate replacement model across all use cases
async function evaluateReplacementModel(
 replacementModel: string,
 deprecatedBaseline: ModelBaseline
): Promise {
 const testCategories = [
 "accuracy",
 "latency",
 "tokens",
 "safety",
 "bias",
 "edge_cases",
 ];

 const report: EvaluationReport = {
 model: replacementModel,
 categories: {},
 summary: {
 regressions: [],
 improvements: [],
 neutral: [],
 },
 };

 for (const category of testCategories) {
 const results = await evaluateCategory(
 replacementModel,
 category
 );

 const baseline = deprecatedBaseline.metrics.deprecated[category];
 const comparison = compareResults(results, baseline);

 report.categories[category] = {
 current: results,
 baseline,
 delta: comparison.delta,
 percentChange: comparison.percentChange,
 status: comparison.status, // "improvement" | "regression" | "neutral"
 };

 if (comparison.status === "regression") {
 report.summary.regressions.push({
 category,
 delta: comparison.delta,
 severity: calculateSeverity(comparison),
 });
 }
 }

 return report;
}

async function categorizeFindings(
 report: EvaluationReport
): Promise<{
 blockers: Finding[];
 warnings: Finding[];
 nonIssues: Finding[];
}> {
 const blockers: Finding[] = [];
 const warnings: Finding[] = [];
 const nonIssues: Finding[] = [];

 for (const regression of report.summary.regressions) {
 if (regression.severity === "critical") {
 // This will break production
 blockers.push(regression);
 } else if (regression.severity === "high") {
 // Needs investigation
 warnings.push(regression);
 } else {
 // Minor, acceptable
 nonIssues.push(regression);
 }
 }

 return { blockers, warnings, nonIssues };
}

Test compatibility of critical integrations:

• Function calling: does the new model use tools correctly?
• Structured outputs: can it generate valid JSON/YAML?
• Context windows: can it handle your longest inputs?
• Token counts: do your assumptions about token costs still hold?

// Test critical integrations
async function testIntegrations(
 newModel: string
): Promise {
 const tests = [
 testFunctionCalling(newModel),
 testStructuredOutput(newModel),
 testContextWindows(newModel),
 testTokenCounting(newModel),
 testParsing(newModel),
 ];

 const results = await Promise.all(tests);

 return {
 functionCalling: results[0],
 structuredOutput: results[1],
 contextWindows: results[2],
 tokenCounting: results[3],
 parsing: results[4],
 allPass: results.every(r => r.pass),
 };
}

async function testFunctionCalling(model: string): Promise {
 const testCases = [
 {
 name: "basic_function_call",
 prompt: "What's the weather in San Francisco?",
 expectedTool: "get_weather",
 },
 {
 name: "multiple_tools",
 prompt: "Get weather and set a reminder",
 expectedTools: ["get_weather", "set_reminder"],
 },
 {
 name: "nested_calls",
 prompt: "Get weather, convert to F, and send SMS",
 expectedChain: ["get_weather", "convert_temperature", "send_sms"],
 },
 ];

 let passed = 0;

 for (const testCase of testCases) {
 const response = await callModel(model, {
 prompt: testCase.prompt,
 tools: getAvailableTools(),
 });

 const toolsUsed = extractToolCalls(response);
 const matches = validateToolUsage(toolsUsed, testCase);

 if (matches) {
 passed++;
 } else {
 console.error(
 `Function call test failed: ${testCase.name}`,
 { expected: testCase, actual: toolsUsed }
 );
 }
 }

 return {
 pass: passed === testCases.length,
 score: passed / testCases.length,
 };
}

Test cost impact:

• Run 10,000 representative requests through new model
• Compare token usage and cost
• If costs increase significantly, factor into decision

Month 3: Rollout & Cutoff (Execution)

Week 1: Shadow mode (if replacement not yet required):

Run the new model in parallel with the old one. Collect metrics but serve users the old model. Validate in real-world conditions before going live.

Week 2: Canary rollout:

// Staged rollout plan
const ROLLOUT_STAGES = [
 {
 name: "internal",
 trafficPercentage: 0,
 userFilter: (u) => u.isInternal,
 duration: "1 week",
 },
 {
 name: "beta",
 trafficPercentage: 5,
 userFilter: (u) => u.betaOptIn,
 duration: "1 week",
 },
 {
 name: "canary",
 trafficPercentage: 10,
 userFilter: (u) => u.id % 100 < 10, // Hash-based
 duration: "3 days",
 },
 {
 name: "half",
 trafficPercentage: 50,
 userFilter: (u) => u.id % 100 < 50,
 duration: "3 days",
 },
 {
 name: "full",
 trafficPercentage: 100,
 userFilter: () => true,
 duration: "continuous",
 },
];

async function routeRequest(
 request: Request,
 user: User
): Promise {
 const currentStage = await getCurrentRolloutStage();
 const shouldUseNewModel = ROLLOUT_STAGES[currentStage].userFilter(
 user
 );

 const model = shouldUseNewModel ? replacementModel : deprecatedModel;

 return await callModel(model, request);
}

// At each stage, validate metrics before advancing
async function validateBeforeAdvance(stage: number): Promise {
 const metrics = await getMetricsForStage(stage);

 return (
 metrics.errorRate < 0.02 &&
 metrics.qualityScore > baselineQuality * 0.95 &&
 metrics.latency < baselineLatency * 1.2 &&
 !metrics.hasUserComplaints
 );
}

Week 3: Full migration:

Once canary stages all pass, go full traffic. Monitor closely for first 24-48 hours.

After cutoff deadline:

• Remove deprecated model from codebase
• Clean up fallback/compatibility code
• Document lessons learned
• Update runbooks and oncall procedures

Fallback Strategies: Hope for the Best, Plan for the Worst

What if the replacement model starts behaving badly in production? You need instant fallbacks:

// Intelligent fallback strategy
async function callModelWithFallback(
 primaryModel: string,
 fallbackModels: string[],
 request: Request,
 options: FallbackOptions = {}
): Promise<{
 response: Response;
 model: string;
 failover: boolean;
}> {
 const { maxRetries = 2, timeout = 30000 } = options;

 let lastError: Error | null = null;
 const modelsToTry = [primaryModel, ...fallbackModels];

 for (let i = 0; i < modelsToTry.length; i++) {
 const model = modelsToTry[i];

 try {
 const response = await Promise.race([
 callModel(model, request),
 new Promise((_, reject) =>
 setTimeout(
 () => reject(new Error("Timeout")),
 timeout
 )
 ),
 ]);

 // Validate response quality
 if (validateResponse(response)) {
 return {
 response,
 model,
 failover: i > 0,
 };
 }

 lastError = new Error("Response validation failed");
 } catch (error) {
 lastError = error as Error;
 console.warn(
 `Model ${model} failed (attempt ${i + 1}):`,
 lastError.message
 );

 // If this is a quota error, don't try fallbacks
 if (
 lastError.message.includes("quota") ||
 lastError.message.includes("rate_limit")
 ) {
 throw lastError;
 }
 }
 }

 // All models failed
 throw new Error(
 `All models exhausted. Last error: ${lastError?.message}`
 );
}

// Example: GPT-3.5 Turbo -> GPT-4 -> GPT-3.5 (cheaper fallback)
async function migrateFromGPT35(): Promise {
 // Deprecated model
 const primaryModel = "gpt-4-turbo";

 // Fallbacks in order of preference
 const fallbackModels = [
 "gpt-4", // Premium fallback
 "gpt-3.5-turbo", // What we're migrating away from (won't work after cutoff, but good during transition)
 "claude-3-5-sonnet", // Alternative provider
 ];

 // Now use this everywhere
 const response = await callModelWithFallback(
 primaryModel,
 fallbackModels,
 request
 );
}

The Migration Checklist

When cutoff day arrives, use this checklist:

One week before:

• All code updated to use new model
• All tests passing with new model
• Canary rollout complete, metrics green
• Fallback logic deployed and tested
• On-call team briefed
• Rollback plan written and rehearsed

Day before:

• Update runbooks with new model name
• Confirm cutoff time in all timezones
• Schedule on-call engineer to monitor
• Set up alerts for errors and quality drop
• Notify support team of potential issues

Cutoff day:

• Monitor error rates, latency, quality scores every minute for first hour
• Check user-facing metrics (churn, support tickets) every 10 minutes
• Be ready to rollback to fallback model instantly
• After 24 hours of stability, stand down heightened monitoring

After cutoff:

• Remove deprecated model from code (don't leave fallbacks forever)
• Update documentation
• Run blameless postmortem (even if migration went well)
• Extract playbook improvements for next time

Real Examples: GPT-3.5 to GPT-4 Migration

When OpenAI deprecated GPT-3.5 Turbo in January 2024, companies that had this playbook succeeded. Those that didn't faced outages.

The successful migrations:

• Established baselines 2 months before cutoff
• Found GPT-4 was 3x slower (needed to optimize prompts to reduce tokens)
• Discovered function calling worked differently (required prompt adjustments)
• Set up canary rollouts 2 weeks before cutoff (caught issues early)
• Executed full migration 3 days before cutoff (extra buffer for problems)

The failed migrations:

• Waited until cutoff week to evaluate GPT-4 (no time to fix issues)
• Assumed GPT-4 was just "faster GPT-3.5" (it's not)
• Deployed without canary (went straight to 100%)
• No fallback strategy (outage when something went wrong)