AI Feature Flags: Ship AI Features Without Shipping Bugs

TL;DR

Traditional feature flags work fine for code. AI feature flags need quality gates Percentage rollouts alone don't catch model failures, you need quality gates that auto-rollback Test the feature flag infrastructure itself, not just the feature behind the flag Combine feature flags with experiment tracking and model evaluation LaunchDarkly/Split + alt.qa = safe AI deployments at scale

Why Traditional Feature Flags Fail for AI

Problem 1: Success Metrics Are Unreliable

With traditional features, success is easy to measure. Did the new login flow increase conversion? Did the redesigned checkout reduce abandonment? You can measure this in hours.

With AI features, success is ambiguous. Did the new recommendation model increase engagement? By how much? Is the increase statistically significant? Did it improve for all user segments or just some?

"A recommendation model that increases engagement 2% overall but decreases it 15% for inactive users isn't a success. Traditional feature flags miss this entirely."

You need multi-dimensional success metrics, not just a single conversion rate.

Problem 2: Rollback Doesn't Actually Help

With code, rolling back is reversible. You revert the code, the old behavior returns instantly. With AI models, rolling back doesn't restore lost user trust. A user had three bad recommendations from your new model. Rolling back the model doesn't unsee those recommendations.

The damage is done. This is why preventing bad model deployments is 100x more important than quick rollbacks.

Problem 3: Gradual Rollouts Hide Systemic Issues

Starting with 5% of traffic seems safe. But your 5% might be biased. Maybe 5% is your most engaged users. Maybe 5% is US-based users. You launch the model to 5% of your most engaged users, they love it, you roll out to 100% of your less engaged users, and it fails spectacularly.

You need stratified rollouts, not just percentage rollouts.

The Right Way to Feature Flag AI Features

Layer 1: Quality Gates Before Rollout

Before a feature flag is even enabled for 1% of users, the model has to pass Practical quality gates:

def evaluate_model_before_rollout(candidate_model, baseline_model):
 """
 Quality gates that must pass before feature flag is enabled
 """

 # Load holdout test set
 X_test, y_test = load_holdout_test_set()

 # Gate 1: Overall accuracy
 candidate_accuracy = accuracy_score(y_test, candidate_model.predict(X_test))
 baseline_accuracy = accuracy_score(y_test, baseline_model.predict(X_test))

 assert candidate_accuracy >= baseline_accuracy, \
 f"Candidate accuracy {candidate_accuracy} < baseline {baseline_accuracy}"

 # Gate 2: Subgroup performance
 for demographic in ["region", "user_tenure", "user_engagement"]:
 for subgroup in df_test[demographic].unique():
 mask = df_test[demographic] == subgroup

 subgroup_candidate = accuracy_score(
 y_test[mask],
 candidate_model.predict(X_test[mask])
 )
 subgroup_baseline = accuracy_score(
 y_test[mask],
 baseline_model.predict(X_test[mask])
 )

 # Allow up to 2% degradation per subgroup
 assert subgroup_candidate >= subgroup_baseline - 0.02, \
 f"Subgroup {subgroup} degraded: {subgroup_candidate} vs {subgroup_baseline}"

 # Gate 3: Fairness and bias
 fairness_metrics = measure_fairness(candidate_model, X_test)
 assert fairness_metrics["demographic_parity"] > 0.95
 assert fairness_metrics["equal_opportunity"] > 0.92

 # Gate 4: Confidence calibration
 predicted_probs = candidate_model.predict_proba(X_test)
 calibration = measure_calibration(y_test, predicted_probs)
 assert calibration["expected_calibration_error"] < 0.05

 print("All quality gates passed. Safe to enable feature flag.")
 return True

These gates are strict. Only models that pass all of them are eligible for rollout.

Layer 2: Canary Deployment (0.1% → 5%)

Once a model passes quality gates, enable the feature flag for tiny percentage:

# Canary deployment: enable for 0.1% of users (or fixed cohort)
feature_flag_config = {
 "feature_name": "new_recommendation_model_v3",
 "enabled": True,
 "rollout_percentage": 0.1,
 "rollout_strategy": "gradual", # Not random, bucketed cohorts
 "cohort": "canary_users_2026_04_04",
 "quality_gates": [
 {
 "metric": "user_engagement",
 "baseline": 0.87,
 "minimum": 0.85, # Allow 2% degradation max
 },
 {
 "metric": "recommendation_diversity",
 "baseline": 0.72,
 "minimum": 0.70,
 },
 {
 "metric": "click_through_rate",
 "baseline": 0.042,
 "minimum": 0.040,
 }
 ],
 "monitoring_interval": "1hour",
 "auto_rollback_threshold": {
 "metric": "user_engagement",
 "degradation_percent": 5, # Rollback if engagement drops >5%
 }
}

Notice the quality gates are built into the feature flag configuration. This connects the flag to automated monitoring and rollback.

Layer 3: Stratified Rollout (5% → 50%)

If canary succeeds, roll out to specific user segments, not just a percentage:

# Stratified rollout
stratified_rollout = {
 "feature_name": "new_recommendation_model_v3",
 "enabled": True,
 "rollout_strategy": "stratified",
 "strata": [
 {
 "name": "us_high_engagement",
 "filter": {"region": "US", "engagement_score": {"min": 0.8}},
 "rollout_percentage": 50,
 "quality_gates": [
 {
 "metric": "user_engagement",
 "minimum": 0.86, # Higher bar for high-engagement users
 }
 ]
 },
 {
 "name": "intl_low_engagement",
 "filter": {"region": {"not": "US"}, "engagement_score": {"max": 0.3}},
 "rollout_percentage": 10, # Slower rollout for risky segment
 "quality_gates": [
 {
 "metric": "user_engagement",
 "minimum": 0.25, # Lower bar for low-engagement users
 }
 ]
 },
 {
 "name": "new_users",
 "filter": {"account_age_days": {"max": 30}},
 "rollout_percentage": 25,
 "quality_gates": [
 {
 "metric": "onboarding_completion",
 "minimum": 0.70,
 }
 ]
 }
 ],
 "monitoring_interval": "1hour",
 "auto_rollback_threshold": {
 "metric": "user_engagement",
 "degradation_percent": 3, # Tighter threshold
 }
}

This is the right way to roll out. Different segments have different quality gates because they have different tolerance for degradation.

Layer 4: Automated Monitoring and Rollback

While the feature flag is enabled, continuously monitor quality metrics:

def monitor_feature_flag_quality(feature_name, duration_hours=24):
 """
 Continuous monitoring with automatic rollback
 """

 start_time = datetime.utcnow()
 rollout_config = get_feature_flag_config(feature_name)

 while True:
 # Every hour, measure quality
 current_metrics = measure_production_metrics(
 feature_flag_users=True,
 metric_names=["user_engagement", "ctr", "recommendation_diversity"]
 )

 baseline_metrics = get_baseline_metrics()

 # Check each quality gate
 for gate in rollout_config["quality_gates"]:
 metric_name = gate["metric"]
 current_value = current_metrics[metric_name]
 baseline_value = baseline_metrics[metric_name]
 minimum_value = gate["minimum"]

 degradation_percent = (1 - current_value / baseline_value) * 100

 if current_value < minimum_value:
 # CRITICAL: Metric below minimum
 auto_rollback(
 feature_name=feature_name,
 reason=f"{metric_name} below minimum: {current_value} < {minimum_value}",
 severity="critical"
 )
 return

 if degradation_percent > rollout_config["auto_rollback_threshold"]["degradation_percent"]:
 # WARNING: Metric degraded significantly
 alert_on_call(
 feature_name=feature_name,
 message=f"{metric_name} degraded {degradation_percent:.1f}%"
 )

 # Auto-rollback if degradation persists for 2 hours
 if time_since_alert > 2:
 auto_rollback(
 feature_name=feature_name,
 reason=f"{metric_name} degraded {degradation_percent:.1f}% for >2 hours",
 severity="warning"
 )
 return

 # Check if we should proceed to next rollout stage
 if is_time_for_next_stage(start_time):
 current_pct = rollout_config["rollout_percentage"]
 next_pct = get_next_rollout_percentage(current_pct)

 if all_metrics_healthy(current_metrics, baseline_metrics):
 update_rollout_percentage(feature_name, next_pct)
 log(f"Rolled out to {next_pct}%")

 sleep(3600) # Check every hour

This is continuous, automated monitoring. Human intervention is for anomalies, not routine rollouts.

Combining Feature Flags with Experiment Tracking

Feature flags and experiment tracking need to be tightly integrated:

from launchdarkly_server_sdk import Context
import mlflow

def serve_recommendation_with_experimentation(user_id):
 """
 Feature flag serves both model selection and experiment tracking
 """

 # Check feature flag
 context = Context(user_id)
 flag_value = ld_client.variation("new_recommendation_model_v3", context)

 if flag_value:
 # User is in treatment group
 model = load_model("recommendation_model_v3")
 variant = "treatment"
 else:
 # User is in control group
 model = load_model("recommendation_model_v2")
 variant = "control"

 # Get recommendations
 recommendations = model.predict(user_id)

 # Log to experiment tracking for later analysis
 mlflow.log_param("variant", variant)
 mlflow.log_param("model_version", model.version)
 mlflow.log_param("user_id", user_id)

 # Also log user interaction for post-experiment analysis
 log_event(
 event_type="recommendation_served",
 user_id=user_id,
 variant=variant,
 model_version=model.version,
 timestamp=datetime.utcnow()
 )

 return recommendations

When you're done with the feature flag rollout, you have clean experiment data showing how treatment and control performed.

Testing the Feature Flag Infrastructure Itself

Most teams test the code behind feature flags but not the flag infrastructure. This is a huge gap.

Test 1: Flag State Consistency

When you turn on a feature flag, users see the flag as enabled within seconds. Test this:

def test_feature_flag_consistency():
 """
 Verify feature flag state is consistent across infrastructure
 """

 # Set flag to enabled
 launchdarkly.set_flag("test_flag", True)
 time.sleep(1)

 # Check flag state from multiple clients
 clients = [
 ld_client_1, # Different datacenter
 ld_client_2, # Different language client
 ld_client_3, # Different service
 ]

 for client in clients:
 flag_state = client.variation("test_flag", context)
 assert flag_state is True, f"Flag inconsistent in {client}"

 # Set flag to disabled
 launchdarkly.set_flag("test_flag", False)
 time.sleep(1)

 # Verify all clients see disabled
 for client in clients:
 flag_state = client.variation("test_flag", context)
 assert flag_state is False, f"Flag inconsistent in {client}"

Test 2: Rollout Percentage Accuracy

When you set a flag to 10% rollout, exactly 10% of users should see it (within 1% margin):

def test_rollout_percentage_accuracy():
 """
 Verify rollout percentages are actually accurate
 """

 # Set flag to 25% rollout
 launchdarkly.set_rollout_percentage("new_model_flag", 25)

 # Check for large sample of users
 sample_size = 10000
 sees_flag = 0

 for user_id in range(sample_size):
 context = Context(user_id)
 flag_state = ld_client.variation("new_model_flag", context)
 if flag_state:
 sees_flag += 1

 actual_percentage = sees_flag / sample_size * 100

 # Should be 25% +/- 2%
 assert 23 <= actual_percentage <= 27, \
 f"Rollout {actual_percentage}% != expected 25%"

Test 3: Targetability Accuracy

When you target specific users or segments, verify they're actually getting the flag:

def test_feature_flag_targeting():
 """
 Verify feature flag targeting rules work correctly
 """

 # Target only US users with engagement > 0.7
 launchdarkly.set_targeting(
 "new_model_flag",
 rules=[
 {
 "attribute": "country",
 "operator": "equals",
 "value": "US"
 },
 {
 "attribute": "engagement",
 "operator": "greater_than",
 "value": 0.7
 }
 ]
 )

 # Test US user with high engagement
 us_user = Context(
 user_id="user_123",
 custom={"country": "US", "engagement": 0.8}
 )
 assert ld_client.variation("new_model_flag", us_user) is True

 # Test US user with low engagement
 us_low_engagement = Context(
 user_id="user_456",
 custom={"country": "US", "engagement": 0.5}
 )
 assert ld_client.variation("new_model_flag", us_low_engagement) is False

 # Test non-US user with high engagement
 intl_user = Context(
 user_id="user_789",
 custom={"country": "UK", "engagement": 0.9}
 )
 assert ld_client.variation("new_model_flag", intl_user) is False

The Complete Deployment Pipeline

it all fits together:

Stage	Duration	Rollout %	Quality Gate	Auto-Rollback if
Canary	4 hours	0.1%	Accuracy >= -2%	Degradation > 5%
Early Adopters	12 hours	5%	Accuracy >= -1%	Degradation > 3%
Regional	24 hours	25%	Accuracy >= baseline	Degradation > 2%
Broad	48 hours	75%	Accuracy >= baseline	Degradation > 1%
Full	Ongoing	100%	Continuous monitoring	Degradation > 0.5%

Notice gates get stricter and rollback thresholds get tighter as you roll out. This is intentional. A small percentage can tolerate more risk than 100%.

The Bottom Line

Feature flags for AI features are not optional. They're essential infrastructure. But traditional feature flags miss the unique challenges of ML: subgroup performance, silent failures, and probabilistic predictions.

Use feature flags with quality gates, stratified rollouts, automated monitoring, and continuous rollback. Only then can you safely ship AI features at scale.

"A slow rollout with strict quality gates catches 95% of model bugs before customers see them. A fast rollout with loose gates ships them to production."

Choose slow.