TL;DR
Independent governance: The team that built the model shouldn't be the only team validating it. You need independent validators who ask hard questions. Model inventory: You need to know what models you have, what they do, what data they use, and who owns them. Testing is impossible if you can't enumerate your models. Validation before deployment: Testing happens before go-live. This seems obvious but it's not. Many shops do "light" validation then iterate in production. Ongoing monitoring: Deployment is not the end of testing. You need continuous monitoring for data drift, performance degradation, and market regime changes. Documentation: You need to write down why you chose each test, what you're testing for, what you found, and what assumptions you're making. This documentation will be reviewed by examiners. Look-ahead bias: It's easy to accidentally use information that wasn't available at prediction time. (Classic mistake: using end-of-day prices to predict same-day movements.) Overfitting: Test enough parameters and you'll find patterns that don't persist. Your model has learned the noise in the test data. Slippage and execution: Backtests assume you can execute trades instantly at quoted prices. Reality has friction: bid-ask spread, market impact, latency. Survivorship bias: Stocks that delisted from the data disappear. Trading strategies that would have bankrupted you are excluded. Regime change: Markets that returned 8% annually for 20 years might return -12% for the next 10. Your backtest can't predict structural breaks. 2008 financial crisis (market crash with liquidity freeze): Model crashes 30%+ in equity markets, but can't exit because bid-ask spreads blow out to 2-3% on normally liquid assets. Volatility spike: VIX goes from 15 to 40 in one trading session (happened March 2020). Models trained on low-vol regimes behave unpredictably. Fed policy shock: Unexpected rate decision or guidance change. Bonds and equities reprice suddenly and asymmetrically. Credit event: Major counterparty defaults or liquidity event. Credit spreads widen 500+ bps in 48 hours. Tail event in your specific market: If you trade FX, test a peg break (like CHF in 2015). If you trade crypto, test 50%+ drawdowns. If you do credit, test covenant breach cascades. Feature importance lies: The model uses a feature but not the way you think. SHAP and LIME values can be misleading if the feature is highly correlated with other features. Contradictory decisions: Two identical loan applications get different decisions. The model is overfit or using spurious patterns. Protected class proxies: The model uses age, zip code, or employment history as a proxy for race or gender. Illegal, even if unintentional. Drift in explanations: The model trained on 2023 data explains decisions differently than when retrained on 2024 data. The underlying patterns shifted. Data drift: Distribution of input features changes. Example: average loan balance is 2x historical because of economic shifts. The model's assumptions break. Concept drift: The relationship between features and target deteriorates. Example: education level used to predict credit risk. Now it doesn't because market dynamics shifted. Model output distribution: The model's predictions skew. Credit model is approving 60% of applications when historical rate is 45%. Why? Data shift? Model degradation? Calibration drift? Actual performance tracking: For trading: is P&L matching backtest expectations? For credit: do approved loans actually default at predicted rates? For pricing: does the model's mark-to-market price match dealer prices? If production performance drops below 80% of backtest expected Sharpe ratio, trigger investigation within 24 hours. If data distributions shift beyond 3-sigma in any critical feature, the model goes into review. If approval rates deviate >10% from forecast, halt new decisions until reviewed. Every 90 days minimum, rerun full validation suite (backtests, stress tests, fairness checks). Retraining is not automatic; it requires sign-off. Train/test split with temporal consistency. Train on 2019-2022 data, validate on 2023 (out-of-sample). Don't mix in 2024 data because you don't know defaults yet. Stratified evaluation by cohort. Measure accuracy separately for <$50K loans, $50K-$250K, $250K+ because the model might work better on certain ranges. Fairness testing. Measure approval rates and default rates separately for age groups, geographies, and employment types. Flag >5% disparity. Stress test: economic downturn. Take your 2023 test data and artificially shift unemployment +2%, wage growth -3%, and housing prices -15%. Rerun predictions. Does default rate spike as expected? Is the spike believable? Explainability audit. Generate 100 random declined applicants. Manually verify that the model's reason for decline makes sense. ("This person has 3 recent late payments" should be more important than "this person lives in California.") Monitoring dashboard. Track monthly: approval rate, actual default rate vs predicted, application volume, demographic distributions. Alert if approval rate drifts >10%. Revalidation schedule. Every 6 months or after any data incident, retrain and revalidate. "The team that built it validated it." Classic SR 11-7 violation. You need independent review. This means different people, ideally different incentives. Backtesting only. Backtest is necessary, not sufficient. You need stress tests and monitoring. Overfitting through hyperparameter search. You tune the model 500 different ways and report the best. That's not validation, that's cheating. No monitoring after deployment. Models degrade silently. Six months later, the model's decisions don't make sense anymore because the world changed. Ignoring transaction costs and slippage. The backtest looks great on clean data. In production with real execution costs, it's break-even or negative. Treating validation as a checkbox. Validation is not a gate you pass once. It's an ongoing process. Backtesting libraries: Zipline, Backtrader, or custom frameworks that handle data alignment, survivorship bias, and execution assumptions. Stress testing frameworks: Start with historical crisis data, then build synthetic scenario generators. Make it easy to run 100+ scenarios at once. Explainability tools: SHAP, LIME, or custom analysis for understanding model decisions. Integrate with your model pipeline so explanations are always fresh. Fairness toolkits: Agarwal et al.'s tools, or IBM Fairness 360. You need to measure and track bias metrics continuously. Monitoring infrastructure: Real-time dashboards tracking model predictions, data distributions, and performance metrics. Alerting on anomalies. Integration with your data warehouse so metrics are always current. Model inventory and governance. Can you list every model used in production? Who owns it? When was it last validated? Documentation of testing approach. Why did you choose your test cases? What were you trying to find? What assumptions are you making? Evidence of backtesting. Show the results. Show the in-sample and out-of-sample performance. Show the equity curve and Sharpe ratio. Stress testing results. What scenarios did you test? What were the losses? Did you fix problems that showed up? Fairness analysis. For credit models: what's the approval rate disparity? What's the default rate disparity by demographic group? Is it within acceptable bounds? Monitoring and alert systems. How do you catch degradation? How quickly would you know if a model started failing? Incident reports. When something went wrong in production, how did you investigate? What did you fix? Real-time model monitoring dashboards, not weekly reports Stress testing against synthetic scenarios, not just historical data Explainability requirements for all credit and trading models Model versioning and the ability to instantly rollback Cross-institutional stress testing (what happens if every bank's model fails at the same time?)