Machine-learning models fail in ways that classical statistical models rarely do. A logistic regression that fits well on training data usually generalises acceptably; a gradient-boosted or deep-learning model that fits well on training data can still fail catastrophically out of sample, out of distribution, or under trivial adversarial perturbation. The result is that AI/ML models need a validation playbook that goes beyond the classical toolkit — not because ML is fundamentally different, but because its failure modes are.
This article sets out that playbook, aligned with the general lifecycle in The Model Validation Lifecycle and the AI/ML supervisory expectations emerging from SR 11-7 practice, EBA discussion papers on ML in IRB and the EU AI Act's high-risk classification.
Failure mode 1: data leakage
Leakage occurs when information from the target variable — or from the future — enters the feature set. Classical statistics has fewer opportunities for leakage because feature engineering is more visible; ML pipelines with hundreds of automatically-generated features have many.
Common leakage patterns:
- Post-event features: fields that are populated only after the target event (e.g. a "restructuring flag" set at the moment of default entering a PD model).
- Look-ahead bias in time series: features computed from data windows that overlap the target period.
- Sample-selection leakage: training on cases that were selected because of the target outcome (e.g. only exposures that were reviewed by a credit committee).
- Target encoding of categorical variables computed on the full dataset rather than within each fold.
The validator's test: for each feature, ask "at time t, when the prediction is made, is this value knowable?" If the answer is no, the feature is a leak.
Failure mode 2: drift
Machine-learning models can degrade silently. Two drift categories deserve monitoring:
- Covariate drift: the distribution of inputs shifts (P(X) changes) while the input-output relationship is stable.
- Concept drift: the relationship between inputs and outputs (P(Y|X)) changes.
Detection tools include PSI on individual features, Jensen–Shannon divergence on multivariate distributions, and rolling performance monitoring against a held-out benchmark. Drift monitoring is a live process, not a one-off validation step.
Failure mode 3: overfitting and challenger benchmarks
The single most important check for an ML model in a regulated context is whether a simpler challenger matches or beats it on out-of-time performance. A gradient-boosted classifier that fails to beat a well-specified logistic regression on true out-of-time data is not evidence of ML value — it is evidence of overfitting to the training window. This principle is written into EBA and SR 11-7 practice for a reason.
Best-practice tests:
- Time-based split (train on t < T, test on t ≥ T), not random split.
- Nested cross-validation for hyperparameter selection.
- Benchmark against a monotonic-constraint version of the same model, a logistic regression, and — for tree ensembles — a single well-pruned tree.
Explainability
Explainability is not a nice-to-have; it is a regulatory expectation for consequential decisions. The current toolkit includes:
- Global explainability: feature-importance rankings from the model (Gini importance, permutation importance).
- Local explainability: SHAP values, LIME, integrated gradients — attributing a specific prediction to its inputs.
- Counterfactual explanations: the smallest input change that flips the prediction.
Validators should test that explanations are stable — small perturbations of the input should not produce wildly different SHAP attributions — and consistent — the aggregate of local explanations should match the global feature-importance ranking. Explanations that are unstable or inconsistent are worse than no explanation, because they create false confidence.
Fairness and disparate impact
Regulated ML models — credit scoring, insurance pricing, employment screening — face fairness obligations under a mix of consumer-protection law, anti-discrimination law and, in the EU, the AI Act's high-risk regime. The metrics are jurisdiction-specific but converge on the same categories:
- Statistical parity: equal approval rates across protected groups.
- Equal opportunity: equal true-positive rates.
- Predictive parity: equal precision (positive predictive value).
- Calibration by group: predicted probabilities equal to realised rates within each group.
Mathematical impossibility theorems (Chouldechova, Kleinberg et al.) show that these criteria are mutually incompatible under most realistic conditions. Validators should therefore test which fairness definition applies, quantify the trade-offs and document the choice — not check all metrics against a single threshold.
Uncertainty quantification
A point prediction without an uncertainty band is a false-precision device. ML uncertainty comes in two flavours:
- Aleatoric: irreducible noise in the outcome given the inputs.
- Epistemic: uncertainty from limited data or model choice, reducible in principle by more data or better models.
Practical estimators: bootstrap resampling of the training set, ensembling across models, Bayesian neural networks with variational approximations, quantile regression forests. For high-stakes decisions, the width of the uncertainty band should feed the downstream action — e.g. widen decision thresholds when uncertainty is high, or trigger human review.
Adversarial and out-of-distribution testing
Two categories of stress test are underused in financial applications.
Adversarial robustness. Small, deliberately-crafted input perturbations that flip predictions are well-documented in image and text models; they exist for tabular models too. A credit scoring model that can be flipped by rounding an income figure is fragile. Robustness testing perturbs inputs within plausible bounds and measures the fraction of predictions that change.
Out-of-distribution (OOD) detection. ML models trained on a distribution X_train tend to be silently wrong on inputs from a different distribution X_prod. OOD detectors — Mahalanobis distance in a suitable embedding, autoencoder reconstruction error, density estimation — flag inputs on which the primary model's prediction should not be trusted. In production, OOD flags trigger fallback behaviour.
Safe fallbacks
An ML system without a safe fallback is one adversarial input, one distribution shift or one silent bug away from a governance incident. Fallback design:
- Primary model produces a prediction and a confidence.
- Fallback model (simpler, more auditable — often a logistic regression or rule-based model) is always available.
- Trigger conditions: OOD flag raised, uncertainty exceeds a threshold, drift metric exceeds a threshold, adversarial detector fires.
- Human-in-the-loop: for the highest-severity trigger paths.
The fallback is not a backup; it is part of the model.
Governance implications
The board-level view of an ML model includes: intended use, performance metrics on out-of-time data, explainability approach, fairness metrics and choices, uncertainty treatment, monitoring plan and fallback logic. The EU AI Act's high-risk classification specifically requires documentation across most of these — the compliance load is real. Regulators have signalled that models used for consequential decisions require the same rigour as internal-model risk components; this is consistent with EBA guidance on ML in IRB.
Limitations
ML brings capability and complexity. A model whose failure modes exceed the validator's toolkit is not a validated model. Institutions should be explicit about which decisions are appropriate for ML today and which are not — and revisit the boundary as tooling matures.
Conclusion
The validation playbook for AI/ML is the classical playbook, extended. Leakage, drift, benchmark challengers, explainability, fairness, uncertainty, adversarial and OOD testing, and safe fallbacks — these are not exotic additions. They are the minimum standard that separates ML models fit for regulated use from ML models that only look impressive on a training set.
Related reading
References and further reading
- Federal Reserve / OCC, Supervisory Guidance on Model Risk Management, SR 11-7.
- European Banking Authority, Discussion paper on machine learning for IRB models.
- European Union, Regulation (EU) 2024/1689 laying down harmonised rules on Artificial Intelligence (AI Act).
- Molnar, C., Interpretable Machine Learning.
About the author
Part of an ongoing series on AI risk and model governance — more about the author.