Model validation is often reduced to a set of backtests attached to an approval memo. That view underestimates what validation is for. The Federal Reserve's Supervisory Guidance on Model Risk Management — SR 11-7, jointly issued with the OCC — defines a model as "a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories" and treats model risk as a first-class risk category, on the same footing as credit, market or liquidity risk. The European Banking Authority's guide to internal models and the ECB's TRIM findings frame the same expectation for European banks; the equivalent for insurers is embedded in Solvency II Article 124 and EIOPA's guidelines on the use of internal models.
Validation, in that light, is independent challenge across the full life of a model: before it goes into production, while it runs, and when it is retired or replaced. This article walks through the six phases that make up that lifecycle and the artefacts an effective validation function produces at each step.
Conceptual soundness: is the model the right tool for the question?
The first phase asks whether the model's theory, methodology and design are appropriate for its purpose. Validators do not simply reproduce the developer's derivations; they interrogate the assumptions that make those derivations valid.
For a credit PD model, conceptual review covers the definition of default (aligned with CRR Article 178 for regulated banks), the choice between logistic regression and gradient boosting, the treatment of low-default portfolios, and the appropriateness of point-in-time versus through-the-cycle calibration for the intended use. For an economic-capital model, it covers the choice of risk measure (VaR versus expected shortfall — see Beyond Historical VaR), the diversification framework, and whether copulas or a factor model best capture cross-portfolio dependence. For a climate physical-risk model, it covers the hazard–exposure–vulnerability chain (see Physical Climate Risk) and whether the chosen resolution matches the loss decision.
Concrete artefacts at this stage: an assumptions register with materiality ratings, a peer-methodology comparison, and a written statement of intended use. If a model's intended use drifts later (a scoring model repurposed for pricing, a stress model repurposed for capital), the assumptions register is what a validator returns to.
Data: quality, representativeness and lineage
No model is better than the data behind it. Data validation covers completeness, accuracy, timeliness, representativeness and lineage. A common failure mode is that developers validate the modelling sample and validators validate the production data — and no one checks that the two match.
Practical tests include:
- Reconciliation between the modelling extract and the source system of record, at both aggregate and transaction level.
- Population stability index (PSI) across the modelling window and the current book, with material shifts (PSI > 0.1) flagged for retraining review.
- Missing-data mechanism analysis — is missingness random, or does it correlate with the outcome (as it often does for defaulted exposures where collateral information is patchy)?
- Feature leakage checks, particularly for machine-learning models where post-event fields can silently enter the training set (see AI/ML Model Risk Validation).
Data governance sits alongside these tests. BCBS 239 defines the principles for effective risk-data aggregation and reporting; a validator with no view on data lineage is validating a fraction of the model.
Implementation: does the code reproduce the design?
Implementation review is where good validators pay for themselves. A model can be conceptually sound and its data clean, and still produce wrong numbers because a monotonic transformation was applied in the wrong order, an off-by-one indexed the wrong maturity bucket, or a floating-point comparison silently truncated tail probabilities.
Independent code review, unit tests on known analytical cases, replication of a sample of production outputs in a separate environment, and reconciliation of stress-scenario outputs across the development and production stacks are all part of this step. For models with vendor components, validators need contractual access to sufficient technical documentation to challenge the vendor's implementation; opaque black boxes are a governance red flag under both SR 11-7 and EBA expectations.
Outcomes analysis: does the model actually work?
Outcomes analysis — often called performance testing or backtesting — is the phase most people associate with validation. It is also the phase most easily gamed if the discrimination and calibration tests are chosen after the results are known.
The core toolkit is well established:
- Discrimination: Gini/AUC, KS, information value, lift charts. For PD models, a Gini above 0.5 on a live sample is a reasonable floor for retail portfolios; wholesale is usually lower.
- Calibration: Hosmer–Lemeshow, binomial and normal tests on realised default rates by grade, and confidence intervals that acknowledge the small-sample nature of low-default portfolios (see IFRS 9 PD/LGD/EAD).
- Stability: rolling PSI on scores and on rating-grade migrations.
- Backtesting for VaR/ES: Kupiec unconditional coverage, Christoffersen conditional coverage, and traffic-light frameworks per Basel market-risk rules.
Outcomes tests should be pre-specified in the validation plan. Choosing the test after seeing the data — or dropping quarters that fail — is one of the classic ways validation functions lose credibility.
Robustness: sensitivity, benchmarking and stress
A model that performs on a benign backtest can still be fragile. Robustness testing probes that fragility.
- Sensitivity analysis perturbs each input and parameter individually and jointly; results should be plausible in direction and magnitude.
- Benchmarking compares outputs to a challenger model of equal or lower complexity. A gradient-boosted PD model that fails to beat a well-specified logistic regression on out-of-time data is not evidence of ML value — it is evidence of overfitting.
- Stress and reverse-stress testing ask what environments break the model and what business consequences follow. Reverse stress is uncomfortable precisely because it names the scenarios in which the model — and the strategy that relies on it — no longer functions.
Governance, findings and remediation
Every phase above generates findings. A validation function without a graded finding taxonomy — critical, high, medium, low — and a tracked remediation register cannot demonstrate independent challenge to a regulator. Findings should specify the residual risk, the mitigating action, the owner and the deadline; open critical findings should trigger use restrictions, model overlays or capital add-ons under Pillar 2 as appropriate.
Retirement is the phase most often skipped. When a model is decommissioned, the validator's role is to check that dependent processes have been re-pointed, that historical outputs remain reproducible for audit, and that the model's data feeds are shut down cleanly. Retired models with live data connections are a persistent source of operational incidents.
Limitations of validation itself
Validation is not free of model risk. Validators bring their own assumptions — that historical data represents the future, that challenger models are meaningfully different, that governance forums will act on findings. The 2007–09 crisis is a reminder that entire industries can validate models consistently and still be wrong together. Independence in reporting lines helps; independence in thought is harder to institutionalise.
Conclusion
The best validation functions are not adversaries of model developers; they are the second pair of eyes that keeps a modelling function honest with itself. Done well, validation shortens the tail of model incidents, gives boards defensible comfort, and produces the audit trail regulators expect. Done as a compliance box-tick, it produces neither.
Related reading
- Beyond Historical VaR
- IFRS 9 PD/LGD/EAD Model Validation
- AI/ML Model Risk Validation
- From Fixed Assumptions to Calibrated Risk Engines
References and further reading
- Board of Governors of the Federal Reserve System and OCC, Supervisory Guidance on Model Risk Management, SR 11-7 / OCC 2011-12.
- European Central Bank, Guide to internal models (revised edition).
- European Banking Authority, Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16).
- European Insurance and Occupational Pensions Authority, Guidelines on the use of internal models (EIOPA-BoS-14/180).
- Basel Committee on Banking Supervision, Principles for effective risk data aggregation and risk reporting, BCBS 239.
About the author
This article is part of an ongoing series on risk, capital and modelling — read more about the author and Quantica Risk Modelling.