Personal Loan Application Risk Scorecard ML Model Audit and Validation
Overview
To evaluate and validate the effectiveness of the XGBoost-based credit risk ML model deployed by a leading consumer lender with a portfolio of 284,000+ active customers, ensuring:Â
- robust predictive performanceÂ
- clear and effective risk segmentation
- long-term model stabilityÂ
Solution
Our methodology for the ML model audit covered all validation domains, from raw data exclusions through to out-of-time stability testing:Â
- Feature Engineering Audit: End-to-end code review, data lineage tracing, and reproducibility testing of all derived variables.
- Statistical Performance Testing: Gini, AUC-ROC, and KS evaluation across train, test, and out-of-time datasets to confirm predictive power.
- Population & Characteristic Stability: PSI and CSI monitoring to detect data drift and feature instability before they impact decisions.
- Missing Value & Exclusion Review: Validation of imputation strategy, business exclusion logic, and population representativeness.
- Decile & Rank-Order Validation: OOT decile analysis to confirm consistent risk segmentation on unseen future data.
- Governance & Documentation Review: Assessment of ML model documentation, coding standards, and compliance with internal risk frameworks.
Impacts Delivered
Based on a comprehensive review of all ML model validation and audit control checkpoints, the model was deemed fit for purpose. The validation was completed successfully with zero critical, high, or medium-risk findings:
- Performance metrics such as Gini, AUC-ROC, and KS demonstrate strong discriminatory power, confirming excellent risk differentiation capability.
- Bivariate analyses show stable relationships between key features and bad rate with no unexpected shifts or reversals.
- Stability metrics indicate minimal data drift, and CSI is well within its thresholds, confirming the model’s reliability on new data.
- OOT decile analysis confirms excellent rank ordering with monotonically decreasing bad rates from the highest to the lowest risk deciles.
