LendWise uses a binary classification model to predict whether a personal loan applicant will default within 90 days. The model was trained on 2023 application data and deployed in January 2024, but the risk team now suspects significant data drift after a new customer acquisition channel launched in Q3.
| Metric | Validation at Launch | Current Holdout (Last 30 Days) | Change |
|---|---|---|---|
| Accuracy | 0.84 | 0.78 | -0.06 |
| Precision | 0.71 | 0.63 | -0.08 |
| Recall | 0.68 | 0.52 | -0.16 |
| F1 Score | 0.69 | 0.57 | -0.12 |
| AUC-ROC | 0.81 | 0.73 | -0.08 |
| Log Loss | 0.44 | 0.58 | +0.14 |
| Default rate in scored population | 12% | 19% | +7 pts |
| Avg predicted default probability | 14% | 13% | -1 pt |
The model is underestimating risk in the most recent population while business conditions and applicant mix have changed. The head of credit wants to know how you would validate whether the model is still reliable, determine whether the issue is covariate drift, concept drift, or calibration failure, and recommend next steps.