Your question is Evaluate Credit Risk Calibration. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Barclays has deployed a probability-of-default model in its unsecured lending workflow to estimate 12-month default risk for Barclaycard applicants. The model ranks applicants well enough for prioritization, but Risk and Finance are concerned that the predicted probabilities used in pricing and approval policy may not reflect true default likelihood.
| Metric | Validation Set | Prior Champion |
|---|---|---|
| AUC-ROC | 0.81 | 0.79 |
| Log Loss | 0.462 | 0.438 |
| Brier Score | 0.149 | 0.136 |
| Expected Calibration Error (ECE) | 0.072 | 0.031 |
| Max Calibration Error | 0.181 | 0.094 |
| Avg predicted PD | 8.9% | 8.1% |
| Observed default rate | 6.2% | 6.3% |
| Predicted PD Band | Volume | Avg Predicted PD | Observed Default Rate |
|---|---|---|---|
| 0-2% | 18,000 | 1.4% | 0.8% |
| 2-5% | 24,000 | 3.6% | 2.7% |
| 5-10% | 21,000 | 7.2% | 5.9% |
| 10-20% | 11,000 | 14.1% | 15.8% |
| 20%+ | 6,000 | 28.4% | 34.7% |
The model appears to discriminate reasonably well, but its probability estimates may be systematically biased. Barclays needs to know whether the model is sufficiently calibrated for approval cutoffs, risk-based pricing, and IFRS 9 forecasting, and what should be changed before wider rollout.