American Credit Acceptance uses a gradient boosting model in its underwriting decision flow to predict 60-day early payment default risk for indirect auto loan applications. The model performed well during development and pilot validation, but its production performance has degraded over the last 4 months after rollout to ACA's dealer network.
| Metric | Development Validation | First 30 Days in Production | Current Production | Change vs Dev |
|---|---|---|---|---|
| AUC-ROC | 0.79 | 0.77 | 0.68 | -0.11 |
| Precision @ approval threshold | 0.74 | 0.72 | 0.61 | -0.13 |
| Recall @ approval threshold | 0.66 | 0.64 | 0.52 | -0.14 |
| F1 Score | 0.70 | 0.68 | 0.56 | -0.14 |
| Log Loss | 0.49 | 0.53 | 0.67 | +0.18 |
| Approval rate | 41% | 42% | 47% | +6 pts |
| Observed 60-day default rate in approved loans | 8.1% | 8.6% | 11.9% | +3.8 pts |
ACA leadership wants to know whether the degradation is caused by data drift, label delay, threshold miscalibration, dealer mix changes, or model overfitting to development data. You need to diagnose the most likely causes and recommend a production monitoring and remediation plan.