Diagnose Production Credit Model Drift

Context

American Credit Acceptance uses a gradient boosting model in its underwriting decision flow to predict 60-day early payment default risk for indirect auto loan applications. The model performed well during development and pilot validation, but its production performance has degraded over the last 4 months after rollout to ACA's dealer network.

Current Performance

The Problem

ACA leadership wants to know whether the degradation is caused by data drift, label delay, threshold miscalibration, dealer mix changes, or model overfitting to development data. You need to diagnose the most likely causes and recommend a production monitoring and remediation plan.

Requirements

1. Interpret what the metric changes imply about model behavior in production.
2. Propose the top hypotheses for why production performance degraded.
3. Explain how you would validate each hypothesis using offline and production data.
4. Recommend specific model, threshold, and monitoring changes.
5. Discuss how you would decide whether to retrain, recalibrate, or roll back.

Constraints

• ACA cannot reduce funded application volume by more than 5% in the next quarter.
• Default losses are materially more expensive than false declines.
• Labels arrive with a 60-90 day delay, so near-real-time monitoring must rely partly on proxy signals.