Diagnose Production Credit Model Drift | Dataford Interview Questions

Your question is Diagnose Production Credit Model Drift. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Context

American Credit Acceptance uses a gradient boosting model in its underwriting decision flow to predict 60-day early payment default risk for indirect auto loan applications. The model performed well during development and pilot validation, but its production performance has degraded over the last 4 months after rollout to ACA's dealer network.

Current Performance

Metric	Development Validation	First 30 Days in Production	Current Production	Change vs Dev
AUC-ROC	0.79	0.77	0.68	-0.11
Precision @ approval threshold	0.74	0.72	0.61	-0.13
Recall @ approval threshold	0.66	0.64	0.52	-0.14
F1 Score	0.70	0.68	0.56	-0.14
Log Loss	0.49	0.53	0.67	+0.18
Approval rate	41%	42%	47%	+6 pts
Observed 60-day default rate in approved loans	8.1%	8.6%	11.9%	+3.8 pts

The Problem

ACA leadership wants to know whether the degradation is caused by data drift, label delay, threshold miscalibration, dealer mix changes, or model overfitting to development data. You need to diagnose the most likely causes and recommend a production monitoring and remediation plan.

Requirements

Interpret what the metric changes imply about model behavior in production.
Propose the top hypotheses for why production performance degraded.
Explain how you would validate each hypothesis using offline and production data.
Recommend specific model, threshold, and monitoring changes.
Discuss how you would decide whether to retrain, recalibrate, or roll back.

Constraints

ACA cannot reduce funded application volume by more than 5% in the next quarter.
Default losses are materially more expensive than false declines.
Labels arrive with a 60-90 day delay, so near-real-time monitoring must rely partly on proxy signals.