Diagnose Overfitting in Loan Default Models

Business Context

NorthStar Bank scores small-business loan applications and wants a model that predicts 90-day default risk before underwriting review. The current model performs well in training but degrades noticeably on new applications, so the team suspects high variance and wants a structured diagnosis and mitigation plan.

Dataset

• Size: 120K historical applications, 42 features
• Target: Binary default within 90 days after origination
• Class balance: 14% default, 86% non-default
• Missing data: 9% missing in bureau features, 4% missing in financial statements, higher missingness for newer applicants

Success Criteria

A good solution should clearly explain the bias-variance tradeoff, diagnose whether the model is suffering from high variance using train/validation behavior, and improve generalization enough to reach ROC-AUC >= 0.82 and PR-AUC >= 0.46 on a held-out test set. The final recommendation should also identify which interventions reduce variance without making the model too biased.

Constraints

• Underwriting requires some interpretability; a black-box-only answer is not sufficient
• Batch scoring must complete in under 5 minutes for 25K daily applications
• Retraining is allowed weekly, not continuously
• Avoid data leakage from post-origination signals

Deliverables

1. Explain the bias-variance tradeoff in practical model-training terms.
2. Train at least one baseline model and one higher-capacity model.
3. Diagnose high variance using train vs. validation metrics, learning curves, and cross-validation.
4. Propose and test mitigation strategies such as regularization, pruning, feature reduction, or more data.
5. Recommend a production-ready model and justify the tradeoffs.