Business Context
NorthStar Lending, a digital consumer lender processing ~250K applications per quarter, wants a credit risk model to predict whether an applicant will default within 12 months. The current team reports very high training accuracy but unstable validation performance, and they want you to diagnose and reduce overfitting before deployment.
Dataset
You are given a historical application dataset with applicant, credit bureau, and early repayment behavior features.
| Feature Group | Count | Examples |
|---|
| Applicant demographics | 6 | age, employment_status, annual_income, housing_status |
| Credit bureau variables | 12 | fico_band, revolving_utilization, delinquencies_24m, inquiries_6m |
| Loan attributes | 7 | loan_amount, term_months, interest_rate, purpose |
| Behavioral features | 5 | first_payment_delay_days, autopay_enabled, early_balance_drop |
| Engineered ratios | 4 | debt_to_income, utilization_to_limit, payment_to_income |
- Size: 120K loans, 34 input features
- Target: Binary — defaulted within 12 months (1) vs did not default (0)
- Class balance: 18% positive, 82% negative
- Missing data: ~9% missing in bureau variables, ~3% missing in income-related fields
Success Criteria
A strong solution should:
- Explain what overfitting is in practical terms using train vs validation behavior
- Build at least one intentionally overfit baseline and one improved model
- Achieve ROC-AUC >= 0.78 and validation-to-test ROC-AUC gap <= 0.03
- Show how regularization and cross-validation improve generalization
Constraints
- Inference must complete in under 50 ms per application
- Risk and compliance teams require interpretable feature effects
- Retraining will happen monthly on a batch pipeline
Deliverables
- Define overfitting and show how to detect it from model performance.
- Train a high-variance baseline model and quantify the generalization gap.
- Build a regularized production candidate and justify the choice.
- Compare train, validation, and test metrics using cross-validation.
- Recommend concrete steps to prevent overfitting in future model updates.
