Business Context

Northstar Bank is building a credit risk model to predict whether a small-business loan will default within 12 months. The Head of Lending does not need mathematical proofs, but does need a clear explanation of why a more complex model can perform worse in production even if it looks better on training data.

Dataset

You are given a historical supervised learning dataset used for binary classification.

• Size: 240K loan applications, 42 features
• Target: default_12m = 1 if the loan defaulted within 12 months, else 0
• Class balance: 14% positive, 86% negative
• Missing data: 10% missing in financial fields, 6% in behavioral features, categorical unknowns in bureau-related fields

Success Criteria

A strong solution should show how to explain the bias-variance tradeoff to a non-technical stakeholder using model results, not theory alone. The final recommendation should justify a model that generalizes well and achieves stable out-of-sample performance.

Constraints

• The explanation must be understandable to business leaders
• The chosen model should remain reasonably interpretable for risk review
• Batch scoring must finish in under 10 minutes for 100K applications
• Retraining is allowed monthly, not daily

Deliverables

1. Train at least three models with different complexity levels and compare train vs validation performance.
2. Demonstrate bias and variance using cross-validation results and learning curves.
3. Provide a stakeholder-friendly explanation of the tradeoff and recommend one model for production.
4. Show how regularization or pruning changes model behavior.
5. Report concrete evaluation metrics and explain which metric matters most for this lending use case.