Business Context

NorthStar Bank uses a binary classification model to predict whether a personal loan applicant will default within 12 months. The model is already in production for pre-approval screening, and the hiring team wants to assess whether you can identify and mitigate common model vulnerabilities before they become business, compliance, or reliability issues.

Dataset

You are given a historical underwriting dataset used to train the current model.

• Rows: 240K applications from the last 24 months
• Target: default_12m (1 if defaulted within 12 months, else 0)
• Class balance: 11.5% positive
• Missing data: 8% missing in employment-related fields, 3% missing in bureau features, and some categories unseen in recent months due to policy changes

Success Criteria

A strong solution should identify the main vulnerabilities of the model, explain how to test for them, and propose practical mitigations. “Good enough” means the model maintains stable out-of-time performance, avoids leakage, is reasonably interpretable for risk review, and does not materially degrade for key applicant segments.

Constraints

• Decisions must be explainable to risk and compliance teams
• Batch scoring latency must stay under 10 minutes for 100K applications
• Retraining is allowed monthly, not daily
• The bank prefers simple, auditable pipelines over highly complex ensembles

Deliverables

1. Train a baseline default classifier and evaluate it on an out-of-time test set.
2. Identify at least five model vulnerabilities you would be prepared to discuss in an interview.
3. Show how you would detect vulnerabilities such as overfitting, leakage, drift, instability, and unfair segment performance.
4. Recommend mitigations and monitoring for production deployment.
5. Explain the tradeoffs between a simpler interpretable model and a stronger but less transparent model.