Context

NorthStar Bank deployed a gradient boosting model to predict whether a small-business loan applicant will default within 12 months. The model replaced a logistic regression scorecard and is now used to auto-approve, reject, or route applications for manual underwriting.

After two quarters in production, leadership sees mixed results: default losses are slightly lower, but approval rates fell and underwriters report too many borderline cases being escalated. You need to assess whether the model is actually performing well and how its accuracy should be verified beyond a single headline metric.

Current Performance

The Problem

The model looks better than the baseline on AUC and accuracy, but production recall dropped materially while manual reviews increased. The bank wants to know whether the model is truly better, what verification steps are missing, and what should be improved before expanding auto-decisioning.

Requirements

1. Interpret the metric tradeoffs between validation and production.
2. Explain whether accuracy is a reliable primary metric here.
3. Identify likely causes of the recall drop and review-rate increase.
4. Recommend how to verify model accuracy using validation, calibration, and error analysis.
5. Propose specific changes to improve business performance.

Constraints

• False negatives are costly because missed defaults create charge-off losses.
• False positives reduce approval volume and revenue.
• Regulators require explainability and stable decision thresholds.