Monitor Deployed Loan Risk Model

Context

Microsoft is using an Azure Machine Learning–deployed gradient boosted classifier to predict default risk for small business credit applications in Dynamics 365 Finance. The model was strong at launch, but six months later the risk team reports more unexpected defaults while the model’s approval rate has stayed nearly unchanged.

Current Performance

The Problem

You need to design a post-deployment monitoring approach and diagnose whether the issue is threshold drift, score miscalibration, feature drift, or a broader change in borrower behavior. Assume labels arrive with a 60-day delay, so some online metrics are only available later.

Requirements

1. Explain which metrics you would monitor daily, weekly, and monthly after deployment.
2. Diagnose what the current metric pattern suggests about model health.
3. Identify likely root causes and how you would validate each one.
4. Recommend concrete actions to improve monitoring and restore performance.
5. State what alert thresholds and rollback criteria you would implement in Azure Machine Learning.

Constraints

1. False negatives are costly because missed high-risk applicants drive loan losses.
2. False positives reduce approvals and hurt revenue.
3. Full retraining takes 10 days and requires model risk review.
4. The business cannot reduce approval volume by more than 3 percentage points without executive sign-off.