Monitor Production Model Drift

Context

ShopLens uses a gradient-boosted binary classifier to predict whether a user session will convert within 24 hours, so the marketing platform can trigger high-value discount offers. The model performed well at launch, but over the last 8 weeks the growth team reports lower campaign ROI despite similar traffic volume.

Current Performance

The Problem

You need to design a production drift monitoring approach that would have detected this degradation early, separated data drift from concept drift, and triggered the right investigation or retraining workflow.

Requirements

1. Explain what the metric changes suggest about drift type and severity.
2. Define the production monitoring dashboard and alert thresholds you would implement.
3. Specify which inputs, outputs, and delayed-label metrics should be tracked daily vs weekly.
4. Recommend how to diagnose whether the issue is feature drift, target drift, calibration drift, or threshold mismatch.
5. Propose a response plan for mild drift vs severe drift.

Constraints

• Labels arrive with a 24-hour delay.
• Discount offers have a fixed weekly budget of $180,000.
• False positives waste discounts; false negatives miss revenue.
• Full retraining can only be done every 2 weeks due to compliance review.