Context

ShopSafe uses a binary classification model to detect fraudulent e-commerce orders before payment approval. The model performed well in offline validation, but after 8 weeks in production, operations reports rising fraud losses and more customer complaints about blocked legitimate orders.

Current Performance

The Problem

The team needs to determine whether the model is still performing well in production and what signals should be monitored beyond a single headline metric like accuracy. You should assess whether the current performance is acceptable, identify likely causes of degradation, and recommend how to evaluate the model continuously in production.

Requirements

1. Interpret the production metrics and explain whether the model is performing well.
2. Identify which metrics matter most for this use case and why.
3. Diagnose what the gap between offline and production performance suggests.
4. Propose a production monitoring plan, including thresholds and alerting.
5. Recommend concrete actions to improve model performance and reduce business risk.

Constraints

• Manual review team can handle at most 1,600 flagged orders per day.
• False positives create customer friction and lost conversion.
• False negatives directly increase fraud losses.
• Fraud labels arrive with a 14-day delay, so some metrics are not available in real time.