Context

ShieldPay runs a card-not-present fraud detection model that scores transactions in real time and sends high-risk cases to a manual review queue. A large enterprise customer says the platform is "missing the fraud we expected it to catch," even though the model still looks strong on aggregate ranking metrics.

Current Performance

The model threshold was raised from 0.42 to 0.63 six weeks ago to reduce analyst workload. The customer’s fraud base rate also increased from 0.35% to 0.52% after expansion into cross-border transactions.

The Problem

You need to determine whether the issue is thresholding, calibration drift, segment-specific underperformance, or a broader model quality problem. The customer wants a clear explanation for why fewer fraud cases are being caught despite strong AUC.

Requirements

1. Interpret the metrics and explain the most likely reason the customer perceives underperformance.
2. Identify at least 3 plausible root causes and how you would validate each.
3. Analyze what the confusion matrix implies for business impact.
4. Recommend specific model, threshold, and monitoring changes.
5. Explain what additional slices or offline evaluations you would request before retraining.

Constraints

• Manual review capacity is capped at 6,000 transactions/day.
• False positives create customer friction and merchant support costs.
• Fraud labels arrive with a 21-day delay.
• The business cannot accept a large increase in review volume without a strong expected ROI.