Context

FinSure runs a binary classification model to flag potentially fraudulent card transactions for customer support review. Over the last 6 weeks, one enterprise customer reported that the model is flagging too many legitimate transactions while still missing some truly suspicious ones.

Current Performance

The Problem

The customer wants to know why the system is flagging the wrong transactions and what should be investigated first. Support operations can handle only a limited number of reviews, and excessive false alarms are causing customer friction.

Requirements

1. Identify which metrics you would inspect first and explain why.
2. Diagnose the most likely causes of the increase in wrong flags.
3. Explain how you would use the confusion matrix and score distributions to validate your hypotheses.
4. Recommend specific actions to improve performance for this customer segment.
5. Discuss whether this looks more like a threshold issue, calibration issue, data drift issue, or label-quality issue.

Constraints

1. Review team capacity is capped at 2,000 transactions per day.
2. A false positive triggers customer outreach and costs about $4 per case.
3. A false negative costs an average of $220 in fraud losses.
4. Retraining requires 10 days, but threshold changes can be deployed same day.