FinSure runs a binary classification model to flag potentially fraudulent card transactions for customer support review. Over the last 6 weeks, one enterprise customer reported that the model is flagging too many legitimate transactions while still missing some truly suspicious ones.
| Metric | Previous Quarter | Current | Change |
|---|---|---|---|
| Precision | 0.78 | 0.54 | -0.24 |
| Recall | 0.72 | 0.69 | -0.03 |
| F1 Score | 0.75 | 0.61 | -0.14 |
| AUC-ROC | 0.86 | 0.84 | -0.02 |
| Log Loss | 0.31 | 0.46 | +0.15 |
| False Positive Rate | 0.9% | 2.8% | +1.9 pts |
| Daily flagged transactions | 1,150 | 2,480 | +1,330 |
| Confirmed fraud rate in flagged set | 78% | 54% | -24 pts |
The customer wants to know why the system is flagging the wrong transactions and what should be investigated first. Support operations can handle only a limited number of reviews, and excessive false alarms are causing customer friction.