Context

FinSure uses a binary classification model to detect potentially fraudulent card transactions before authorization. The current model has reduced chargeback losses, but the risk team believes it is still missing too many fraud cases while sending too many legitimate transactions to manual review.

Current Performance

The model scores 5,000,000 transactions per month. Fraud prevalence is 0.8% (40,000 actual fraud cases). At the current decision threshold of 0.60, the offline holdout results are:

The Problem

Manual review capacity is limited, and each false positive creates customer friction. However, each missed fraud event is expensive. The VP of Risk wants to know whether the model is actually performing well for this use case, whether the threshold is appropriate, and what should be improved before the next release.

Requirements

1. Interpret the current metrics in the context of fraud detection and class imbalance.
2. Explain whether accuracy would be a useful primary metric here and why.
3. Analyze the tradeoff between precision and recall at the current threshold.
4. Recommend how you would validate the model before changing the threshold or retraining.
5. Propose concrete improvements to increase business value, not just offline metrics.

Constraints

• Manual review team can handle at most 40,000 alerts per month.
• Average cost of a false positive review is $4.
• Average loss from a missed fraud case is $210.
• Fraud labels arrive with a 30-day delay due to chargeback confirmation.