Context

FinShield uses a binary classification model to score card transactions for fraud and sends high-risk transactions to a manual review queue. Over the last month, risk leaders noticed either too many legitimate transactions being reviewed or too many fraudulent transactions slipping through, and they want to know whether the current alert threshold is set too high or too low.

Current Performance

The model outputs a fraud probability, and the production threshold is currently 0.80.

The Problem

You need to determine whether the current threshold of 0.80 is too conservative or too permissive, quantify the tradeoff, and recommend a better operating point.

Requirements

1. Interpret what the current precision-recall tradeoff implies about the threshold.
2. Compare threshold options using both model metrics and business costs.
3. Explain how you would validate whether the score distribution is well calibrated.
4. Identify what additional error analysis you would run before changing the threshold.
5. Recommend a threshold strategy and monitoring plan.

Constraints

• Fraud analysts can review at most 5,000 alerts/day.
• Average loss per missed fraud transaction is $240.
• Average operational cost per false positive review is $6, plus customer friction.
• Chargeback labels arrive with a 21-day delay, so evaluation is not fully real-time.