Tune Fraud Alert Threshold | Dataford Interview Questions

Your question is Tune Fraud Alert Threshold. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Context

FinShield uses a binary classification model to score card transactions for fraud and sends high-risk transactions to a manual review queue. Over the last month, risk leaders noticed either too many legitimate transactions being reviewed or too many fraudulent transactions slipping through, and they want to know whether the current alert threshold is set too high or too low.

Current Performance

The model outputs a fraud probability, and the production threshold is currently 0.80.

Metric	Threshold = 0.80	Threshold = 0.65	Threshold = 0.50
Precision	0.92	0.78	0.61
Recall	0.41	0.68	0.84
F1 Score	0.57	0.73	0.71
False Positive Rate	0.003	0.009	0.021
Alerts/day	1,900	4,600	8,900
True fraud caught/day	820	1,360	1,680
Missed fraud/day	1,180	640	320
Review capacity/day	5,000	5,000	5,000

The Problem

You need to determine whether the current threshold of 0.80 is too conservative or too permissive, quantify the tradeoff, and recommend a better operating point.

Requirements

Interpret what the current precision-recall tradeoff implies about the threshold.
Compare threshold options using both model metrics and business costs.
Explain how you would validate whether the score distribution is well calibrated.
Identify what additional error analysis you would run before changing the threshold.
Recommend a threshold strategy and monitoring plan.

Constraints

Fraud analysts can review at most 5,000 alerts/day.
Average loss per missed fraud transaction is $240.
Average operational cost per false positive review is $6, plus customer friction.
Chargeback labels arrive with a 21-day delay, so evaluation is not fully real-time.