Context

NovaPay uses a binary classification model to score card transactions for fraud. Transactions with scores above a threshold are declined in real time; transactions below the threshold are approved and can still convert into completed purchases. Leadership wants a threshold that balances customer conversion against fraud loss.

Current Performance

The model was trained on 48M transactions from the last 6 months. On a recent holdout set of 5,000,000 transactions, fraud prevalence is 0.40% (20,000 fraud, 4,980,000 legitimate). Average gross margin on an approved legitimate order is $18, average loss on an approved fraudulent order is $240, and each false decline causes an estimated $7 long-term customer friction cost in addition to the lost margin.

The Problem

The risk team prefers lower fraud losses, while growth prefers fewer false declines and higher conversion. The current production threshold is 0.50, but the CFO believes the company may be leaving value on the table.

Requirements

1. Determine which threshold you would recommend and why.
2. Quantify the tradeoff between conversion impact and fraud loss.
3. Explain whether precision/recall alone are sufficient for this decision.
4. Describe how calibration and score distribution should influence threshold selection.
5. Propose how you would validate the threshold before full rollout.

Constraints

• False declines directly reduce checkout conversion.
• Fraud labels arrive with a 30-day delay.
• The business can tolerate at most a 0.9% false positive rate in production.