Sony PlayStation Store uses a binary classifier to detect potentially fraudulent digital purchases before fulfillment. The Risk team is deciding between two candidate models for production: Model P (precision-oriented) and Model R (recall-oriented).
Evaluation was run on a labeled holdout set of 200,000 transactions with a fraud rate of 2.0% (4,000 fraud, 196,000 legitimate).
| Metric | Model P | Model R |
|---|---|---|
| Precision | 0.91 | 0.63 |
| Recall | 0.54 | 0.86 |
| F1 Score | 0.68 | 0.73 |
| AUC-ROC | 0.89 | 0.90 |
| False Positive Rate | 0.11% | 1.03% |
| Flagged transactions | 2,374 | 5,460 |
| Missed fraud cases | 1,840 | 560 |
| Estimated manual review cost / flagged case | $2.50 | $2.50 |
| Estimated loss / missed fraud case | $120 | $120 |
Sony can only send flagged purchases to a limited review queue before digital content is delivered. The business wants to reduce fraud losses without creating too much customer friction from unnecessary holds on legitimate purchases.