MediScan built a binary classification model to detect a rare cancer from routine blood test results. The model is being considered for use as an initial screening tool, where missed positive cases are far more costly than sending some healthy patients to follow-up testing.
The test set contains 100,000 patients, with a disease prevalence of 1%.
| Metric | Model A | Naive Baseline |
|---|---|---|
| Accuracy | 98.7% | 99.0% |
| Precision | 43.3% | 0.0% |
| Recall | 52.0% | 0.0% |
| F1 Score | 47.2% | 0.0% |
| False Negatives | 480 | 1,000 |
| False Positives | 680 | 0 |
A product manager argues that the model should not be deployed because its 98.7% accuracy is lower than the naive baseline's 99.0%. However, the clinical team believes accuracy is the wrong metric because the dataset is highly imbalanced and the business cost of false negatives is much higher than false positives.