SafeDrive uses a binary classification model to detect imminent collision risk from driver-assistance sensor data and trigger an in-cabin emergency alert. The current model performs well on aggregate accuracy, but operations teams report too many nuisance alerts while safety reviewers are concerned about missed dangerous events.
| Metric | Validation Set | Prior Model | Change |
|---|---|---|---|
| Accuracy | 0.962 | 0.948 | +0.014 |
| Precision | 0.412 | 0.355 | +0.057 |
| Recall | 0.781 | 0.846 | -0.065 |
| F1 Score | 0.539 | 0.500 | +0.039 |
| AUC-ROC | 0.913 | 0.901 | +0.012 |
| False Positive Rate | 0.031 | 0.044 | -0.013 |
| False Negative Rate | 0.219 | 0.154 | +0.065 |
| Alert Rate | 4.8% | 6.9% | -2.1 pts |
The validation set contains 200,000 driving windows, with 6,000 true safety-critical events (3.0% prevalence). At the current threshold, the model produces 4,800 true positives, 1,200 false positives, 1,200 false negatives, and 192,800 true negatives.
Leadership wants a recommendation on how to think about false positives versus false negatives in this safety system and whether the current threshold is appropriate.