Context

VisionWatch runs a live dashboard that shows safety detections from warehouse cameras. A binary object detection model scores each candidate event from 0 to 1, and the dashboard only displays detections above a chosen threshold. Operations managers complain that the current setting creates too many distracting alerts, while safety leads worry that important events are being hidden.

Current Performance

Validation set: 12,000 candidate events, 600 true incidents (5% prevalence).

Additional model metrics on the same validation set:

The Problem

You need to recommend the right threshold for a live dashboard where users can only review about 250 alerts per day. Missing a true safety incident is estimated to cost 20x more than showing a false alert, but repeated false alerts reduce trust and dashboard usage.

Requirements

1. Recommend a threshold and justify it using the metrics above.
2. Explain the precision-recall tradeoff for this dashboard use case.
3. Assess whether calibration matters for threshold selection here.
4. Describe what additional offline and online validation you would run before changing the threshold.
5. Propose how you would monitor performance after rollout.

Constraints

• Review capacity is capped at 250 alerts/day.
• Threshold changes can be deployed daily, but model retraining happens monthly.
• The dashboard is used for real-time operational decisions, so alert fatigue is a material business risk.