Context

VisionOps uses a computer vision pipeline to detect and track forklifts, pallets, and workers in warehouse video. Before rendered tracks and bounding boxes are shown to human operators, the team wants a validation layer that filters out incorrect overlays because bad renders reduce operator trust and trigger unnecessary escalations.

Current Performance

The Problem

The detector appears strong on box precision, but operators still report that some rendered tracks are visibly wrong: boxes drift, IDs switch after occlusion, and confidence scores are over-trusted. You need to design an evaluation and validation approach that determines whether a rendered track or box is correct before it reaches operators.

Requirements

1. Explain which offline metrics best capture “render correctness” for both boxes and tracks.
2. Diagnose why high box precision is not sufficient for operator-facing quality.
3. Propose a pre-operator validation strategy using confidence, temporal consistency, and thresholding.
4. Recommend how to calibrate scores and choose thresholds under business constraints.
5. Describe what error slices you would inspect first and how you would measure improvement.

Constraints

• Operators can review at most 12,000 rendered events per day.
• Missing a true safety event is more costly than sending an extra review.
• The validation layer must add under 40 ms latency per frame.