Design Real-Time Video Analytics Platform

Product Context

VisionGrid provides video analytics for retail chains, warehouses, and campuses. Customers upload or stream camera feeds and expect near-real-time detection, search, and alerting for events such as person entry, vehicle counting, safety violations, and suspicious activity.

Scale

Task

Design the end-to-end ML system for this platform. Address the following:

1. Clarify the product requirements and define the primary ML tasks, outputs, and users.
2. Estimate system scale and propose a multi-stage architecture for ingest, candidate retrieval, ranking, and alert generation.
3. Choose models for each stage and explain online vs batch inference decisions.
4. Design the training, feature, and feedback pipelines, including how labels are created from delayed human review.
5. Define offline and online evaluation, monitoring, and rollout strategy.
6. Identify major failure modes, especially around feature drift, training-serving skew, camera heterogeneity, and operational outages.

Constraints

• Cameras vary widely in resolution, frame rate, lighting, and placement; many are low quality.
• Raw video retention is limited: 7 days hot, 30 days warm, then derived features only for compliance and cost.
• Some customers require on-prem or edge inference for privacy; others allow cloud processing.
• False negatives on safety alerts are costly, but excessive false positives cause alert fatigue.
• Serving cost must stay below $0.015 per camera-hour on average.