Design Multimodal Data Labeling Router

Product Context

LabelFlow is a managed data labeling platform used by enterprise ML teams to annotate text, images, audio, video, and document/PDF data. Customers upload tasks, and the platform must route each item to the right annotator queue, pre-label model, and quality-control workflow while keeping turnaround time and cost low.

Scale

Task

Design an end-to-end ML system for a multi-input-type labeling platform. Your design should address:

1. How to represent heterogeneous inputs and build a multi-stage decision system for task routing, annotator matching, and optional pre-label generation.
2. The online and offline architecture, including feature stores, model training, batch vs real-time inference, and feedback logging.
3. How to support cold-start for new customers, new task types, and new annotators while meeting latency and cost constraints.
4. How to evaluate the system offline and online across quality, throughput, cost, and fairness.
5. The main failure modes, especially feature drift, training-serving skew, and quality regressions by modality or customer segment.

Constraints

• Some customers require data residency and cannot send raw data across regions.
• Human-in-the-loop quality is the product: wrong routing increases rework cost and SLA misses.
• Raw video/audio inference is expensive; heavy multimodal models cannot run synchronously on every request.
• New task schemas appear weekly, and label taxonomies can differ by customer.
• The platform must provide auditable assignment decisions for enterprise compliance.