Design Low-Latency Support Triage

Product Context

Sparksoft Support Cloud routes inbound customer conversations to the best help content, automation, or human queue. The platform serves enterprise support teams that need fast responses while keeping inference and infrastructure costs predictable.

Scale

Task

Design an end-to-end ML system for Sparksoft Support Cloud that, for each incoming customer message, selects the best next action: retrieve relevant help content, rank likely resolution paths, and optionally re-rank for business rules such as SLA priority, language, and compliance. Your design should explicitly balance cost, performance, and reliability rather than optimizing only model quality.

Address the following:

1. Clarify the product objective, prediction target, and success metrics for automated support triage.
2. Propose a multi-stage architecture (retrieval → ranking → re-ranking) and explain which parts run online vs. batch.
3. Size the system and give a latency and cost budget across stages, including feature serving and model inference.
4. Choose models for each stage and justify why they fit the scale and cost constraints.
5. Define offline and online evaluation, including how you would measure cost-efficiency and guard against regressions.
6. Identify key failure modes such as feature drift, training-serving skew, stale indexes, and degraded fallbacks.

Constraints

• 40% of requests come from the top 200 enterprise tenants, creating strong traffic skew.
• New knowledge-base articles must become retrievable within 10 minutes.
• Some tenants prohibit cross-tenant training data leakage and require regional data residency.
• GPU capacity is limited; most serving must run on CPU, with selective use of heavier models.
• The system must degrade gracefully to deterministic routing rules if ML components are slow or unavailable.