Design AI Customer Support Routing

Product Context

HelpFlow is a SaaS customer support platform used by large e-commerce and fintech companies. Design an AI-driven support system that can understand incoming user issues, retrieve relevant knowledge, rank next-best actions, and decide whether to answer automatically, suggest an agent reply, or route to a human specialist.

Scale

Task

1. Clarify the product requirements and define what decisions the ML system makes versus what remains rule-based or human-controlled.
2. Propose a scalable multi-stage architecture for intake, retrieval, ranking, re-ranking, and final action selection.
3. Choose models for each stage and explain tradeoffs across quality, latency, and cost.
4. Design the offline and online data pipelines, including labels, feedback loops, and feature storage.
5. Define evaluation, experimentation, monitoring, and rollback strategy.
6. Identify major failure modes, especially feature drift, training-serving skew, and unsafe or low-confidence automation.

Constraints

• Live chat responses must meet p99 < 700ms; agent-assist suggestions should ideally appear in < 300ms.
• Some tenants require data isolation and cannot share raw ticket text across customers.
• Personally identifiable information must be redacted before long-term storage or model training.
• Knowledge content changes frequently; freshness matters for refunds, policy changes, and outages.
• Wrong auto-responses are costly, so the system must support confidence thresholds and safe fallback to humans.