Design AI Support Reply Ranking

Product Context

AssistFlow is a client-facing AI support platform used by enterprise customers. End users type a question in a web or mobile chat, and the system must retrieve relevant knowledge, rank candidate answers, and return a high-quality response with low latency.

Scale

Task

Design an end-to-end ML system for this application, from ingestion through production monitoring. Your design should address:

1. How data is ingested, cleaned, versioned, and made available for training and serving
2. A multi-stage online architecture for retrieval, ranking, and optional final re-ranking or answer synthesis
3. Model choices for each stage, including how you handle tenant isolation, freshness, and cold start
4. Training, offline evaluation, and online experimentation strategy
5. Monitoring, alerting, rollback, and failure handling in production

Constraints

• Tenant data must remain logically isolated; cross-tenant leakage is a hard compliance failure
• 35% of documents are updated within 24 hours, so freshness matters
• GPU budget is limited: average inference cost must stay below $0.012 per request
• Some customers require explainability: the system must expose citations and confidence signals
• The product must degrade gracefully if the generative stage is unavailable; retrieval-only answers are acceptable
• Training labels are partially delayed and noisy because many users do not explicitly rate responses