Product Context

HelpHub is a SaaS customer support platform used by mid-market e-commerce merchants. Today, most incoming support tickets are manually reviewed by agents; the company wants to move toward ML-assisted triage first, then fully automated resolution for low-risk cases.

Scale

Task

Design an end-to-end ML system that decides whether a ticket should be:

1. routed to a human,
2. shown an agent-assist recommendation, or
3. fully auto-resolved with a suggested action or response.

Your design should address:

1. Requirements and scope: define what decisions are automated vs kept human-in-the-loop, and what success means.
2. System architecture: propose the online and offline architecture, including retrieval, ranking, and final decisioning.
3. Modeling choices: choose models for candidate retrieval, ranking, and automation eligibility, with clear tradeoffs.
4. Data and training: define labels, feedback loops, feature pipelines, retraining cadence, and how you avoid training-serving skew.
5. Evaluation and launch: explain offline metrics, online experimentation, guardrails, and a staged rollout from assistive to autonomous mode.
6. Failure modes: identify key risks such as bad auto-resolutions, feature drift, policy violations, and outages.

Constraints

• Only low-risk intents (refund status, order tracking, password reset, FAQ-style issues) are eligible for full automation at launch.
• Some tickets contain PII and payment data; raw text retention is limited by compliance policy.
• Merchants require auditability: every automated action must log the evidence and model version used.
• Cost target is under $0.004 per resolved ticket on average, so expensive LLM calls cannot be used on every request.
• New policies and macros are updated hourly, so the system must handle freshness without full retraining for every change.