Context

HelpHive uses a multiclass text classifier to route incoming customer support tickets into four queues: Billing, Technical, Shipping, and Account Access. The team launched the model to reduce manual triage time, but operations leaders noticed that some urgent Account Access tickets are still being misrouted, creating customer delays.

Current Performance

Per-Class Metrics

The Problem

The VP of Support is concerned that overall accuracy looks strong, but the model may be failing on the most operationally sensitive class. You need to determine which metrics best reflect system quality and whether the model is acceptable for production use.

Requirements

1. Explain which metrics you would prioritize for this system and why.
2. Interpret the gap between overall accuracy (0.86) and macro recall (0.69).
3. Diagnose what the per-class results suggest about model behavior.
4. Recommend specific evaluation and validation steps before the next release.
5. Propose metric targets aligned to the business risk of misrouting Account Access tickets.

Constraints

• 12,000 tickets/day
• Misrouted Account Access tickets add 18 minutes average delay
• Manual review capacity is limited to 1,500 tickets/day
• The team wants a simple metric dashboard for non-technical stakeholders