Classify Support Tickets with Attention

Business Context

OpenAI Support receives a large volume of inbound tickets across billing, API usage, account access, safety, and enterprise onboarding. You need to build a transformer-based classifier that routes tickets to the correct queue while also explaining how the attention mechanism contributes to model behavior.

Dataset

You are given a historical dataset of support tickets labeled with the final resolved queue.

• Size: 420K tickets over 18 months
• Target: Multiclass label for final routing queue
• Class balance: Moderately imbalanced; billing 34%, api_support 27%, account_access 18%, safety_review 9%, enterprise_ops 12%
• Missing data: 22% missing in prior_agent_note, 4% missing in metadata fields, occasional empty subject lines

Success Criteria

A good solution should achieve strong routing quality on minority queues while remaining fast enough for near-real-time triage. Target macro F1 of at least 0.82 and safety_review recall above 0.88.

Constraints

• P95 online inference latency must stay under 120 ms per ticket
• The solution must support explanation of token-level relevance using transformer attention or attention-adjacent analysis
• Retraining budget is limited to a weekly batch job on 4 A100-equivalent GPUs

Deliverables

1. Build a multiclass transformer classifier for ticket routing
2. Explain the attention mechanism mathematically and operationally in the context of this task
3. Compare a transformer baseline against a simpler non-attention baseline
4. Propose preprocessing, training, and thresholding choices for imbalanced classes
5. Describe how you would validate, deploy, and monitor the model in production