Explain Attention in Support Ticket Transformers

Business Context

HelpDeskPro is replacing a legacy TF-IDF ticket router with a Transformer-based classifier for customer support tickets. Before deployment, the NLP team wants candidates to demonstrate practical understanding of self-attention and how it affects implementation, interpretability, and performance.

Data

You are given 850,000 historical English support tickets from SaaS customers. Each ticket contains a subject and body, with lengths ranging from 8 to 900 tokens (median: 96). Labels cover 6 routing queues: Billing, Account Access, Bug Report, Feature Request, Integration, and Other. The class distribution is moderately imbalanced, with Bug Report and Account Access making up 58% of all tickets.

Success Criteria

A strong solution should correctly explain self-attention mathematically and operationally, connect it to token interactions in support text, and show how to implement and inspect it in a modern Transformer pipeline. Good enough means the proposed model reaches at least 0.84 macro-F1 on a held-out set while keeping p95 inference latency under 120 ms for sequences up to 256 tokens.

Constraints

• Deployment target is a single T4 GPU and CPU fallback for batch jobs
• Explanations must be understandable to ML engineers and product stakeholders
• Long tickets must be truncated or chunked consistently
• The solution should use standard Python NLP tooling and be production-oriented

Requirements

1. Explain self-attention using queries, keys, values, scaling, and softmax.
2. Show how self-attention helps model dependencies in support tickets better than bag-of-words features.
3. Build a Transformer-based text classification pipeline in Python.
4. Include preprocessing for noisy ticket text, URLs, stack traces, and repeated signatures.
5. Evaluate both model quality and attention behavior on representative examples.