Business Context
NimbusDesk, a B2B SaaS support platform, receives thousands of inbound support tickets each day across products that launch faster than labeled training data can be created. The operations team wants a zero-shot NLP system that can assign tickets to the correct queue before a supervised classifier is available.
Data Characteristics
- Volume: ~80,000 historical tickets per month, but only 5-10% are manually labeled for new categories
- Text length: 8-220 words per ticket, median 42 words
- Language: English only for the first release
- Label distribution: Highly skewed; common classes include billing, login, API, and bug report, while new classes appear weekly
- Input format: Short subject line plus free-text description, often noisy and inconsistent
Success Criteria
A good solution should route tickets with macro-F1 73 on established categories and provide useful predictions for unseen categories using only label names or short label descriptions. Top-2 recall should be high enough for human-in-the-loop triage, with inference latency under 150 ms per ticket.
Constraints
- New categories may have zero labeled examples at launch
- The model must run in a Python service on a single GPU or CPU fallback
- Predictions should be explainable enough for support managers to audit category assignments
Requirements
- Explain what zero-shot learning is in NLP and when it is preferable to supervised fine-tuning.
- Build a zero-shot classifier for support ticket routing using modern Python tooling.
- Define a preprocessing pipeline for short, noisy ticket text.
- Compare a zero-shot NLI approach against a TF-IDF baseline.
- Describe how you would evaluate performance on both seen and unseen categories.
- Propose when you would transition from zero-shot to few-shot or fully supervised training.
