Business Context
LexiPrep, an interview preparation platform, wants to automatically route user-submitted AI questions to the correct framework family so editors can review content faster. A common failure case is confusion between framework ecosystems such as NGA and ADK, especially when users ask comparative questions in short, ambiguous text.
Data
You are given a corpus of 180,000 historical question titles and descriptions from AI learning forums and internal content libraries.
- Task: classify each question into one of 4 labels:
NGA, ADK, Both/Comparison, Other AI Framework
- Text length: 8-220 words, median 34 words
- Language: English only
- Label distribution: 28% NGA, 24% ADK, 18% Both/Comparison, 30% Other AI Framework
- Noise: abbreviations, inconsistent capitalization, product aliases, and partial framework names are common
Success Criteria
A production-ready solution should achieve macro-F1 >= 0.86, with F1 >= 0.90 on the Both/Comparison class because those questions are most valuable for editorial review.
Constraints
- Inference latency must stay below 80 ms per query
- The model must run on a single CPU-backed API pod or one small GPU
- Editors need probability scores for fallback manual review on low-confidence predictions
Requirements
- Build a multi-class text classification pipeline for framework-related questions.
- Design preprocessing for abbreviations, aliases, punctuation noise, and short-text ambiguity.
- Implement a modern Python solution using a transformer fine-tuning workflow.
- Evaluate class-wise performance and propose a confidence-thresholding strategy.
- Explain how your system distinguishes direct comparison questions from single-framework questions.
- Describe how you would monitor drift as new framework names enter the taxonomy.
