Classify AI Framework Questions

Business Context

LexiPrep, an interview preparation platform, wants to automatically route user-submitted AI questions to the correct framework family so editors can review content faster. A common failure case is confusion between framework ecosystems such as NGA and ADK, especially when users ask comparative questions in short, ambiguous text.

Data

You are given a corpus of 180,000 historical question titles and descriptions from AI learning forums and internal content libraries.

• Task: classify each question into one of 4 labels: NGA, ADK, Both/Comparison, Other AI Framework
• Text length: 8-220 words, median 34 words
• Language: English only
• Label distribution: 28% NGA, 24% ADK, 18% Both/Comparison, 30% Other AI Framework
• Noise: abbreviations, inconsistent capitalization, product aliases, and partial framework names are common

Success Criteria

A production-ready solution should achieve macro-F1 >= 0.86, with F1 >= 0.90 on the Both/Comparison class because those questions are most valuable for editorial review.

Constraints

• Inference latency must stay below 80 ms per query
• The model must run on a single CPU-backed API pod or one small GPU
• Editors need probability scores for fallback manual review on low-confidence predictions

Requirements

1. Build a multi-class text classification pipeline for framework-related questions.
2. Design preprocessing for abbreviations, aliases, punctuation noise, and short-text ambiguity.
3. Implement a modern Python solution using a transformer fine-tuning workflow.
4. Evaluate class-wise performance and propose a confidence-thresholding strategy.
5. Explain how your system distinguishes direct comparison questions from single-framework questions.
6. Describe how you would monitor drift as new framework names enter the taxonomy.