Business Context

NovaDesk sells AI-powered customer support automation to mid-market enterprises. Sales leadership wants an NLP system that detects skeptical customer language in call transcripts and emails so account teams can respond with evidence, case studies, and human escalation before deals stall.

Data

You have 180,000 historical text interactions labeled by sales enablement specialists into 4 classes: High Skepticism (8%), Moderate Skepticism (22%), Neutral/Exploratory (46%), and Positive/Confident (24%). Inputs include call transcript segments, follow-up emails, and meeting notes. Text is primarily English, with 6% mixed English-Spanish content. Length ranges from 10 to 1,200 tokens, with a median of 140 tokens. Labels are noisy because some skepticism is implicit rather than explicit.

Success Criteria

A good solution should achieve macro-F1 >= 0.82, recall >= 0.90 on High Skepticism, and support actionable confidence scoring so sales reps know when to trigger a proof-oriented response playbook. The model should also produce interpretable outputs for skeptical stakeholders.

Constraints

• Inference latency under 150 ms per interaction
• Deployable on a single GPU or CPU fallback
• No external API calls due to customer confidentiality
• Must support weekly retraining with newly labeled interactions

Requirements

1. Build a multi-class text classification system to detect customer confidence level toward AI capabilities.
2. Design a preprocessing pipeline for transcripts, emails, and noisy CRM notes.
3. Implement a modern Python solution using a transformer baseline and a simpler interpretable benchmark.
4. Address class imbalance and noisy labels.
5. Define how you would calibrate probabilities and choose thresholds for escalation.
6. Propose an evaluation plan, including offline metrics and qualitative error analysis.
7. Explain how this system would help build confidence when customers are skeptical about AI capabilities.