Context

NimbusCRM uses a binary classification model to score inbound leads for its B2B sales team. Leads with scores above a threshold are routed to account executives for immediate follow-up; the rest go to lower-cost email nurture. The current threshold was set conservatively at launch, but sales leadership believes too many high-value leads are being missed.

Current Performance

Validation set: 50,000 leads from the last quarter. Positive class = lead converts to a qualified sales opportunity within 30 days.

Additional facts:

• Base rate of qualified opportunities: 9.8%
• Model AUC-ROC: 0.81
• Brier score: 0.17
• Sales team capacity: 9,000 routed leads/month
• Average AE follow-up cost per routed lead: $45
• Average gross profit from a converted qualified opportunity: $4,200

The Problem

You need to recommend the best operating threshold for routing leads to sales, not just the threshold with the best offline metric.

Requirements

1. Evaluate the threshold tradeoff using both model metrics and business impact.
2. Recommend a threshold and justify it quantitatively.
3. Explain whether calibration matters before finalizing the threshold.
4. Describe what error analysis you would run on false positives and false negatives.
5. Propose improvements if no single threshold is satisfactory.

Constraints

• Sales cannot handle more than 9,000 routed leads/month.
• Missing a high-intent lead is costly, but excessive low-quality routing reduces AE productivity.
• Threshold policy should be simple enough to operationalize within one quarter.