ShopEase uses a binary classification model to predict which customer support tickets should be escalated to a senior retention team because they are likely to result in subscription cancellation within 30 days. The model performs reasonably well offline, but business leaders are unsure how to interpret the metrics in terms of staffing, saved revenue, and customer experience.
| Metric | Current Model | Previous Rules System |
|---|---|---|
| Accuracy | 0.91 | 0.88 |
| Precision | 0.62 | 0.41 |
| Recall | 0.78 | 0.55 |
| F1 Score | 0.69 | 0.47 |
| AUC-ROC | 0.84 | 0.69 |
| Tickets flagged per month | 12,000 | 18,500 |
| Actual churn-risk tickets per month | 9,500 | 9,500 |
| Avg monthly revenue per saved customer | $120 | $120 |
The VP of Customer Success wants to know whether this model is good enough to expand nationwide. They do not want a metric-by-metric definition; they want a business interpretation of what the current performance means, what tradeoffs exist, and whether threshold changes would improve outcomes.