Diagnose Customer Churn Model Gaps

Context

StreamWave uses a binary classification model to predict which subscribers are likely to cancel in the next 30 days so the CRM team can send retention offers. The team recently expanded the model from one market to three, but campaign ROI dropped even though top-line validation accuracy still looks acceptable.

Current Performance

The Problem

The model still appears "good" on accuracy, but it is missing a large share of actual churners. Marketing can contact at most 30,000 customers per month, and each retention offer costs $12. A missed churner is estimated to cost $180 in lost gross margin.

Requirements

1. Interpret what the metric pattern says about model behavior and why accuracy is misleading here.
2. Identify likely reasons recall fell sharply while AUC-ROC declined only modestly.
3. Use the provided results to explain the business impact of false negatives vs. false positives.
4. Propose a structured error analysis plan to identify meaningful churn patterns the model is missing.
5. Recommend specific model, threshold, or validation changes to improve performance under the campaign-capacity constraint.

Constraints

• CRM team can contact at most 30,000 users per month
• Offer cost is $12 per contacted user
• Model retraining can happen only once per month
• Leadership wants measurable lift within one quarter