Validate Customer Churn Classifier

Context

StreamWave uses a binary classification model to predict which subscribers are likely to cancel in the next 30 days so the retention team can send save offers. A logistic regression model was trained on 1.2M historical users, then deployed last quarter. Leadership is concerned because the model shows strong overall accuracy, but retention campaign results have been weaker than expected.

Current Performance

The Problem

The current model appears better on accuracy and AUC-ROC, but it identifies far fewer churners than the previous baseline. The retention team has budget to contact only 50,000 users per month, so the company needs a validation approach that reflects business goals rather than relying on a single metric.

Requirements

1. Explain which concepts and metrics you would use to validate this model properly.
2. Interpret whether the current model is actually better than the baseline for the churn use case.
3. Use the metrics to diagnose what the model is optimizing for and what risks that creates.
4. Recommend how you would validate threshold choice, data splits, and model generalization.
5. Propose improvements to evaluation so future model launches are compared consistently.

Constraints

• Monthly retention outreach capacity is capped at 50,000 users.
• Missing a true churner is estimated to cost $85 in lost margin.
• Contacting a non-churner costs $4 in incentive spend and support overhead.