Context

StreamWave uses a binary classification model to predict whether a subscriber will churn in the next 30 days so the retention team can send discounts and outreach. A logistic regression model was recently deployed, and leadership is concerned because the headline accuracy looks strong, but churn campaigns are not improving retention as expected.

Current Performance

The Problem

The model appears accurate overall, but it identifies only a small share of actual churners. The retention team has budget to contact at most 6,000 customers per month, so threshold choice and metric interpretation matter more than accuracy alone.

Requirements

1. Explain how you would evaluate this classification model beyond accuracy.
2. Interpret what the current metric values imply about model behavior.
3. Use the confusion matrix and class imbalance to explain why business outcomes may be disappointing.
4. Recommend how you would validate the model before changing the threshold or retraining.
5. Propose specific improvements to increase business value while respecting outreach capacity.

Constraints

• Monthly outreach capacity: 6,000 customers
• Cost of unnecessary retention offer: $8 per false positive
• Estimated value of saving a true churner: $120
• Churn labels arrive with a 30-day delay