Context

StreamWave uses a binary classification model to predict whether a subscriber will churn in the next 30 days so the CRM team can send retention offers. A logistic regression model was recently deployed, and leadership wants to know if it is actually performing well enough to use in production.

Current Performance

The model was evaluated on a holdout set of 10,000 subscribers with a 20% churn rate.

Confusion matrix counts at the current threshold of 0.50:

The Problem

The VP of Growth sees 84% accuracy and believes the model is strong. However, the retention team is concerned that nearly half of churners may be missed. You need to assess whether this model is truly performing well and whether the current operating threshold is appropriate.

Requirements

1. Interpret the model's performance using the provided metrics, not accuracy alone.
2. Explain what the confusion matrix implies for business outcomes.
3. Identify the main weaknesses of the current model behavior.
4. Recommend how you would validate the model further before broader rollout.
5. Propose concrete improvements to the model or decision threshold.

Constraints

• Retention offers cost $8 per targeted user.
• A missed churner costs an estimated $120 in lost contribution margin.
• The CRM team can contact at most 1,800 users per week.