Context

StreamWave uses a binary classification model to predict which paid subscribers are likely to cancel in the next 30 days so the CRM team can send retention offers. The current model was deployed 2 months ago, but leadership is concerned that campaign ROI is below target despite seemingly strong overall accuracy.

Current Performance

The Problem

The model still shows high accuracy, but it is identifying far fewer churners than expected. As a result, many at-risk users are not receiving retention offers, and the retention team wants to know whether the issue is threshold choice, class imbalance, drift, or segment-specific underperformance.

Requirements

1. Interpret the current metrics and explain why accuracy may be misleading here.
2. Diagnose the most likely reasons recall dropped from 0.71 to 0.36.
3. Use the confusion matrix and business constraints to assess whether the current threshold is appropriate.
4. Identify which additional validation or error analysis you would run.
5. Recommend specific model or process improvements to improve performance.

Constraints

• Retention budget supports at most 15,000 offers per month.
• Each offer costs $8.
• Saving a true churner is worth $95 in expected annual gross profit.
• Product and pricing changed 6 weeks after deployment.