Evaluate Churn Model Utility

Context

Streamly, a subscription video platform, uses a binary classifier to predict which paid users are likely to churn in the next 30 days so the growth team can send retention offers. The model scores all active subscribers weekly, but leadership is unsure whether the model is actually useful because campaign costs have risen while retained revenue has not improved as expected.

Current Performance

The Problem

The growth team wants to know whether this model is good enough to drive retention spend, whether the threshold is wrong, or whether the model is poorly calibrated for decision-making.

Requirements

1. Interpret whether the model is useful for a retention team, not just whether it has good offline metrics.
2. Assess the tradeoff between ranking quality, threshold choice, and campaign ROI.
3. Diagnose what the gap between predicted and actual churn implies.
4. Recommend how you would validate incremental business value.
5. Propose concrete model and policy improvements.

Constraints

• Weekly retention budget is capped at $300,000.
• Only one offer can be shown per user per month.
• False positives waste incentive spend and may train users to wait for discounts.
• Missing true churners reduces subscriber growth and LTV.