Choose Metrics for Churn Prediction

Context

Streamly, a subscription video platform, uses a binary classification model to predict whether a customer will churn in the next 30 days so the retention team can send discounts or outreach. The current model has strong overall ranking performance, but the marketing team says the chosen success metric may not reflect the real business tradeoff.

Current Performance

The Problem

The team currently reports accuracy to leadership because it is easy to explain, but only 12% of customers churn. Retention offers cost $8 per contacted customer, while saving a true at-risk customer generates $95 in expected gross profit. The retention team can contact at most 40,000 customers per month.

Requirements

1. Explain why accuracy may be misleading for this churn problem.
2. Identify which evaluation metric(s) should be prioritized and why.
3. Discuss how business costs and outreach capacity affect metric choice.
4. Recommend whether Streamly should optimize for precision, recall, F1, AUC-ROC, or another approach.
5. Propose how you would choose a decision threshold for deployment.

Constraints

• Monthly contact capacity is fixed at 40,000 customers.
• False positives waste retention budget and may annoy loyal users.
• False negatives mean losing subscribers who could have been retained.