Choose Metrics for Churn Model

Context

StreamBox has deployed a binary classification model to predict whether a subscriber will churn in the next 30 days. The product team wants to use the model to trigger retention offers, but leadership is concerned that the team is reporting only accuracy while the business cost of missed churners is much higher than the cost of sending an unnecessary offer.

Current Performance

The Problem

The current model looks better on accuracy and AUC-ROC, but it is catching fewer actual churners. The retention team has budget to contact only 25,000 users per month, so the company needs to decide which metrics should drive model selection and thresholding.

Requirements

1. Explain which metrics you would use to evaluate this churn model and why.
2. Interpret the current results relative to the previous model and identify the main tradeoff.
3. Use the provided metrics and confusion matrix to assess whether the current model is suitable for deployment.
4. Recommend how you would choose a decision threshold given the retention budget.
5. Suggest concrete next steps to improve evaluation and model performance.

Constraints

• Monthly retention budget covers at most 25,000 offers.
• Each unnecessary offer costs $4.
• Each missed churner is estimated to cost $65 in lost gross margin.
• Churn labels are finalized 30 days after prediction, so feedback is delayed.