Optimize Underperforming Churn Model

Context

StreamBox uses a binary classification model to predict which paid subscribers are likely to cancel in the next 30 days so the CRM team can send retention offers. A gradient boosting model replaced a logistic regression baseline six weeks ago, but retention lift from the campaign has been weaker than expected.

Current Performance

The Problem

The model still shows acceptable overall accuracy, but it is missing a large share of churners and the retention team believes too few at-risk users are being targeted. Leadership wants to know whether the issue is thresholding, calibration, segment drift, feature quality, or model overfitting.

Task

1. Diagnose what the current metrics suggest about model behavior.
2. Identify likely root causes for the drop in recall and F1.
3. Explain what additional validation or error analysis you would run.
4. Recommend specific actions to improve performance.
5. Discuss whether you would change the decision threshold before retraining.

Constraints

1. CRM can contact at most 35,000 users per week.
2. A false positive retention offer costs $4.
3. A false negative churn miss costs an estimated $95 in lost contribution margin.
4. Full retraining and deployment takes 10 days.