Detect Overfitting in Churn Model

Context

StreamWave uses a gradient boosting classifier to predict which subscribers are likely to cancel within 30 days so the retention team can send discounts. The model performed well during training, but leadership is concerned that offline results may not generalize because recent campaign performance has been weaker than expected.

Current Performance

The Problem

The data science manager wants to know whether this model is overfitting, how confident you are in that diagnosis, and what should be changed before the next deployment. You should focus on interpreting the gap between training and validation performance rather than proposing a brand-new modeling approach.

Requirements

1. Determine whether the model is overfitting and explain which metrics support your conclusion.
2. Identify at least 3 likely causes of the performance gap.
3. Explain what additional validation checks you would run to confirm the diagnosis.
4. Recommend 3-5 concrete changes to reduce overfitting.
5. Discuss the business impact of deploying the current model as-is.

Constraints

• Retention budget allows offers to only 8% of active subscribers each week.
• False positives waste discount spend; false negatives miss preventable churn.
• Retraining can happen weekly, but feature engineering changes take one sprint.