ChurnGuard, a SaaS company, recently deployed a logistic regression model to predict customer churn based on user behavior and engagement metrics. After three months in production, the model's performance has raised concerns as it consistently flags a low number of churn cases.
| Metric | At Launch (3 months ago) | Current | Change |
|---|---|---|---|
| Precision | 0.90 | 0.88 | -2.2% |
| Recall | 0.40 | 0.35 | -12.5% |
| F1 Score | 0.55 | 0.50 | -9.1% |
| AUC-ROC | 0.85 | 0.80 | -5.9% |
| Daily Churn | 1,000 | 900 | -10% |
| Total Users | 50,000 | 50,000 | 0% |
Despite maintaining a high precision of 88%, the model's recall has dropped from 40% to 35%, resulting in a significant number of missed churn cases. This decline in recall raises alarms for the customer success team, as they rely on accurate predictions to proactively engage customers at risk of leaving.