In developing a customer churn prediction model for a SaaS company, the data science team has encountered unexpected fluctuations in model performance, suggesting potential issues with data reliability. The model is trained on historical usage data, customer demographics, and engagement metrics to predict churn rates effectively.
| Metric | Current Value | Baseline Value | Change |
|---|---|---|---|
| Accuracy | 0.78 | 0.85 | -8.2% |
| Precision | 0.75 | 0.80 | -6.25% |
| Recall | 0.70 | 0.78 | -10.3% |
| F1 Score | 0.72 | 0.79 | -8.86% |
| Churn Rate | 12% | 10% | +20% |
The significant drop in performance metrics indicates that the model struggles to generalize on recent data, raising concerns about the reliability of the data sources used for training and validation. The team must diagnose the underlying issues affecting data quality and model performance.