NewsPulse uses a text classification model to route incoming customer support messages into one of two classes: urgent escalation or standard queue. The model performs well Monday through Friday, but support leaders report a sharp quality drop on weekends, causing delayed handling of urgent tickets.
| Metric | Weekdays | Weekends | Change |
|---|---|---|---|
| Accuracy | 0.91 | 0.78 | -0.13 |
| Precision (urgent) | 0.84 | 0.69 | -0.15 |
| Recall (urgent) | 0.88 | 0.57 | -0.31 |
| F1 Score (urgent) | 0.86 | 0.62 | -0.24 |
| AUC-ROC | 0.93 | 0.81 | -0.12 |
| Avg. urgent ticket rate | 18% | 27% | +9 pts |
| Daily ticket volume | 24,000 | 11,000 | -54% |
Weekend recall for urgent tickets has fallen to 57%, meaning the model misses a large share of truly urgent messages when staffing is already thinner. Product and operations teams want to know whether this is caused by data drift, label quality, traffic mix changes, threshold issues, or a pipeline problem.