HelpHive uses a text classification model to label incoming customer support tickets as Urgent or Non-Urgent so urgent cases can be prioritized by agents. The model was recently deployed, but operations leaders report that too many urgent tickets are still reaching the standard queue.
The model was evaluated on a held-out test set of 10,000 labeled tickets.
| Metric | Value |
|---|---|
| Accuracy | 0.91 |
| Precision (Urgent) | 0.78 |
| Recall (Urgent) | 0.52 |
| F1 Score (Urgent) | 0.62 |
| AUC-ROC | 0.87 |
| Log Loss | 0.31 |
| Urgent ticket rate | 12% |
| Decision threshold | 0.50 |
Although overall accuracy looks strong, the support team is concerned because many truly urgent tickets are not being flagged. Missing an urgent ticket can delay resolution for outages, billing failures, or security issues, while false positives mainly create extra review work.