Context

AcmeCloud uses a binary classification model to route incoming support tickets as urgent or non-urgent. The model is a logistic regression classifier used in production for 6 weeks, but support managers report that too many urgent tickets are still reaching the standard queue.

Current Performance

The Problem

Leadership sees high overall accuracy and assumes the model is performing well, but the operations team is concerned because many truly urgent tickets are not being escalated. You need to assess whether the model is actually effective for the business goal and recommend how to improve evaluation and model performance.

Requirements

1. Interpret the current metrics and explain whether the model is acceptable for production use.
2. Explain why accuracy may be misleading in this setting.
3. Use the metric values to diagnose the main weakness of the model.
4. Recommend which metrics should be prioritized going forward and why.
5. Propose specific actions to improve model performance and validation.

Constraints

• Missing an urgent ticket delays response by 4-6 hours on average.
• Reviewing a false positive urgent ticket adds only ~3 minutes of agent time.
• The support team can handle up to 25% of daily tickets in the urgent queue without hiring.