Context

ShopFlow uses a binary classifier to predict whether a customer support ticket should be escalated to the specialist operations team. The new gradient-boosted model replaced a logistic regression baseline in a shadow test, but leadership cares about operational improvement, not just offline lift. Specialist capacity is fixed, and missed escalations create SLA breaches and customer churn risk.

Current Performance

The Problem

The new classifier finds more truly urgent tickets, but it also sends many more tickets to specialists and exceeds daily review capacity. You need to determine whether the model is actually improving operations or simply shifting work downstream.

Requirements

1. Interpret whether the new model is better for operations, not just offline metrics.
2. Quantify the trade-off between false negatives and false positives using the data provided.
3. Assess whether the current decision threshold is appropriate given capacity limits.
4. Identify likely failure modes and what additional analysis you would run before launch.
5. Recommend concrete changes to model usage, evaluation, or thresholding.

Constraints

• Specialist team can handle at most 1,300 escalations per day.
• A false negative causes an average of $42 in downstream cost.
• A false positive costs $6 in specialist handling time.
• Ticket labels are finalized with a 7-day delay.