Context

ShopFlow uses a binary classifier to predict whether an order should be sent to manual fraud review before fulfillment. The customer is unhappy with current outcomes: analysts say too many reviewed orders are legitimate, while operations says some fraudulent orders are still shipping.

Current Performance

The Problem

The customer asks whether they should tune the decision threshold, redesign the review workflow, or accept the current model behavior because the model itself may already be near its practical limit. You need to interpret the metrics and recommend the best path.

Requirements

1. Determine whether threshold tuning alone can satisfy both fraud and operations goals.
2. Explain what the unchanged AUC but changing precision/recall implies.
3. Recommend when a workflow change is preferable to model tuning.
4. Identify whether calibration issues affect threshold decisions.
5. State when accepting current behavior is reasonable.

Constraints

• Manual review capacity cannot exceed 1,600 orders/day this quarter.
• False negatives cost about $120 per missed fraudulent order.
• False positives cost $6 in analyst time plus customer-delay risk.
• Retraining the model would take 4 weeks; threshold or workflow changes can ship in days.