Context

ShopNow uses two ML models in production: a classifier to detect fraudulent orders and a regression model to forecast weekly demand for top-selling products. Product leaders are asking which metric should be used to evaluate each model because recent reviews focused on a single score without considering business cost.

Current Performance

The Problem

The fraud team says each false negative costs about $240 in chargebacks, while each false positive costs $8 in manual review and customer friction. The inventory team says over-forecasting creates holding cost, but under-forecasting causes stockouts and lost margin on high-demand SKUs.

Requirements

1. Explain when precision, recall, and F1-score should be prioritized for the fraud model.
2. Explain why accuracy is misleading in this setting.
3. Explain when RMSE is the right metric for the demand model versus classification metrics.
4. Recommend the primary metric for each business problem and justify it using the cost structure.
5. Suggest one follow-up analysis for threshold tuning or error segmentation.

Constraints

• Fraud review team can handle only 1,200 alerts/day.
• Fraud prevalence is low and varies by season.
• Demand forecast errors on top 5% of SKUs matter more than long-tail items.