Interpret F1 for Imbalanced Classification

Context

ShopSafe is building a binary classifier to detect fraudulent orders before fulfillment. Fraud is rare, so leadership is concerned that the current evaluation dashboard may overstate model quality by focusing on accuracy.

Current Performance

The team evaluated a logistic regression model on 100,000 recent orders. Only 1,000 orders were actually fraudulent.

Confusion matrix counts:

The Problem

The product manager sees 97.2% accuracy and believes the model is ready for launch. However, the risk team argues that the model is still weak because it misses most fraudulent orders. You need to explain what the F1 score means, why it matters here, and whether it is a better summary metric than accuracy.

Requirements

1. Define precision, recall, and F1 score using the numbers above.
2. Explain how the F1 score is calculated and interpret the value 0.290.
3. Compare F1 score with accuracy for this imbalanced dataset.
4. Explain why a model can have high accuracy but poor fraud detection performance.
5. Recommend 2-3 actions to improve the model or evaluation process.

Constraints

• Each missed fraud order costs about $120 on average.
• Each false positive triggers a manual review costing $4.
• Review capacity is limited to 400 flagged orders per day.