Context

HealthShield uses a binary classifier to detect rare insurance fraud claims before payout. The target class is highly imbalanced: only 1% of 500,000 historical claims are fraudulent, and the current model is being judged internally using overall accuracy.

Current Performance

The Problem

Leadership sees 98.6% accuracy and assumes the model is production-ready, but the fraud team argues that accuracy is misleading because the positive class is only 1%. You need to evaluate whether the model is actually useful, explain the tradeoffs created by the imbalance, and recommend how to improve both evaluation and model performance.

Requirements

1. Explain why accuracy is a poor primary metric in this setting.
2. Interpret the current precision, recall, F1, ROC-AUC, and PR-AUC values.
3. Use the confusion matrix to quantify business impact from false positives and false negatives.
4. Recommend how you would validate the model and choose an operating threshold.
5. Propose concrete modeling or data strategies to improve performance on the minority class.

Constraints

• Manual investigation capacity is limited to 8,000 claims per month.
• Each false positive review costs $12 in analyst time.
• Each missed fraud claim costs an average of $1,800.
• The evaluation approach must be understandable to non-technical stakeholders.