Choose Metrics for Imbalanced Claims

Context

SureShield Insurance built a binary classification model to predict whether a newly submitted claim will become a high-cost fraudulent claim requiring special investigation. The dataset is highly imbalanced: only 1.8% of historical claims are labeled fraud. The team initially optimized for accuracy, but fraud losses remain high and investigators say too many risky claims are being missed.

Current Performance

The Problem

The VP of Claims wants a recommendation on which evaluation metric should be the primary decision metric for model selection and threshold tuning. The answer must reflect the severe class imbalance and the business cost asymmetry: a missed fraudulent claim costs about $12,000 on average, while investigating a legitimate claim costs about $85.

Requirements

1. Identify which metric(s) should be prioritized over accuracy and explain why.
2. Interpret what the current metrics imply about model behavior on an imbalanced dataset.
3. Recommend how to evaluate threshold tradeoffs given limited investigator capacity.
4. Propose a validation approach for comparing future models.
5. Suggest concrete next steps to improve business impact, not just headline metrics.

Constraints

• Investigation team can review at most 1,500 claims per week.
• False negatives are far more expensive than false positives.
• The model score is used to rank claims, then a threshold determines review.