Context

BlueLedger uses a gradient-boosted classifier to predict the probability that a card payment will become a confirmed fraud or chargeback within 60 days. Scores above 0.80 are auto-declined, scores from 0.50 to 0.80 go to manual review, and lower scores are approved.

The model still ranks transactions reasonably well, but risk leaders are concerned that the predicted probabilities are not trustworthy enough for a high-stakes payments decision. In the last quarter, several score bands materially underpredicted realized fraud rates, creating avoidable losses and inconsistent review policies.

Current Performance

Calibration by Score Band

Task

1. Assess whether the model is well calibrated for this decision setting.
2. Explain which metrics and plots you would use, and why ranking metrics alone are insufficient.
3. Diagnose the business risk created by the current miscalibration across score bands.
4. Recommend how to recalibrate or redesign decision thresholds.
5. Describe how you would validate calibration by segment before deployment.

Constraints

• False declines create customer churn and merchant complaints.
• Missed fraud has an average loss of $420 per transaction.
• Manual review capacity is fixed at 18,000 transactions per day.