Context

LedgerFlow uses a gradient-boosted binary classifier to flag card transactions for post-authorization fraud review. The AI audit team found that a specific $1,240 electronics purchase was flagged as high risk, but the customer claims it was legitimate and wants a clear explanation.

Current Performance

For the disputed transaction, the model score was 0.87. Top contributing signals shown in the audit tool were: new device (+0.19), merchant-country mismatch (+0.16), transaction amount 3.4x user median (+0.14), two declined attempts in prior 10 minutes (+0.11), and shipping address changed same day (+0.09).

The Problem

You need to assess whether the flag was reasonable, explain how to communicate the decision to the customer without overstating model certainty, and recommend how to evaluate whether the audit explanation system is trustworthy.

Requirements

1. Interpret the production metrics and explain what they imply about customer-facing audit decisions.
2. Assess whether the disputed transaction was flagged for sensible reasons.
3. Explain how you would communicate the result to a disagreeing customer.
4. Identify weaknesses in the current evaluation setup, especially around explanation quality and calibration.
5. Recommend metric, threshold, and validation improvements.

Constraints

• False positives create customer friction and regulatory complaints.
• Fraud losses average $420 per missed fraudulent transaction.
• Manual review capacity is limited to 4,000 transactions/day.
• Explanations must be understandable to non-technical support agents.