Business Context

You’re the analytics lead at ResolveAI, a fintech/legaltech platform that helps insurers and claimants settle auto injury and property damage claims. The product serves 3.5M claims/year across the US and UK, and ~40% of claims now receive an AI-generated initial settlement offer that a human adjuster can accept, edit, or override. ResolveAI’s customers care about cycle time and cost, but they are also under increasing scrutiny from regulators and plaintiffs’ attorneys for disparate outcomes across protected classes.

Over the last quarter, ResolveAI expanded the model to recommend offers for more complex injury claims. Customer success reports that cycle time improved (median time-to-offer down from 3.2 days to 1.1 days), but a large insurer paused rollout after an internal audit suggested the AI’s offers were “systematically lower” for certain demographic groups after controlling for claim severity. The CEO asks you to design a single, reportable fairness metric (with supporting cuts) that can be used in:

• quarterly business reviews with insurers,
• internal model release gates,
• and (if needed) regulatory responses.

Metric Scenario

Stakeholders disagree on what “fair” means:

• Legal/Compliance wants parity across protected classes (race/ethnicity proxies, gender, age band, disability status where available).
• Claims Ops wants consistency with historical adjuster outcomes and reduced variance.
• Finance worries that “fairness” metrics could be gamed by overpaying.
• ML argues that fairness must be evaluated conditional on claim severity and jurisdiction.

You have to propose a metric that is: (a) hard to game, (b) decomposable into root causes, (c) stable enough to track weekly, and (d) aligned with business outcomes (retention, expansion, regulatory risk).

Available Data

What You Need To Produce

1. Define a primary KPI for “AI settlement fairness” that can be tracked over time and compared across model versions.
2. Explain why your definition is appropriate for this domain (settlement negotiation, policy limits, jurisdictional variation) and what it does not capture.
3. Provide a calculation approach including: severity/jurisdiction adjustment, handling missing protected-class data, and minimum sample thresholds.
4. Decompose the metric into actionable drivers (where disparity is coming from: data, model, workflow, or downstream negotiation).
5. Propose benchmarks/targets for launch gates (e.g., “block release if fairness degrades by X”), and specify guardrail metrics to prevent “fairness via overpayment.”
6. Recommend concrete actions you would take if the metric worsens after a new model release.

Constraints:

• You must support weekly reporting, but also provide a quarterly view suitable for audits.
• Protected-class attributes are incomplete and may be inferred via proxies in some jurisdictions.
• The metric must be robust to the fact that the final settlement is influenced by negotiation, attorney involvement, and litigation.