Context

You’re a senior data scientist at ShieldSure, a P&C insurer writing auto and homeowners policies across the US. ShieldSure receives ~1.2M claims/year and uses a model to predict ultimate claim severity (final paid loss) at FNOL (first notice of loss). The prediction drives two high-stakes decisions: (1) whether to route a claim to the Special Investigations Unit (SIU) or a standard adjuster, and (2) how much initial reserve to set (impacts capital requirements and quarterly financial reporting).

The challenge: ultimate settlement labels mature slowly. Simple claims settle in weeks, but litigation-heavy bodily injury claims can take 6–18 months, and a long tail can take 2–3 years. Executives want to ship an updated model now because inflation and repair costs have shifted, but the team cannot wait a year to know whether the model is truly better.

Current System

• Model: Gradient-boosted trees predicting probability that ultimate severity exceeds $25,000 (binary), plus a separate regression head for expected ultimate paid.
• Training data: Claims opened Jan 2021–Dec 2023, but only claims with closed status are used as ground truth.
• Deployment: Real-time scoring at FNOL; decisions must be made within <300 ms.

Observed Performance (Using Only Matured/Closed Claims)

You evaluate on claims opened in Q1 2024 that are already closed by end of Q3 2024 (so labels are available). This is ~38% of Q1 claims and is heavily skewed toward simpler claims.

Operations reports that after a limited shadow test, v2 would have routed ~18% more claims to SIU for the same threshold, which is not feasible given SIU capacity.

The Problem

Leadership is asking: “Is v2 actually better, or are we fooling ourselves because we’re only evaluating on early-settling claims?” You suspect label delay / censoring bias: the subset of claims that close quickly is not representative of the full distribution, especially for high-severity and litigated claims.

You need to propose a validation strategy that provides credible evidence of improvement before ultimate labels mature, while protecting the business from reserve misestimation and SIU overload.

Requirements (What you must do)

1. Diagnose what is potentially wrong with the closed-claims-only evaluation and how it can bias AUC/log loss/precision-recall.
2. Propose an offline validation design that uses partially matured data (open claims) without leaking future information.
3. Define leading indicators / proxy labels you would use (e.g., early payments, litigation flags) and how you would quantify their relationship to ultimate severity.
4. Recommend a production evaluation plan (e.g., shadow mode, champion–challenger, backtesting) that allows safe rollout.
5. Explain how you would handle calibration and thresholding given SIU capacity constraints and delayed feedback.

Constraints

• SIU can investigate at most 1,500 claims/week; exceeding this creates backlog and regulatory risk.
• Under-reserving increases financial restatement risk; over-reserving ties up capital.
• Claim handling processes changed in 2024 (new vendor network), so historical closure patterns may not match.
• You must provide a go/no-go recommendation within 6 weeks.