Context

NorthRiver Health deployed a gradient boosting model to predict inpatient sepsis risk within the next 12 hours and trigger rapid-response review. After a 6-week pilot across 3 hospitals, leadership sees mixed results: the model improved early detection, but clinicians report too many alerts and uneven performance across units.

Current Performance

During the pilot, 18,400 admissions were scored. Sepsis prevalence was 6.5% (1,196 cases). At the current threshold, the model generated 2,613 alerts, including 732 true positives and 1,881 false positives.

The Problem

You need to assess whether this AI project should be considered successful in a healthcare setting, where clinical benefit, alert burden, calibration, and patient safety all matter—not just discrimination metrics.

Requirements

1. Interpret whether the pilot met success criteria from both a model and clinical operations perspective.
2. Explain what the metric shifts from validation to production suggest.
3. Identify the most important risks in deploying this model more broadly.
4. Recommend how you would validate performance across patient segments and care settings.
5. Propose concrete next steps to improve the model and deployment policy.

Constraints

• False negatives can delay treatment for a life-threatening condition.
• False positives contribute to alert fatigue and clinician distrust.
• The rapid-response team can review at most 2,000 alerts per month without adding staff.