Validate Aircraft Defect Risk Model

Context

Airbus Group has deployed a binary classification model to predict whether a completed work order in the Skywise maintenance analytics environment will lead to an in-service aircraft defect within 30 days. The model is used to prioritize engineering review for high-risk maintenance events on the A320 fleet, but engineering teams report that too many later-confirmed defects were not escalated.

Current Performance

The Problem

The model appears more conservative in production: precision improved, but recall dropped sharply. Airbus engineering leadership wants to know whether the current model is still the right choice, whether the threshold is mis-set, and how to validate any replacement before rollout.

Requirements

1. Interpret the metric pattern and explain what it suggests about model behavior.
2. Diagnose likely causes for the recall drop and propose a validation plan.
3. Recommend how you would compare this model against at least one alternative.
4. Explain which metrics should drive model selection given the operational cost of missed defects.
5. Propose concrete changes to improve performance and safely validate them.

Constraints

• Engineering review capacity is limited to 1,100 alerts per week.
• Missing a true defect is far more costly than reviewing a false alert.
• Labels are delayed because defects are confirmed only after inspection and service events.