Validate Aircraft Defect Risk Model | Dataford Interview Questions

Your question is Validate Aircraft Defect Risk Model. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Context

Airbus Group has deployed a binary classification model to predict whether a completed work order in the Skywise maintenance analytics environment will lead to an in-service aircraft defect within 30 days. The model is used to prioritize engineering review for high-risk maintenance events on the A320 fleet, but engineering teams report that too many later-confirmed defects were not escalated.

Current Performance

Metric	Validation Set	Last 8 Weeks in Production	Change
Accuracy	0.91	0.89	-0.02
Precision	0.74	0.78	+0.04
Recall	0.68	0.49	-0.19
F1 Score	0.71	0.60	-0.11
AUC-ROC	0.86	0.82	-0.04
Log Loss	0.29	0.37	+0.08
High-risk alerts/week	1,240	910	-330
Confirmed defects/week	1,350	1,420	+70

The Problem

The model appears more conservative in production: precision improved, but recall dropped sharply. Airbus engineering leadership wants to know whether the current model is still the right choice, whether the threshold is mis-set, and how to validate any replacement before rollout.

Requirements

Interpret the metric pattern and explain what it suggests about model behavior.
Diagnose likely causes for the recall drop and propose a validation plan.
Recommend how you would compare this model against at least one alternative.
Explain which metrics should drive model selection given the operational cost of missed defects.
Propose concrete changes to improve performance and safely validate them.

Constraints

Engineering review capacity is limited to 1,100 alerts per week.
Missing a true defect is far more costly than reviewing a false alert.
Labels are delayed because defects are confirmed only after inspection and service events.