Problem
Scenario
You've deployed a model and the team sees a sudden drop in production performance after a period of stable results. Offline training and validation looked healthy before launch, but recent monitoring suggests the model is making worse decisions in production. You need to figure out whether this is a data issue, a serving issue, a threshold or calibration problem, or a real change in the population the model is scoring.
Question
If a deployed model's accuracy drops suddenly, what steps do you take to diagnose and fix the issue?
Observed Signals
What This Tests
- Interpret whether the drop is threshold-related or ranking-related
- Use confusion matrix changes to reason about business impact
- Check calibration drift and feature drift
- Prioritize diagnosis before choosing retrain, recalibrate, or rollback
Practicing as: Machine Learning Engineer interview at CapgeminiHi, I'll play your Capgemini interviewer for the Machine Learning Engineer role. Answer the question above like we're in the room, and I'll respond the way a real interviewer would.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
