Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Diagnose Sudden Accuracy Drop

HardModel Evaluation00:00
Practice interviewer
In session
5 left
00:00

Your question is Diagnose Sudden Accuracy Drop. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Scenario

You've deployed a model and the team sees a sudden drop in production performance after a period of stable results. Offline training and validation looked healthy before launch, but recent monitoring suggests the model is making worse decisions in production. You need to figure out whether this is a data issue, a serving issue, a threshold or calibration problem, or a real change in the population the model is scoring.

Question

If a deployed model's accuracy drops suddenly, what steps do you take to diagnose and fix the issue?

Observed Signals

Recall·0.58AUC-ROC·0.79Precision·0.69Top feature PSI·0.31Calibration error·0.11Production accuracy·0.86 to 0.74

What This Tests

  • Interpret whether the drop is threshold-related or ranking-related
  • Use confusion matrix changes to reason about business impact
  • Check calibration drift and feature drift
  • Prioritize diagnosis before choosing retrain, recalibrate, or rollback