Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Debug Training to Production Gap

HardModel Evaluation00:00
I
Practice interviewer
Your interviewer
In session
I
Interviewer

Welcome to your interview.

The question is on your right: Debug Training to Production Gap. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

You need to log in / sign up to chat or submit.

Problem

Scenario

You've shipped a model that looked strong in offline training and validation, but it is underperforming in production. The team wants you to figure out why the offline results did not carry over.

Question

How do you debug a model that looks good offline but fails in production?

Observed Signals

Production AUC-ROC·0.74Validation AUC-ROC·0.92Offline recall @ 0.42·0.76Offline precision @ 0.42·0.81Production recall @ 0.42·0.41Production precision @ 0.42·0.52