Problem
Scenario
You've shipped a model that looked strong in validation, but it is underperforming after deployment. You need to figure out why the offline results did not hold up in the real world.
Question
Tell me about a time your model performed well in validation but failed in deployment. How did you debug it?
What changed
Production AUC-ROC·0.76Validation AUC-ROC·0.92Production Recall @ 0.40·0.49Validation Recall @ 0.40·0.74Production Precision @ 0.40·0.54Validation Precision @ 0.40·0.81
Practicing as: AI Engineer interview at Figure AIHi, I'll play your Figure AI interviewer for the AI Engineer role. Answer the question above like we're in the room, and I'll respond the way a real interviewer would.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
