Problem
Scenario
You have a classifier that outputs probabilities, and stakeholders want to know whether those scores can be trusted for decision-making. The model may rank examples well, but the question is whether a predicted 0.8 really means about an 80% chance of the event.
Question
How would you assess whether a model is well-calibrated?
Observed Signals
ECE·0.061AUC-ROC·0.84Log loss·0.46Brier score·0.142Top decile observed rate·0.58Top decile predicted risk·0.67
Practicing as: Research Scientist interview at Amperos HealthHi, I'll play your Amperos Health interviewer for the Research Scientist role. Answer the question above like we're in the room, and I'll respond the way a real interviewer would.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
