Your question is Assess Model Calibration. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
You have a model that outputs probabilities for a binary decision, and the team wants to know whether those scores can be trusted as probabilities. The model looks acceptable on ranking metrics, but people are unsure whether a score of 0.8 really means an 80% chance of the positive class.
How would you assess whether a model is calibrated?