Your question is Measure Model Calibration. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
You have a model that outputs probabilities, and the team wants to know whether those scores mean what they say. The model may rank cases well, but the question is whether a predicted 0.80 really corresponds to about an 80% event rate.
How would you measure whether a model is calibrated?