Evaluate Calibration in Search Ranking

Scenario

You own a gradient-boosted ranking model that scores historical record hints shown in a genealogy search experience. The model outputs a click probability used both to rank hints and to suppress low-confidence items below a 0.10 threshold. Offline ranking metrics looked strong, but product partners now report that downstream surfaces using the score as a probability are over-triggering because users click far less often than the model predicts in some score bands. You need to determine whether the model is well calibrated, not just well ranked.

Performance Data

Question

How would you measure and diagnose calibration for this model, and what would you recommend if ranking performance remains acceptable but the predicted probabilities are systematically too high in production?