Your question is Improve ACE of a Classifier. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Open-ended: how would you improve ACE (adaptive calibration error) of a classifier?
Asked in the technical phone screen stage. Candidate says it maps to a Kilian Weinberger calibration paper.
Explain how you would diagnose poor calibration, choose a recalibration method, and integrate it into an end-to-end ML workflow. Address validation data construction, adaptive binning, multiclass behavior, distribution shift, offline evaluation, online monitoring, and rollback criteria. Discuss how calibration changes may affect ranking quality, threshold decisions, abstention, and downstream costs. Be explicit about training-serving skew and how you would prevent calibration from being fitted on biased or leaked data.