Your question is Testing Lift After Model Change. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
You're evaluating whether a product change actually improved an outcome metric, rather than just moving due to noise or selection effects. You need to decide how to attribute the change to the new model or feature.
How would you test whether a change in a model or feature caused a statistically significant improvement in the outcome metric?