Your question is Novelty Effects in UI Test. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
You work on a consumer social product and your team has redesigned a highly visible navigation surface that every active user sees on app open. The new UI is cleaner and more prominent, and the team believes it will increase the share of users who start a core engagement action from the home surface. Because the change is visually obvious, you are concerned that any early lift could be driven by novelty rather than a durable behavior change. You need to design an experiment that can separate a short-term reaction from sustained impact.
How would you design and analyze this experiment so that you can estimate the true long-term impact of the UI change while accounting for novelty effects, and what decision rule would you use to ship or not ship?