Scenario

You work on a consumer social product and your team has redesigned a highly visible navigation surface that every active user sees on app open. The new UI is cleaner and more prominent, and the team believes it will increase the share of users who start a core engagement action from the home surface. Because the change is visually obvious, you are concerned that any early lift could be driven by novelty rather than a durable behavior change. You need to design an experiment that can separate a short-term reaction from sustained impact.

Constraints

• Eligible traffic: 1.2M daily active users per day across iOS and Android
• Maximum experiment duration: 21 days
• 50% of eligible users can be exposed to treatment
• Day-7 retention cannot decline by more than 0.5 percentage points
• Crash-free sessions cannot decline by more than 0.2 percentage points

Question

How would you design and analyze this experiment so that you can estimate the true long-term impact of the UI change while accounting for novelty effects, and what decision rule would you use to ship or not ship?