Context
AcmeFit, a subscription fitness app, wants to know whether changing the homepage Sign Up button from gray to green increases new-user registration. This is a low-cost UI change, but the team wants a disciplined experiment before shipping globally.
Hypothesis Seed
The product manager believes the current gray button has low visual salience on mobile landing pages. A greener, higher-contrast button may increase the proportion of visitors who complete registration after viewing the landing page.
Constraints
- Eligible traffic: 120,000 unique landing-page visitors per day
- Platform mix: 70% mobile web, 30% desktop web
- Current landing-page-to-sign-up conversion rate: 8.0%
- Maximum experiment duration: 14 days
- The team only wants to ship if the change produces a meaningful lift; a false positive is worse than a false negative because the button color would be rolled out to all acquisition traffic and could hurt paid marketing efficiency.
- The experiment platform supports user-level randomization and daily monitoring for safety, but the team does not want ad hoc peeking for significance.
Deliverables
- Define the null and alternative hypotheses, and state whether you would use a one-sided or two-sided test.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Include the unit of analysis and a clear minimum detectable effect (MDE).
- Calculate the required sample size per arm using the baseline conversion rate, your chosen alpha/power, and the MDE. Translate that into expected runtime given available traffic.
- Choose the unit of randomization, allocation strategy, duration, and any stratification or blocking you would use.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparisons policy, SRM checks, and what decision rule you would use to ship / not ship / iterate.
