Power Analysis for Small Lift Experiment

Business Context

StreamCart is testing a new personalized recommendation widget on its mobile home screen. Product expects only a small improvement in purchase conversion, so the main question is whether the experiment can detect that lift with reasonable power.

Problem Statement

You are planning an A/B test on purchase conversion rate. The baseline conversion rate is 8.0%, and the team wants to detect a minimum meaningful lift of 0.4 percentage points (from 8.0% to 8.4%). Assume a two-sided test with significance level 5% and target power 80%.

Given Data

Requirements

1. State the null and alternative hypotheses.
2. Compute the required sample size per group for a two-sample proportion test.
3. Estimate the total test duration in days.
4. If the team can only run for 7 days, compute the approximate power with that smaller sample.
5. Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

• Users are independently randomized and counted once.
• Equal allocation to control and treatment.
• Normal approximation for two proportions is appropriate.
• No major seasonality or novelty effects beyond the planned test window.