Power Analysis for Small Lift Experiment | Dataford Interview Questions

Your question is Power Analysis for Small Lift Experiment. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

StreamCart is testing a new personalized recommendation widget on its mobile home screen. Product expects only a small improvement in purchase conversion, so the main question is whether the experiment can detect that lift with reasonable power.

Problem Statement

You are planning an A/B test on purchase conversion rate. The baseline conversion rate is 8.0%, and the team wants to detect a minimum meaningful lift of 0.4 percentage points (from 8.0% to 8.4%). Assume a two-sided test with significance level 5% and target power 80%.

Given Data

Metric	Value
Baseline conversion rate $p_1$	0.080
Expected treatment conversion rate $p_2$	0.084
Absolute lift $\Delta$	0.004
Relative lift	5.0%
Significance level $\alpha$	0.05
Target power $1-\beta$	0.80
Daily eligible users	120,000
Traffic split	50% / 50%

Requirements

State the null and alternative hypotheses.
Compute the required sample size per group for a two-sample proportion test.
Estimate the total test duration in days.
If the team can only run for 7 days, compute the approximate power with that smaller sample.
Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

Users are independently randomized and counted once.
Equal allocation to control and treatment.
Normal approximation for two proportions is appropriate.
No major seasonality or novelty effects beyond the planned test window.