Power Analysis for Checkout A/B Test

Business Context

CartJet is a large e-commerce marketplace (~8M weekly active users) planning a redesign of its checkout page to reduce friction on mobile web. The product team is willing to run an A/B test for up to 3 weeks, but traffic is expensive: every extra day of experimentation delays rollout and has real revenue opportunity cost.

Historically, the purchase conversion rate (session → purchase) on mobile web is about 3.80%. Finance has asked the experimentation team to only run tests that have a high chance of detecting economically meaningful improvements.

You are the data scientist responsible for proposing a power analysis process and producing a concrete sample size recommendation.

Problem Statement

Design a power analysis for a two-arm A/B test on a binary metric (conversion). You want to detect a minimum detectable effect (MDE) of a +0.30 percentage point absolute lift (from 3.80% to 4.10%). The test will be analyzed using a two-sided hypothesis test at α = 0.05 with target power = 80%.

Given Data

Requirements

1. Describe your process for conducting power analysis in this setting (what inputs you need, how you choose MDE, α, power, and what assumptions you check).
2. Compute the required sample size per group using a normal approximation for a two-proportion z-test.
3. Convert the sample size into total eligible sessions needed (accounting for the 0.92 eligibility rate).
4. Suppose CartJet can route 1.2M mobile-web sessions/day into the experiment. Estimate the runtime in days.
5. Provide one risk/caveat where the theoretical power calculation can be misleading in practice, and how you’d mitigate it.

Assumptions and Constraints

• Random assignment at the session level; each session is an independent Bernoulli trial.
• No interference between users (SUTVA) and stable instrumentation.
• Use a two-sided test (product wants to detect regressions too).
• Normal approximation is acceptable (large n), but you should sanity-check whether it’s valid.