Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Power Analysis for Checkout A/B Test

MediumStatistics & Probability00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Business Context

CartJet is a large e-commerce marketplace (~8M weekly active users) planning a redesign of its checkout page to reduce friction on mobile web. The product team is willing to run an A/B test for up to 3 weeks, but traffic is expensive: every extra day of experimentation delays rollout and has real revenue opportunity cost.

Historically, the purchase conversion rate (session → purchase) on mobile web is about 3.80%. Finance has asked the experimentation team to only run tests that have a high chance of detecting economically meaningful improvements.

You are the data scientist responsible for proposing a power analysis process and producing a concrete sample size recommendation.

Problem Statement

Design a power analysis for a two-arm A/B test on a binary metric (conversion). You want to detect a minimum detectable effect (MDE) of a +0.30 percentage point absolute lift (from 3.80% to 4.10%). The test will be analyzed using a two-sided hypothesis test at α = 0.05 with target power = 80%.

Given Data

ItemValue
Baseline conversion rate (control)0.0380
Target conversion rate (treatment)0.0410
Absolute MDE0.0030
Significance level (two-sided)0.05
Target power0.80
Allocation50% control / 50% treatment
Expected fraction of sessions eligible for experiment (tracking + bot filtering)0.92

Requirements

  1. Describe your process for conducting power analysis in this setting (what inputs you need, how you choose MDE, α, power, and what assumptions you check).
  2. Compute the required sample size per group using a normal approximation for a two-proportion z-test.
  3. Convert the sample size into total eligible sessions needed (accounting for the 0.92 eligibility rate).
  4. Suppose CartJet can route 1.2M mobile-web sessions/day into the experiment. Estimate the runtime in days.
  5. Provide one risk/caveat where the theoretical power calculation can be misleading in practice, and how you’d mitigate it.

Assumptions and Constraints

  • Random assignment at the session level; each session is an independent Bernoulli trial.
  • No interference between users (SUTVA) and stable instrumentation.
  • Use a two-sided test (product wants to detect regressions too).
  • Normal approximation is acceptable (large n), but you should sanity-check whether it’s valid.