Interview Guides

Power Analysis for Small Lift Experiment | Dataford Interview Questions - Dataford - Ace your Interview

Power Analysis for Small Lift Experiment

Medium

Statistics & Probability

Asked at 3 companies3A/B TestingPower AnalysisSample Size

Also asked at

Problem

Business Context

StreamCart is testing a new personalized recommendation widget on its mobile home screen. Product expects only a small improvement in purchase conversion, so the main question is whether the experiment can detect that lift with reasonable power.

Problem Statement

You are planning an A/B test on purchase conversion rate. The baseline conversion rate is 8.0%, and the team wants to detect a minimum meaningful lift of 0.4 percentage points (from 8.0% to 8.4%). Assume a two-sided test with significance level 5% and target power 80%.

Given Data

Metric	Value
Baseline conversion rate $p_1$	0.080
Expected treatment conversion rate $p_2$	0.084
Absolute lift $\Delta$	0.004
Relative lift	5.0%
Significance level $\alpha$	0.05
Target power $1-\beta$	0.80
Daily eligible users	120,000
Traffic split	50% / 50%

Requirements

State the null and alternative hypotheses.
Compute the required sample size per group for a two-sample proportion test.
Estimate the total test duration in days.
If the team can only run for 7 days, compute the approximate power with that smaller sample.
Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

Users are independently randomized and counted once.
Equal allocation to control and treatment.
Normal approximation for two proportions is appropriate.
No major seasonality or novelty effects beyond the planned test window.

Sample Data

Example 1

Input

{"alpha":0.05,"power_target":0.8,"absolute_lift":0.004,"baseline_rate":0.08,"relative_lift":0.05,"treatment_rate":0.084,"daily_eligible_users":120000,"traffic_split_control":0.5,"traffic_split_treatment":0.5,"planned_days_for_power_check":7}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	0.080
Expected treatment conversion rate $p_2$	0.084
Absolute lift $\Delta$	0.004
Relative lift	5.0%
Significance level $\alpha$	0.05
Target power $1-\beta$	0.80
Daily eligible users	120,000
Traffic split	50% / 50%

Requirements

State the null and alternative hypotheses.
Compute the required sample size per group for a two-sample proportion test.
Estimate the total test duration in days.
If the team can only run for 7 days, compute the approximate power with that smaller sample.
Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

Users are independently randomized and counted once.
Equal allocation to control and treatment.
Normal approximation for two proportions is appropriate.
No major seasonality or novelty effects beyond the planned test window.

Sample Data

Example 1

Input

{"alpha":0.05,"power_target":0.8,"absolute_lift":0.004,"baseline_rate":0.08,"relative_lift":0.05,"treatment_rate":0.084,"daily_eligible_users":120000,"traffic_split_control":0.5,"traffic_split_treatment":0.5,"planned_days_for_power_check":7}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Minimum Detectable Effect for Signup TestMedium Sample Size Planning for Signup TestMedium

Small Lift Significance for Feature LaunchMedium

Next question

Power Analysis for Small Lift Experiment

Medium

Statistics & Probability

Asked at 3 companies3A/B TestingPower AnalysisSample Size

Also asked at

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	0.080
Expected treatment conversion rate $p_2$	0.084
Absolute lift $\Delta$	0.004
Relative lift	5.0%
Significance level $\alpha$	0.05
Target power $1-\beta$	0.80
Daily eligible users	120,000
Traffic split	50% / 50%

Requirements

State the null and alternative hypotheses.
Compute the required sample size per group for a two-sample proportion test.
Estimate the total test duration in days.
If the team can only run for 7 days, compute the approximate power with that smaller sample.
Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

Users are independently randomized and counted once.
Equal allocation to control and treatment.
Normal approximation for two proportions is appropriate.
No major seasonality or novelty effects beyond the planned test window.

Sample Data

Example 1

Input

{"alpha":0.05,"power_target":0.8,"absolute_lift":0.004,"baseline_rate":0.08,"relative_lift":0.05,"treatment_rate":0.084,"daily_eligible_users":120000,"traffic_split_control":0.5,"traffic_split_treatment":0.5,"planned_days_for_power_check":7}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	0.080
Expected treatment conversion rate $p_2$	0.084
Absolute lift $\Delta$	0.004
Relative lift	5.0%
Significance level $\alpha$	0.05
Target power $1-\beta$	0.80
Daily eligible users	120,000
Traffic split	50% / 50%

Requirements

State the null and alternative hypotheses.
Compute the required sample size per group for a two-sample proportion test.
Estimate the total test duration in days.
If the team can only run for 7 days, compute the approximate power with that smaller sample.
Explain how small expected lift affects sample size and what trade-offs you would discuss with the product team.

Assumptions

Users are independently randomized and counted once.
Equal allocation to control and treatment.
Normal approximation for two proportions is appropriate.
No major seasonality or novelty effects beyond the planned test window.

Sample Data

Example 1

Input

{"alpha":0.05,"power_target":0.8,"absolute_lift":0.004,"baseline_rate":0.08,"relative_lift":0.05,"treatment_rate":0.084,"daily_eligible_users":120000,"traffic_split_control":0.5,"traffic_split_treatment":0.5,"planned_days_for_power_check":7}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Minimum Detectable Effect for Signup TestMedium Sample Size Planning for Signup TestMedium

Small Lift Significance for Feature LaunchMedium

Next question