Interview Guides

Non-Significant A/B Test Launch Decision | Dataford Interview Questions - Dataford - Ace your Interview

Non-Significant A/B Test Launch Decision

Medium

Statistics & Probability

Asked at 7 companies7Hypothesis TestingA/B TestingStatistical Significance

Also asked at

Problem

Business Context

StreamCart tested a new mobile checkout button intended to improve purchase conversion. After a 14-day A/B test, the PM wants to launch because the treatment looks directionally better, but the experiment did not reach statistical significance.

Problem Statement

Assess whether the observed lift is strong enough to justify launch, and quantify what the current test result does and does not tell us.

Given Data

Group	Users	Purchases	Conversion Rate
Control	52,400	6,026	11.50%
Treatment	52,100	6,105	11.72%

Additional inputs:

Parameter	Value
Test duration	14 days
Significance level	0.05
Desired power	0.80
PM's minimum practical lift to launch	0.50 percentage points

Requirements

State the null and alternative hypotheses for a two-sided test.
Run a two-proportion z-test and compute the pooled proportion, standard error, z-statistic, and p-value.
Compute a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at the 5% level.
Compare the observed lift with the PM's minimum practical threshold.
Estimate the required sample size per group to detect a 0.50 percentage point absolute lift with 80% power at 5% significance.
Based on the evidence, explain what you would recommend to the PM: launch, hold, or continue testing.

Assumptions

Users were randomly assigned and counted once.
No sample ratio mismatch or major instrumentation issues.
Conversion is a Bernoulli outcome and normal approximation is valid.
The primary metric was pre-specified before the test started.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":52100,"control_rate":0.115,"treatment_rate":0.1172,"test_duration_days":14,"control_conversions":6026,"treatment_conversions":6105,"minimum_practical_lift":0.005}

Output(none)

Problem

Business Context

Problem Statement

Assess whether the observed lift is strong enough to justify launch, and quantify what the current test result does and does not tell us.

Given Data

Group	Users	Purchases	Conversion Rate
Control	52,400	6,026	11.50%
Treatment	52,100	6,105	11.72%

Additional inputs:

Parameter	Value
Test duration	14 days
Significance level	0.05
Desired power	0.80
PM's minimum practical lift to launch	0.50 percentage points

Requirements

State the null and alternative hypotheses for a two-sided test.
Run a two-proportion z-test and compute the pooled proportion, standard error, z-statistic, and p-value.
Compute a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at the 5% level.
Compare the observed lift with the PM's minimum practical threshold.
Estimate the required sample size per group to detect a 0.50 percentage point absolute lift with 80% power at 5% significance.
Based on the evidence, explain what you would recommend to the PM: launch, hold, or continue testing.

Assumptions

Users were randomly assigned and counted once.
No sample ratio mismatch or major instrumentation issues.
Conversion is a Bernoulli outcome and normal approximation is valid.
The primary metric was pre-specified before the test started.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":52100,"control_rate":0.115,"treatment_rate":0.1172,"test_duration_days":14,"control_conversions":6026,"treatment_conversions":6105,"minimum_practical_lift":0.005}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Diagnose Misleading Signup A/B TestMedium AA/B Test vs Observational LiftEasy

Experiment Significance for Pricing BannerMedium

Next question

Non-Significant A/B Test Launch Decision

Medium

Statistics & Probability

Asked at 7 companies7Hypothesis TestingA/B TestingStatistical Significance

Also asked at

Problem

Business Context

Problem Statement

Assess whether the observed lift is strong enough to justify launch, and quantify what the current test result does and does not tell us.

Given Data

Group	Users	Purchases	Conversion Rate
Control	52,400	6,026	11.50%
Treatment	52,100	6,105	11.72%

Additional inputs:

Parameter	Value
Test duration	14 days
Significance level	0.05
Desired power	0.80
PM's minimum practical lift to launch	0.50 percentage points

Requirements

State the null and alternative hypotheses for a two-sided test.
Run a two-proportion z-test and compute the pooled proportion, standard error, z-statistic, and p-value.
Compute a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at the 5% level.
Compare the observed lift with the PM's minimum practical threshold.
Estimate the required sample size per group to detect a 0.50 percentage point absolute lift with 80% power at 5% significance.
Based on the evidence, explain what you would recommend to the PM: launch, hold, or continue testing.

Assumptions

Users were randomly assigned and counted once.
No sample ratio mismatch or major instrumentation issues.
Conversion is a Bernoulli outcome and normal approximation is valid.
The primary metric was pre-specified before the test started.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":52100,"control_rate":0.115,"treatment_rate":0.1172,"test_duration_days":14,"control_conversions":6026,"treatment_conversions":6105,"minimum_practical_lift":0.005}

Output(none)

Problem

Business Context

Problem Statement

Assess whether the observed lift is strong enough to justify launch, and quantify what the current test result does and does not tell us.

Given Data

Group	Users	Purchases	Conversion Rate
Control	52,400	6,026	11.50%
Treatment	52,100	6,105	11.72%

Additional inputs:

Parameter	Value
Test duration	14 days
Significance level	0.05
Desired power	0.80
PM's minimum practical lift to launch	0.50 percentage points

Requirements

State the null and alternative hypotheses for a two-sided test.
Run a two-proportion z-test and compute the pooled proportion, standard error, z-statistic, and p-value.
Compute a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at the 5% level.
Compare the observed lift with the PM's minimum practical threshold.
Estimate the required sample size per group to detect a 0.50 percentage point absolute lift with 80% power at 5% significance.
Based on the evidence, explain what you would recommend to the PM: launch, hold, or continue testing.

Assumptions

Users were randomly assigned and counted once.
No sample ratio mismatch or major instrumentation issues.
Conversion is a Bernoulli outcome and normal approximation is valid.
The primary metric was pre-specified before the test started.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":52100,"control_rate":0.115,"treatment_rate":0.1172,"test_duration_days":14,"control_conversions":6026,"treatment_conversions":6105,"minimum_practical_lift":0.005}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Diagnose Misleading Signup A/B TestMedium AA/B Test vs Observational LiftEasy

Experiment Significance for Pricing BannerMedium

Next question