A/B Test vs Observational Lift | Dataford Interview Questions

Business Context

StreamCart changed its mobile checkout button color and wants to know whether the change truly improved purchase conversion. The team has two sources of evidence: a randomized A/B test and a prior observational analysis from users who self-selected into the new experience.

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

Group	Users	Purchases	Conversion Rate
Control (old button)	8,000	880	11.0%
Treatment (new button)	8,200	984	12.0%

2) Observational analysis from a prior soft launch

Segment	Old Button Users	Old Purchases	Old Rate	New Button Users	New Purchases	New Rate
High-intent users	1,500	270	18.0%	4,000	800	20.0%
Low-intent users	6,500	610	9.38%	2,000	184	9.2%
Overall	8,000	880	11.0%	6,000	984	16.4%

Use a significance level of $\alpha = 0.05$ .

Requirements

Compute the estimated lift from the randomized A/B test.
Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
Compute a 95% confidence interval for the A/B test lift.
Compare the A/B test estimate with the observational overall lift.
Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

Randomization in the A/B test was implemented correctly.
Each user appears once and outcomes are independent.
The normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

Group	Users	Purchases	Conversion Rate
Control (old button)	8,000	880	11.0%
Treatment (new button)	8,200	984	12.0%

2) Observational analysis from a prior soft launch

Segment	Old Button Users	Old Purchases	Old Rate	New Button Users	New Purchases	New Rate
High-intent users	1,500	270	18.0%	4,000	800	20.0%
Low-intent users	6,500	610	9.38%	2,000	184	9.2%
Overall	8,000	880	11.0%	6,000	984	16.4%

Use a significance level of $\alpha = 0.05$ .

Requirements

Compute the estimated lift from the randomized A/B test.
Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
Compute a 95% confidence interval for the A/B test lift.
Compare the A/B test estimate with the observational overall lift.
Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

Randomization in the A/B test was implemented correctly.
Each user appears once and outcomes are independent.
The normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

Group	Users	Purchases	Conversion Rate
Control (old button)	8,000	880	11.0%
Treatment (new button)	8,200	984	12.0%

2) Observational analysis from a prior soft launch

Segment	Old Button Users	Old Purchases	Old Rate	New Button Users	New Purchases	New Rate
High-intent users	1,500	270	18.0%	4,000	800	20.0%
Low-intent users	6,500	610	9.38%	2,000	184	9.2%
Overall	8,000	880	11.0%	6,000	984	16.4%

Use a significance level of $\alpha = 0.05$ .

Requirements

Compute the estimated lift from the randomized A/B test.
Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
Compute a 95% confidence interval for the A/B test lift.
Compare the A/B test estimate with the observational overall lift.
Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

Randomization in the A/B test was implemented correctly.
Each user appears once and outcomes are independent.
The normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

Group	Users	Purchases	Conversion Rate
Control (old button)	8,000	880	11.0%
Treatment (new button)	8,200	984	12.0%

2) Observational analysis from a prior soft launch

Segment	Old Button Users	Old Purchases	Old Rate	New Button Users	New Purchases	New Rate
High-intent users	1,500	270	18.0%	4,000	800	20.0%
Low-intent users	6,500	610	9.38%	2,000	184	9.2%
Overall	8,000	880	11.0%	6,000	984	16.4%

Use a significance level of $\alpha = 0.05$ .

Requirements

Compute the estimated lift from the randomized A/B test.
Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
Compute a 95% confidence interval for the A/B test lift.
Compare the A/B test estimate with the observational overall lift.
Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

Randomization in the A/B test was implemented correctly.
Each user appears once and outcomes are independent.
The normal approximation is appropriate because sample sizes are large.