A/B Test vs Observational Lift | Dataford Interview Questions

Your question is A/B Test vs Observational Lift. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

StreamCart changed its mobile checkout button color and wants to know whether the change truly improved purchase conversion. The team has two sources of evidence: a randomized A/B test and a prior observational analysis from users who self-selected into the new experience.

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

Group	Users	Purchases	Conversion Rate
Control (old button)	8,000	880	11.0%
Treatment (new button)	8,200	984	12.0%

2) Observational analysis from a prior soft launch

Segment	Old Button Users	Old Purchases	Old Rate	New Button Users	New Purchases	New Rate
High-intent users	1,500	270	18.0%	4,000	800	20.0%
Low-intent users	6,500	610	9.38%	2,000	184	9.2%
Overall	8,000	880	11.0%	6,000	984	16.4%

Use a significance level of $\alpha = 0.05$ .

Requirements

Compute the estimated lift from the randomized A/B test.
Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
Compute a 95% confidence interval for the A/B test lift.
Compare the A/B test estimate with the observational overall lift.
Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

Randomization in the A/B test was implemented correctly.
Each user appears once and outcomes are independent.
The normal approximation is appropriate because sample sizes are large.