Business Context

StreamCart changed its mobile checkout button color and wants to know whether the change truly improved purchase conversion. The team has two sources of evidence: a randomized A/B test and a prior observational analysis from users who self-selected into the new experience.

Problem Statement

Explain the difference between A/B testing and observational analysis using the data below, then quantify what each approach suggests about the checkout change.

Given Data

1) Randomized A/B test

2) Observational analysis from a prior soft launch

Use a significance level of $\alpha = 0.05$.

Requirements

1. Compute the estimated lift from the randomized A/B test.
2. Run a two-proportion z-test for the A/B test and state whether the result is statistically significant.
3. Compute a 95% confidence interval for the A/B test lift.
4. Compare the A/B test estimate with the observational overall lift.
5. Explain, using the segment data, why the observational result may be biased and why A/B testing is better for causal inference here.

Assumptions

• Randomization in the A/B test was implemented correctly.
• Each user appears once and outcomes are independent.
• The normal approximation is appropriate because sample sizes are large.