Business Context
StreamCart tested a new checkout page. The overall result looks neutral, but mobile and desktop segments point in opposite directions, so the product team wants a statistically grounded recommendation.
Problem Statement
Assess whether the experiment is truly inconclusive or whether the contradiction is explained by segment mix. Use the overall data and the device-level breakdown to evaluate the evidence.
Given Data
Overall experiment
| Group | Sample Size | Conversions | Conversion Rate |
|---|
| Control | 20,000 | 2,300 | 11.50% |
| Treatment | 20,000 | 2,380 | 11.90% |
Device-level breakdown
| Segment | Group | Sample Size | Conversions | Conversion Rate |
|---|
| Mobile | Control | 14,000 | 1,540 | 11.00% |
| Mobile | Treatment | 8,000 | 960 | 12.00% |
| Desktop | Control | 6,000 | 760 | 12.67% |
| Desktop | Treatment | 12,000 | 1,420 | 11.83% |
Use a two-sided significance level of 0.05.
Requirements
- State the null and alternative hypotheses for the overall test.
- Compute the overall difference in conversion rates and perform a two-proportion z-test.
- Construct a 95% confidence interval for the overall lift.
- Compare the mobile and desktop segment results qualitatively and explain why the data may appear contradictory.
- Recommend what you would do next if the evidence is inconclusive or mixed.
Assumptions
- Users were randomly assigned within the experiment.
- Each observation is an independent user-level conversion outcome.
- The normal approximation is valid because sample sizes and conversion counts are large enough.
- Segment assignment (mobile vs desktop) is observed, not randomized.
