Business Context
StreamCart, a subscription video platform, changed its signup page to reduce friction and increase paid trial starts. The product team ran a randomized A/B test before deciding whether to launch the new design to all traffic.
Problem Statement
Use the experiment results to determine whether the new signup page improved conversion rate enough to justify rollout.
Given Data
| Group | Users Exposed | Trial Starts | Conversion Rate |
|---|
| Control (old signup) | 18,400 | 2,116 | 11.50% |
| Treatment (new signup) | 18,900 | 2,325 | 12.30% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Desired power for planning | 0.80 |
| Minimum detectable effect for planning | 0.8 percentage points |
| |
Requirements
- State the null and alternative hypotheses for the conversion-rate comparison.
- Compute the observed conversion rates and the absolute lift.
- Run a two-proportion z-test using the pooled standard error.
- Calculate the two-sided p-value and determine whether the result is statistically significant at α=0.05.
- Construct a 95% confidence interval for the difference in conversion rates.
- Assess whether the observed lift is practically meaningful for rollout.
- Briefly explain how you would think about sample size adequacy for this test.
Assumptions
- Users were randomly assigned and each user appears once.
- Conversion is binary: a user either starts a paid trial or does not.
- No major traffic-source mix shift occurred during the experiment.
- The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.
