Business Context
StreamCart, a subscription video platform, launched a new signup page feature that shows personalized plan recommendations. The product team wants to know whether the feature improves paid conversion from visitor to subscriber.
Problem Statement
You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.
Given Data
A 14-day user-level A/B test was run with random assignment at the first eligible visit.
| Group | Users Exposed | Paid Conversions | Conversion Rate |
|---|
| Control (current signup flow) | 52,400 | 6,131 | 11.70% |
| Treatment (personalized recommendations) | 51,900 | 6,492 | 12.51% |
Additional parameters:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | One-tailed |
| Baseline conversion rate for planning | 11.7% |
| Minimum detectable effect | 0.8 percentage points |
| Desired power | 80% |
Requirements
- State the null and alternative hypotheses for this experiment.
- Identify the correct unit of randomization and primary metric.
- Compute the observed lift in conversion rate.
- Run a two-proportion z-test and calculate the p-value.
- Construct a 95% confidence interval for the treatment-control difference.
- Determine whether the result is statistically significant at α=0.05.
- Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
- Briefly list two guardrail metrics or experimental risks you would monitor before rollout.
Assumptions
- Randomization was implemented correctly and independently across users.
- Each user appears once in the analysis dataset.
- No major tracking bugs or sample ratio mismatch occurred.
- The experiment duration is long enough to cover day-of-week effects.
