Business Context
StreamCart, a subscription video platform, tested a new signup page intended to increase paid trial starts. The product manager sees a positive lift, but before rollout wants to know whether the result is trustworthy.
Problem Statement
Assess whether the observed A/B test result is statistically credible and identify the main checks you would use to judge trustworthiness. Use the data below to evaluate statistical significance, confidence interval, and sample ratio mismatch.
Given Data
| Metric | Control | Treatment |
|---|
| Assigned users | 52,400 | 47,600 |
| Trial starts | 6,026 | 5,950 |
| Conversion rate | 11.50% | 12.50% |
Additional test settings:
| Parameter | Value |
|---|
| Planned traffic split | 50% / 50% |
| Test duration | 14 days |
| Significance level | 0.05 |
Requirements
- State the null and alternative hypotheses for conversion rate.
- Run a two-proportion z-test and compute the p-value.
- Compute a 95% confidence interval for the treatment minus control lift.
- Check whether the traffic allocation suggests a sample ratio mismatch (SRM) using a chi-square goodness-of-fit test.
- Based on the numbers, explain whether the result is trustworthy and what else you would inspect before shipping.
Assumptions
- User-level randomization was intended to be independent.
- Each user is counted once.
- Normal approximation is acceptable because both groups have large sample sizes and many conversions/non-conversions.
- Ignore secondary metrics for the numerical calculations, but discuss them qualitatively in the interpretation.
