Business Context
Chime wants to test a redesigned sign-up flow on its landing page to increase completed account applications without hurting downstream quality. A 14-day randomized experiment was run on new visitors to Chime's landing page.
Problem Statement
Evaluate whether the new sign-up flow improves the primary conversion metric enough to justify rollout, and quantify the uncertainty around the lift.
Given Data
| Group | Visitors | Completed Sign-Ups | Conversion Rate |
|---|
| Control (current flow) | 52,400 | 6,131 | 11.7004% |
| Treatment (new flow) | 52,050 | 6,534 | 12.5533% |
Additional experiment parameters:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Planned power | 0.80 |
| Baseline conversion for planning | 11.7% |
| Minimum detectable effect for planning | 0.8 percentage points |
Requirements
- State the null and alternative hypotheses for the sign-up conversion rate.
- Compute the observed conversion rates and absolute lift.
- Run a two-proportion z-test using the pooled standard error.
- Calculate the two-sided p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Determine whether the result is statistically significant at the 5% level.
- Briefly assess whether the observed lift is practically meaningful for Chime.
- Using the planning assumptions, estimate the required sample size per variant for 80% power.
Assumptions
- Visitors were randomly assigned at the user level and counted once.
- No major instrumentation issues or sample-ratio mismatch occurred.
- The normal approximation is valid because both groups have large sample sizes.
- The primary metric is completed sign-up conversion on the landing-page flow.
