Business Context
PixelCraft, a design collaboration app, tested a new onboarding button style proposed by the product design team. The PM wants a simple explanation of whether the observed lift is "real," and whether the test was large enough to detect a meaningful improvement.
Problem Statement
You need to explain p-value and statistical power in plain language, but also quantify them using the experiment results below. Treat this as a standard A/B test on signup conversion.
Given Data
| Group | Sample Size | Signups | Conversion Rate |
|---|
| Control (old button) | 8,000 | 960 | 12.0% |
| Treatment (new button) | 8,200 | 1,066 | 13.0% |
Additional test design inputs:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Minimum meaningful lift for planning | 1.0 percentage point |
| Baseline conversion rate for planning | 12.0% |
Requirements
- State the null and alternative hypotheses.
- Compute the observed difference in conversion rates.
- Calculate the two-proportion z-statistic and p-value.
- Explain the p-value in non-technical language appropriate for a product designer.
- Estimate the test's statistical power to detect a 1.0 percentage point absolute lift (from 12.0% to 13.0%) at α=0.05.
- Explain statistical power in non-technical language and say whether this experiment was adequately powered.
- Give a recommendation on whether PixelCraft should roll out the new button.
Assumptions
- Users were randomly assigned and counted once.
- The normal approximation for proportions is valid.
- Ignore segmentation and multiple-testing issues for the core calculation.
