Your question is P-Value and Power for Button Redesign. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
PixelCraft, a design collaboration app, tested a new onboarding button style proposed by the product design team. The PM wants a simple explanation of whether the observed lift is "real," and whether the test was large enough to detect a meaningful improvement.
You need to explain p-value and statistical power in plain language, but also quantify them using the experiment results below. Treat this as a standard A/B test on signup conversion.
| Group | Sample Size | Signups | Conversion Rate |
|---|---|---|---|
| Control (old button) | 8,000 | 960 | 12.0% |
| Treatment (new button) | 8,200 | 1,066 | 13.0% |
Additional test design inputs:
| Parameter | Value |
|---|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Minimum meaningful lift for planning | 1.0 percentage point |
| Baseline conversion rate for planning | 12.0% |