Explaining P-Value and Power

Business Context

PixelCraft, a design collaboration app, tested a new onboarding illustration intended to increase first-session completion. A product designer saw that the experiment result was “not significant” and asked what the p-value means, and why the team also cares about statistical power.

Problem Statement

Use the experiment data below to explain both p-value and statistical power in plain language, while also computing the formal test result. Then assess whether the test was large enough to reliably detect the target improvement.

Given Data

Assume a two-sided test comparing two independent proportions.

Requirements

1. State the null and alternative hypotheses.
2. Compute the pooled proportion, standard error, z-statistic, and p-value.
3. Decide whether the observed lift is statistically significant at $\alpha = 0.05$.
4. Explain the p-value in plain English for a non-technical product designer.
5. Estimate whether the current test had enough power to detect a 2.0 percentage point lift.
6. Explain statistical power in plain English and why a non-significant result does not necessarily mean “no effect.”

Assumptions

• Users were randomly assigned.
• Each user is counted once.
• The normal approximation for proportions is appropriate.
• Use equal-variance planning assumptions for the power/sample size check.