Business Context

PixelCraft, a design collaboration app, tested a new onboarding screen created by the Product Design team. The designer wants a plain-language explanation of p-value and statistical power, grounded in the actual experiment results.

Problem Statement

You need to explain whether the new onboarding improved activation, what the p-value says about the observed result, and what statistical power says about the test's ability to detect a meaningful improvement.

Given Data

Additional assumptions for power analysis:

Requirements

1. State the null and alternative hypotheses.
2. Calculate the observed difference in activation rates.
3. Compute the test statistic and p-value.
4. Explain the p-value in plain English for a non-technical Product Designer.
5. Estimate the statistical power to detect a lift from 15.0% to 16.5% with the given sample sizes.
6. Explain power in plain English and say whether this experiment was adequately powered.
7. Give a business recommendation on whether to ship, rerun, or collect more data.

Assumptions

• Users were randomly assigned to control and treatment.
• Each user appears once in the analysis.
• Normal approximation is appropriate because sample sizes are large.
• Ignore segmentation and multiple-testing issues for this exercise.