Business Context
Splice tested a new call-to-action on the Sounds browse page to increase the share of visitors who start a free trial. A Product Growth Analyst needs to explain the result to a non-technical PM using both intuition and the underlying numbers.
Problem Statement
You need to explain what a p-value means and what statistical power means in plain English, then compute both for this experiment and say whether the test was strong enough to trust.
Given Data
| Metric | Control | Treatment |
|---|
| Visitors | 20,000 | 20,000 |
| Free-trial starts | 1,800 | 1,980 |
| Conversion rate | 9.0% | 9.9% |
Additional assumptions for power analysis:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Minimum effect of interest | 0.8 percentage points |
| Baseline conversion rate for planning | 9.0% |
Requirements
- State the null and alternative hypotheses.
- Compute the observed lift, pooled proportion, standard error, z-statistic, and p-value.
- Explain the p-value in plain English for a non-math audience.
- Estimate the test's statistical power to detect a true lift of 0.8 percentage points at the given sample size.
- Explain power in plain English and assess whether this experiment was adequately powered.
- Give a business recommendation for whether Splice should roll out the new CTA on the Sounds browse page.
Assumptions
- Users were randomly assigned and counted once.
- The normal approximation for proportions is appropriate.
- Ignore segmentation and multiple-testing issues for the core calculation, but mention them in interpretation.
