Whenever you're ready, walk me through how you'd approach this.
Splice tested a new call-to-action on the Sounds browse page to increase the share of visitors who start a free trial. A Product Growth Analyst needs to explain the result to a non-technical PM using both intuition and the underlying numbers.
You need to explain what a p-value means and what statistical power means in plain English, then compute both for this experiment and say whether the test was strong enough to trust.
| Metric | Control | Treatment |
|---|---|---|
| Visitors | 20,000 | 20,000 |
| Free-trial starts | 1,800 | 1,980 |
| Conversion rate | 9.0% | 9.9% |
Additional assumptions for power analysis:
| Parameter | Value |
|---|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Minimum effect of interest | 0.8 percentage points |
| Baseline conversion rate for planning | 9.0% |