Business Context
Splice tested a new call-to-action on the Sounds landing page to increase free-trial signup starts. After only a few days, the treatment looks better, but traffic is still limited because the experiment was launched to a small share of users.
Problem Statement
You need to decide how to handle an experiment result that looks promising but is based on a small sample. Quantify the evidence, assess uncertainty, and recommend whether Splice should roll out, stop, or continue the test.
Given Data
| Group | Users Exposed | Trial Starts | Conversion Rate |
|---|
| Control | 2,400 | 288 | 12.0% |
| Treatment | 600 | 90 | 15.0% |
Additional inputs:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Desired power | 0.80 |
| Minimum detectable effect for planning | 3.0 percentage points |
Requirements
- State the null and alternative hypotheses for the conversion-rate lift.
- Run a two-proportion z-test using the current data and compute the p-value.
- Compute a 95% confidence interval for the treatment-control difference.
- Explain whether the result is statistically significant and whether the current sample is sufficient for a decision.
- Estimate the required per-group sample size to detect a 3.0 percentage point lift from a 12.0% baseline at 80% power and 5% significance.
- Give a business recommendation for what to do next if the result looks promising but the sample is still small.
Assumptions
- Users were randomly assigned and counted once.
- Independence between users is reasonable.
- Use a two-sided test.
- Ignore seasonality and segmentation effects for this calculation, but mention them in interpretation.
