StreamCart, a grocery delivery app, tested a new personalized reorder widget on the home screen. The feature appears to increase 7-day reorder conversion slightly, and the PM wants to know whether statistical significance alone is enough to ship.
You need to evaluate whether the observed lift is statistically significant, quantify the uncertainty, and decide whether the feature should be launched given a small implementation cost and potential UX risk.
The experiment ran for 21 days with user-level randomization.
| Group | Sample Size | 7-Day Reorders | Conversion Rate |
|---|---|---|---|
| Control | 52,400 | 6,288 | 12.00% |
| Treatment | 52,100 | 6,460 | 12.40% |
Additional business inputs:
| Input | Value |
|---|---|
| Significance level | 0.05 |
| Monthly exposed users if launched | 4,000,000 |
| Incremental gross profit per additional reorder | $1.80 |
| Monthly engineering/maintenance cost | $22,000 |