Rippling Growth wants to launch a new lifecycle email journey in Rippling HR Cloud to drive admins to complete payroll setup. The team can either run a standard A/B test between two email variants, or reserve a true holdout group that receives no email so they can estimate incremental lift over the existing baseline.
Decide whether a holdout group is justified for this experiment, using the observed pilot data and the statistical trade-off between measurement value and lost short-term conversions.
A 2-week pilot randomly split eligible admins into three groups:
| Group | Users | Payroll setup completions within 14 days | Conversion Rate |
|---|---|---|---|
| Holdout (no email) | 8,000 | 640 | 8.0% |
| Control email (current journey) | 16,000 | 1,440 | 9.0% |
| Treatment email (new journey) | 16,000 | 1,632 | 10.2% |
Additional business inputs:
| Metric | Value |
|---|---|
| Significance level | 0.05 |
| Desired power | 0.80 |
| Value per additional payroll setup completion | $180 |
| Next full rollout audience over 1 quarter | 240,000 admins |