Context
Rippling’s Growth team wants to know whether a lifecycle campaign in Rippling Email Campaigns is creating net-new demand for payroll and benefits demos, or merely converting companies that would have signed up anyway. You need to design an experiment that measures true incrementality before the team scales spend.
Hypothesis Seed
The proposed change is to send a targeted outbound campaign to eligible admin users from SMB companies that have started but not completed a demo request flow for Rippling Payroll. The team believes the campaign will increase incremental demo-request conversion by nudging undecided prospects, not just accelerating users who were already likely to convert.
Constraints
- Eligible population: 120,000 admin users per week
- Baseline 14-day demo-request conversion among eligible users: 4.0%
- Maximum experiment window: 4 weeks of enrollment, with 14 additional days for conversion measurement
- Business wants at least a 10% relative lift to justify scaling the campaign
- False positives are costly because scaled outreach has real send cost and can create unsubscribes / brand fatigue
- False negatives are also meaningful because this is a high-intent acquisition surface, but less costly than over-crediting a non-incremental campaign
Deliverables
- Define the experiment to measure incremental lift, including the right control group and the unit of randomization.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. State the MDE explicitly.
- Calculate the required sample size per arm and determine whether the test fits within the traffic and time constraints.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparisons policy, and how you will diagnose issues like sample ratio mismatch.
- State a clear ship / don’t-ship / iterate rule that respects both the primary metric and guardrails, and explain how you would interpret a result that is statistically significant but operationally small.
