Context
BCG Digital Ventures is testing a redesigned onboarding checklist in its venture studio portfolio growth platform, Venture Growth Hub, to increase activation of newly invited operators and founders. The team believes the new checklist clarifies next steps and should improve 7-day activation, but traffic is limited and leadership wants a decision within one month.
Hypothesis Seed
The treatment replaces the current static onboarding panel with a personalized checklist ordered by predicted completion likelihood. Because activation is noisy and user quality varies meaningfully by acquisition source and prior product engagement, the team is considering CUPED using pre-experiment engagement as a covariate to reduce variance and detect smaller lifts.
Constraints
- Eligible traffic: 12,000 newly invited users per week
- Maximum experiment window: 28 days total, including ramp
- Allocation target: 50/50 after a 2-day instrumentation ramp
- Baseline 7-day activation rate: 24%
- Pre-period covariate available for 92% of users: prior 14-day workspace engagement score
- Leadership cares more about false positives than false negatives because shipping a weak onboarding flow would affect multiple venture launches
- The feature should not materially hurt invite-to-first-session rate or increase support-contact rate
Deliverables
- Define the null and alternative hypotheses, the primary metric, at least three guardrails, and an explicit MDE.
- Design the experiment end-to-end, including unit of randomization, duration, allocation, and whether CUPED is appropriate here versus another variance-reduction method.
- Calculate the required sample size with and without CUPED using a realistic variance-reduction assumption, and translate that into expected runtime given the traffic constraint.
- Pre-register the analysis plan: statistical test, CUPED adjustment formula, peeking policy, SRM checks, and how you will treat users missing the pre-period covariate.
- State a clear ship / don’t-ship / iterate decision rule that respects guardrails, including what you would do if the result is statistically significant but smaller than the MDE.
