Context
FlowNote, a team-collaboration app, wants to simplify new-user onboarding by replacing a 5-step setup flow with a 2-step guided template picker. Early dogfooding suggests more users complete onboarding, but PMs worry that faster activation may attract lower-intent users who churn after the first week.
Hypothesis Seed
The new onboarding flow reduces friction and should increase activation, defined as completing setup and creating a first project within 24 hours. However, the company will only ship if the activation gain does not come at the cost of materially worse long-term retention.
Constraints
- Eligible traffic: 24,000 new signups per day globally
- Only new users are eligible; existing users cannot be exposed
- Maximum experiment runtime: 28 days, because the onboarding team must decide before the next quarterly release
- Randomization must happen at signup
- False positives are costly because retention loss harms downstream revenue; false negatives are acceptable if the team can iterate next quarter
- You need enough time to observe Day-28 retention for the full analysis cohort
Deliverables
- Define the hypothesis, primary metric, guardrails, and a clear MDE for both activation and retention risk.
- Calculate the required sample size and show whether the available traffic can support the test within 28 days.
- Choose the unit of randomization, allocation, duration, and any stratification or ramp plan.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparison treatment, and how you will interpret a result where activation improves but retention declines.
- State a ship / do-not-ship / iterate rule that explicitly respects guardrails and addresses common pitfalls such as novelty effects, SRM, and interference across invited teammates.
