Context
FitLoop, a subscription fitness app, wants to increase free-to-paid conversion. The growth team proposes replacing the current 4-screen onboarding with a shorter personalized onboarding flow that recommends a workout plan immediately after signup.
Hypothesis Seed
The team believes reducing onboarding friction and showing a personalized plan earlier will increase 7-day trial-start conversion among new users. However, there is concern that a faster flow may attract lower-intent users, hurting downstream paid conversion or increasing refund requests.
Constraints
- Eligible traffic: 120,000 new signup users per week
- Maximum experiment duration: 3 weeks, because the quarterly launch decision is fixed
- Randomization must happen at first signup experience and remain sticky across devices if the user logs in
- False positives are costly because onboarding changes require design, engineering, and lifecycle CRM updates; false negatives are also meaningful because signup growth has stalled
- The team wants a decision after one experiment, not an open-ended iteration cycle
Task
- Define a clear null and alternative hypothesis for this onboarding experiment, including whether you would use a one-sided or two-sided test.
- Specify the primary metric, 2-4 guardrail metrics, and 1-3 secondary metrics. Include the unit of analysis, expected baseline, and an explicit MDE for the primary metric.
- Calculate the required sample size per arm and determine whether the test can be completed within 3 weeks given available traffic. Show the math and state any assumptions.
- Choose the unit of randomization, allocation strategy, duration, and any stratification. Explain how you would handle issues like peeking, multiple comparisons, and any mismatch between unit of randomization and unit of analysis.
- Describe the main pitfalls you would watch for in this experiment, including novelty effects, sample ratio mismatch, and any possible interference or SUTVA concerns. End with a pre-registered ship / don't-ship rule that respects guardrails.
