Context
FitStream, a consumer fitness app, wants to test a redesigned onboarding flow that asks users for goals and preferred workout times before showing the home screen. The growth team believes the extra personalization step will improve downstream activation, but it may also add friction and reduce immediate signup completion.
Hypothesis Seed
The proposed change is expected to increase the share of new users who become meaningfully activated in their first week, even if it slightly lowers top-of-funnel completion. Your task is to decide what the right primary metric should be and design the experiment so the company can make a launch decision with confidence.
Constraints
- Eligible traffic: 120,000 new signup starts per day
- 85% of traffic is mobile app, 15% web
- Maximum experiment duration: 21 days
- Team wants a decision within one product cycle; delaying a good change is costly, but shipping a false positive that hurts retention is more costly
- Historical baselines for new users:
- Signup completion rate: 62%
- Day-1 activation rate: 28%
- Day-7 retained-and-activated rate: 12%
- 7-day subscription trial start rate: 6%
- Engineering can support either user-level randomization or session-level randomization
Deliverables
- Define the hypothesis and choose the single best primary metric for this growth experiment, explaining why it is better than plausible alternatives such as signup completion or trial start rate.
- Specify 2-4 guardrail metrics, an explicit MDE, and calculate the required sample size and expected runtime using the available traffic.
- Choose the unit of randomization, allocation, duration, and any stratification needed across platform or geography.
- Pre-register the analysis plan: statistical test, peeking policy, treatment of secondary metrics, and how you will handle sample ratio mismatch or instrumentation issues.
- State a clear ship / don’t-ship / iterate rule that respects guardrails and practical significance, not just statistical significance.
