Context
FitLoop, a subscription fitness app, wants to increase new-user acquisition by making its referral prompt more prominent in the post-workout flow. The growth team believes the change will drive more invites sent, but leadership is concerned that an aggressive prompt could hurt workout completion, retention, or paid conversion.
Hypothesis Seed
The treatment replaces a small referral link with a full-screen referral card shown immediately after workout completion, with one-tap sharing and a stronger incentive message. The team expects this to increase referral invite rate and downstream referred sign-ups, but the experiment must be designed with clear guardrails so growth does not come at the expense of core engagement or monetization.
Constraints
- Eligible traffic: 120,000 completed-workout users per day
- 70% of users are on mobile app, 30% on web
- Maximum experiment duration: 14 days
- Allocation may be 50/50 after a 1-day safety ramp
- Baseline referral invite rate among eligible users: 8.0%
- Baseline 7-day workout return rate: 34%
- Baseline subscription conversion within 7 days: 4.5%
- Product leadership considers a false positive costly because a harmful prompt could reduce retention; a false negative is acceptable if it avoids shipping a bad experience
Deliverables
- Define the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Be explicit about why each guardrail is needed and what threshold would block launch.
- State the null and alternative hypotheses, choose the unit of randomization, and justify the unit of analysis.
- Calculate the required sample size for the primary metric using a pre-specified MDE, and translate it into expected runtime under the traffic constraints.
- Pre-register an analysis plan covering the statistical test, peeking policy, multiple comparisons policy, and how you will check for sample ratio mismatch.
- Give a clear ship / don’t-ship / iterate rule that respects the guardrails, including what you would do if the primary metric improves but a guardrail degrades.
