Context
FitLoop, a consumer fitness app, wants to test a redesigned referral prompt shown after a user completes a workout. The growth team believes the new prompt will increase invites sent and downstream new-user signups, but leadership is concerned about harming workout completion and user retention.
Hypothesis Seed
The treatment replaces a small text link with a full-screen referral card that highlights a free 14-day premium trial for both inviter and invitee. The team expects this to improve referral conversion by making the value proposition clearer, but it may also feel interruptive and reduce core engagement.
Constraints
- Eligible traffic: 120,000 workout-completing users per day
- 70% of users are on mobile; 30% on web
- Maximum experiment runtime: 14 days, after which marketing needs a launch decision
- False positives are costly because a bad rollout could hurt retention during peak New Year acquisition season
- False negatives are also meaningful because referrals are one of the app's cheapest growth channels
- Engineering can support only one primary success metric for the ship decision
Task
- Define the experiment hypothesis, the single primary metric, and 2-4 guardrail metrics. Be explicit about metric formulas, units, and why each guardrail matters.
- Choose an MDE and calculate the required sample size and expected runtime using the traffic above. Show the math with concrete assumptions.
- Specify the experiment design: unit of randomization, allocation, duration, and any stratification or ramp plan.
- Pre-register the analysis plan: statistical test, peeking policy, handling of multiple metrics, and how you will deal with any mismatch between unit of randomization and unit of analysis.
- State a clear ship / don’t-ship / iterate rule that respects the guardrails, and call out the main risks such as novelty effects, sample ratio mismatch, and interference from social referrals.
