Context
FitFlow, a mobile fitness app, is testing a new feature that lets users send a one-tap "nudge" to friends who have not completed their daily workout. Product leadership wants to know whether the feature improves engagement without creating notification fatigue or hurting retention.
Hypothesis Seed
The team believes adding the nudge button on the friends tab will increase next-day workout completion by making social accountability more salient. However, the feature could also increase notification opt-outs, app uninstalls, or reduce user satisfaction if it feels spammy.
Constraints
- Eligible traffic: 120,000 active mobile users per day
- 70% of users have at least one friend and are eligible for the experiment
- Maximum test duration: 14 days; a launch decision is required before the next release train
- Randomization must be stable across iOS and Android app sessions
- False positives are costly because a bad social feature can increase churn; false negatives are moderately costly because this is a promising engagement lever
Deliverables
- Define a clear primary metric for this experiment and explain why it best captures the intended user value.
- Propose 2-4 guardrail metrics, with explicit thresholds, to ensure the feature does not harm the user experience or business.
- Calculate the required sample size and expected duration using a stated MDE, baseline, alpha, and power.
- Choose the unit of randomization and analysis approach, and explain how you will handle interference risk from social connections.
- Pre-register an analysis plan including the statistical test, peeking policy, multiple-comparisons policy, and a ship / don't-ship rule that respects guardrails.
Assume the current baseline next-day workout completion rate among eligible users is 28%. The PM says the smallest worthwhile improvement is 1.5 percentage points absolute. Historical data shows notification opt-out is 4.0%, 14-day retention is 36%, and uninstall rate is 0.8% among eligible users.
