Context
LoopChat, a consumer messaging app, wants to launch a new growth feature: after a user sends their first message of the day, they see a lightweight prompt to invite friends via contacts. The growth team believes this will increase new-user acquisition, but product leadership is concerned it could hurt engagement if the prompt feels spammy.
Hypothesis Seed
The treatment adds an in-app invite prompt with one-tap sharing and personalized copy. The team expects it to increase successful invites and downstream activated signups, while keeping core engagement and retention stable.
Constraints
- Eligible traffic: 120,000 existing active users per day
- Average eligibility rate: 1 exposure opportunity per eligible user per day
- Baseline successful invite rate: 8.0% of eligible users send at least one invite within 24 hours
- Maximum experiment duration: 21 days
- Decision deadline is fixed because marketing needs a launch recommendation before the next campaign
- False positives are costly because an annoying growth surface could reduce long-term retention; false negatives are acceptable if modest
Tasks
- Define the primary metric, 2-4 guardrail metrics, and a realistic MDE for this growth feature.
- Calculate the required sample size and determine whether the test can be completed within 21 days given the available traffic.
- Choose the unit of randomization, allocation, duration, and any stratification/blocking you would use.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you will handle any mismatch between unit of randomization and analysis.
- State a clear ship / don’t-ship / iterate rule that respects guardrails, and identify the main pitfalls for this experiment.
