Context
Discord is testing a new feature called Quick Invite Cards, which lets users share a richer invite card into servers and DMs. Product leadership believes the feature could increase server joins and 7-day activation, but because invites spread through social interactions, the wrong randomization unit could contaminate the test.
Hypothesis Seed
The team expects Quick Invite Cards to improve the rate at which exposed users join a server after receiving an invite, and to increase downstream 7-day activation of new joiners. However, if one user in control receives an invite generated by a treated user, interference may violate SUTVA and bias the estimate.
Constraints
- Eligible traffic: 1.2M invite-sending users per day across medium and large servers
- Average eligible clusters: 60,000 servers/day with at least 20 active members
- Baseline invite-to-join conversion: 12.0%
- Maximum experiment duration: 21 days, including ramp
- Leadership wants 80% power at = 0.05 to detect a 5% relative lift in invite-to-join conversion
- False positives are costly because rollout changes invite surfaces across clients; false negatives are acceptable up to one quarter if safety is preserved
- The team can randomize by user, server, or server-community cluster, but engineering prefers the simplest design that remains valid
Tasks
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrails, and at least one secondary metric.
- Choose the unit of randomization and justify it explicitly in the presence of network effects and SUTVA risk.
- Calculate the required sample size using the stated baseline and MDE, then translate it into expected duration under your chosen design.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparison policy, SRM checks, and how you will handle any mismatch between unit of randomization and unit of analysis.
- State a clear ship / dont-ship / iterate rule that respects guardrails, even if the primary metric is significant.
