Context
Andreessen Horowitz is considering a growth change in the a16z.com article share flow: after a reader shares a post, the product would show a stronger prompt to invite collaborators into a private reading circle. The team believes this could increase sharing and repeat engagement, but the product has strong network effects because one user’s invite changes another user’s experience.
Hypothesis Seed
The proposed treatment adds a more prominent post-share invite module with suggested contacts and a one-tap “start a reading circle” CTA. The belief is that increasing invites will create more active reading circles, which in turn lifts 7-day retained sharing behavior. However, because invited users may interact with treated users, a naive user-level A/B test may violate independence assumptions.
Constraints
- Eligible traffic: 180,000 weekly active readers per week on a16z.com, of which 36% use the share flow at least once
- Average cluster size from historical invite graphs: 1.8 users in the first week after an invite
- Maximum decision window: 28 days
- False positives are costly because shipping a noisy invite mechanic can create spammy behavior and damage brand trust
- False negatives are also meaningful because the team has only one major Q3 growth slot for this surface
Deliverables
- Define a testable hypothesis, the primary metric, at least three guardrails, and an explicit MDE.
- Design the experiment, including the unit of randomization that accounts for network effects, traffic allocation, duration, and any stratification or clustering strategy.
- Calculate the required sample size with real numbers and determine whether the test is feasible within 28 days.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparisons policy, and how you will detect/handle sample ratio mismatch.
- State a clear ship / don’t ship / iterate rule that respects guardrails, and explain what you would do if the primary metric improves but a network-effect-related pitfall makes interpretation ambiguous.
