Context
The Growth team at a16z.com plans to launch a new Portfolio Discovery module on the homepage that recommends portfolio companies and related content. Leadership wants to know whether the launch drives incremental growth in engaged portfolio exploration or simply cannibalizes clicks and sessions that would have happened through existing navigation, search, or newsletter traffic.
Hypothesis Seed
The team believes the new module will increase total weekly engaged portfolio-company visits per eligible user by surfacing relevant companies earlier in the journey. The key risk is cannibalization: users may click the new module instead of using existing discovery paths, producing a surface-level lift in module clicks but no net increase in total portfolio exploration.
Constraints
- Eligible traffic: 180,000 logged-in or cookied homepage users per week
- Homepage eligibility rate: 70% of total site visitors
- Maximum experiment duration: 4 weeks including a 2-day ramp
- Decision deadline is fixed because the module is tied to a quarterly content launch
- False positives are costly: shipping a cannibalizing module would consume homepage real estate and editorial resources
- False negatives are also meaningful: the team will likely deprioritize a broader personalization roadmap if this test fails
Deliverables
- Define a hypothesis that distinguishes incremental growth from cannibalized usage, and specify the primary metric, guardrails, and at least one secondary diagnostic metric.
- Calculate the required sample size for the primary metric using an explicit MDE, and determine whether the test can be completed within the 4-week window.
- Choose the unit of randomization and explain how you will avoid contamination across sessions and surfaces.
- Pre-register an analysis plan: statistical test, handling of multiple metrics, peeking policy, and the exact ship / don’t-ship rule.
- Identify the main experimentation risks, including at least: novelty effect, sample ratio mismatch, and interference/SUTVA concerns if users share content or return across devices.
