Context
Andreessen Horowitz is considering a faster-growth approach for a consumer startup in its portfolio: changing the primary homepage call-to-action on a16z.com from a generic newsletter signup prompt to a more prominent, benefit-led email capture module. The goal is to learn how experimentation should guide fast product decisions without creating noisy or misleading wins.
Hypothesis Seed
The proposed variant makes the signup value proposition clearer and reduces friction by moving the email form higher on the page. The growth team believes this will increase visitor-to-newsletter-signup conversion, with minimal harm to content engagement quality.
Constraints
- Eligible traffic: 120,000 homepage visitors per day
- 70% mobile, 30% desktop
- Maximum decision window: 14 days
- 50/50 allocation after a brief instrumentation ramp
- False positives are costly because a low-quality signup gain could hurt downstream engagement and brand trust
- False negatives are acceptable if the experiment is underpowered for tiny effects; the team only wants to ship a materially useful improvement
Task
- Define a clear null and alternative hypothesis, the primary metric, 2-4 guardrail metrics, and a realistic MDE.
- Calculate the required sample size using explicit assumptions (alpha, power, baseline conversion, MDE), and determine whether the test can finish within 14 days.
- Choose the unit of randomization and explain whether analysis should happen at the visitor, session, or pageview level.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you will check for instrumentation issues such as sample ratio mismatch.
- State a concrete ship / don’t ship / iterate rule that respects both the primary metric and guardrails, and note key pitfalls such as novelty effects or interference across repeated visitors.
