Context
Assume Andreessen Horowitz is testing a new annual subscription price and checkout presentation for an a16z premium research product. Leadership wants to know whether the change creates real incremental revenue or merely pulls forward conversions that would have happened anyway.
Hypothesis Seed
The proposed treatment increases the annual plan price from $199 to $219 and adds a “best value” framing on the pricing page. The growth team believes this may raise short-term revenue per visitor by improving monetization from high-intent users, but it could also reduce conversion, increase refunds, or shift users from monthly to annual plans without improving longer-term net revenue.
Constraints
- Eligible traffic: 120,000 unique pricing-page visitors per week
- Current checkout funnel baseline: 8.0% purchase conversion from pricing-page visitor to paid subscription
- Baseline 60-day net revenue per pricing-page visitor: $15.20
- Maximum experiment runtime: 28 days, because finance needs a recommendation before the next planning cycle
- False positives are costly: shipping a bad price could depress acquisition and brand trust for a full quarter
- False negatives are also meaningful: the team will defer packaging work if the test is inconclusive
- You may assume 85% of visitors are new prospects and 15% are returning prospects
Deliverables
- State the null and alternative hypotheses, including whether you would use a one-sided or two-sided test.
- Define the primary metric, secondary metrics, and guardrails that distinguish durable revenue lift from temporary behavior shifting.
- Calculate sample size and minimum detectable effect, then determine whether the test is feasible within 28 days given the available traffic.
- Choose the unit of randomization, allocation, duration, and any stratification or blocking strategy.
- Pre-register the analysis plan, including test choice, peeking policy, multiple-comparison handling, and how you would diagnose pitfalls such as novelty effects, sample ratio mismatch, and interference across users.
