Context
StreamCart, a subscription video platform, tested a redesigned pricing page intended to simplify plan selection and increase paid starts. The experiment showed a higher checkout conversion rate, but total revenue per visitor was flat.
Hypothesis Seed
The new pricing page emphasizes the entry-level plan, reduces copy, and makes the monthly plan more visually prominent than the annual plan. Product believes this will increase paid conversion by reducing friction, but finance is concerned it may shift users toward cheaper plans and dilute revenue.
Constraints
- Eligible traffic: 180,000 unique pricing-page visitors per day
- Maximum runtime: 14 days; leadership needs a ship/no-ship decision before the next marketing campaign
- Allocation target: 50/50 after a 1-day ramp
- Baseline paid conversion rate: 8.0%
- Baseline revenue per visitor (RPV): $4.80
- Historical standard deviation of user-level RPV: $22
- Small revenue losses are costly: do not ship if RPV declines by more than 1% relative, even if conversion improves
- False positives are more expensive than false negatives because pricing-page changes affect monetization broadly
Task
Design the experiment and explain how you would investigate the apparent contradiction: conversion up, revenue flat.
- State the null and alternative hypotheses, define the primary metric, guardrails, and a realistic MDE for the primary decision metric.
- Calculate the required sample size with explicit assumptions (alpha, power, baseline, variance/proportion, MDE) and translate it into expected runtime given the traffic constraint.
- Choose the unit of randomization and analysis, and pre-register an analysis plan covering the statistical test, peeking policy, multiple-comparison treatment, and ship/no-ship rule.
- Describe what you would investigate if paid conversion is significantly up but RPV is flat: plan mix shifts, discounting, annual vs monthly cannibalization, segment heterogeneity, novelty effects, and instrumentation issues.
- List the key experiment risks and how you would mitigate them before trusting the result.
