Diagnose Flat Revenue Pricing Test

Context

StreamCart, a subscription video platform, tested a redesigned pricing page intended to simplify plan selection and increase paid starts. The experiment showed a higher checkout conversion rate, but total revenue per visitor was flat.

Hypothesis Seed

The new pricing page emphasizes the entry-level plan, reduces copy, and makes the monthly plan more visually prominent than the annual plan. Product believes this will increase paid conversion by reducing friction, but finance is concerned it may shift users toward cheaper plans and dilute revenue.

Constraints

Eligible traffic: 180,000 unique pricing-page visitors per day
Maximum runtime: 14 days; leadership needs a ship/no-ship decision before the next marketing campaign
Allocation target: 50/50 after a 1-day ramp
Baseline paid conversion rate: 8.0%
Baseline revenue per visitor (RPV): $4.80
Historical standard deviation of user-level RPV: $22
Small revenue losses are costly: do not ship if RPV declines by more than 1% relative, even if conversion improves
False positives are more expensive than false negatives because pricing-page changes affect monetization broadly

Task

Design the experiment and explain how you would investigate the apparent contradiction: conversion up, revenue flat.

State the null and alternative hypotheses, define the primary metric, guardrails, and a realistic MDE for the primary decision metric.
Calculate the required sample size with explicit assumptions (alpha, power, baseline, variance/proportion, MDE) and translate it into expected runtime given the traffic constraint.
Choose the unit of randomization and analysis, and pre-register an analysis plan covering the statistical test, peeking policy, multiple-comparison treatment, and ship/no-ship rule.
Describe what you would investigate if paid conversion is significantly up but RPV is flat: plan mix shifts, discounting, annual vs monthly cannibalization, segment heterogeneity, novelty effects, and instrumentation issues.
List the key experiment risks and how you would mitigate them before trusting the result.

Constraints

Eligible traffic: 180,000 unique pricing-page visitors per day

Maximum runtime: 14 days; leadership needs a ship/no-ship decision before the next marketing campaign

Allocation target: 50/50 after a 1-day ramp

Baseline paid conversion rate: 8.0%

Baseline revenue per visitor (RPV): $4.80

Historical standard deviation of user-level RPV: $22

Small revenue losses are costly: do not ship if RPV declines by more than 1% relative, even if conversion improves

False positives are more expensive than false negatives because pricing-page changes affect monetization broadly

Task

Design the experiment and explain how you would investigate the apparent contradiction: conversion up, revenue flat.

State the null and alternative hypotheses, define the primary metric, guardrails, and a realistic MDE for the primary decision metric.

Calculate the required sample size with explicit assumptions (alpha, power, baseline, variance/proportion, MDE) and translate it into expected runtime given the traffic constraint.

Choose the unit of randomization and analysis, and pre-register an analysis plan covering the statistical test, peeking policy, multiple-comparison treatment, and ship/no-ship rule.

Describe what you would investigate if paid conversion is significantly up but RPV is flat: plan mix shifts, discounting, annual vs monthly cannibalization, segment heterogeneity, novelty effects, and instrumentation issues.

List the key experiment risks and how you would mitigate them before trusting the result.

Problem

Context

Hypothesis Seed

Constraints

Task

Diagnose Flat Revenue Pricing Test

Problem

Context

Hypothesis Seed

Constraints

Task