Context
Quora’s growth team is considering a more prominent signup prompt on the logged-out Question Page to increase new-user activation. The team wants to use experimentation, not intuition, to decide whether the change should ship.
Hypothesis Seed
The proposed treatment replaces the current lightweight signup module with a more persistent prompt shown after a logged-out reader scrolls 40% down a question page. The belief is that this will increase the rate at which anonymous readers create an account and become activated users, but it may also hurt reader engagement if the prompt feels intrusive.
Constraints
- Eligible traffic: 1.2M logged-out Question Page visitors per day globally
- Only 60% of that traffic is on surfaces where the new prompt can be reliably rendered
- Maximum experiment runtime: 14 days, after which the PM must make a ship / no-ship decision
- False positives are costly because an intrusive prompt could reduce content consumption and long-term SEO traffic
- False negatives are also meaningful because signup and activation are core product-led growth goals
Task
- Define a clear hypothesis for the experiment and specify the primary metric, guardrails, and at least one secondary metric. Be explicit about the unit of randomization and the unit of analysis.
- Choose an MDE, calculate the required sample size with real numbers, and determine whether the experiment can be completed within 14 days given the available traffic.
- Propose the experiment design, including allocation, duration, any stratification, and how you would pre-register the analysis plan.
- Specify the statistical test you would use, your policy on peeking and multiple comparisons, and how you would handle common experimentation risks such as sample ratio mismatch, novelty effects, and interference.
- State a concrete ship / don’t-ship rule that respects both the primary metric and guardrails.
