Context
StreamRead, a mobile news app, tested a redesigned article recommendation module on the home feed. Early results suggest the variant increases article click-through but reduces time spent on site, and the PM wants a launch recommendation.
Hypothesis Seed
The new module surfaces more visually prominent headlines and shorter article previews. The team believes this will increase the share of users who click into at least one article, but there is concern that it may drive lower-quality clicks that reduce downstream engagement.
Constraints
- Eligible traffic: 600,000 daily active users per day on the home feed
- Maximum experiment duration: 14 days, after which the team must decide ship / iterate / stop
- Only 80% of DAU is eligible because logged-out users cannot be consistently tracked
- False positives are costly: shipping a clickbait-like experience could hurt long-term retention and ad revenue
- False negatives are also meaningful: the feed team has only one launch slot this quarter
- The experiment platform supports 50/50 user-level randomization and daily safety monitoring, but no adaptive bandits
Deliverables
- State the null and alternative hypotheses, and decide whether the primary test should be one-sided or two-sided.
- Define the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Be explicit about the unit of analysis and the minimum detectable effect (MDE).
- Calculate the required sample size using real numbers, then translate it into expected runtime given the available traffic.
- Propose the experiment design: unit of randomization, allocation, duration, stratification, and a pre-registered analysis plan covering peeking and multiple comparisons.
- Explain how you would make the launch decision if clicks are significantly up but time on site is significantly down, including what thresholds would trigger ship, don’t ship, or iterate.
Assume the current baseline home-feed click-through rate is 32%, average engaged minutes per user-day is 11.0, and the business considers a +1.5% relative lift in CTR meaningful only if engagement and monetization are not materially harmed.
