Context
Splice is considering a change to the Sounds search results page: a more prominent inline preview player and clearer CTA on each sample card. The Growth team believes this will help users evaluate sounds faster and increase the rate at which a search session leads to a sound being added to a project or downloaded.
Hypothesis Seed
The proposed treatment makes sound discovery easier on Splice Sounds search. The expectation is a modest lift in search-to-download conversion, but there is concern that the new UI could reduce search depth, hurt preview engagement quality, or create a short-lived novelty effect.
Constraints
- Eligible traffic: 180,000 unique logged-in Splice users per day visit Sounds search
- Average baseline search-to-download conversion: 12.0% per eligible user-day
- Max experiment window: 21 days, after which Product must decide whether to ship, iterate, or drop
- Allocation target: 50/50 after a brief instrumentation ramp
- False positives are costly because shipping a noisy win to a core discovery surface could reduce long-term trust and retention
- False negatives are also meaningful because search is a major growth lever, but the team prefers to be conservative
Deliverables
- Define a clear null and alternative hypothesis for this experiment, and state whether you would use a one-sided or two-sided test.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Include the unit of analysis and an explicit minimum detectable effect (MDE).
- Calculate the required sample size per arm and translate it into how long the experiment needs to run given the available traffic. Show the math and state assumptions.
- Choose the unit of randomization, allocation plan, and duration. Explain how you will handle peeking, multiple comparisons, and any mismatch between unit of randomization and unit of analysis.
- Describe the ship / don’t-ship rule and call out key pitfalls such as sample ratio mismatch, novelty effects, and any interference or SUTVA concerns relevant to Splice Sounds discovery.
