Context
AdSpark runs mobile app install campaigns for advertisers across iOS and Android. The ads team has built a new campaign optimization policy that dynamically reallocates budget toward placements predicted to drive more installs at lower cost.
Hypothesis Seed
The new optimizer is expected to improve install conversion rate by showing ads in higher-intent contexts, but it may also reduce advertiser efficiency if it increases cost per install or concentrates spend in a few publishers. You need to design the A/B test before launch.
Constraints
- Eligible traffic: 1.2M ad impressions/day across mobile inventory
- Average click-through rate: 1.8%
- Baseline click-to-install rate: 12%
- Baseline install rate per impression: 0.216%
- Average revenue per 1,000 impressions (RPM): $14.50
- Maximum experiment duration: 21 days
- Allocation must start with a 5% treatment ramp for 1 day, then move to full allocation if instrumentation is healthy
- False positives are expensive because advertisers may shift budget based on misleading performance gains; false negatives are also costly because the quarter's growth target depends on campaign efficiency improvements
Task
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrails, and an explicit MDE that is meaningful for the ads business.
- Calculate the required sample size and expected runtime using the provided baselines. Show the math and state whether the test is feasible within 21 days.
- Choose the unit of randomization and explain how you will handle interference across auctions, users, advertisers, or publishers.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparison policy, SRM checks, and how you will treat any mismatch between unit of randomization and unit of analysis.
- State a clear ship / don't ship / iterate decision rule that respects guardrails, not just the primary metric.
