Bandit vs A/B for Splice CTA | Dataford Interview Questions

Your question is Bandit vs A/B for Splice CTA. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

Splice is testing three homepage CTA variants on the Splice Sounds landing page to increase trial starts. Product wants to know whether to run a standard fixed-horizon A/B/n test or switch to a multi-armed bandit that reallocates traffic during the experiment.

Given Data

After the first 20,000 visitors, traffic was split evenly across three variants.

Variant	Visitors	Trial Starts	Observed Conversion Rate
A (current CTA)	6,667	800	12.00%
B ("Start creating")	6,666	860	12.90%
C ("Try Sounds free")	6,667	920	13.80%

Additional planning inputs:

Quantity	Value
Remaining visitors expected this month	180,000
Significance level for fixed A/B/n follow-up	0.05
Desired power	0.80
Minimum detectable lift vs control	1.5 percentage points
Value per incremental trial start	$18

Problem Statement

Decide whether Splice should continue with a standard fixed-horizon experiment or switch to a multi-armed bandit. Your answer should quantify both the learning objective and the traffic-allocation objective.

Requirements

Compute the posterior mean conversion rate for each arm using a Beta(1,1) prior.
Estimate the probability each arm is best, and identify the current leader.
Calculate the expected number of incremental trial starts over the remaining 180,000 visitors if Splice:
- keeps equal allocation across all 3 arms, or
- sends all remaining traffic to the current best observed arm.
Compute the approximate per-variant sample size needed for a fixed-horizon test to detect a 1.5 percentage point lift from 12.0% to 13.5%.
Based on the numbers, recommend bandit vs standard A/B/n, and explain when each is preferable.

Assumptions

Visitors are independent and randomly assigned in the initial phase.
Conversion is binary: visitor starts a trial or not.
No interference across users and no major seasonality shift during the month.
For sample-size planning, use a normal approximation for two proportions and equal-sized groups.