Context
A Meta marketing analytics team is evaluating two creative variants for an Instagram Feed paid social campaign for a large advertiser. The advertiser wants to know whether a new video-first creative improves downstream conversion efficiency versus the current static-image creative before rolling it out to the full budget.
Hypothesis Seed
The treatment creative is a short-form vertical video optimized for mobile attention, while control is the current static image. The team believes the new creative will increase purchase conversion rate by improving thumb-stop behavior and click quality, not just top-of-funnel engagement.
Constraints
- Eligible delivery: 1.2 million impressions/day on Instagram Feed
- Estimated reach: 480,000 unique people/day
- Campaign budget allows a maximum 14-day experiment
- Randomization must happen inside Meta Experiments / split test framework
- False positives are costly because the advertiser may shift several million dollars of spend to an inferior creative
- False negatives are also costly because the campaign is seasonal and there may not be time for a second full test
- You may assume baseline post-click purchase conversion rate is 3.0% and baseline CTR is 1.2%
Deliverables
- Define the experiment hypothesis, the primary metric, 2-4 guardrail metrics, and a clear MDE.
- Calculate the required sample size and determine whether the test can finish within 14 days given available traffic.
- Choose the unit of randomization, allocation, duration, and any stratification or variance-reduction approach you would use.
- Pre-register the analysis plan: statistical test, handling of multiple comparisons, peeking policy, and how you will diagnose sample ratio mismatch.
- State a ship / don’t-ship / iterate rule that respects both the primary metric and guardrails, and name the main pitfalls for creative testing on Meta surfaces.
