Test Instagram Ad Creative Variants

Context

A Meta marketing analytics team is evaluating two creative variants for an Instagram Feed paid social campaign for a large advertiser. The advertiser wants to know whether a new video-first creative improves downstream conversion efficiency versus the current static-image creative before rolling it out to the full budget.

Hypothesis Seed

The treatment creative is a short-form vertical video optimized for mobile attention, while control is the current static image. The team believes the new creative will increase purchase conversion rate by improving thumb-stop behavior and click quality, not just top-of-funnel engagement.

Constraints

• Eligible delivery: 1.2 million impressions/day on Instagram Feed
• Estimated reach: 480,000 unique people/day
• Campaign budget allows a maximum 14-day experiment
• Randomization must happen inside Meta Experiments / split test framework
• False positives are costly because the advertiser may shift several million dollars of spend to an inferior creative
• False negatives are also costly because the campaign is seasonal and there may not be time for a second full test
• You may assume baseline post-click purchase conversion rate is 3.0% and baseline CTR is 1.2%

Deliverables

1. Define the experiment hypothesis, the primary metric, 2-4 guardrail metrics, and a clear MDE.
2. Calculate the required sample size and determine whether the test can finish within 14 days given available traffic.
3. Choose the unit of randomization, allocation, duration, and any stratification or variance-reduction approach you would use.
4. Pre-register the analysis plan: statistical test, handling of multiple comparisons, peeking policy, and how you will diagnose sample ratio mismatch.
5. State a ship / don’t-ship / iterate rule that respects both the primary metric and guardrails, and name the main pitfalls for creative testing on Meta surfaces.