Context
NimbusCRM is testing new landing-page and ad-copy messaging for its B2B demo request funnel. The growth team believes the new message may attract fewer casual browsers and more qualified prospects, but leadership is concerned it could simply inflate top-of-funnel clicks without improving downstream sales quality.
Hypothesis Seed
The treatment changes headline and CTA copy from feature-focused messaging to ROI-focused messaging. The claim is that this will improve qualified lead rate among visitors who click through to the demo form, not just click-through rate (CTR).
Constraints
- Eligible traffic: 120,000 ad clicks to the landing page per week
- Baseline landing-page-to-qualified-lead conversion rate: 8.0%
- Baseline ad-to-landing-page CTR: 2.5%
- Maximum experiment duration: 21 days
- Sales team can review at most 3,500 submitted leads per week without operational strain
- False positives are costly because low-quality leads waste SDR time; false negatives are acceptable if they avoid degrading lead quality
- The team needs a ship / don’t-ship decision by the end of the 3-week window
Tasks
- Define the null and alternative hypotheses, including whether the test should be one-sided or two-sided.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Be explicit about the unit of analysis and the minimum detectable effect (MDE).
- Calculate the required sample size and determine whether the test can be completed within 21 days given available traffic and operational constraints.
- Propose the experiment design: unit of randomization, allocation, duration, stratification, and how you will handle the fact that CTR is immediate while lead quality is delayed.
- Pre-register the analysis plan, including the statistical test, peeking policy, multiple-comparison policy, and checks for sample ratio mismatch or other validity threats. Then state a clear ship / don’t-ship rule that respects the guardrails.
