You are evaluating an online experiment for a digital commerce product, and early results look promising. Before acting on them, you want to make sure the observed lift is not driven by common experimentation pitfalls such as early stopping, novelty, interference across users, or traffic allocation issues.
What are some common pitfalls in online experiments, and how would you avoid them in the way you design, monitor, analyze, and interpret the test?
Ability to name major online experimentation pitfallsUnderstanding of how those pitfalls bias estimatesUse of guardrails and pre-registered decision rulesAwareness of SRM, peeking, novelty, and interference