You work on a mobile game team that is testing a new in-game feature. Early experiment results look promising, but you are concerned that players may be reacting to the feature because it is new rather than because it creates lasting value. The team wants to know whether the observed lift is durable enough to trust.
How do you think about novelty effects when evaluating a new feature? How would you design the experiment and interpret the results so you can distinguish a short-term novelty bump from a real long-term improvement?
Understanding of novelty effects in online experimentsAbility to define a steady-state primary metric and guardrailsPower analysis with an explicit MDEAwareness of peeking and other experimentation pitfalls