Business Context

HelloFresh launched a redesign of the My Menu experience in the app at the start of Week 5. In the same period, the team noticed a drop in weekly engagement and wants to know whether the decline came from the product change or normal seasonal behavior.

Problem Statement

Use a difference-in-differences approach to test whether the redesign caused a statistically significant decline in engagement after controlling for seasonality using a holdout market.

Given Data

Engagement is measured as the weekly proportion of active subscribers who opened My Menu at least once.

Assume the holdout market captures the same seasonal forces affecting the treated market. Test at a significance level of 5%.

Requirements

1. Compute the pre/post change in engagement for the treated market.
2. Compute the pre/post change in engagement for the holdout market.
3. Estimate the difference-in-differences effect.
4. Calculate the standard error, z-statistic, and two-sided p-value for the DiD estimate.
5. Construct a 95% confidence interval.
6. Conclude whether the observed decline is more consistent with seasonality alone or with a negative product impact.

Assumptions

• Subscriber-level outcomes are independent within each cell.
• The holdout market was not exposed to the redesign.
• Parallel trends holds: absent the redesign, treated and holdout markets would have experienced the same seasonal change.
• Large-sample normal approximation is appropriate for engagement proportions.