Fixed vs Random Effects for SaaS Retention

Business Context

CloudDesk is a B2B SaaS company (project management) with ~8,000 paying customer accounts and ~1.6M monthly active end users. The company shipped an “AI Summary” feature intended to reduce churn by making it easier for teams to catch up on work.

The rollout was not randomized: larger, higher-paying accounts were prioritized by Customer Success and got the feature earlier. You have monthly account-level data for 6 months (3 pre + 3 post relative to each account’s rollout month), and you want to estimate the causal impact of enabling AI Summary on monthly logo retention (whether the account is still active and paying at month end).

A key modeling decision: should you use a fixed-effects model (account-specific intercepts) or a random-effects / mixed-effects model (account intercepts treated as random draws)?

Problem Statement

You are asked to (a) explain the difference between fixed-effects and random-effects models in this context, (b) decide which you would use for the primary estimate of the feature’s effect, and (c) compute and interpret the estimated treatment effect using the summary statistics below.

Given Data

You fit two linear probability models (LPMs) on the same panel dataset (48,000 account-month rows = 8,000 accounts × 6 months). The outcome is retained (0/1). The key regressor is ai_enabled (0/1). You also include month dummies to absorb global seasonality.

Additional dataset facts:

Requirements

1. Conceptual: Explain, in your own words, what changes between a fixed-effects model and a random-effects (mixed) model for account-level panel data.
2. Assumptions: State the key identification assumption behind random effects (often phrased as an exogeneity/orthogonality condition) and why it is questionable here.
3. Quantitative: For each model (A and B), compute the 95% confidence interval for _treat and test whether the effect is statistically different from 0 at  = 0.05.
4. Model choice: In this SaaS rollout scenario, which estimate would you use as the primary number for decision-making and why?
5. Business interpretation: Translate the chosen _treat into an expected change in monthly churn probability and discuss whether it is practically meaningful at CloudDesk’s scale.

Assumptions and Constraints

• Treat the reported coefficients/SEs as coming from large-sample asymptotics (use normal critical values).
• The LPM is used for interpretability; you may comment on limitations (heteroskedasticity, predictions outside [0,1]) but do not need to refit a logistic model.
• Month fixed effects are included in both models.
• The rollout targeting suggests account-level unobservables (e.g., account health) may correlate with ai_enabled.