/ \
A/B Testing Product Sense
& Statistics & Metrics
\ /
\ /
\ /
Behavioral & Culture
### SQL & Data Manipulation
Data scientists at **Spotify** work directly with high-volume streaming logs and event data. Technical screens and live coding rounds test your ability to query, transform, and derive insights from raw datasets using SQL and Python (Pandas).
**Be ready to go over:**
- **SQL Window Functions** – Mastery of `ROW_NUMBER()`, `RANK()`, `DENSE_RANK()`, `LEAD()`, `LAG()`, and cumulative aggregation frames over partitioned user stream logs.
- **Data Aggregation & Complex Joins** – Joining multi-table schema efficiently (e.g., streaming logs, track metadata, user subscription states) using correct inner, left, and outer join logic.
- **Pandas Manipulation** – Replicating complex data processing steps in Python, including filtering, grouping, reshaping data, and vectorized operations on dataframes.
- **Advanced concepts (less common)** – Query optimization techniques, partition strategies, handling sparse matrix transformations, and window frame specifications (`ROWS BETWEEN`).
**Example scenarios:**
- Write a SQL query using window functions to identify users who listened to the same artist for more than 3 consecutive days.
- Given a raw table of stream events, calculate the 30-day rolling conversion rate of free-tier users converting to premium subscriptions using Pandas.
### A/B Testing & Statistical Inference
Experimentation is a core part of product development at **Spotify**. You will be evaluated on your ability to design sound experiments, calculate sample sizes, analyze results, and avoid common analytical traps in live environments.
**Be ready to go over:**
- **Statistical Significance & Hypothesis Testing** – Formulating null and alternative hypotheses, selecting correct test statistics (t-tests, z-tests, chi-square), interpreting p-values, and calculating confidence intervals.
- **Experimentation Pitfalls** – Identifying and mitigating sample ratio mismatch (SRM), novelty effects, network interference/spillover, and p-hacking.
- **Variance Reduction & Advanced Testing** – Understanding methods like CUPED (Controlled-Experiment Using Pre-Experiment Data) to reduce variance and shorten test duration.
- **Advanced concepts (less common)** – Multi-armed bandits for real-time recommendation optimization, sequential testing frameworks, and causal inference methods (difference-in-differences, synthetic controls).
**Example scenarios:**
- You are testing a new algorithmic playlist recommendation engine. How do you design an A/B test when user streams might spill over through social sharing?
- An A/B test result shows a p-value of 0.04 with a slight lift in streams, but the Sample Ratio Mismatch check fails. How do you investigate and handle this?
### Product Metric Design & Metric Drop Diagnosis
This area evaluates your product sense and analytical troubleshooting abilities. You must demonstrate how to turn ambiguous user behaviors into clear quantitative frameworks and systematically diagnose unexpected metric shifts.
**Be ready to go over:**
- **Product Metric Design** – Defining primary, secondary, and guardrail metrics for new or existing features (e.g., measuring engagement for artist tools like **Marquee** or **Discovery Mode**).
- **Metric Drop Diagnosis** – Utilizing a structured diagnostic framework to isolate root causes when core metrics (such as DAU, completion rates, or sign-ups) suddenly decline.
- **User Funnel Analysis** – Identifying bottlenecks across sign-up, onboarding, playlist creation, and subscription conversion funnels.
- **Advanced concepts (less common)** – Long-term user lifetime value (LTV) modeling, implicit metric feedback loops, and counterfactual analysis.
**Example scenarios:**
- Daily Active Users (DAU) for Spotify's desktop client dropped by 4% over the weekend. Walk me through your step-by-step diagnostic workflow.
- How would you design a composite metric framework to evaluate overall artist success on **Spotify**?