Measure Onboarding Cohort Retention Lift

Business Context

Splice recently launched a new onboarding flow in the Splice mobile app and web experience to help new users find relevant sounds, create their first pack, and start a trial faster. Product leadership wants to know whether the change is improving long-term retention, not just Day 1 activation.

Metric Scenario

The new onboarding launched on April 1 for all new signups in the US, UK, and Canada. In the 8 weeks before launch, weekly new-user volume averaged 42,000 signups; in the 8 weeks after launch, it averaged 45,000. Early metrics improved: onboarding completion rose from 61% to 74%, first sample download within 24 hours rose from 38% to 49%, and trial start rate rose from 22% to 27%. However, 30-day paid retention is only partially observable for the most recent cohorts, and leadership is concerned that the flow may be increasing shallow activation without improving durable usage.

Requirements

1. Define the primary cohort-based retention metric you would use for this onboarding change, including cohort entry date, retained-user definition, and observation window.
2. Explain how you would compare pre-launch and post-launch cohorts while accounting for incomplete maturity, seasonality, and acquisition mix shifts.
3. Describe how you would decompose retention movement to determine whether gains come from better activation, stronger content engagement, or higher trial-to-paid conversion.
4. Identify the leading indicators you would use before long-term retention fully matures, and explain why they are predictive.
5. Call out at least three pitfalls or confounders that could make the onboarding change look better or worse than it is.

Data Available

• user_signup with signup timestamp, platform, country, acquisition_channel, experiment flags
• onboarding_events with step completion, genre selections, skipped steps, tutorial completion
• content_engagement with sample downloads, likes, pack follows, searches, DAU activity
• subscription_events with trial start, trial end, paid conversion, cancellation, renewal
• creator_profile and catalog_interactions with genres, content types, and recommendation exposure