Business Context
Instacart’s grocery delivery app serves ~12M monthly active customers across the US and Canada, processing millions of orders per week. A recent redesign of the cart and checkout experience was intended to reduce time-to-checkout and increase order conversion, but leadership is seeing conflicting signals: overall Order Conversion Rate is flat, while revenue per visitor is down ~3% week-over-week. The Checkout PM believes the redesign may have introduced friction for certain user segments (e.g., first-time buyers on iOS, EBT-eligible customers, or users with substitutions).
You are the analytics lead asked to design a funnel analysis that can be used both for (a) one-time diagnosis of the redesign impact and (b) ongoing monitoring as the team iterates weekly.
Metric Scenario
Stakeholders (PM, Design, Payments, and Ops) are asking:
- “Where exactly are users dropping off—cart, address, delivery slot selection, payment, or place order?”
- “Is this a true behavior change or a tracking/definition issue?”
- “Which fixes should we prioritize this sprint to recover revenue?”
Complications:
- Users can browse and add items while logged out, then log in at checkout.
- Some steps are optional depending on context (e.g., tipping shown only in some markets; substitutions only if items are out of stock).
- Delivery slot availability can block checkout, and is influenced by operational capacity.
- Users may switch devices (web to mobile) or return later (multi-session funnel).
Data Available
| Source | What it contains | Grain |
|---|
app_events | Client events: screen_view, tap, errors, latency | event-level |
cart_events | add/remove items, cart value, item count | event-level |
checkout_sessions | checkout_id, user_id, start/end timestamps, device, app_version | checkout attempt |
orders | order_id, checkout_id, submitted_at, subtotal, fees, tip, promos | order |
payments | payment_method, auth/3DS outcomes, declines, retries | payment attempt |
delivery_slots | slots shown, slots available, slot selected, capacity flags | request-level |
user_dim | country, state, new vs existing, membership tier | user-level |
Your Task (What you must produce)
- Define the funnel: enumerate the steps from “shopping intent” to “order placed,” including how you handle optional steps (tips, substitutions) and blocking states (no delivery slots).
- Specify inclusion/exclusion rules: what is the funnel population (all visitors, shoppers with cart > $0, checkout starters)? How do you handle bots, internal testers, and logged-out users?
- Choose the unit of analysis: user-level vs checkout-attempt-level vs session-level. Explain trade-offs and when you’d use each.
- Design the measurement approach:
- event instrumentation needed at each step
- how you will dedupe events and handle retries
- time windowing (same-session vs 7-day completion)
- Compute and present funnel metrics: step-to-step conversion, overall conversion, time between steps, and error rates.
- Decompose and diagnose: propose at least 4 segment cuts and how each would change your interpretation (e.g., payments vs ops-driven drop-off).
- Actionability: based on likely outcomes, propose concrete product/engineering actions and what guardrails you’d monitor to avoid regressions.
Assume you have one week to deliver an analysis readout to the VP of Product and a dashboard spec for ongoing tracking.
