Business Context
You’re the analytics lead for ShopSwift, a large e-commerce marketplace (≈ 35M MAUs, 4.5M DAUs, $6B annual GMV) operating in the US and UK. The company is rolling out a one-page checkout redesign intended to reduce friction and increase conversion. The redesign is being ramped from 10% to 50% of traffic over the next two weeks, and leadership wants a dashboard that can be used in daily standups and weekly exec reviews.
Two days into the ramp, the VP of Product messages: “I’m seeing conversion down in a few views. Is this real? Should we pause the rollout?” Meanwhile, the Head of Payments is worried about an increase in payment declines and potential processor issues. Customer Support reports a spike in “can’t place order” tickets.
You have to walk the interview panel through a dashboard you built (or would build) for this scenario: who it’s for, what decision it enables, and how it avoids misleading stakeholders.
Metric Scenario
The dashboard must answer:
- Is checkout performance improving or degrading overall?
- If it’s changing, where in the funnel is it happening (shipping, payment, review, confirmation)?
- Is the change driven by mix shifts (device, country, traffic source, new vs returning) or by true product impact?
- What are the leading indicators that predict next-week revenue impact?
- What guardrails ensure we don’t “win conversion” by harming cancellations, fraud, or customer experience?
Data Available
| Source | What it contains | Granularity |
|---|
checkout_events | Step events: view_shipping, submit_shipping, view_payment, submit_payment, place_order, error codes | event-level |
orders | order_id, user_id, order_ts, subtotal, shipping_fee, tax, discount, status (placed/canceled/returned) | order-level |
payments | payment_attempt_id, order_id, processor, auth_result, decline_reason, latency_ms, 3DS_required | attempt-level |
sessions | session_id, user_id, device, app_version, referrer, country, experiment_bucket | session-level |
support_tickets | ticket_id, created_ts, category, linked_order_id, free-text tags | ticket-level |
Requirements (what you must produce)
- Define the primary KPI your dashboard will anchor on and justify why it’s the right executive KPI for checkout health.
- Specify the audiences (e.g., VP Product vs Payments Ops vs On-call Engineering) and what decision each audience can make from the dashboard (pause rollout, rollback, hotfix, processor reroute, etc.).
- Design the dashboard layout: the top-line tile(s), the funnel section, and the diagnostic drill-downs. Be explicit about time windows (hourly vs daily), comparison baselines, and alert thresholds.
- Provide a metric decomposition plan that would let you isolate whether the issue is UX friction, payment processing, traffic mix, or instrumentation.
- List guardrail metrics and explain trade-offs (e.g., conversion up but cancellations/fraud up).
- Describe how you would ensure data quality and trust (late events, duplicate orders, bot traffic, experiment bucketing correctness).
Constraints:
- You have 48 hours to ship a first version used by executives.
- The dashboard must support near-real-time monitoring (≤60 minutes latency).
- The redesign cannot be evaluated purely on revenue because of delayed cancellations/returns; you need leading indicators.
