Business Context

You’re the analytics partner for ShopSwift, a high-volume e-commerce marketplace (~9M MAUs, 1.4M DAUs) selling consumer electronics and home goods across the US and EU. Checkout is mission-critical: the company does ~$2.5B annual GMV and leadership tracks purchase conversion as a top KPI. A recent mobile checkout redesign shipped to 100% of users after positive usability tests.

Two weeks after launch, the VP of Product flags that mobile purchase conversion dropped from 3.20% to 2.95% (-7.8% relative), representing an estimated $3–5M monthly GMV risk. The design team’s core assumption was: “Reducing visual clutter and moving shipping options behind a ‘See delivery options’ accordion will reduce cognitive load and increase conversion.” They believe the conversion drop is due to seasonality and want to keep the new design.

You’re asked to use quantitative data to validate or disprove the design assumption and recommend what to do next.

Metric Scenario

The redesign made three notable changes on mobile:

1. Shipping options are collapsed by default (accordion).
2. Promo code entry moved from the payment step to the cart.
3. The primary CTA changed from “Place order” to “Confirm purchase”.

Customer support also reports a small increase in tickets tagged “shipping cost surprise,” but the design team argues it’s anecdotal.

Data Available

You have a Snowflake warehouse with event instrumentation and order outcomes.

Your Task (what you must produce)

1. Define the primary KPI you will use to evaluate the redesign (and at least 2 guardrails). Be explicit about numerator/denominator and inclusion rules.
2. Decompose the KPI into a funnel and identify which step(s) explain the conversion decline. Provide at least 4 segmentation cuts that could reveal the driver.
3. Disprove or support the design assumption using quantitative evidence. Your answer should clearly state what pattern in the data would falsify the assumption (e.g., “less clutter helped, but shipping accordion increased drop-off at step X”).
4. Propose 3 testable hypotheses linking the UI changes to metric movement (e.g., shipping fee surprise, promo code behavior, CTA semantics), and specify what metrics/events you’d query for each.
5. Recommend concrete next actions (e.g., rollback, hotfix, targeted experiment) and what success criteria you’d use over the next 1–2 weeks.

Constraints

• You have 5 business days to deliver a recommendation.
• You cannot run a brand-new long experiment immediately, but you can propose a small holdout or targeted A/B if justified.
• Any fix must maintain compliance with EU pricing transparency requirements (shipping fees must be clearly disclosed before purchase).