Monitor Marketplace Ranking After Launch

Product Context

ShopNow is a large e-commerce marketplace. Its search and recommendation surfaces use an ML ranking system to order products for shoppers, and the team wants a robust post-deployment monitoring design that catches feature drift, model quality regressions, and training-serving skew before they materially hurt revenue.

Scale

Task

Design the end-to-end ML system and monitoring strategy for this ranking stack after deployment. Address the following:

1. Clarify the product objective, prediction target, and what “model quality” means online versus offline.
2. Propose a multi-stage architecture (retrieval → ranking → re-ranking) and explain which features are computed batch vs near-real-time vs fully online.
3. Design a monitoring framework for feature drift, label drift, calibration, delayed feedback, and training-serving skew across the pipeline.
4. Define how you would evaluate the system offline and online, including alert thresholds, dashboards, and rollback criteria.
5. Identify likely failure modes at this scale and explain how the system degrades safely when features or models are stale, missing, or unhealthy.

Constraints

• p99 latency must stay under 120ms globally.
• The business requires fresh inventory and price changes reflected within 10 minutes.
• Some labels are delayed: purchases can occur hours after impression.
• Cost matters: the online ranker must primarily run on CPU, with limited GPU budget for offline training only.
• The system must support auditable monitoring for regulated categories where ranking changes can affect seller exposure.