Design a Personalized Product Recommender

Scenario

You are designing a personalized recommendation system for a large e-commerce marketplace. The system powers the home feed and product detail page recommendations, and it is expected to improve conversion, basket size, and repeat engagement. Users should see relevant products based on their behavior, context, and catalog changes, while new products must become discoverable quickly. The system must operate reliably in production and support rapid iteration by data science and engineering teams.

Scale

Question

How would you design this end-to-end machine learning system for production, including the data pipeline, model stages, serving architecture, evaluation approach, and how you would handle drift, skew, and operational failures at scale?