You are designing a recommendation system for a trading platform that surfaces personalized insights such as market commentary, analyst notes, macro updates, and strategy signals. Users open the platform throughout the trading day and expect relevant, timely insights that match their instruments, sectors, and trading style. The surface is important for engagement and for helping users act on fast-moving market information. Freshness matters because an insight that is useful at market open may be stale an hour later.
| Signal | Value |
|---|---|
| DAU | 2M |
| Peak QPS (feed requests) | 18K |
| Active insights catalog | 12M |
| New insights per day | 450K |
| Market-hours traffic share | 70% of daily requests in 8 hours |
| Per-request latency budget (p99) | 180ms |
How would you design this end-to-end system so it can retrieve, rank, and serve personalized trading insights at this scale while staying fresh, measurable, and reliable? Explain the architecture, model choices, serving strategy, evaluation plan, and the failure modes you would expect in production.