Product Context

Anthropic wants to scale a training pipeline that produces ranking and retrieval models used to improve Claude.ai response quality, prompt routing, and internal recommendation surfaces such as example prompts and tool suggestions. You are given a proposed pipeline that works today; the question is where it will break at 10x scale and how you would redesign it.

Scale

Task

Assume the current architecture is: application logs and feedback events land in Kafka, batch ETL builds features in a warehouse, daily training jobs produce a retrieval model and a ranker, models are registered and deployed to online serving, and online predictions are logged back for future training.

Design the 10x-scale version and explain where the current design will fail first.

1. Clarify the functional and non-functional requirements for both training and serving.
2. Identify the main bottlenecks at 10x scale across data ingestion, feature computation, training, indexing, and online serving.
3. Propose an end-to-end architecture covering offline training, online retrieval/ranking/re-ranking, and the feedback loop.
4. Choose models for each stage and justify tradeoffs between quality, freshness, cost, and latency.
5. Define how you would evaluate the system offline and online, including rollout strategy.
6. Call out failure modes such as feature drift, training-serving skew, stale indexes, and logging gaps, with detection and mitigation.

Constraints

• Some labels are delayed or noisy: explicit thumbs-up/down arrives quickly, but longer-term satisfaction signals may take days.
• Anthropic needs reproducible training datasets for audits, while also supporting near-real-time feature freshness.
• Raw prompts may have retention and privacy constraints; not all data can be stored indefinitely.
• Online serving must degrade gracefully if retrieval or ranking models are unavailable.