Design ML-Powered API Rate Limiter

Product Context

NovaAPI is a public API platform used by mobile apps, SaaS vendors, and internal services. The company wants to replace static per-key quotas with an ML-driven rate limiter that predicts abusive or bursty traffic in real time and decides whether to allow, delay, challenge, or reject requests while minimizing impact on legitimate customers.

Scale

Task

Design an end-to-end ML system for adaptive rate limiting. Your design should address:

1. How to define the prediction problem and translate model outputs into rate-limit actions (allow, soft-throttle, hard-throttle, challenge).
2. A multi-stage online architecture for high-throughput decisioning, including fast candidate policy retrieval, ML scoring, and final policy re-ranking / rule enforcement.
3. The offline and streaming data pipelines for features, labels, training, and feedback loops, including delayed labels for abuse outcomes.
4. Model choices for each stage, with clear tradeoffs between latency, interpretability, and recall of abusive traffic.
5. Evaluation strategy: offline metrics, online experiments, and operational guardrails.
6. Failure modes, especially feature drift, training-serving skew, regional outages, and false positives on high-value customers.

Constraints

• The system must preserve existing contractual limits for enterprise customers; ML can only tighten or relax within configured bounds.
• Some abuse labels arrive hours later from downstream fraud investigations or chargebacks.
• PII cannot be used directly in model features; features must satisfy internal privacy policy.
• The platform must fail open for a small allowlisted set of critical internal services, but fail closed for clearly malicious traffic patterns.
• Serving cost target is under $0.00002 per request at peak scale.