Custom Rate Limiter for LLM Endpoints | Dataford Interview Questions

Explain implementing a custom rate limiter class in Python (e.g., Token Bucket or Leaky Bucket) to manage rate limits across external LLM provider endpoints.

Implement a token bucket simulation. The function run_rate_limiter(events, capacity, refill_rate) receives timestamped requests as [timestamp, tokens_requested] pairs and returns a Boolean for each request. Start with a full bucket, refill continuously up to capacity, and accept a request only when enough tokens are available. Events are ordered by nondecreasing timestamp.

Constraints

0 <= events.length <= 1000
0 <= timestamp <= 10^9
Events are sorted by nondecreasing timestamp
1 <= capacity <= 10^9
0 < refill_rate <= 10^9
1 <= tokens_requested <= capacity
All timestamps and request sizes are integers

Function Signature

def run_rate_limiter(events, capacity, refill_rate):

Your question is Custom Rate Limiter for LLM Endpoints. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Explain implementing a custom rate limiter class in Python (e.g., Token Bucket or Leaky Bucket) to manage rate limits across external LLM provider endpoints.

Constraints

0 <= events.length <= 1000
0 <= timestamp <= 10^9
Events are sorted by nondecreasing timestamp
1 <= capacity <= 10^9
0 < refill_rate <= 10^9
1 <= tokens_requested <= capacity
All timestamps and request sizes are integers

Function Signature

def run_rate_limiter(events, capacity, refill_rate):

Your question is Custom Rate Limiter for LLM Endpoints. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.