Rate-Limited Concurrent Web Scraper

Explain how you would build a lightweight web scraper that respects rate limits while still maximizing throughput. Describe the concurrency model, request scheduling, retries, and how you would react to 429 responses.

Token bucket style rate limiting

A rate limiter controls how many requests may be sent over time. A token bucket model is a practical choice because it allows short bursts up to capacity while enforcing a steady average rate.

tokens = min(capacity, tokens + refill_rate * elapsed)
if tokens >= 1:
    tokens -= 1
    send_request()

Async concurrency for I/O-bound work

Web scraping is usually network-bound rather than CPU-bound, so asynchronous I/O can keep many requests in flight without creating many threads. This improves throughput while still allowing centralized rate-limit enforcement.

async def worker(queue):
    while True:
        url = await queue.get()
        await limiter.acquire()
        await fetch(url)

Adaptive retry and backoff

Transient failures such as timeouts, 5xx responses, or 429s should not be retried immediately. Exponential backoff with jitter reduces synchronized retries and helps the scraper recover without overwhelming the target.

delay = min(max_delay, base * (2 ** attempt)) + random.uniform(0, jitter)

Per-host scheduling and fairness

Different hosts may expose different limits, so a single global queue is often insufficient. Keeping separate host-level budgets prevents one domain from starving others and avoids violating stricter limits on a subset of targets.

host = urlparse(url).netloc
limiters[host].acquire()

Observability-driven throughput tuning

Maximizing throughput safely requires measuring request rate, latency, error rate, and 429 frequency. These signals let the scraper adjust concurrency and pacing instead of relying on a fixed guess.

metrics = {
    'rps': rps,
    'p95_latency': p95,
    'rate_limited': count_429
}