Rate Limiting
IntermediateA cap on how many requests a client may make per time window — the traffic rule whose 429 errors shape how all scraping and API work is paced.
In depth
Rate limiting restricts how often a client can hit a server: so many requests per second, minute, or day, tracked per IP address, API key, account, or session. Exceed the budget and the server throttles you — most visibly with HTTP 429 Too Many Requests, often carrying a Retry-After header stating when to come back.
How limits are enforced
- Token/leaky bucket: the dominant algorithms — a budget that refills steadily and drains per request, allowing short bursts while capping sustained rates.
- Fixed and sliding windows: simpler counters per time slice, common on APIs with published quotas.
- Layered keys: serious targets limit on several dimensions at once — per IP, per session, per endpoint — so evading one limit still trips another. Silent variants exist too: instead of a 429, some sites quietly slow, degrade, or poison responses past a threshold.
Living within limits
For API consumers: respect Retry-After, back off exponentially with jitter on 429s, and cache aggressively. For scrapers, rate limits are the arithmetic behind proxy pools: if a target tolerates a modest rate per IP and you need much more throughput, the required pool size is roughly your target rate divided by the per-IP tolerance. Distributing requests across rotating IPs keeps each address inside the site's comfort zone — politeness, parallelized.
Read the ceiling, don't slam into it
Probe a target's tolerance gently and stay well under it. Traffic that rides exactly at the limit looks engineered; human-shaped traffic has slack, variance, and pauses.
Examples
- An API returns 429 with Retry-After: 60 when a client exceeds its per-minute quota.
- A scraper spreads requests across 200 rotating IPs to keep each address under a site's per-IP tolerance.
- A login endpoint rate-limits per account and per IP separately to blunt credential-stuffing.
Common use cases
FAQs
'Too Many Requests' — you exceeded the server's rate limit. Check for a Retry-After header giving the wait time, back off (exponentially, with jitter), and reduce your steady-state rate so it stops recurring.
Limits are usually tracked per IP, so spreading requests across a rotating pool keeps each address under the threshold. Required pool size is roughly your desired total rate divided by the per-IP tolerance — with headroom, since limits are rarely documented.
No — a rate limit is temporary friction that clears when you slow down; a ban is a durable trust judgment. But repeatedly slamming into limits erodes your IP's reputation and invites exactly the escalation that ends in bans.