How to rate limit a scraper in Laravel so a site stops blocking you

Sites that dislike being scraped rarely say so politely. They answer 403 and stop serving your address for a while.

The reflex is to add a sleep. That helps until you have two workers, at which point each one is politely waiting while the pair of them doubles the rate.

Two problems, configured together

// config/larascraper.php
'throttle' => [
    'archive.search' => [
        'interval'  => 10,    // seconds between requests, across all processes
        'lock_base' => 120,   // first lockout after an address is refused
        'lock_max'  => 3600,  // ceiling; each further refusal doubles the wait
        'max_wait'  => 30,    // give up rather than queue longer (0 = no limit)
    ],
],

class ArchiveSearchScraper extends Scraper
{
    protected ?string $throttleKey = 'archive.search';
}

Pacing keeps you under the rate that gets noticed. Lockout is what happens when you were noticed anyway.

Pacing has to be shared, or it is not pacing

The state lives in one cache entry per key, larascraper:throttle:{key}, using Laravel's cache. That is the whole point: a queue worker, a web request and an Artisan command hitting the same target share one schedule instead of each keeping a private one.

Which cache store you use decides the reach. file or array is per machine; Redis, Memcached or the database extends it across servers. Nothing needs sweeping, since the entry expires on its own once the last lockout has elapsed.

Reserving a turn, rather than reading the last one

This is the part worth understanding, because the naive version fails exactly when you need it.

The obvious implementation reads when the last request went out and waits the difference. Three workers waking at the same instant read the same answer, compute the same wait, sleep the same seconds, and fire together. You have built a burst of three wearing the costume of a paced request, and it happens under load, which is when you are most likely to be watched.

Each request instead reserves its turn before waiting. Three simultaneous workers get three different departure times and then wait them out in parallel.

The consequence is that a busy target grows a queue, and max_wait bounds it:

'max_wait' => 30,   // 0 means never refuse

A turn further away than that throws ThrottledException instead of parking a worker for four minutes, and the turn is left for whoever comes next.

Catch it where the scraper distinguishes the site said no from we never asked:

use EduLazaro\Larascraper\Exceptions\ThrottledException;

try {
    $result = ArchiveSearchScraper::run($query);
} catch (ThrottledException) {
    return $this->releaseBackToQueue();   // not an empty result set
}

Reporting a queue as zero results is how a scheduling problem gets written into a database as a fact about the world, and nobody ever finds it.

Lockout: the address, not the request

When an exit is refused with a 403 or a 429, it stops being used for that key for lock_base seconds, and the next attempt goes out through a different one. Refused again while still remembered, it waits double, up to lock_max.

The moment it succeeds, the escalation is forgotten. An address that recovers should not be carrying the penalty of a bad afternoon three hours later.

Requests made with no proxy are an exit like any other, locked out under the label direct.

The retry policy that matters more than the pacing

Only the transient statuses 408, 429, 500, 502, 503 and 504 are retried. Everything else fails fast.

403 is the exception to the exception, and it is the interesting one:

• It is retried only while another exit is still free. With every address locked out, the fetch fails instead of hammering the one that just refused you.
• That retry skips the retry delay. Waiting is for a target that might recover in a moment, and a refusal is not that. The pacing interval still applies.

This is where most homegrown scrapers earn their ban. A refusal comes back, the retry logic treats it as transient, and the scraper goes back to the same address three more times with increasing enthusiasm. The retries are what get you blocked, not the crawl.

Keys are not hosts, on purpose

The same domain will serve a listing happily while refusing a download endpoint, and a lockout earned by the second should not stop the first. Scrapers that should share a budget share a key.

Without $throttleKey the URL host is used, which is a sane default rather than a good decision. Name the key.

Keys with no entry in throttle are not paced or locked out at all and never touch the cache, so none of this costs anything until you ask for it.

Inspecting it

use EduLazaro\Larascraper\Support\Throttle;

$throttle = new Throttle('archive.search');

$throttle->lockedOut();                      // ['203.0.113.10:8080', 'direct']
$throttle->available('203.0.113.11:8080');   // true
$throttle->nextFreeIn($labels);              // seconds until the first frees up
$throttle->forget();                         // wipe pacing and lockouts

lockedOut() is the first thing to look at when a crawl has gone quiet. If every exit is in there, you are not being slow, you are being refused.

The proxy pool underneath

Lockout only has somewhere to go if there is more than one exit:

'proxies' => [
    '203.0.113.10:8080',
    'http://user:secret@203.0.113.11:8080',
    'socks5://203.0.113.12:1080',
    ['url' => '203.0.113.13:8080', 'user' => 'user', 'pass' => 'secret'],
],

One is picked at random per request. Entries can be plain strings or arrays, mixed freely. An explicit ->proxy() call or the $proxy property always wins over the list.

Credentials written inline are split out of the URL before use, because Chrome ignores them in --proxy-server and the browser runner has to pass them to page.authenticate() instead. Percent-encode special characters in credentials (p:w becomes p%3Aw), or the split will disagree with you about where the password ends.

And check the user agent before blaming anything else

A 403 on the first request is usually not rate limiting at all. Measured against a site that blocks scrapers, all over plain HTTP:

The HTTP driver already sends a real Chrome user agent from config('larascraper.http_user_agent'), precisely so this does not happen. Sending nothing is not neutral: Guzzle fills in GuzzleHttp/7, which announces that a script is calling.

On the browser driver, do not write one by hand. The user agent is asked of the Chrome that just launched, with the one word that gives headless away removed, so version and platform stay true. A made-up user agent is a louder signal than an honest headless one, because Client Hints are filled in by the real Chrome and cannot be talked out of it: a browser claiming one version while emitting another is contradicting itself, and a real one never does.

The full configuration surface is in the proxies and throttling chapter.