Proxies and throttling

Sites that dislike being scraped rarely say so politely. They answer 403 and stop serving that address for a while. Two mechanisms keep a scraper welcome, and they work together: spreading requests across exits, and pacing them.

A single proxy

->proxy('200.20.14.84:40200')
->proxy('200.20.14.84:40200', 'username', 'password')   // with auth

The class-property equivalent is $proxy, $proxyUser and $proxyPass.

Note that proxy() declares its first argument as string, not ?string. Passing a config value that turns out to be null raises a TypeError, and a TypeError extends Error, so a catch (Exception) around the call will not catch it. Either cast at the call site or catch Throwable:

->proxy((string) config('scrapers.proxy.url'), config('scrapers.proxy.user'), config('scrapers.proxy.pass'))

An empty string is ignored cleanly, so that cast is also how you make the proxy optional.

A pool of proxies

Publish the config file and list several. One is picked at random per request, so a site that blocks a single address does not block every scrape:

// config/larascraper.php
'proxies' => [
    '203.0.113.10:8080',
    'http://user:secret@203.0.113.11:8080',
    'socks5://203.0.113.12:1080',
    ['url' => '203.0.113.13:8080', 'user' => 'user', 'pass' => 'secret'],
],

Entries may be plain strings or arrays, mixed freely. Credentials written inline are split out of the URL before use, because Chrome ignores them in --proxy-server and the browser runner has to pass them to page.authenticate() instead. Percent-encode special characters in credentials (p:w becomes p%3Aw).

An explicit ->proxy() call, or the $proxy property, always wins over the list. Leave proxies empty to keep the previous behaviour.

This is worth setting up before you need it. A single proxy is a single point of failure, and proxies expire; the day one does, every scraper you own stops at once.

Throttling and lockout

Both are configured together, per throttle key:

// config/larascraper.php
'throttle' => [
    'cendoj.search' => [
        'interval'  => 10,    // seconds between requests, across all processes
        'lock_base' => 120,   // first lockout after an address is refused
        'lock_max'  => 3600,  // ceiling; each further refusal doubles the wait
        'max_wait'  => 30,    // give up rather than queue longer (0 = no limit)
    ],
],

A scraper declares its key with a property:

class CendojSearchScraper extends Scraper
{
    protected ?string $throttleKey = 'cendoj.search';
}

Pacing

Requests sharing a key are spaced interval seconds apart, across every process that makes them. A queue worker, a web request and an Artisan command share one schedule instead of each keeping its own, so parallel work does not turn into a burst.

Each request reserves its turn before waiting, rather than reading when the last one went out and waiting on that. The difference matters exactly when it is needed most: three workers waking at the same instant read the same answer, wait the same seconds, and fire together, which is a burst of three dressed up as a paced request. Reserving gives them three different departure times, which they then wait out in parallel.

Reserving means a busy target grows a queue, and max_wait bounds it. A turn further away than that throws ThrottledException instead of parking the worker, and the turn is left for whoever comes next. At 0, the default, nothing is ever refused.

Catch ThrottledException where the scraper distinguishes "the site said no" from "we never asked". Reporting a queue as an empty result set is how a network problem gets recorded as a fact about the world.

Lockout

When an exit is refused with a 403 or a 429, it stops being used for that key for lock_base seconds and the next attempt goes out through another one. Refused again while still remembered, it waits double, up to lock_max. The moment it succeeds the escalation is forgotten, because an address that recovers should not carry the penalty of a bad afternoon.

Requests made with no proxy are an exit like any other, locked out under the label direct.

Because a lockout is about the address rather than the request, a 403 is retried only while some other exit is still free. With all of them locked out the fetch fails instead of hammering the one that just refused. That retry also skips the retry() delay: waiting is for a target that might recover in a moment, and a refusal is not that. The pacing interval still applies.

Keys are not hosts

Deliberately. The same domain can serve a listing happily while refusing a download endpoint, and a lockout earned by the second should not stop the first. Scrapers that should share a budget simply share a key.

Without $throttleKey the URL host is used, which is only a sane default. Prefer naming the key. Keys with no entry in throttle are neither paced nor locked out and never touch the cache, so none of this costs anything until you ask for it.

Where the state lives

One cache entry per key, larascraper:throttle:{key}, through Laravel's cache. That is what makes it shared between processes: with the file or array driver it is per machine, and a shared store such as Redis, Memcached or the database extends it across servers. Nothing needs sweeping, since the entry expires on its own once the last lockout has elapsed.

It can be inspected or cleared directly, which is what you want when a scraper is mysteriously idle:

use EduLazaro\Larascraper\Support\Throttle;

$throttle = new Throttle('cendoj.search');

$throttle->lockedOut();                    // ['203.0.113.10:8080', 'direct']
$throttle->available('203.0.113.11:8080'); // true
$throttle->nextFreeIn($labels);            // seconds until the first one frees up
$throttle->forget();                       // wipe pacing and lockouts for this key