How to crawl thousands of pages concurrently with Larascraper

A scraper that fetches one page in 300 ms fetches ten thousand pages in fifty minutes, and almost all of that is waiting. The CPU is idle, the network is idle between requests, and the process is blocked on curl doing nothing.

The usual PHP answer is a queue: dispatch ten thousand jobs, run twenty workers, add Redis, add supervision, add a way to know when it finished. That is a lot of infrastructure to buy something that is really just overlapped I/O.

Larascraper's Spider does it in one process.

What a Spider is

A Scraper fetches one page. A Spider walks a whole source and drives many Scrapers, with concurrency, rate limiting, per-item error isolation and one shared session, in a single class.

It is deliberately imperative. There is no declarative wiring to learn, no targets(), collect(), shouldVisit() or onError() to override. You write one handle() that drives the crawl:

use EduLazaro\Larascraper\Spider;
use EduLazaro\Larascraper\Support\ScraperResponse;

class GamesSpider extends Spider
{
    protected int $concurrency = 20;   // up to 20 detail pages in flight
    protected int $delay = 250;        // ms between waves

    public function handle(): int
    {
        // 1. Log in once. The shared Session captures the cookie, so every
        //    scraper this spider runs afterwards inherits the session.
        LoginScraper::make()->useSession($this->session)->handleToResponse();

        // 2. Discover the work, then drop what is already stored.
        $ids = collect(GameListScraper::run()->data)
            ->reject(fn ($id) => Game::where('remote_id', $id)->exists());

        // 3. Fan out.
        $this->pool($ids, GameScraper::class, $this->save(...));

        return $ids->count();
    }

    protected function save(mixed $data, mixed $id, ScraperResponse $response): void
    {
        if ($response->success) {
            Game::updateOrCreate(['remote_id' => $id], $data);
        }
    }
}

$count = GamesSpider::run();

The unit scraper is an ordinary Scraper. Note that items are params, not URLs, so the scraper builds its own:

class GameScraper extends Scraper
{
    protected string $driver = 'http';

    protected function handle(int $id): ScraperResponse
    {
        return $this->scrape("https://shop.com/games/{$id}")
            ->crawl(GameCrawler::class)
            ->run();
    }
}

How the overlap actually works

Each item runs its scraper inside a PHP Fiber. When the scraper fetches on the http driver, the fetch does not block: it suspends the fiber with a fully resolved request spec.

The scheduler collects every currently suspended fiber's spec into one Http::pool() wave, which is overlapped curl_multi network I/O under a single handler, and then resumes each fiber with its settled response. As fibers finish they free their slot and the scheduler refills from $items to keep $concurrency in flight.

Two consequences worth understanding.

A multi-step scraper's second fetch rides the same wave as another item's first fetch. There is no barrier between waves for a given item; the pool is a moving window, not a batch.

And dependent requests within one item stay sequential, which is what you want. Logging in and then reading a protected page in the same handle() cannot overlap and must not. Only independent items overlap.

No queue, no workers, no Redis, no supervisor. It is one process and a Fiber.

The two constraints

Concurrency needs the http driver. The overlap comes from Http::pool(), so the browser driver is not pooled. Each Puppeteer run is an isolated Chromium; those items still run, correctly, just one at a time.

This is the strongest practical argument for checking whether you need a browser at all. The same measurement in both directions: a browser page took 1.6 seconds against 0.10 to 0.38 over HTTP, and only the HTTP one can then be run twenty at a time. The combined difference on a large crawl is not a factor of four, it is a factor of eighty.

The pattern that gets you there is a hybrid: drive the listing with the browser, because that is where the controls are, and give the detail pages to an HTTP scraper the Spider can pool.

The shared session needs the http driver too, for the same reason. On the browser driver the Session is a documented no-op.

Per-item error isolation is the real feature

Over ten thousand pages, some will fail. A timeout, a 500, a layout that changed on one section of the site, one document that is malformed. If any of those aborts the crawl, you do not have a crawler, you have a lottery.

A RequestException, or any other Throwable, from one item is caught and turned into a failed ScraperResponse that is still handed to your collector:

protected function save(mixed $data, mixed $id, ScraperResponse $response): void
{
    if (! $response->success) {
        Log::warning("item {$id} failed: {$response->error}");
        return;
    }

    Game::updateOrCreate(['remote_id' => $id], $data);
}

One bad item is data, not an abort. And it arrives with a code, so a crawl that half-worked tells you which half.

The retriable statuses 408, 429, 500, 502, 503 and 504 are retried in the concurrent path too, at the scheduler level: a retriable wave result is re-issued in a later wave rather than resumed, so it matches the sequential retry set exactly.

Make it resumable, and be a good guest

Two lines do most of the work here.

Filter before you pool. The ->reject(...) in the example is not a detail, it is what makes the crawl resumable. Kill it at item 7,000 and the next run starts at 7,000, because the first 7,000 are already stored:

$ids = collect($allIds)->reject(fn ($id) => Game::where('remote_id', $id)->exists());

$items may also be a lazy generator, so an open-ended crawl never materializes the full list up front.

Pace it. $delay puts milliseconds between waves, and $concurrency caps how many are in flight. Twenty concurrent requests against a small institutional site is not a crawl, it is a load test that nobody consented to.

For anything you will run repeatedly, put a throttle key on the scraper and give it an interval in config. It is enforced across every process, so a queue worker, a web request and an Artisan command share one schedule instead of each keeping its own. It also gives you the lockout behaviour: an address refused with a 403 stops being used for a while and the next attempt goes out through another proxy, with the escalation forgotten the moment it succeeds. That is in Proxies and throttling.

When not to use a Spider

For one page, a plain Scraper is the entire answer. Reach for a Spider when you are crawling many pages from one source and you want concurrency, per-item isolation, one accumulating session, and a single place that says what the crawl is.

The full API, including how items map to handle() parameters and what pool() accepts, is in Spiders and sessions.