Spiders and sessions

A Scraper fetches one page. A Spider is the orchestrator on top: it walks a whole source, a bulletin, a paginated index, a range of ids, and drives many unit Scrapers, threading one shared session through the run.

Reach for a Spider when you are crawling many pages from one source and want concurrency, rate limiting, per-item error isolation and one accumulating login or CSRF session in a single place. For one page, a plain Scraper is the whole answer.

A Spider is imperative

There is no declarative wiring: no $scraper property, no targets(), collect(), bootSession(), shouldVisit() or onError() to override. You extend Spider and write one handle() that drives the crawl.

use EduLazaro\Larascraper\Spider;
use EduLazaro\Larascraper\Support\ScraperResponse;

class GamesSpider extends Spider
{
    protected int $concurrency = 20;   // up to 20 detail pages in flight
    protected int $delay = 250;        // ms between pool waves

    public function handle(): int
    {
        // 1. Log in once. The shared Session captures the cookie, so every
        //    scraper the spider runs afterwards inherits the session.
        LoginScraper::make()->useSession($this->session)->handleToResponse();

        // 2. Discover the work, then filter out what is already stored.
        $ids = collect(GameListScraper::run()->data)
            ->reject(fn ($id) => Game::where('remote_id', $id)->exists());

        // 3. Fan out, concurrently.
        $this->pool($ids, GameScraper::class, $this->save(...));

        return $ids->count();
    }

    protected function save(mixed $data, mixed $id, ScraperResponse $response): void
    {
        if ($response->success) {
            Game::updateOrCreate(['remote_id' => $id], $data);
        }
    }
}

$count = GamesSpider::run();

The unit Scrapers are ordinary Scrapers. Each item is the scraper's handle() params, not a URL, so the scraper builds its own:

class GameScraper extends Scraper
{
    protected string $driver = 'http';   // required for concurrency

    protected function handle(int $id): ScraperResponse
    {
        return $this->scrape("https://shop.com/games/{$id}")
            ->crawl(GameCrawler::class)
            ->run();
    }
}

The members

pool

protected function pool(
    iterable $items,
    string|PendingScraper $scraper,
    callable $collect,
    ?int $concurrency = null,
): void

• $items are per-run params, not URLs. An array item is spread as the scraper's arguments (['id' => 5, 'lang' => 'en'] or [5, 'en']); a scalar is passed as the single argument. It may be a lazy generator, so an open-ended crawl never materializes the full list.
• $scraper is a class-string, made fresh through the container per item, or a configured template from Scraper::with(...), cloned per item so no mutable state is shared. The Session is threaded into every run.
• $collect is any callable, called once per item as ($data, $item, $response). Branch on $response->success inside it.
• $concurrency caps runs in flight, defaulting to the property.

How the concurrency works

Each item runs its scraper inside a PHP Fiber. When a scraper fetches on the http driver, the fetch suspends the fiber with a fully resolved request spec instead of blocking. The scheduler gathers every currently suspended fiber's spec into one Http::pool() wave, overlapped curl_multi network I/O under a single handler, then resumes each fiber with its settled response.

As fibers finish they free their slot and the scheduler refills from $items to keep $concurrency in flight, with $delay milliseconds between waves. A multi-step scraper's later fetch rides the same wave as another item's first fetch.

No queue, no workers, no extra infrastructure. It is one process.

Per-item error isolation

A RequestException, or any other Throwable, from one item is caught and turned into a failed ScraperResponse (success = false, error set, data = null) that is still handed to $collect. One bad item never aborts the crawl; you see it as ! $response->success in the collector.

The retriable statuses 408, 429, 500, 502, 503 and 504 are retried in the concurrent path too, at the scheduler level: a retriable wave result is re-issued in a later wave rather than resumed, so it matches the sequential retry set.

Concurrency needs the http driver

The overlap comes from Http::pool(), so it applies to the http driver only. The browser driver is not pooled, because each run is an isolated Chromium; those items run one at a time, still correctly, just not overlapped.

The overlap is also across items. Dependent multi-step requests within one item, log in and then read a protected page in the same handle(), stay sequential, which is exactly what you want.

Where the old hooks went

They are not framework hooks anymore, they are code you write in handle():

• Boot the session: run a login scraper at the top with ->useSession($this->session).
• Skip stored work: filter $items before pool(), so you never fetch what you already have. That is also what makes a crawl resumable.
• Handle a bad page: check $response->success inside $collect.

The shared Session

Session is a small mutable cookie jar that a whole crawl shares. One object is created once and threaded by reference into every Scraper of the run, so a login or CSRF cookie established on the first request rides along to every request that follows, and the jar keeps accumulating.

Cookies are held per host, last-wins on name collisions, so two hosts never leak into each other:

$session->cookiesFor('shop.test');            // ['name' => 'value', ...]
$session->store('shop.test', ['sid' => '9']); // merge, last-wins
$session->all();

A Spider creates and threads it for you. Driving scrapers by hand, make one session-aware with useSession() on an instance or withSession() statically:

use EduLazaro\Larascraper\Support\Session;

$session = new Session();

GameScraper::withSession($session)->run($firstUrl);
GameScraper::make()->useSession($session)->handleToResponse([$nextUrl]);

The jar is merged under any explicit per-call cookies(...), so an explicit cookie always wins. It only receives Set-Cookie after a successful fetch, so a failed or 5xx response never clobbers the good session cookies that later targets rely on. Cookies stay transport state: they live on $this->request->cookies and never surface on the content-only ScraperResponse.

Driver caveat. The shared jar works on the http driver only. On the browser driver it is a documented no-op, since each Puppeteer run is an isolated browser and that driver rejects explicit cookies. Which is the other reason the unit Scraper in the example sets protected string $driver = 'http';.