How to keep a login session across a whole crawl in Laravel

The first request logs in. The second one gets the login page again.

The usual fix is to hand the cookies back and forth by hand, and it works for two requests. By the tenth, every method signature has grown a $cookies parameter, and somewhere in the middle a request that failed has overwritten the good session with nothing.

One jar, passed by reference

use EduLazaro\Larascraper\Support\Session;

$session = new Session();

LoginScraper::withSession($session)->run($credentials);
ReportScraper::withSession($session)->run($reportUrl);   // already authenticated

Session is a small mutable cookie jar threaded by reference into every scraper of a run. A cookie established on the first request rides along to every request after it, and the jar keeps accumulating as the crawl goes on. Nothing is returned, re-passed or merged by you.

Two methods, plus all() to look inside:

$session->cookiesFor('shop.test');              // ['sid' => '9', ...] for one host
$session->store('shop.test', ['sid' => '9']);   // merge, last-wins

Cookies are held per host, taken from the request URL, so two hosts in the same crawl never leak into each other. That is not a detail on a crawl that follows links off-site: a session cookie for the portal has no business travelling to the CDN it embeds.

In a Spider, you get it for free

For a real crawl you rarely construct one. A Spider creates the Session in run(), threads it into every scraper pool() drives, and exposes it so you can log in at the top:

use EduLazaro\Larascraper\Spider;

class ReportsSpider extends Spider
{
    protected int $concurrency = 20;

    public function handle(): int
    {
        // Log in once. Every scraper below inherits the cookie.
        LoginScraper::make()->useSession($this->session)->handleToResponse();

        $ids = collect(ReportListScraper::run()->data)
            ->reject(fn ($id) => Report::where('remote_id', $id)->exists());

        $this->pool($ids, ReportScraper::class, $this->save(...));

        return $ids->count();
    }
}

This used to be a framework hook called bootSession(). It is now just the first line of handle(), which is better: there is nothing to look up about when it runs relative to everything else, because you can see it running first.

useSession() is the chainable instance form, withSession() the static one that mirrors with(). Use whichever reads better where you are.

Two rules that stop it going wrong

A failed fetch never writes to the jar. Cookies are only taken from a successful response. A 500 or a timeout mid-crawl cannot clobber the good session cookies that the remaining thousand requests depend on. This is the failure that is worst to debug when it is absent, because the crawl keeps running and simply stops being authenticated, and the symptom appears hundreds of items later as pages that look like they were never logged in.

An explicit cookie always wins. The jar is merged under any per-call cookies(...), so overriding one value for one request does not require emptying the jar or fighting it.

Where the cookies are, and where they are not

They live on the request layer, inside handle():

$cookies = $this->request->cookies;   // ['session' => '...']

They are deliberately not on the ScraperResponse, which carries exactly three fields: data, success and error. That object is the scraper's judgement about content, and a cookie is transport. If you need one on the way out, fold it into data on purpose.

The same separation is why a 200 can be success = false, error = 'captcha'. The status is a fact about the request; success is a fact about what came back.

The caveat that costs an afternoon

The shared jar works on the http driver and does nothing on the browser driver.

Not a bug and not an oversight: each Puppeteer run is an isolated Chromium with its own profile, and that driver rejects explicit cookies outright, so Runner::supportsCookies() is false there. A Session on the browser driver is a documented no-op.

Which means the unit scraper a Spider drives usually wants to say so:

class ReportScraper extends Scraper
{
    protected string $driver = 'http';

    protected function handle(int $id): ScraperResponse
    {
        return $this->scrape("https://portal.test/reports/{$id}")
            ->crawl(ReportCrawler::class)
            ->run();
    }
}

There is a second reason for that line, and it points the same way: concurrency comes from Http::pool(), so it only applies to the HTTP driver too. Browser runs happen one at a time. Session continuity and concurrency are both HTTP driver properties, and a crawl that needs either needs both.

If the login genuinely requires a browser, because it is behind JavaScript or a captcha, do that part in the browser, read the cookie off $this->request yourself, and store() it into the jar before fanning out over HTTP.

The shape that works

Log in once at the top of handle(), filter the work before you fetch any of it, fan out with pool(), and check $response->success inside the collector so one bad page is data rather than an abort:

protected function save(mixed $data, mixed $id, ScraperResponse $response): void
{
    if ($response->success) {
        Report::updateOrCreate(['remote_id' => $id], $data);
    }
}

Three former hooks, bootSession, shouldVisit and onError, are those three lines. Nothing was lost by removing them, and the order stopped being something you had to remember.

More on the jar and the spider that carries it is in the spiders and sessions chapter.