Migrating from v2

Version 3 moves one thing: the fetch chain lives inside the scraper now, not at the call site. Everything else in this chapter follows from that.

The migration is mechanical, but it has a nasty property worth knowing before you start. None of the breaking changes are visible to php -l. Every file still parses, the container still resolves, the command still starts, and then it dies on the first fetch. Plan on running each scraper once, for real, rather than trusting a clean lint.

The shape of the change

In v2 the caller built the request and the scraper read the result off itself:

// v2 — call site
BeatGameScraper::scrape($url)
    ->retry(3, 20)
    ->proxy($proxy, $user, $pass)
    ->waitForSelector('#__NEXT_DATA__')
    ->run(beatId: $id);

// v2 — scraper
public function handle(int $beatId): array
{
    return $this->parse($this->html);
}

In v3 the caller passes data and the scraper owns the request:

// v3 — call site
BeatGameScraper::run($url, $id, $proxy, $user, $pass);

// v3 — scraper
public function handle(string $url, int $beatId, string $proxy = '', ?string $proxyUser = null, ?string $proxyPass = null): array
{
    $fetch = $this->scrape($url)
        ->retry(3, 20)
        ->waitForSelector('#__NEXT_DATA__');

    if ($proxy !== '') {
        $fetch->proxy($proxy, $proxyUser, $proxyPass);
    }

    $fetch->run();

    return $this->parse((string) $this->request->html);
}

This is better where it counts: the scraper now declares its own requirements instead of trusting five call sites to remember them. But it does mean the URL becomes a handle() parameter, and every call site changes.

The three that break at runtime

1. $this->crawler no longer exists

This is the big one, because in a typical project most scrapers use it. In v2 the package handed you a Symfony DomCrawler over the fetched page. In v3 it is gone.

You do not have to rewrite your parsing. Build the crawler yourself at the top of handle() and every line below it keeps working:

use Symfony\Component\DomCrawler\Crawler as DomCrawler;

$crawler = new DomCrawler((string) $this->request->html);

$crawler->filter('#archive-news-list .wrapper-inner')->each(function ($article) {
    // unchanged
});

Two hundred lines of filter() calls migrate with one added line.

The other option, worth taking when a scraper is due for a tidy anyway, is to move the parsing into a Crawler class and chain ->crawl(MyCrawler::class). That is the shape the package is built around now, and it makes the parsing testable against a fixture.

2. $this->url is no longer populated

$url is a typed property that the package stopped filling in. Reading it raises:

Typed property EduLazaro\Larascraper\Scraper::$url must not be accessed before initialization

It bites the scrapers that stored the source URL in whatever they saved:

'link' => $this->url,     // v2
'link' => $url,           // v3: it is a handle() parameter now

3. Closures lose the variables the fetch used to own

This is the one that fails silently, so it is the one to grep for. Moving the fetch inside handle() changes what is in scope, and a use (...) list that was complete in v2 is not anymore:

// v2: the inner scraper read the proxy from config at the call site
$crawler->filter('.item')->each(function ($node) use (&$count) {
    MegaArticleScraper::scrape($link)->proxy(config('...'))->run(type: 'news');
});

// v3: the proxy arrives as a parameter, so the closure has to be given it
$crawler->filter('.item')->each(function ($node) use (&$count, $proxy, $proxyUser, $proxyPass) {
    MegaArticleScraper::run($link, 'news', $proxy, $proxyUser, $proxyPass);
});

Forget the use and nothing throws. The variable is simply empty, the proxy quietly stops being applied, and you find out weeks later when the target starts refusing your server's address.

The trap that hides all three

While you are in there, look at how the failure is caught.

proxy() declares its first argument as string, not ?string. A config value that is absent arrives as null and raises a TypeError, and TypeError extends Error, not Exception. So this does not catch it:

try {
    MyScraper::run($url, config('scrapers.proxy.url'));
} catch (Exception $e) {          // does NOT catch TypeError
    Log::error($e->getMessage());
}

The command dies with a non-zero exit, the scheduler swallows it, and the scrape has been down for two months before anyone notices. Cast at the call site, which also makes the proxy properly optional:

MyScraper::run($url, (string) config('scrapers.proxy.url'), config('scrapers.proxy.user'), config('scrapers.proxy.pass'));

An empty string is ignored cleanly by the fetch chain.

Reading the result

$this->request is a RequestResponse with status, error, html, file, contentType and cookies. See Responses and failures.

Worth adopting while you are here

The migration is a good moment to pick up two things v3 added that v2 had no answer for.

Say when a scrape found nothing. fail('no_results') and ok() separate "the request worked" from "it brought back what I asked for". A 200 that is really a captcha wall or an empty listing is the failure mode that costs the most, precisely because nothing throws. If your call sites currently do MyScraper::run($url); and discard the return value, they cannot tell the difference. See Responses and failures.

Set up a proxy pool. v3 reads a list from config/larascraper.php and picks one per request, so a single expired proxy no longer takes down every scraper you own. See Proxies and throttling.

A migration order that works

1. Publish the config file and add the proxy pool, if you use proxies at all.
2. Take one scraper. Move the chain from the call site into handle(), add the URL and the proxy arguments as parameters.
3. Replace $this->crawler with a DomCrawler built from $this->request->html, and $this->url with the parameter.
4. Grep that scraper for function ( and check every use (...) list against the variables the fetch now owns.
5. Run it. Not lint it.
6. Repeat, then fix the call sites, casting config values to string.

Doing all the scrapers before running any of them is how you end up with several silent failures at once and no idea which change caused which.