Migrating from Larascraper v2 to v3

Larascraper v3 changes one thing, and the rest follows from it: the fetch chain lives inside the scraper now, instead of at the call site.

In v2 the caller built the request and the scraper read the result off itself:

BeatGameScraper::scrape($url)
    ->retry(3, 20)
    ->proxy($proxy, $user, $pass)
    ->waitForSelector('#__NEXT_DATA__')
    ->run(beatId: $id);

In v3 the caller passes data, and the scraper owns its own request:

BeatGameScraper::run($url, $id, $proxy, $user, $pass);

That is better where it counts. A scraper that knows a page needs a proxy and a wait should not be relying on five different call sites to remember it. But the migration has a property worth knowing before you start.

Nothing fails until it runs

None of the breaking changes are visible to php -l. Every file parses. The container resolves. The command starts, prints its first line, and then dies on the first fetch.

I migrated seven scrapers, linted them all clean, and hit three separate runtime failures in a row. Plan on running each scraper for real, one at a time. A clean lint means nothing here.

1. $this->crawler is gone

This is the one that touches the most code. In v2 the package handed you a Symfony DomCrawler over the fetched page, and most scrapers are built on it. In v3 the property does not exist.

The first instinct is to rewrite the parsing against whatever replaced it. Do not. Build the crawler yourself at the top of handle() and every line below keeps working exactly as it did:

use Symfony\Component\DomCrawler\Crawler as DomCrawler;

$crawler = new DomCrawler((string) $this->request->html);

$crawler->filter('#archive-news-list .wrapper-inner')->each(function ($article) {
    // unchanged
});

Two hundred lines of filter() calls migrate with one added line. The HTML moved from $this->html to $this->request->html, which is where the whole transport layer lives now: status, cookies, content type, captured file.

If a scraper is due for a tidy anyway, the other option is to move the parsing into a Crawler class and chain ->crawl(MyCrawler::class). That is the shape v3 is built around, and it makes the parsing testable against a saved fixture with no network at all. But it is a refactor, not a migration step, and doing both at once means you cannot tell which one broke.

2. $this->url is no longer populated

$url is still declared on the base class, still typed, and no longer filled in. Reading it gives you:

Typed property EduLazaro\Larascraper\Scraper::$url must not be accessed before initialization

It shows up in the scrapers that store the source URL in whatever they save, which is most of the ones that write to a database:

'link' => $this->url,     // v2
'link' => $url,           // v3: it is a handle() parameter now

Trivial once you know. Confusing for a minute if you do not, because the error names a property you never touched.

3. Closures lose what the fetch used to own

This is the only one that fails silently, so it is the one to go looking for rather than wait for.

Moving the fetch inside handle() changes what is in scope. A use (...) list that was complete in v2 is not anymore, because the values it needs are now parameters instead of things the call site read from config:

// v2: the inner scraper read the proxy from config, at the call site
$crawler->filter('.item')->each(function ($node) use (&$count) {
    MegaArticleScraper::scrape($link)->proxy(config('...'))->run(type: 'news');
});

// v3: the proxy arrives as a parameter, so the closure has to be handed it
$crawler->filter('.item')->each(function ($node) use (&$count, $proxy, $proxyUser, $proxyPass) {
    MegaArticleScraper::run($link, 'news', $proxy, $proxyUser, $proxyPass);
});

Forget the use and nothing throws. The variable is empty, the proxy quietly stops being applied, and the scraper keeps working, from your server's own address, until the target notices and starts refusing it. That is a failure you find out about weeks later.

Grep every migrated scraper for function ( and check each use (...) against the variables the fetch now owns. It takes a minute and it is the cheapest minute of the whole migration.

The trap that hides all three

While you are in there, look at how the caller catches failure.

proxy() declares its first argument as string, not ?string. A config key that is absent on that machine arrives as null and raises a TypeError. And TypeError extends Error, not Exception. So this does not catch it:

try {
    MyScraper::run($url, config('scrapers.proxy.url'));
} catch (Exception $e) {          // does NOT catch TypeError
    Log::error($e->getMessage());
}

The command dies with a non-zero exit code. The scheduler swallows it. There is no entry in the log, because the handler that would have written one was never reached.

I had a news scraper down for two months on exactly this, on a server whose .env was missing PROXY_URL. It ran every day at 13:00, exited non-zero every day at 13:00, and nobody looked, because a scraper that finds nothing new looks identical to a source that published nothing new.

The fix is a cast at the call site, which also makes the proxy properly optional, since an empty string is ignored cleanly by the fetch chain:

MyScraper::run(
    $url,
    (string) config('scrapers.proxy.url'),
    config('scrapers.proxy.user'),
    config('scrapers.proxy.pass'),
);

And catch Throwable when you mean everything.

What to pick up while you are here

Two things v3 added that answer problems v2 had no answer for, and the migration is the natural moment to adopt them.

Say when a scrape found nothing. fail('no_results') and ok() separate "the request worked" from "it brought back what I asked for". A 200 that is really a captcha wall, a login redirect or an empty listing is the failure mode that costs the most, precisely because nothing throws:

if ($rows === []) {
    return $this->fail('no_results');
}

That is worth nothing if the call site does MyScraper::run($url); and discards the return value, which four of mine did. The response object is the only thing that knows what happened.

Set up a proxy pool. v3 reads a list from config/larascraper.php and picks one per request, so a single expired proxy stops being able to take down every scraper you own at once. Mine expired in August and did exactly that.

An order that works

1. Publish the config file and add the proxy pool, if you use proxies at all.
2. Take one scraper. Move the chain from the call site into handle(); add the URL and proxy arguments as parameters.
3. Replace $this->crawler with a DomCrawler over $this->request->html, and $this->url with the parameter.
4. Grep that scraper for function ( and check the use (...) lists.
5. Run it. Not lint it.
6. Repeat. Then fix the call sites, casting config values to string.

Migrating all of them before running any of them gets you several silent failures at once and no way to tell which change caused which.

The full reference version of this, with the before-and-after table and the rest of the v3 surface, is in the migration chapter of the documentation.