Crawlers

A Crawler is parsing and nothing else. It receives a document and extracts data, with no idea how that document was fetched. That is what makes it reusable across scrapers, and testable against a fixture with no network involved.

namespace App\Scrapers\Crawlers;

use EduLazaro\Larascraper\Crawler;

class BikeCrawler extends Crawler
{
    protected function handle(): array
    {
        return [
            'name'  => $this->filter('h1')->text(''),
            'price' => $this->filter('.price')->text(''),
            'url'   => $this->filter('a.buy')->attr('href'),
        ];
    }
}

Reading the document

filter() and html() require a string input. Called on anything else they throw a LogicException that points you at raw().

Always pass a default to text(). ->text('') returns an empty string when the node is missing; ->text() with no argument throws on an empty node list, which turns a missing optional field into a failed scrape.

The input is not always HTML

The input is typed mixed on purpose. It can be an HTML string, an XML string, an arbitrary text payload, or an array of named parts when a single fetch yields more than one document:

class SplitCrawler extends Crawler
{
    protected function handle(): array
    {
        ['meta' => $metaXml, 'body' => $bodyHtml] = $this->raw();

        return [...];
    }
}

Server rendered JSON

Modern frameworks put the page's data in the initial HTML as a JSON blob, which is a far more stable thing to parse than a class name a build step generated:

class GameCrawler extends Crawler
{
    protected function handle(): array
    {
        if (! preg_match('#<script id="__NEXT_DATA__" type="application/json">(.*?)</script>#s', $this->html(), $m)) {
            throw new ScrapeException('no_data');
        }

        $data = json_decode($m[1], true)['props']['pageProps']['game']['data'] ?? [];

        return ['name' => $data['game_name'] ?? null];
    }
}

Hashed CSS classes change with every deploy of the target site. A payload key rarely does, because their own frontend reads it.

Signalling a content failure

A 200 can still be a captcha wall, a block page, a redirect to a login, or an empty result set. When the Crawler sees that the page did not give it what it needed, it throws a ScrapeException whose message is the error code:

use EduLazaro\Larascraper\Crawler;
use EduLazaro\Larascraper\Exceptions\ScrapeException;

class BikeCrawler extends Crawler
{
    protected function handle(): array
    {
        if ($this->filter('h1.product-title')->count() === 0) {
            throw new ScrapeException('no_product');
        }

        return ['name' => $this->filter('h1.product-title')->text('')];
    }
}

The crawl(...)->run() terminal catches it and folds it into a ScraperResponse with success = false and error = 'no_product'. It never bubbles out of run(); the caller branches on $result->success.

This is the single most valuable habit in the package. A scraper that returns an empty array on a block page reads to the rest of your application as a fact about the world: the site has no products. An error code reads as what it is.

Running one on its own

The standard entry point is run($input), consistent with Scraper::run() and Spider::run():

$data   = BikeCrawler::run($html);                                  // HTML
$items  = FeedCrawler::run($xmlString);                             // XML
$parts  = SplitCrawler::run(['meta' => $metaXml, 'body' => $body]); // multi-part

create($input)->parse() and (new BikeCrawler($input))->parse() still work; parse() is kept as a legacy alias of run().

That is what makes parsing cheap to test. Store a copy of the page as a fixture and assert on the parsed output:

public function test_it_reads_the_price(): void
{
    $data = BikeCrawler::run(file_get_contents(__DIR__ . '/fixtures/bike.html'));

    $this->assertSame('349.00', $data['price']);
}

When the target site changes its markup, that test fails locally and immediately, instead of your production crawl quietly collecting nulls.