← Larascraper 16 / 17

Commands and testing

Artisan commands


`php artisan larascraper:install`	Install the Node packages and the Chrome binary the browser driver needs. Options: `--publish`, `--no-npm`, `--no-browser`, `--captcha`. See Installation.
`php artisan make:scraper MyScraper`	Generate a scraper class in `app/Scrapers`.
`php artisan list:scrapers`	List every scraper in `app/Scrapers`.
`php artisan vendor:publish --tag=larascraper-config`	Publish `config/larascraper.php`.

Trying a scraper by hand

Tinker is the fastest loop while you are still working out selectors:

php artisan tinker

$result = \App\Scrapers\BikeScraper::run('https://shop.com/bikes/4');

dd($result->success, $result->error, $result->data);

When the data comes back empty, look at the layer below before touching selectors:

$scraper = \App\Scrapers\BikeScraper::make();
$result  = $scraper->handleToResponse(['https://shop.com/bikes/4']);

$scraper->request->status;        // did the fetch even succeed
substr($scraper->request->html, 0, 500);

A 200 with HTML that says "enable JavaScript" means the wrong driver. A 403 means the user agent or the address. Empty HTML with a 200 usually means an action timed out before the content arrived.

Testing the parsing

A Crawler only knows about documents, so it can be tested with no network at all. Save a copy of the page as a fixture and assert on the parsed output:

public function test_it_reads_the_price(): void
{
    $html = file_get_contents(__DIR__ . '/fixtures/bike.html');

    $data = \App\Scrapers\Crawlers\BikeCrawler::run($html);

    $this->assertSame('349.00', $data['price']);
}

This is the test worth having. When the target changes its markup, it fails locally and immediately instead of your production crawl quietly collecting nulls. Refresh the fixture when you deliberately adapt to a redesign, and the diff tells you exactly what changed on their side.

Test the content-failure branch too, since that is the one that never runs in development:

public function test_a_block_page_is_a_failure_not_an_empty_product(): void
{
    $this->expectException(\EduLazaro\Larascraper\Exceptions\ScrapeException::class);

    \App\Scrapers\Crawlers\BikeCrawler::run('<html><body>Access denied</body></html>');
}

Testing a scraper end to end

For the scraper itself, the useful assertions are about decisions, not about the network: that a captcha page produces success = false with the right code, that a missing optional field does not fail the run, that a 404 raises a RequestException. Structure the scraper so those decisions live in handle() and in the Crawler, and both are reachable without a live site.

If you do run against the network in CI, keep it to a smoke test on a page you control, and expect it to be the flakiest thing in the suite.

Watching a scraper in production

The single most useful habit: do not discard the response. A call site like MyScraper::run($url); cannot report anything, because it threw away the only object that knew what happened.

$result = MyScraper::run($url);

if (! $result->success) {
    Log::warning("scrape failed: {$result->error}", ['url' => $url]);
}

And define what success means for your case. If zero rows is a problem, make the scraper say so with fail('no_results'), so a source that quietly stops publishing is distinguishable from a source that quietly started blocking you.

Throttle state is also inspectable when a scraper seems idle for no reason:

use EduLazaro\Larascraper\Support\Throttle;

(new Throttle('shop.listing'))->lockedOut();   // which exits are refused

LaraClaude

Larascraper is supported by LaraClaude, a Laravel toolkit plugin for Claude Code. It ships a /lc:generate-scraper skill:

/lc:generate-scraper BikeScraper https://shop.com/bikes

Given a name and a target URL it checks that Larascraper and its Node side are installed, reads the installed Scraper API so it only uses methods your version actually has, generates the class with make:scraper, fills handle() and a Crawler from the real page markup rather than from guesses, wires up the actions the page needs, and runs the scraper once to confirm the fields come back populated.

Install the plugin with /plugin install github:edulazaro/laraclaude.

PreviousMigrating from v2 NextReference

On this page