How to test a scraper in Laravel without hitting the network

Edu Lazaro / 26 August 2026 / 5 min read

A scraper test that fetches a real page tests three things at once: your parsing, the site's markup, and whether the internet worked in the last ten seconds. Two of those are not yours, and they are the two that fail.

So the suite goes red on a morning you changed nothing, everyone learns to ignore it, and the day the parsing genuinely breaks nobody notices.

The split that makes it testable

The reason scrapers are usually untestable is that fetching and parsing are the same method. Separate them and the half that actually contains your logic has no network in it at all.

A Crawler only knows about a document. It receives one, returns data, and has no idea how the document arrived:

namespace App\Scrapers\Crawlers;

use EduLazaro\Larascraper\Crawler;

class BikeCrawler extends Crawler
{
    protected function handle(): array
    {
        return [
            'name'  => $this->filter('h1')->text(''),
            'price' => $this->filter('.price')->text(''),
            'specs' => $this->filter('ul.specs li')->each(fn ($li) => trim($li->text(''))),
        ];
    }
}

Which means the test is a function call:

public function test_it_reads_a_product_page(): void
{
    $data = BikeCrawler::run(file_get_contents(__DIR__.'/fixtures/bike.html'));

    $this->assertSame('Trek Marlin 7', $data['name']);
    $this->assertCount(6, $data['specs']);
}

No HTTP client to fake, no server to stand up, no Http::fake() array of URLs to keep in step with the scraper. run($input) is the standard entry, matching Scraper::run() and Spider::run(). create($input)->parse() still works and parse() is kept as a legacy alias.

Test the failure codes, not just the happy path

The valuable assertions are the ones about pages that are not what you wanted, because those are what production is full of:

public function test_a_block_page_is_reported_as_a_failure(): void
{
    $this->expectException(ScrapeException::class);
    $this->expectExceptionMessage('no_product');

    BikeCrawler::run(file_get_contents(__DIR__.'/fixtures/captcha-wall.html'));
}

A ScrapeException thrown in a crawler is caught by the terminal and folded into the response as success = false with that code, so it never bubbles out of run(). Testing it at the crawler level is testing it before the folding, which is where the decision lives.

Keep a fixture per failure mode: the captcha wall, the empty result set, the "your search is not valid" page, the layout after a redesign. That last one is worth keeping even after you fix it, as the regression test for the fix.

Where to get fixtures

Save them from the scraper you already have:

$html = ProbeScraper::run($url)->data;
file_put_contents(__DIR__.'/fixtures/bike.html', $html);

Two rules that keep this from becoming its own problem. Trim them: a 900 KB page with every inline script is unreadable in a diff, and the fifty lines you actually parse are the fixture. Scrub them: real pages carry session identifiers, and occasionally names and addresses, and a test fixture is the last place anybody looks for a leak.

The input does not have to be HTML

Crawler input is deliberately generic, which matters because a lot of scraping is not HTML:

$items = FeedCrawler::run($xmlString);                             // XML
$data  = SplitCrawler::run(['meta' => $metaXml, 'body' => $html]); // several parts

Inside, $this->filter($css, 'xml') filters as XML, $this->raw() gives you the untouched input for json_decode or a regex, and $this->html() gives the document as a string. filter() and html() require a string, and throw a clear LogicException pointing at raw() when the input is not one.

Parsing a JSON payload out of a page is the same shape, and is usually the better scraper anyway: a hashed CSS class changes on the next deploy and a payload key rarely does, because the site's own code reads it.

The half you cannot test this way

Fetching is the site's behaviour, and no fixture makes an assertion about it true tomorrow. Do not try. What you can do is make it cheap to check by hand:

php artisan tinker

$result = \App\Scrapers\BikeScraper::run('https://shop.test/bikes/4');

dd($result->success, $result->error, $result->data);

Three fields, and between them they say what happened. success is the scraper's judgement about content, error is its code, and data is what came out. HTTP facts are deliberately elsewhere, on $this->request inside handle(), because a 200 can perfectly well be success = false, error = 'captcha' and collapsing those two would lose exactly the distinction you need.

php artisan list:scrapers shows what exists in app/Scrapers, which is the fastest check that the PHP side is installed correctly.

What to run in CI

Every crawler against its fixtures. Fast, deterministic, and the only part that fails because of something you did.

Nothing that fetches. Not because fetch tests are worthless, but because a red suite that is usually not your fault is worse than no suite: it trains people to ignore the colour, and that costs you the day it means something.

If you want to know when a target changes its markup, that is a scheduled job that fetches one page and alerts on a no_product. It is monitoring, and it belongs where monitoring goes, not in the suite that gates a deploy.

More on the crawler API and driving scrapers by hand is in the commands and testing chapter.

written by Edu Lazaro · August 2026

On this page