How to scrape a JavaScript site with Larascraper

The page renders in your browser and comes back empty from file_get_contents. The obvious conclusion is that you need a headless browser, and the obvious conclusion is wrong about half the time.

Here is how to tell, and what to do in each case.

First, check whether you actually need one

A page can be "JavaScript" in two quite different ways, and only one of them needs Chrome.

The data is in the response, and JavaScript only paints it. This is most of the modern web. A Next.js page ships its data as JSON inside a __NEXT_DATA__ script tag in the initial HTML. Nuxt does the same with __NUXT_DATA__. Anything server-rendered arrives complete and then gets hydrated.

The data genuinely is not there. The page loads a shell and fetches content afterwards, or it only appears once you interact.

Test it before you assume, with the HTTP driver and a value you know is on the page:

class ProbeScraper extends Scraper
{
    protected string $driver = 'http';

    protected function handle(string $url): string
    {
        return (string) $this->scrape($url)->run()->data;
    }
}

$html = ProbeScraper::run($url)->data;

str_contains($html, 'Elden Ring');          // is the content there?
str_contains($html, '__NEXT_DATA__');       // is the data there as JSON?

If either is true, you are done, and you should stop here.

Why it matters more than it looks

Measured on the same page, through the same proxy: 1.6 seconds through Chromium, 0.10 to 0.38 seconds through plain HTTP. Four to fifteen times faster, plus no Node, no Chrome binary, no memory spike, and one less thing to install in a container.

At one page that is a curiosity. At forty thousand pages it is the difference between a crawl that finishes overnight and one that does not.

There is a second reason. A __NEXT_DATA__ payload is typed and complete, with fields their own frontend needs and their markup may never show. Parsing it is also more stable than parsing markup: a hashed CSS class changes on their next deploy, and a payload key rarely does, because their own code reads it.

I had a scraper built on hashed class names die overnight when the target migrated to Next.js. The rewrite that replaced it is shorter than the original, faster, and has not broken since:

class GameCrawler extends Crawler
{
    protected function handle(): array
    {
        if (! preg_match('#<script id="__NEXT_DATA__" type="application/json">(.*?)</script>#s', $this->html(), $m)) {
            throw new ScrapeException('no_data');
        }

        $game = json_decode($m[1], true)['props']['pageProps']['game']['data']['game'][0] ?? [];

        return [
            'name' => $game['game_name'] ?? null,
            'main' => $game['comp_main_avg'] ?? null,
        ];
    }
}

The site's redesign, which looked like a disaster, handed me better data than the markup ever had.

And check the user agent before blaming JavaScript

An empty response is not always JavaScript. Quite often it is a 403 that you did not look at closely.

Measured against a site that blocks scrapers, everything over plain HTTP, no browser involved:

No fingerprinting, no Cloudflare, no JavaScript challenge. One header. Which makes sense: a site that wants Google to index its pages cannot afford to be genuinely hard to fetch.

Larascraper's HTTP driver already sends a real Chrome user agent by default, from config('larascraper.http_user_agent'), precisely so this does not happen to you. Sending nothing is not neutral: Guzzle fills in GuzzleHttp/7, which announces that a script is calling.

So before you reach for Chrome, look at $this->request->status. A 403 is an address or a header problem, and a browser will not fix either.

When you do need the browser

Some pages really do build their content client-side, or only reveal it after you interact. That is what the browser driver is for, and in Larascraper it is the default, so you write nothing:

class ResultsScraper extends Scraper
{
    protected function handle(string $url): ScraperResponse
    {
        return $this->scrape($url)
            ->click('#accept-cookies')
            ->type('#search', 'zelda')
            ->press('Enter', waitForNavigation: true)
            ->waitForSelector('.results')
            ->scrollToBottom()
            ->crawl(ResultsCrawler::class)
            ->run();
    }
}

Actions run in order, in a single browser session, after navigation and before the final HTML is captured. The waits happen inside Node, where the page is alive, so timing behaves.

Three things are worth getting right from the start.

Arm the wait before the click. Use waitForNavigation: true on the action itself, or clickAndWait(), rather than a separate ->waitForNavigation() afterwards. Otherwise the navigation can finish before the wait begins, which is the classic scraper that works on your laptop and fails on a fast server.

Not every wait should be fatal. An element that legitimately may not appear, an empty result set, an optional banner, should not kill the run:

->waitForSelector('.banner', ['optional' => true, 'timeout' => 2000])

And when either of two things can happen, wait for both and let the first one win:

->waitForSelector(['.results', '.no-results'])

Guard optional steps. A click('#accept-cookies') on a page with no cookie banner is a failed action, and therefore a failed fetch. Ask the live page first:

use EduLazaro\Larascraper\Support\Condition;

->when(
    Condition::selectorExists('#cookie-banner'),
    fn ($b) => $b->click('#accept-cookies'),
)

The interesting case: use both

The driver belongs to the fetch, not to the scraper, so one handle() can use each where it earns its cost. Drive the page with the browser only for the part that needs a browser, then pull the results over HTTP:

protected function handle(string $listUrl): array
{
    $urls = $this->scrape($listUrl)          // browser: the list is behind a control
        ->select('#year', '2026')
        ->wait(2500)
        ->waitForSelector('a.result')
        ->crawl('a.result')
        ->texts();

    return collect($urls)
        ->map(fn ($url) => $this->scrape($url)->driver('http')->run()->data)   // http: fast
        ->all();
}

One Chromium launch for the listing, hundreds of cheap HTTP fetches for the detail pages. On a crawl of any size that is the whole game, and if the detail scraper is its own class you can then run those concurrently through a Spider, which the browser driver could never do.

More on both drivers, including what each one refuses to do, is in the drivers chapter.