How to scrape infinite scroll in Laravel

The list loads twenty items, and twenty more appear when you scroll. Your scraper gets the first twenty.

The fix is three lines. Before you write them, spend one minute on the check that removes the problem entirely, because it works more often than it has any right to.

First: is there an endpoint

Infinite scroll is a UI over a paginated API. The page is calling something to get those next twenty items, and that something usually answers you too.

Open the network tab, scroll once, and look at the XHR. If you see a call like /api/items?page=2, you are done: fetch it directly on the HTTP driver, in a loop, with no browser involved.

That path is four to fifteen times faster, needs no Node and no Chrome, and can run concurrently in a Spider, which the browser driver cannot. It also gives you typed, complete data instead of parsed markup, and it does not break when a hashed CSS class changes on the next deploy.

Reach for the browser when the endpoint is signed, session-bound or genuinely absent. Not before.

The browser version

When you do need it, one scroll gets one page:

->scrollToBottom()
->wait(800)

Which is exactly the trap. scrollToBottom() scrolls once and triggers one load. A list of forty pages needs forty of them, and you do not know that the number is forty.

Loop on the page's own state

use EduLazaro\Larascraper\Support\Condition;

class FeedScraper extends Scraper
{
    protected function handle(string $url): ScraperResponse
    {
        return $this->scrape($url)
            ->repeatUntil(
                Condition::selectorMissing('.loading-spinner'),
                fn ($b) => $b->scrollToBottom()->wait(800),
                max: 40,
                delay: 300,
            )
            ->crawl(FeedCrawler::class)
            ->run();
    }
}

The condition is evaluated by Puppeteer against the live page each pass, so you are asking the page whether it is still loading rather than guessing from PHP, which is not in the browser and cannot see any of this.

Pick whichever marker the site actually gives you:

The end marker is the best of the three when it exists, because it is unambiguous. A spinner that is missing might mean the list is exhausted or might mean the request has not started yet, which is why the wait(800) inside the body is doing real work: it gives the fetch time to begin before the condition is re-checked.

Why not a fixed count of scrolls

->scrollToBottom()->wait(800)
->scrollToBottom()->wait(800)
->scrollToBottom()->wait(800)   // and hope

Because it is wrong in both directions at once. On a short list you pay for scrolls that do nothing, and on a long one you silently truncate. Silent truncation is the bad half: the scraper succeeds, the data looks fine, and you find out months later that every list over sixty items was cut off.

repeatUntil() is bounded too, and deliberately: max defaults to 5, is clamped to at least 1, and there is no unbounded mode. The difference is that it stops on the condition and uses max only as a backstop, so the usual case is correct and the pathological case is contained.

Set max above what you expect and treat hitting it as a signal. If a crawl regularly stops at exactly max, the list is longer than you thought and you are truncating again, just further out.

Be polite between passes

delay throttles the time between iterations, and infinite scroll is exactly where that matters: each pass fires a request, and a tight loop turns one page view into forty rapid calls from one address.

max: 40,
delay: 300,

If the site blocks addresses, pair it with a throttle key so the pacing is shared across every process that touches the same target, rather than each worker keeping its own idea of polite.

A failed pass is not a failed run

If the body of the loop throws mid iteration, because an element was missing on a transient render, that counts as one attempt. The loop re-checks the condition and goes again, bounded by max. Only when every attempt is spent and the condition still never held does the last error surface.

Which means a single bad pass in a forty pass scroll does not lose you the thirty-nine good ones.

Then check what you actually got

The scroll loop can succeed and still under-collect, and nothing about the run will tell you. Count in the crawler and say so:

class FeedCrawler extends Crawler
{
    protected function handle(): array
    {
        $items = $this->filter('.feed-item');

        if ($items->count() === 0) {
            throw new ScrapeException('no_items');
        }

        return $items->each(fn ($item) => [
            'title' => $item->filter('h3')->text(''),
            'url'   => $item->filter('a')->attr('href'),
        ]);
    }
}

And where the site tells you the total, compare against it. A list that advertises 847 results and yields 60 is a scrape that failed while reporting success, which is the only kind of failure that gets into a database and stays there.

More on the loop, the conditions and the branch form is in the conditional flow chapter.