How to follow pagination in Laravel when the page count is unknown

Follow the "Next" link until there is no "Next" link. That is the whole algorithm, and it goes wrong in three specific ways that all look like success.

The shape

use EduLazaro\Larascraper\Support\Condition;

class ListScraper extends Scraper
{
    protected function handle(string $url): ScraperResponse
    {
        return $this->scrape($url)
            ->repeatUntil(
                Condition::selectorMissing('a.next'),
                fn ($b) => $b->clickAndWait('a.next')->waitForSelector('.results'),
                max: 200,
                delay: 500,
            )
            ->crawl(ResultsCrawler::class)
            ->run();
    }
}

Two things to notice before the failure modes.

The exit condition describes the page, and is checked by Puppeteer against the live document each pass, because PHP is not inside the browser.

clickAndWait(), not click(). A click that triggers a navigation needs the wait armed before it fires, or on a fast server the navigation completes before you start waiting for it and the run dies at the timeout. In a loop this is worse than usual: it works for the first few pages and fails somewhere in the middle, which reads like the site rate limiting you.

But this collects one page

The chain ends with one crawl() over the final document. You have just clicked through 200 pages to parse the last one.

Which is fine when the last page is what you want, and rarely is. When you want all of them, the loop belongs outside the fetch, in PHP, where you can keep what each page produced:

protected function handle(string $url): array
{
    $all = [];

    for ($page = 1; $page <= 200; $page++) {
        $result = $this->scrape("{$url}?page={$page}")
            ->driver('http')
            ->crawl(ResultsCrawler::class)
            ->run();

        if (! $result->success || $result->data === []) {
            break;
        }

        $all = [...$all, ...$result->data];
    }

    return $all;
}

This is the version to reach for by default. It runs on the HTTP driver, which is several times faster and can go concurrent in a Spider, and each page is an independent fetch, so one bad page does not take the other 199 with it.

Use the in-browser repeatUntil() version when the pagination is genuinely stateful: a POST-backed form, a session-bound cursor, a "Load more" button with no URL of its own.

The three ways it goes wrong

A site that serves page 500 of a 40 page list. Plenty of applications clamp out of range pages to the last one, or return the first, rather than 404ing. A loop that stops when the response fails will never stop. Stop on content: an empty result set, or the same first item as the previous page.

$firstUrl = $result->data[0]['url'] ?? null;

if ($firstUrl === $previousFirstUrl) {
    break;              // the site is repeating itself
}

The last page is a legitimately empty page. Some paginators render a final page with a header and no rows. If your crawler throws no_results on zero rows, and it should, then treat that particular failure as the terminator rather than an error:

if ($result->error === 'no_results') {
    break;
}

An error page that renders where the results go. This is the one that cost me a day. A court portal accepts only 10, 20, 30 or 50 as its results-per-page value; anything else makes it render "the search is not valid" inside the results container. Same selector, same status, zero rows. The loop terminates on the first page, reports success, and records that the archive is empty.

The defence is to name the error explicitly in the crawler, so a refusal and an absence are different codes:

if ($this->filter('.error-message')->count() > 0) {
    throw new ScrapeException('invalid_query');
}

if ($this->filter('.result-row')->count() === 0) {
    throw new ScrapeException('no_results');
}

Then break on no_results and let invalid_query be a loud failure. Without that distinction, every stop condition you write is guessing.

Prefer a real total when you can get one

Most listings tell you how many results there are, and it is worth parsing even if you do not use it to drive the loop, because it gives you an assertion:

$expected = (int) $this->filter('.result-count')->text('0');

Compare it against what you collected at the end. A crawl that advertised 847 and returned 60 succeeded and is wrong, and that is the only kind of wrong that gets written to a database and stays there for months.

Once you have URLs, stop clicking

The pagination loop's real job is often just to discover URLs. As soon as you have them, the sequential click-through has done its work and the detail pages want a Spider:

class ArchiveSpider extends Spider
{
    protected int $concurrency = 20;
    protected int $delay = 250;

    public function handle(): int
    {
        $ids = collect($this->discoverIds())
            ->reject(fn ($id) => Document::where('remote_id', $id)->exists());

        $this->pool($ids, DocumentScraper::class, $this->save(...));

        return $ids->count();
    }
}

Filtering before pool() rather than inside it is what makes a crawl resumable: you never fetch what you already stored, so an interrupted run continues instead of starting over.

Bound it, always

repeatUntil() is bounded by construction, with max defaulting to 5 and no unbounded mode. Your own for loop is not, so give it a ceiling anyway, and treat reaching that ceiling as a signal rather than a normal ending. A crawl that regularly stops at exactly max is not finishing, it is being cut off.

More on the loop and its conditions is in the conditional flow chapter; the concurrent half is in spiders and sessions.