How to submit a search form in a headless browser from Laravel

Filling a search form in a headless browser is four lines until it is not. The form has a multi-select backed by a JavaScript widget, a checkbox list inside a collapsed dropdown, and a results area that sometimes says "no results" instead of having results.

Here is the version that survives all three.

The four lines

use EduLazaro\Larascraper\Scraper;
use EduLazaro\Larascraper\Support\ScraperResponse;

class SearchScraper extends Scraper
{
    protected function handle(string $term): ScraperResponse
    {
        return $this->scrape('https://portal.test/search')
            ->type('[name=q]', $term)
            ->select('[name=year]', '2026')
            ->press('Enter', waitForNavigation: true)
            ->waitForSelector('.results')
            ->crawl(ResultsCrawler::class)
            ->run();
    }
}

Actions run in order, in a single browser session, after navigation and before the final HTML is captured. The waits happen inside Node, where the page is alive, so timing behaves the way you would expect it to.

Every selector is a plain CSS selector passed to Puppeteer, so attribute selectors work and are usually the right choice on a form: [name=q] survives a redesign that #search-input-2 does not.

Arm the wait before the click

->press('Enter', waitForNavigation: true)   // yes
->clickAndWait('#submit')                   // yes

->click('#submit')->waitForNavigation()     // race

The last one arms the wait after the click. On a fast server the navigation can finish before the wait begins, and then you wait for a navigation that already happened until the timeout kills the run. This is the classic scraper that works on a laptop and fails in production, and it is entirely a scheduling artefact.

Multi-selects deselect what you just chose

The native way to pick several options is to call select repeatedly, and on a <select multiple> each call replaces the selection rather than adding to it. You end up with the last value only, and no error anywhere.

Pass an array and they are applied in a single call:

->select('#court', ['11', '12', '13'])   // all three

A string selects one option, an array selects several without earlier values falling off.

When type() and select() cannot reach the field

Modern forms are full of controls that look like a select and are not: a visible widget writing into an <input type="hidden">, a checkbox list rendered inside a collapsed dropdown that a native click cannot reach because it is not on screen.

Two escape hatches, and they are the ones that make a real portal scrapeable:

->setValue('[name=court_ids]', '11,12,13')   // hidden input behind a widget
->check('[name="areas[]"]')                   // every matching checkbox
->uncheck('#include-archived')

setValue() writes the value directly and fires input and change, so the page's own listeners run and the form state updates as though a person had done it.

check() and uncheck() tick every match and fire a bubbling change, which reaches widget-backed checkboxes that a click cannot. Boxes already in the requested state are left alone, and a selector that matches nothing is a silent no-op rather than a failure, so an optional filter does not break the run.

The wait that has to accept both outcomes

A search returns results, or it returns nothing, and both are valid. Waiting only for .results means every legitimately empty search burns the full timeout and then fails.

Wait for either and let the first one win:

->waitForSelector(['.results', '.no-results'])

The list is grouped into one comma selector, so the wait resolves on whichever lands first. For an element that may genuinely never appear, an optional banner, a cookie notice, make the timeout non-fatal and keep it short:

->waitForSelector('.promo', ['optional' => true, 'timeout' => 2000])

Without timeout, an optional wait still sits there for the global timeout before giving up, which on a large crawl is the difference between hours.

Guard the steps that are conditional

A cookie banner appears once per profile, and a fresh browser profile per run means it appears every time, until the day the site changes and it does not. An unguarded click('#accept-cookies') on a page without the banner is a failed action, which fails the fetch.

use EduLazaro\Larascraper\Support\Condition;

->when(
    Condition::selectorExists('#cookie-banner'),
    fn ($b) => $b->click('#accept-cookies'),
)

The condition is evaluated by Puppeteer against the live page, because PHP is not inside the browser. You describe what to check and Node checks it.

The failure that cost me a day

A court records portal takes a results-per-page parameter. I set it to 100, which is a perfectly ordinary number, and got a page with no results on it.

The field accepts 10, 20, 30 or 50. Anything else makes the application render "the search is not valid" in the element where the results go. Same selector, same layout, same status code. My wait resolved, my crawler found zero rows, and the scraper reported an empty result set, calmly, for every query.

The lesson is not about that portal. It is that a 200 with a wait that resolved is not success, and if your crawler only knows how to count rows it will report a broken search as a fact about the world. Say so explicitly:

class ResultsCrawler extends Crawler
{
    protected function handle(): array
    {
        if ($this->filter('.error-message')->count() > 0) {
            throw new ScrapeException('invalid_query');
        }

        if ($this->filter('.result-row')->count() === 0) {
            throw new ScrapeException('no_results');
        }

        return $this->filter('.result-row')->each(fn ($row) => [
            'title' => $row->filter('h3')->text(''),
            'url'   => $row->filter('a')->attr('href'),
        ]);
    }
}

ScrapeException is caught by the terminal and folded into the response as success = false with that code, so it never bubbles out of run(). The caller branches on $result->error and can tell "the site refused my query" from "there is nothing there", which are two very different things to record in a database.

That distinction is the single most valuable thing to build into a scraper early, and it costs three lines.

Then move off the browser

Once the form has produced a list of URLs, the browser has done its job. The detail pages almost never need it:

protected function handle(string $term): array
{
    $urls = $this->scrape('https://portal.test/search')   // browser: the form needs it
        ->type('[name=q]', $term)
        ->press('Enter', waitForNavigation: true)
        ->waitForSelector(['.results', '.no-results'])
        ->crawl('a.result')
        ->texts();

    return collect($urls)
        ->map(fn ($url) => $this->scrape($url)->driver('http')->run()->data)   // http
        ->all();
}

One Chromium launch for the form, cheap HTTP fetches for everything behind it, and the detail scraper can then run concurrently in a Spider, which the browser driver cannot do at all.

The full action list is in the page actions chapter.