← Larascraper 10 / 17

Page actions

Sometimes the content you need only appears after interacting with the page: accepting a cookie banner, filling and submitting a form, paginating, expanding a "show more" section, or scrolling to trigger lazy loading.

Actions chain on the fetch chain before the terminal. They are sent to Puppeteer and executed in order, in a single browser session, right after navigation and before the final HTML is captured. The waits happen inside Node, where the page is alive, so timing works naturally:

protected function handle(string $url): ScraperResponse
{
    return $this->scrape($url)
        ->click('#accept-cookies')
        ->type('#search', 'zelda')
        ->press('Enter', waitForNavigation: true)   // submit and wait for the new page
        ->waitForSelector('.results')
        ->scrollToBottom()                          // trigger lazy loading
        ->wait(800)
        ->crawl(ResultsCrawler::class)              // parse the final HTML
        ->run();
}

Actions need a page, so they are browser-driver only. Combining them with ->driver('http') throws a LogicException.

The actions


`->click($selector)`	Click an element, waiting for it first.
`->click($selector, waitForNavigation: true)` / `->clickAndWait($selector)`	A click that triggers a page load, and the wait for it.
`->type($selector, $text)`	Type into an input, waiting for it first.
`->select($selector, $value)`	Choose an option by value in a `<select>`. Pass an array to select several at once on a `<select multiple>`, applied in a single call so earlier values are not deselected.
`->setValue($selector, $value)`	Set an element's value directly, firing `input` and `change`. For hidden inputs populated by a custom widget, or fields `type()` and `select()` cannot reach.
`->check($selector)` / `->uncheck($selector)`	Tick or untick every match, firing a bubbling `change`. Works on widget-backed checkboxes hidden in a collapsed dropdown where a native click cannot reach. Boxes already in state are left alone, and no match is a silent no-op.
`->hover($selector)`	Hover over an element.
`->press($key)`	Press a key (`Enter`, `Tab`, `Escape`). Pass `waitForNavigation: true` when it submits a form.
`->waitForSelector($selector, $options = [])`	Wait until an element appears. See below.
`->waitForNavigation()`	Wait for a navigation to finish.
`->wait($ms)`	Wait a fixed number of milliseconds.
`->scroll('bottom'\|'top')` / `->scrollToBottom()`	Scroll the page, for infinite scroll and lazy loading.
`->submit($selector)`	Submit a form, including its hidden fields and tokens.
`->visit($url, $waitUntil = 'networkidle2')`	Navigate mid-flow, resolved against the current page.
`->gotoAttr($selector, $attr = 'href', $waitUntil = 'networkidle2')`	Navigate to the URL held in an element's attribute, for an `<object data="…">` or `<embed src="…">` viewer where the next URL is not a link.
`->reload($waitUntil = 'networkidle2')`	Reload the page, for instance to regenerate a captcha image before solving it.

If an action fails, a selector that never appears within the timeout for example, the fetch fails, which raises a RequestException after the retries, exactly like an HTTP error.

Selectors

Every $selector is a plain CSS selector passed to Puppeteer, so anything CSS supports works, attribute selectors included:

->type('[name=email]', 'me@example.com')      // by name attribute
->type('input[name=captcha]', $code)          // tag plus name
->click('[name=submit]')
->select('[name=lang]', 'en')                 // a <select name="lang">

[name=x], [name="x"] and input[name=x] all work, as do [data-id=5] and [type=submit].

Prefer these to generated class names. A hashed class from a build step changes on the target's next deploy; a name attribute is part of how their own form works.

Waits that should not be fatal

waitForSelector() takes options, and both of them matter more than they look:

->waitForSelector('.results')                                     // required
->waitForSelector('.banner', ['optional' => true, 'timeout' => 2000])
->waitForSelector(['.results', '.no-results'])                    // whichever lands first

'optional' => true swallows a timeout and continues, for elements that legitimately may never appear: an empty result set, a banner that only shows sometimes. Pair it with a short 'timeout' so an absent element does not burn the whole global allowance.
An array of selectors is grouped into one comma selector, so the wait resolves on whichever appears first. This is how you wait for "results or the no-results message" without guessing which one you are getting.

Navigation waits

visit(), gotoAttr() and reload() accept a Puppeteer wait condition. The default 'networkidle2' is right for most pages, but some servers keep connections open and never reach network idle, where 'networkidle2' would burn the whole timeout. For those, pass 'domcontentloaded' and let a following waitForSelector() be the real "content is ready" signal:

->visit($url, 'domcontentloaded')->waitForSelector('.results')

Arm the wait before the click

For a click or key press that loads a new page, use waitForNavigation: true on that action, or clickAndWait(), rather than a separate ->waitForNavigation() afterwards. That arms the wait before the click, which avoids the race where the navigation finishes before the wait starts. It is the single most common source of a scraper that works locally and fails on a fast server.

Driving controls instead of guessing URLs

Actions are not only for cookie walls. Quite often the data you are missing is behind a control on the page rather than behind a different URL, and the site's own API is the part that is locked down.

A concrete case: a listing page whose unfiltered view had already been fully harvested, returning 355 items of which 354 were known. The page carried a year <select>. Driving it surfaced 202 items, 84 of them new:

$this->scrape('https://example.com/stats')
    ->select('#showyear', '2026')
    ->wait(2500)
    ->waitForSelector('a[href^="/item/"]')
    ->crawl('a[href^="/item/"]')
    ->texts();

Meanwhile the site's search API, which accepts a year range in its payload, answers Session expired or invalid fingerprint to anything that is not their own frontend. The clean path was closed and the visible control was open. Look at what the page offers a human before deciding a source is exhausted.

PreviousProxies and throttling NextConditional flow

On this page