Configuration

You tune a scraper in three interchangeable places, and which one you reach for is about scope, not capability.

Class properties, for a scraper that always works this way:

use EduLazaro\Larascraper\Scraper;

class BikeScraper extends Scraper
{
    protected string $driver = 'browser';
    protected int $timeout = 10000;                     // ms
    protected int $tries = 5;                           // attempts
    protected int $retryDelay = 10;                     // seconds between attempts
    protected array $headers = ['Accept-Language' => 'en'];
    protected ?string $userAgent = null;
    protected ?string $proxy = '200.20.14.84:40200';
    protected ?string $proxyUser = 'username';
    protected ?string $proxyPass = 'password';
    protected ?string $throttleKey = 'shop.listing';
}

The fetch chain, per request:

protected function handle(string $url): ScraperResponse
{
    return $this->scrape($url)
        ->proxy('200.20.14.84:40200', 'username', 'password')
        ->timeout(10000)
        ->headers(['Accept-Language' => 'en'])
        ->retry(3, 5)
        ->crawl(BikeCrawler::class)
        ->run();
}

with(...), for a one-off override from the outside:

BikeScraper::with(timeout: 45000, tries: 1)->run($url);

Timeout

Milliseconds, 20000 by default. On the browser driver it covers navigation and every action wait, so a page with several waitForSelector() calls shares one budget:

->timeout(10000)

Individual waits can override it. waitForSelector($sel, ['timeout' => 2000]) keeps an optional element from burning the whole allowance.

Headers

->headers([
    'Accept-Language' => 'en',
    'X-Custom-Header' => 'Hello',
])

Every header is sent untouched, on either driver, with one exception: User-Agent, which is applied through the user-agent mechanism rather than sent as an extra header. On the browser driver that is the only way it takes effect at all, because once Chrome has a user-agent override an extra header for it is ignored.

Retries

$tries (default 3) and $retryDelay (default 15 seconds), or ->retry($attempts, $seconds) on the chain:

->retry(3, 5)   // 3 attempts, 5s apart

Only the transient statuses 408, 429, 500, 502, 503 and 504 are retried. Anything else fails fast, and if it survives the retries it throws a RequestException. Retrying a 404 or a 401 only wastes time; neither is going to change its mind.

A 403 is the exception to the exception. It is retried only while another proxy is still free to try, because the refusal is about the address rather than the request, and that retry skips the delay. See Proxies and throttling.

User agent

On the browser driver you normally set nothing. The user agent is asked of the Chrome that just launched, and the one word that gives headless away is dropped:

Mozilla/5.0 (X11; Linux x86_64) … HeadlessChrome/148.0.0.0 Safari/537.36   ← what it is
Mozilla/5.0 (X11; Linux x86_64) … Chrome/148.0.0.0 Safari/537.36           ← what it says

Version and platform stay true, which is the entire point. Client Hints (Sec-CH-UA, navigator.userAgentData) are filled in by the real Chrome and cannot be talked out of it, so a made-up user agent is a louder signal than an honest headless one: a browser claiming one version while emitting another is contradicting itself, and a real one never does. The build number needs no faking either, since Chrome froze it at 0.0.0 years ago, so Chrome/148.0.0.0 is character for character what a real Chrome 148 sends.

It costs one CDP call, around 0.3 ms against a launch of roughly 220 ms, so it is asked every time rather than cached. A cache would go stale on the next Chrome upgrade and put the mismatch straight back.

Override it when you actually mean to, asking for the mobile version of a site for instance:

protected ?string $userAgent = 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) …';

->userAgent('Mozilla/5.0 (iPhone; …)')   // per fetch
->userAgent(null)                        // hand the question back to Chrome

The http driver has nobody to ask, so its default is written out in config, where you can bump it without waiting for a package release. Unlike the derived one, it ages:

// config/larascraper.php
'http_user_agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) …Chrome/148.0.0.0…',

Emptying the key does not fall through to sending nothing. It falls back to HttpRunner::DEFAULT_USER_AGENT, because sending nothing is not silence: Guzzle fills in GuzzleHttp/7, which announces that a script is calling. To claim something else, name it; there is no way down to nothing.

The browser driver ignores that key on purpose. Letting a config string override what Chrome reports would restore the exact contradiction it replaced.

Precedence, in full:

The config file

php artisan vendor:publish --tag=larascraper-config

The vision engine that reads scanned PDFs also reads from this file when the keys are present: openai_key, vision_model (default gpt-4o-mini), vision_lang (default spa) and vision_dpi (default 150). The key also falls back to services.openai.key and then to the OPENAI_API_KEY environment variable. See Files and PDFs.

Everything works without publishing anything. The file is for when you want a pool, throttling, or a newer user agent.