Scraping in PHP splits into two toolchains, and the seam is where the bugs live

Scraping in PHP has a fault line running through it, and everyone builds on both sides of it.

Static pages go through Guzzle. Fast, cheap, no dependencies, and the code is a fetch and a parse. Pages that render in JavaScript cannot be fetched at all, so they go through a headless browser, which in PHP means shelling out to a Node script, passing arguments as JSON on the command line, and parsing whatever comes back on stdout.

Two toolchains. Two error models: one throws a Guzzle exception, the other gives you a non-zero exit code and a string. Two ways to express "wait for this to appear", one of which does not exist. Two places to configure timeouts. And glue in between, which is where the bugs are, because the glue is the part nobody tests.

The worst property of the split is that it is not stable. A site you scrape with Guzzle today ships a frontend rewrite next quarter and now needs the browser. That is not a config change, it is a rewrite of that scraper into the other toolchain, with a different error model and different parsing.

The driver is a detail, not an architecture

The fix is to stop letting transport decide code shape. A scraper describes what to get; how the bytes arrive is a swappable detail:

$this->scrape($url)->crawl(BikeCrawler::class)->run();          // browser
$this->scrape($url)->driver('http')->crawl(BikeCrawler::class)->run();  // plain HTTP

Same class, same crawler, same ScraperResponse with the same success and error. When that site ships its rewrite you delete ->driver('http').

Two decisions inside that are worth pulling out.

The browser is the default. The cheaper option would be the obvious default, and it is the wrong one, because the failure modes are asymmetric. Defaulting to HTTP against a JavaScript page gives you a successful fetch of an empty shell: no exception, a crawler returning nulls, and a scraper that looks like it works. Defaulting to the browser against a static page is merely slower. Make the default the one whose failure is loud.

Combining actions with the HTTP driver throws. ->type(...)->driver('http') is not silently ignored and does not silently upgrade you to a browser. It is a contradiction in the code, and quietly picking an interpretation would hide a real mistake behind behaviour you did not ask for.

Parsing takes generic input on purpose

A Crawler gets a DOM query builder, which covers most cases:

'name' => $this->filter('h1.title')->text(),

But committing to "input is HTML" would have been a mistake, because a meaningful share of scraping targets are not HTML at all: an endpoint returning JSON, a feed returning XML, a page whose real payload is a JSON blob inside a script tag.

So $this->filter($selector, 'xml') parses as XML, and $this->raw() hands over the untouched input for json_decode or a regex. The abstraction covers the common case and gets out of the way for the rest, rather than forcing everything through a DOM parser that has to pretend JSON is a document.

Retries have to be bounded, and have to distinguish two failures

Flaky flows are the norm: a captcha, a session that needs warming, a page that intermittently does not render. So there is a retry loop, and a retry loop pointed at somebody else's server is a thing you can do real harm with.

->repeatUntil(
    Condition::selectorMissing('#captcha'),
    fn ($b) => $b->solveCaptcha('#captcha-img', '#captcha-input')->clickAndWait('#verify'),
    max: 5,
    delay: 1500,
)

max and delay are not optional, because an unbounded retry against a site that is down is indistinguishable from a small denial of service, and the person running it usually does not notice.

The subtler decision is that the loop separates two kinds of failure. A throw inside an attempt is expected) the captcha was misread, the click landed early (so it counts as a failed attempt and the loop tries again. But a configuration error, a solver name that does not exist or a 4xx from an API key, aborts immediately. Retrying a wrong API key five times with a delay just makes you wait longer for the same answer.

Concurrency in a single-threaded language

At scale the constraint is that PHP has no threads, and scraping is almost entirely waiting on the network.

The usual PHP answers are to fan out into queued jobs, which means N workers, N browsers and a coordination problem, or to give up and go sequential, which for five thousand pages means hours of doing nothing but waiting.

Fibers give a third option. Each scraper runs until it needs bytes and then suspends. The Spider collects all the suspended fetches into one wave, sends them together, and resumes each Fiber with its response:

protected int $concurrency = 20;

Which means real concurrency lives on the HTTP driver: you cannot batch twenty headless browsers this way, and pretending otherwise would be a lie in the API. It is also why a shared cookie Session threads through every request: a login obtained once has to survive the whole crawl, or every one of those five thousand requests re-authenticates.

What it replaced

One class shape, one error model, one place to configure a timeout, and the transport as an argument rather than an architecture.

The full API (crawlers, actions, waits, captchas, file capture and spiders) is on the Larascraper page. Source on GitHub, package on Packagist.