Responses and failures

Scraping has two kinds of failure and they deserve different treatment. The network being down is exceptional. A page returning a captcha is not: pages block, layouts change, searches come back empty. Larascraper draws the line there, and the caller only ever catches one exception.

The ScraperResponse

run() always returns a ScraperResponse, a value object with exactly three fields:

handle() never builds one by hand

run() normalizes whatever handle() returns:

fail() and ok() are for the decision you make in handle(); ScrapeException is for a failure raised deep in nested code, typically inside a Crawler. They produce the same thing.

use EduLazaro\Larascraper\Scraper;
use EduLazaro\Larascraper\Support\ScraperResponse;

class LawScraper extends Scraper
{
    protected function handle(string $url): array|ScraperResponse
    {
        $text = trim($this->scrape($url)->capture()->file()->text());

        if ($text === '') {
            return $this->fail('no_text');   // success = false, error = 'no_text'
        }

        return ['text' => $text];            // success = true
    }
}

Where the HTTP facts live

The ScraperResponse deliberately carries no HTTP facts. No status, no cookies, no raw html, no captured binary. Those belong to the request layer, and inside handle() they are on $this->request, a RequestResponse with status, error, html, file, contentType and cookies:

protected function handle(string $url): array
{
    $data = $this->scrape($url)->crawl(BikeCrawler::class)->run()->data;

    $status  = $this->request->status;      // 200
    $cookies = $this->request->cookies;     // ['session' => '...']

    return ['data' => $data, 'status' => $status];
}

The separation is the point. success is the scraper's own judgement about content: a status = 200 can perfectly well be success = false, error = 'captcha'. If a caller needs an HTTP fact, fold it into data on the way out.

The one exception you catch

Request-level failure means the network was down, or a status the fetcher treats as failure survived the bounded retries. That throws a RequestException, which carries the RequestResponse, so you can branch on the status. This is the only exception that reaches the caller.

Scrape-level failure comes back as success = false plus error. Never an exception.

use App\Scrapers\BikeScraper;
use EduLazaro\Larascraper\Exceptions\RequestException;

try {
    $result = BikeScraper::run($url);          // always a ScraperResponse

    if (! $result->success) {
        // $result->error is 'captcha' / 'no_results' / ...
        return;
    }

    $bike = $result->data;
} catch (RequestException $e) {
    report("HTTP {$e->response->status}: {$e->getMessage()}");
}

Do not throw the response away

The most expensive bug in a scraping system is the one that does not raise anything. A scraper whose target quietly started returning a block page keeps running, keeps exiting zero, and keeps writing nothing. Nobody notices for weeks, because nothing failed.

Two habits prevent it, and both are one line:

Read the response. A call site that does this is blind by construction:

MyScraper::run($url);       // return value discarded

Say what "worked" means. If a scrape that finds no rows is not a success, make the scraper say so, and then a caller that checks success reports it:

if ($rows === []) {
    return $this->fail('no_results');
}

There is a third trap worth naming, because it defeats a try/catch that looks correct. catch (Exception) does not catch a TypeError, or anything else extending Error, so a configuration value arriving as null where a string was declared kills the command through a handler that appears to cover it. Catch Throwable when you mean everything.