Why your Laravel scraper works by hand and fails in cron

The scraper runs from your terminal. The scheduler runs it and it fails, usually with something about Node not being found, sometimes with nothing useful at all.

Nothing about your code is wrong. Cron runs a non-interactive, non-login shell, and that shell never read the file that put Node on your path.

The NVM trap

If Node came from NVM, it lives under ~/.nvm/versions/node/... and is put on the path by a snippet in your shell profile. An interactive terminal reads that profile. Cron does not.

So node exists for you and does not exist for the scheduler, which is exactly the confusing part: which node from your prompt proves nothing about the environment the job runs in.

Fix it by making the snippet available to non-interactive shells. Edit ~/.bash_profile:

nano ~/.bash_profile

and make sure this is at the top:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

Then:

source ~/.bash_profile

That is the direct fix, and it works. The better answer is that NVM does not belong on a production server. It exists to switch between Node versions while developing, which is not something a server does, and in exchange it makes the interpreter's location depend on shell configuration. A system Node is one moving part fewer, and this class of failure disappears with it.

Prove it rather than guessing

Before changing anything, find out what the job's environment actually looks like, from the job:

* * * * * cd /var/www && /usr/bin/php artisan schedule:run >> /var/www/storage/logs/cron.log 2>&1

The 2>&1 is not optional. Without it the error goes to cron's mail, which on most servers goes nowhere, and you are left with a job that fails silently. A scheduled scraper with no captured stderr is the reason this problem takes a day instead of ten minutes.

Then have the job tell you where things are:

* * * * * which node; echo $PATH >> /tmp/cronenv.log 2>&1

Compare that PATH with your own. The difference is the bug.

The same symptom, three more causes in Docker

Containers produce the identical failure for reasons that have nothing to do with cron, and it is worth checking all of them because the error message does not distinguish between them.

Chrome was never downloaded in that container. larascraper:install installs the Node packages and the Chrome binary Puppeteer needs. If node_modules is already present, for instance mounted in from the host, Puppeteer skips its automatic Chrome download, so the command installs the binary explicitly. Run it in the environment the scraper executes in, not on the host:

docker compose exec app php artisan larascraper:install

The container was recreated and Chrome went with it. The binary lands in that environment's cache, and a recreate without a volume for it wipes it. Either persist the cache or run the install as part of the image build.

A system Chrome exists and you would rather use it. Point Puppeteer at it and skip the download:

php artisan larascraper:install --no-browser

with PUPPETEER_EXECUTABLE_PATH set to the binary. This is usually the right shape for a slim production image.

If the Node packages are missing entirely, the scraper fails fast with a message that says so, rather than failing obscurely somewhere further along. That is deliberate, and it is the one version of this failure that diagnoses itself.

The captcha packages are separate

tesseract.js and jimp are not installed by default, so a project that never solves a captcha stays lean. A scraper that calls solveCaptcha() in production while the install ran without the flag fails with a message pointing at:

php artisan larascraper:install --captcha

Worth knowing because it is environment-dependent in the same way: it works locally, where you ran the flag months ago and forgot.

Do not use the browser if you do not need it

The whole class of problem belongs to the browser driver. Node, Chrome, the binary cache, the path. The HTTP driver has none of it, and a good proportion of pages that look like they need a browser do not:

class ProbeScraper extends Scraper
{
    protected string $driver = 'http';

    protected function handle(string $url): string
    {
        return (string) $this->scrape($url)->run()->data;
    }
}

If the value you want is in that HTML, or the data is sitting in a __NEXT_DATA__ script tag, you are done: no Node, no Chrome, nothing to install in the container, and it fetches several times faster besides.

The system packages for the PDF engines are a different list and Composer cannot install those either. ghostscript for text(), poppler-utils for pdftotext and page rasterization, tesseract-ocr for local OCR. Note that the cloud vision engine still needs poppler-utils, because it rasterizes pages locally before sending them anywhere.

A checklist for the next time

Installation, the flags and the environment notes are in the installation chapter.