Processing pipeline

Processing pipeline

Every top-level file goes through an asynchronous processing pipeline that runs in the queue, so the user's upload returns instantly while heavier work (image variants, video transcoding, PDF previews, text extraction, embeddings) happens in the background.

The flow is always the same:

1. A top-level File is created. FileObserver::created sets processing_status to pending and dispatches ProcessFileJob.
2. ProcessFileJob runs on the queue and calls ProcessFileAction.
3. ProcessFileAction marks the file processing (firing FileProcessingStarted), resolves the applicable steps, runs them in ascending priority() order, then marks the file completed and fires FileProcessed.

The observer only enqueues a job for files whose type is image, video, document, or audio. Variants (files with a parent_id) never enter the pipeline: their generating action marks them completed and the observer just fires VariantGenerated. See the Data model section for the processing_status column and the Events section for the dispatched events.

// You do not call this yourself. Creating a file triggers it.
$file = $user->addFile($uploaded, 'documents');
// $file->processing_status is now ProcessingStatus::PENDING
// A ProcessFileJob is queued. Once the worker runs it,
// processing_status moves to PROCESSING then COMPLETED (or FAILED).

The ProcessingStatus enum (EduLazaro\Laracrate\Enums\ProcessingStatus) has four cases: PENDING, PROCESSING, COMPLETED, FAILED, plus helpers isTerminal() and isInProgress(). If the queue is not running in development, the file stays pending until a worker picks it up. That is expected behavior, not a bug.

Default steps

Each step is a small class implementing EduLazaro\Laracrate\Contracts\FileActionInterface. The package registers these globally in LaracrateServiceProvider, ordered here by priority():

The text steps write two sidecar artifacts next to the binary on the same disk: .json (extracted full text plus per-page content) and .chunks.jsonl (chunks and embeddings). Each later step gates on the artifact the previous one produced, so the chain stops cleanly if extraction yields nothing. These sidecars are purged when the file is force deleted. See the Images, variants and watermarks, Video and PDF previews, and Text extraction, embeddings and search (RAG) sections for what each step actually does and the config keys it reads.

Priority bands

priority() returns an ascending integer. Lower numbers run first. Follow this convention so your steps slot in at the right point:

Failure and retry behavior

The pipeline is fail-fast. If any step throws, ProcessFileAction marks the file failed, stores the exception message in processing_error, fires FileProcessingFailed, and rethrows. Later steps do not run.

The rethrow lets the queue retry. ProcessFileJob declares:

• public int $tries = 3;
• public array $backoff = [10, 30, 60]; (seconds between attempts)
• public int $timeout = 600;
• public bool $deleteWhenMissingModels = true;

$deleteWhenMissingModels matters when a file is replaced before the worker reaches its job (for example setFile() swapping an avatar): Laravel silently discards the orphaned job instead of failing three times with ModelNotFoundException. Any job you add that receives a model should set it too. The job's queue name and connection come from laracrate.queue.name and laracrate.queue.connection (see the Configuration section).

Writing a step

A step decides whether it applies (supports(), optional) and what to do (handle()), and declares its order (priority()). The supports(File $file): bool method is optional: if you omit it, handle() always runs. Scope by file type, collection, or model inside supports(); throw from handle() to fail the pipeline.

namespace App\Pipeline\Steps;

use EduLazaro\Laracrate\Contracts\FileActionInterface;
use EduLazaro\Laracrate\Enums\FileType;
use EduLazaro\Laracrate\Models\File;

class ScanForVirusesStep implements FileActionInterface
{
    public function supports(File $file): bool
    {
        return $file->type === FileType::DOCUMENT;
    }

    public function priority(): int
    {
        return 5; // run before everything else
    }

    public function handle(File $file): void
    {
        // Inspect $file->key on $file->disk. Throw to fail the pipeline.
    }
}

Steps run on the queue worker, so this is also the only place you should shell out to external binaries (ffmpeg, imagick, pdftoppm). Keep that work out of the observer and out of CreateFileAction.

There are two extension points.

(a) Register a step globally

Add it to the FileActionRegistry from your own service provider's boot(). It then applies to every file (subject to its own supports()), and remove() lets you drop a packaged default by class name.

use App\Pipeline\Steps\ScanForVirusesStep;
use EduLazaro\Laracrate\Pipeline\Steps\Image\OptimizeImageStep;
use EduLazaro\Laracrate\Support\FileActionRegistry;

public function boot(): void
{
    app(FileActionRegistry::class)
        ->add(new ScanForVirusesStep())
        ->remove(OptimizeImageStep::class);
}

(b) Declare a step per collection or per model

Declare the step class under a collection's actions key in config/laracrate.php. ProcessFileAction resolves it from the container and merges it with the global steps before sorting by priority. Steps under models.{fileable_type}.actions are added (not substituted) for files whose fileable_type matches, so you can layer model-specific steps on top of collection-wide ones.

'collections' => [
    'documents' => [
        // ... disk, access, etc.
        'actions' => [
            \App\Pipeline\Steps\ClassifyDocumentStep::class, // all documents
        ],
        'models' => [
            \App\Models\Lawsuit::class => [
                'actions' => [
                    \App\Pipeline\Steps\DetectDeadlinesStep::class, // lawsuits only
                ],
            ],
        ],
    ],
],

Per-collection action classes must implement FileActionInterface. If a configured class does not, ProcessFileAction logs a warning and skips it. For introspection you can call app(FileActionRegistry::class)->applicableFor($file) to see which global steps would run for a given file.