Search and RAG

Text extraction, embeddings and search (RAG)

Laracrate can turn your uploaded files into searchable text and vector embeddings, so you can run keyword, semantic, or hybrid search across everything a model owns. This is the foundation for retrieval-augmented generation (RAG): extract text from a PDF, contract, audio recording or scanned image, chunk it, embed it, store it, then query it. Everything runs in the processing pipeline (see the Processing pipeline section), so the user's upload stays instant.

The flow has four stages, each a separate action that hands off through portable sidecar files in storage:

ExtractText  ->  ChunkText  ->  GenerateEmbedding  ->  PersistChunks
   |               |                |                     |
{key}.json   {key}.chunks.jsonl   (+embedding)        ChunkStore

ExtractTextAction writes a .json sidecar with the full text plus per-page segments. ChunkTextAction splits it into .chunks.jsonl (one JSON object per line). GenerateEmbeddingAction rewrites that JSONL with an embedding field on each chunk. PersistChunksAction reads the final JSONL and stores it through the active ChunkStore driver. The JSONL is the canonical, portable artifact: if you switch backends later, you can rebuild every store by re-running PersistChunksAction over your files without re-extracting or re-embedding.

Enabling extraction and embeddings

There is a master switch and a per-collection opt-in. Both must be on for a file to be embedded.

First, flip the master switch in config/laracrate.php (it defaults to off, because most uploads do not need this):

'embeddings' => [
    'enabled' => true,
    // ...
],

Then opt in per collection with extract and embed:

'collections' => [
    'documents' => [
        'disk'         => 'documents',
        'access'       => 'signed',
        'extract' => true, // run the extractor chain, write {key}.json
        'embed'   => true, // chunk + embed + persist to the ChunkStore
    ],
],

extract alone gives you searchable plain text and the .json sidecar (readable via $file->extractedText(), see the end of this section) without any embedding API cost. Add embed to generate vectors and enable semantic search.

The extractor chain and fallback

Extraction runs through a TextExtractorRegistry: an ordered list of extractors. For a given file, the registry collects every extractor whose supports($file) returns true (chainFor($file)), then ExtractTextAction tries them in order. If an extractor returns fewer characters than embeddings.min_text_per_file (default 100), the action keeps the best result so far and tries the next one. The first extractor to clear the threshold wins. This is how a native PDF reader can fall back to OCR for a scanned PDF.

The bundled extractors:

All class names live under EduLazaro\Laracrate\Extractors\. The OCR, audio and video extractors call paid third-party APIs and incur per-file cost (the source documents rough estimates, for example a 10-page PDF OCR is around 0.004 USD on Claude Haiku, audio transcription is around 0.006 USD per minute). They also fail safe: if no API key is configured for the selected provider, supports() returns false and the chain moves to the next extractor instead of erroring out.

OcrImageTextExtractor and VideoTranscribeExtractor emit two segments with distinct context values, text (verbatim OCR or transcript) and description (a visual summary), so the chunker produces a separate embedding for each. Video visual frame description is opt-in: add the pseudo-type 'video.visual' to the collection's extract array to enable it (LARACRATE_VIDEO_FRAME_INTERVAL controls the seconds between frames, default 30).

Registering extractors

By default (when embeddings.extractors is empty) the registry loads only PdfTextExtractor then PlainTextExtractor. To control the order and which extractors run, set embeddings.extractors to a list of fully qualified class names. A recommended chain for legal or scanned documents:

'embeddings' => [
    'enabled'    => true,
    'extractors' => [
        \EduLazaro\Laracrate\Extractors\PdfTextExtractor::class,    // native PDFs, free
        \EduLazaro\Laracrate\Extractors\OcrPdfTextExtractor::class, // scanned PDFs, paid OCR
        \EduLazaro\Laracrate\Extractors\OcrImageTextExtractor::class,
        \EduLazaro\Laracrate\Extractors\PlainTextExtractor::class,
    ],
],

Order matters: put the fast, free extractor first and the paid OCR fallback second, so OCR only runs when the cheap path returns too little text.

To register a custom extractor at runtime instead, resolve the registry and add an instance (for example in a service provider's boot()):

use EduLazaro\Laracrate\Support\TextExtractorRegistry;

app(TextExtractorRegistry::class)
    ->add(new \EduLazaro\Laracrate\Extractors\PdfTextExtractor())
    ->add(new \App\Extractors\MyOcrExtractor());

A custom extractor implements EduLazaro\Laracrate\Contracts\TextExtractor with two methods: supports(File $file): bool and extract(File $file): ExtractedContent. Build the return value with ExtractedContent::singlePage($text, $metadata) or ExtractedContent::fromPages($pages, $metadata), where each page is ['page_number' => int, 'text' => string] (and optionally 'context' => string).

OCR configuration

The OCR extractors pick their provider from ocr.provider (anthropic by default, or openai):

LARACRATE_OCR_PROVIDER=anthropic
LARACRATE_OCR_MODEL=claude-haiku-4-5
LARACRATE_ANTHROPIC_API_KEY=sk-ant-...
# or for OpenAI:
# LARACRATE_OCR_PROVIDER=openai
# LARACRATE_OPENAI_API_KEY=sk-...

API key resolution order, per provider, is: the value passed to the extractor constructor, then config('laracrate.ocr.{provider}.api_key'), then LARACRATE_{PROVIDER}_API_KEY, then the generic ANTHROPIC_API_KEY / OPENAI_API_KEY. Model resolution is: constructor argument, then config('laracrate.ocr.{provider}.model'), then LARACRATE_OCR_MODEL, then the provider default (claude-haiku-4-5 for Anthropic, gpt-4o-mini for OpenAI). Audio transcription is OpenAI only (Whisper) and reads LARACRATE_OPENAI_API_KEY with model LARACRATE_AUDIO_MODEL (default whisper-1).

Chunking knobs

ChunkTextAction splits the extracted text into overlapping chunks. The token figures are approximate (the chunker counts roughly four characters per token):

If any extracted page carries a context, the chunker splits per page and propagates the context to each chunk; otherwise it concatenates all pages and tracks which source page each chunk spans.

Embedding providers

The embedding provider is resolved from the EduLazaro\Laracrate\Contracts\EmbeddingProvider binding. The contract is embed(array $texts): array (a vector per input string, same order), plus dimensions(), model() and name().

Both live under EduLazaro\Laracrate\Embeddings\. To use your own provider (a self-hosted model, Anthropic, BGE-M3, etc.), bind it in your service provider:

use EduLazaro\Laracrate\Contracts\EmbeddingProvider;

$this->app->bind(EmbeddingProvider::class, \App\Embeddings\MyProvider::class);

Note the query side: when you search semantically, the active ChunkStore embeds the query with this same provider, so the query vectors and the stored chunk vectors must come from the same model.

Chunk stores

A ChunkStore is the persistence and search backend. The contract (EduLazaro\Laracrate\Contracts\ChunkStore) has five methods:

public function store(File $file, array $chunks): int;
public function getByFile(File $file): \Illuminate\Support\Collection;
public function search(string $query, array $filters = [], array $options = []): \Illuminate\Support\Collection;
public function deleteByFile(File $file): void;
public function driverName(): string;

The driver is selected by chunks.driver (default mysql):

Both classes are under EduLazaro\Laracrate\Chunks\. With the mysql driver, chunks are rows in laracrate_file_chunks (the text column has a FULLTEXT index, though keyword matching currently uses SQL LIKE) and the embedding is stored alongside; search pulls a candidate pool and ranks it in PHP. With meilisearch, chunks are pushed as documents into the configured index with embeddings injected as _vectors.{embedder} (userProvided mode), and laracrate_file_chunks is not written: Meilisearch becomes the single source of chunks while the .chunks.jsonl sidecar stays the portable backup.

Wiring up Meilisearch

Add the env config:

LARACRATE_CHUNKS_DRIVER=meilisearch
LARACRATE_MEILISEARCH_INDEX=laracrate_file_chunks
LARACRATE_MEILISEARCH_EMBEDDER=default

Then bind a Meilisearch\Client in your AppServiceProvider (Laracrate resolves it from the container and falls back to MysqlChunkStore with a warning if it is not bound):

use Meilisearch\Client;

public function register(): void
{
    $this->app->singleton(Client::class, fn () => new Client(
        config('services.meilisearch.host', 'http://127.0.0.1:7700'),
        config('services.meilisearch.key'),
    ));
}

The index is created and configured on demand: MeilisearchSync::ensureIndex() sets the filterable, sortable and searchable attributes and registers the embedder as userProvided with your configured dimensions.

A custom store (Qdrant, pgvector, ...)

Implement the five-method contract and bind it directly. Your binding wins over the built-in driver switch:

use EduLazaro\Laracrate\Contracts\ChunkStore;

$this->app->singleton(ChunkStore::class, \App\Chunks\QdrantChunkStore::class);

Searching

Run a search through the active ChunkStore. The signature is search(string $query, array $filters = [], array $options = []) and it returns a Collection of result rows.

use EduLazaro\Laracrate\Contracts\ChunkStore;

$results = app(ChunkStore::class)->search('termination clause', [
    'fileable_type' => \App\Models\Matter::class,
    'fileable_id'   => $matter->id,
    'tenant_id'     => $org->id,
], [
    'limit'          => 10,
    'semantic_ratio' => 0.7,
]);

foreach ($results as $hit) {
    // $hit['file_id'], $hit['chunk_index'], $hit['text'],
    // $hit['score'], $hit['matched'] ('keyword' | 'semantic' | 'hybrid'),
    // $hit['metadata']
}

Supported filter keys: file_ids (array of ints), fileable_type, fileable_id, tenant_type, tenant_id, collection, context, category. Options are limit (default 10) and semantic_ratio (a float from 0 to 1, default 0.7).

semantic_ratio is the cost and quality dial:

• 0: keyword only. The query is not embedded, so there is no embedding API call. Use this for cheap, exact-match search or when embeddings are off.
• >0: the query is embedded with your EmbeddingProvider and blended with keyword results. 1.0 is pure semantic. Anything in between is hybrid.

To read the extracted plain text of a single file (no search backend involved), use the model accessor. extractedText() returns the full text from the .json sidecar, or null if extraction has not run:

$text = $file->extractedText();          // full concatenated text, or null
$content = $file->extractedContent();    // ExtractedContent DTO: fullText, pages[], metadata

extractedContent() returns an ExtractedContent object whose pages array preserves per-page (or per-segment) text, useful for citing a source page back to the user.