Late Chunking: Solving Context Loss in Laravel RAG Pipelines

Stop feeding your vector database garbage. Shredding a document into 500-token snippets blindfolds your retrieval system. Chunk 40 loses the context of the introduction, gutting the semantic meaning. You need late chunking vector search to keep the global narrative alive. This approach forces the model to digest the entire text first, generating hidden states for the whole document before pooling them into chunks. It is the difference between reading a book and scanning random sentences from a shredder.

Laravel 13 makes this straightforward via the new AI SDK and pgvector. By leveraging late chunking, you build a context-first ingestion pipeline that handles transformer hidden states without drifting into fragmented scripts. This moves beyond naive RAG into a system that respects the structural integrity of your information.

We will move away from the "split then embed" workflow toward an asynchronous orchestration using Jina AI’s late chunking models. You will set up a Laravel job that takes a full document, runs the embedding pass, and maps those contextualized vectors back to chunks in PostgreSQL. This eliminates the context loss that plagues most current implementations.

The Context Shredder: Why Traditional Chunking Fails

Traditional RAG pipelines chop text into arbitrary blocks, destroying the semantic thread. When you see the word "liability" in a shredded legal contract, the context of who is liable to whom is often lost.

The problem with naive splitting

Recursive character splitters with arbitrary token limits create orphans. If paragraph one defines a "Global Storage Interface" and paragraph ten contains configuration details, a naive embedding of paragraph ten won't "know" it refers to that interface. The vector lacks the semantic gravity needed for accurate pgvector similarity searches. You lose the relationships that make data meaningful.

How late chunking keeps the story straight

Late chunking flips the flow: feed the entire document—up to 8192 tokens—into the transformer. The model's self-attention mechanism ensures every token "sees" every other token before pooling happens.

// Traditional: Shred, then embed
$chunks = $splitter->split($bigDocument);
foreach ($chunks as $chunk) {
    $vector = $ai->embeddings()->create($chunk);
    $db->save($vector);
}

// Late Chunking: Embed the whole, then pool
$fullEmbeddings = $ai->embeddings()->createWithHiddenStates($bigDocument);
$semanticChunks = $fullEmbeddings->poolByBoundaries($manualOffsets);

In this "embed-then-pool" pattern, a sentence's vector represents those words within the context of the entire file. The model encodes global context into local representations, ensuring retrieval stays sharp even with vague language.

Prerequisites: Wiring the Laravel AI SDK

Ensure you are running Laravel 13 for tight SDK integration. Your PostgreSQL instance must have the pgvector extension active (CREATE EXTENSION vector;). Get a Jina AI API key to access their v2/v3 models built for late chunking. Add it to your .env:

JINA_API_KEY=jina_your_secret_key_here
AI_DEFAULT_PROVIDER=jina

Prepare your schema with the AI SDK scaffolding:

php artisan ai:setup

This generates migrations for the vectors table, bridging Eloquent models to the latent space.

Step 1: Configuring Jina AI for Late Chunking Vector Search

We must instruct the model to retain its hidden states across the entire document sequence.

Setting the late_chunking parameter

In config/ai.php, define a dedicated driver. The late_chunking flag tells the transformer to retain hidden states for the full sequence rather than clearing them after each fragment.

'drivers' => [
    'jina-late' => [
        'driver' => 'jina',
        'api_key' => env('JINA_API_KEY'),
        'model' => 'jina-embeddings-v3',
        'options' => [
            'late_chunking' => true,
            'task' => 'retrieval.passage',
        ],
    ],
],

Defining the embedding dimension

Jina v3 models output vectors with 1024 dimensions. Your pgvector columns must match this exactly to avoid ingestion errors. Update your migration:

$table->vector('embedding', 1024);

Test the connection to ensure the Jina bridge is active:

php artisan ai:test-connection jina-late

Step 2: Defining Semantic Boundaries in PHP

To maintain context, identify natural boundaries while keeping the document string whole. You must record exact character offsets for these segments within the full document.

Avoiding character-count traps

Instead of str_split(), identify natural pauses (paragraphs/sentences). Record the start and end positions. These offsets allow Jina AI to pool embeddings correctly after processing the full text.

Using the Spatie Markdown-to-Sentences package

Use spatie/markdown-to-sentences to find logical breaks without complex regex. Create a BoundaryMap to track these offsets.

namespace App\\\Services\\\Search;

use Spatie\\\MarkdownToSentences\\\SentenceSegmenter;

class BoundaryMap
{
    public function __construct(
        public string $fullText,
        public array $offsets = []
    ) {}

    public static function fromText(string $text): self
    {
        $sentences = (new SentenceSegmenter())->segment($text);
        $offsets = [];
        $currentPos = 0;

        foreach ($sentences as $sentence) {
            $start = strpos($text, $sentence, $currentPos);
            if ($start === false) continue;
            
            $end = $start + strlen($sentence);
            $offsets[] = [$start, $end];
            $currentPos = $end;
        }

        return new self($text, $offsets);
    }
}

Passing the full text allows the model to calculate attention weights globally before generating final vectors for specific chunks.

Step 3: Executing the Context-Aware Ingestion Job

Embedding calls should occur in background jobs to prevent timeouts. The ProcessLateChunking job orchestrates the flow from raw text to PostgreSQL.

Handling the API response payload

Send the full document string. The Jina AI v3 model processes the sequence, returning vectors that carry the semantic weight of the surrounding text.

public function handle(): void
{
    $results = AI::embeddings()
        ->model('jina-embeddings-v3')
        ->input($this->document->content)
        ->withConfig(['task' => 'text-matching', 'late_chunking' => true])
        ->get();

    $this->storeChunks($results);
}

Mapping vectors to chunk segments

Iterate through the response to create document_chunks records. Use the vector cast in your Eloquent model for proper formatting.

protected function storeChunks(Collection $results): void
{
    foreach ($results as $item) {
        $this->document->chunks()->create([
            'content' => $item->text,
            'embedding' => $item->embedding,
            'metadata' => [
                'char_count' => strlen($item->text),
                'model' => 'jina-v3',
            ],
        ]);
    }
}

Step 4: Scaling with Kafka and Event-Driven Ingestion

For high-volume ingestion, decouple file processing from embedding calls using Kafka. This prevents API rate-limit issues and manages the firehose of data.

// In your DocumentService.php
public function dispatchToPipeline(Document $document)
{
    $producer = app('kafka.producer');
    $producer->send('document-ingestion', json_encode([
        'document_id' => $document->id,
        'path' => $document->file_path,
    ]));
}

A long-running consumer dispatches Bus::batch(). Use a circuit breaker for API calls to handle 429 errors gracefully. Monitor the job_batches table to manage worker nodes.

Step 5: Testing Retrieval Precision with pgvector

Verify accuracy by comparing late-chunked embeddings against naive recursive splitting.

A/B testing retrieval accuracy

Run a query in Tinker that relies on implicit context or pronouns.

$query = "How does the system handle backpressure during high-load spikes?";

// Late Chunking
$lateResults = LateChunkedDocument::query()
    ->nearestNeighbor('embedding', AI::vectorize($query))
    ->take(3)
    ->get()
    ->map(fn($doc) => $doc->content);

// Naive approach
$naiveResults = NaiveDocument::query()
    ->nearestNeighbor('embedding', AI::vectorize($query))
    ->take(3)
    ->get()
    ->map(fn($doc) => $doc->content);

dump(['late' => $lateResults, 'naive' => $naiveResults]);

Visualizing the context retention

Late chunked vectors retrieve relevant segments even if they lack specific keywords, as they carry the document's global topic. A "throttling" sentence will match a "mitigation" query because it was encoded alongside the "mitigation" heading.

Troubleshooting Contextual Drift

Log token counts before dispatching. Jina v3 limits are 8,192 tokens; exceeding this leads to truncation. Ensure boundary offsets use mb_strlen() for multi-byte character accuracy.

// Verify indices
$boundaries = $this->getSentenceBoundaries($text);
Log::info('Boundary check', [
    'text_length' => mb_strlen($text),
    'last_boundary' => end($boundaries)
]);

Monitor RAM usage in php.ini. Handling 1024-dimension vectors in bulk can spike memory. Use dedicated queues with higher timeouts for remote embedding calls.

Beyond the Document Shredder

Late chunking ensures document segments retain their executive summary context. Laravel 13 streamlines the orchestration between the AI SDK and pgvector.

Note that high-dimensional embeddings (1024D) are resource-intensive. During work on the MAHI healthcare platform, building HNSW indices on Vespa.ai triggered OOM crashes. Tuning max-links-per-node and limiting indexing concurrency was required to stabilize memory usage.

Context-aware ingestion is critical for production-grade RAG. Monitor pgvector query precision to ensure your LLM has the context it needs to avoid hallucinations.

Sources & Further Reading

• jina.ai — https://jina.ai/news/late-chunking-in-rerankers-and-embedding-models/
• laravel.com — https://laravel.com/docs/13.x/ai
• github.com — https://github.com/pgvector/pgvector
• arxiv.org — https://arxiv.org/abs/2409.04701
• huggingface.co — https://huggingface.co/jinaai/jina-embeddings-v3

Frequently Asked Questions

What is late chunking vector search and how does it differ from traditional chunking?

Late chunking vector search processes the entire document through a transformer model before splitting it into chunks. Traditional chunking shreds documents first, losing the semantic relationship between distant sections. By generating hidden states for the full text sequence, late chunking ensures each chunk's vector includes context from the whole document, leading to significantly higher retrieval accuracy in RAG systems by preventing context loss and maintaining global narrative integrity.

How do you implement late chunking in Laravel 13?

Implementation involves configuring the Laravel 13 AI SDK with a driver that supports hidden states, such as Jina AI’s v3 model. You must enable the late_chunking flag in your configuration, define precise character offsets for your semantic boundaries, and store the resulting 1024-dimension vectors in a PostgreSQL table with the pgvector extension. This setup allows the model to process global attention before mapping contextualized embeddings back to specific document segments.