The Local Embedding Edge: Replacing OpenAI with Self-Hosted SLMs in Laravel RAG Pipelines

Stop sending your proprietary data to third-party APIs for every query. If your monthly embedding bill is ballooning, it is time to deploy a self-hosted embedding model RAG. While OpenAI’s `text-embedding-3-small` is convenient, it tethers your privacy and latency to external infrastructure. Running an Ollama instance—`ollama pull mxbai-embed-large`—demonstrates that modern Small Language Models (SLMs) are production-ready engines that keep vectors inside your VPC.

Transitioning requires managing a self-hosted architecture without overloading PHP workers. You must handle vector dimension shifts—moving from OpenAI's 1536-dimensional vectors to 1024 or 768 dimensions in pgvector. By using a Python sidecar pattern, you can bypass HTTP overhead and use native concurrency to feed your vector store while Laravel orchestrates the batches. This guide builds a faster, cheaper, and entirely private pipeline using the Laravel 13 AI SDK.

The Prerequisites: Setting the Local Stage

Shifting vector processing locally requires hardware capable of concurrent requests and a specifically configured environment.

Infrastructure Requirements

Use a machine with a dedicated NVIDIA GPU (8GB+ VRAM) or Apple Silicon. Install Ollama as your local inference server. Pull the model for local embeddings for laravel:

ollama pull nomic-embed-text

Verify the service is active at localhost:11434. A "Ollama is running" message confirms the engine is ready.

The Laravel Foundation

Your Laravel 13 application needs PostgreSQL with the pgvector extension. Ensure your database user can execute CREATE EXTENSION IF NOT EXISTS vector;.

Adjust your schema for dimensionality. nomic-embed-text uses 768 dimensions, while OpenAI defaults to 1536. Update your schema to prevent PostgreSQL rejection:

Schema::table('documents', function (Blueprint $table) {    $table->dropColumn('embedding');    $table->vector('embedding', 768)->nullable();});

Step 1: Spinning Up the Local Inference Engine

Fire up your ollama embedding server to move away from cloud-based costs. If you have Ollama installed, this setup takes moments.

Pulling the Weights

You need a lean model with high performance. Run:

ollama pull nomic-embed-text

This model supports an 8192 context window and outperforms older standards. Once resident in VRAM, expect response times under 20ms for single sentences, outperforming public network latency.

Testing the Socket

Verify the endpoint manually with a curl request to port 11434:

curl http://localhost:11434/api/embeddings -d '{  "model": "nomic-embed-text",  "prompt": "Laravel and pgvector are a perfect match."}'

A return of 768 dimensions confirms you are ready. This local loopback is the foundation of your high-speed pipeline.

Step 2: Configuring the Laravel 13 AI SDK for Local Models

Hook the inference engine into your application logic by defining a custom provider in Laravel 13.

Extending the Provider List

In config/ai.php, define a provider that treats Ollama as a first-class citizen. Ollama mimics the OpenAI API structure, but explicit naming prevents confusion.

'providers' => [    'ollama' => [        'base_url' => env('AI_BASE_URL', 'http://localhost:11434/v1'),        'api_key' => env('AI_API_KEY', 'ollama'),        'type' => 'openai',    ],],

If the default driver struggles with specific response formats, create an OllamaEmbeddingProvider class and register it in AppServiceProvider using AI::extend().

Environment Variables

Update .env to point to your local hardware. Use a placeholder API key to satisfy the HTTP client.

AI_PROVIDER=ollamaAI_BASE_URL=http://localhost:11434/v1AI_EMBEDDING_MODEL=nomic-embed-textAI_API_KEY=local-dev-no-key-needed

For Docker (Sail) users, use host.docker.internal to reach the host service. Test connectivity via Tinker with AI::withModel('nomic-embed-text')->embed('Testing').

Step 3: Managing the Dimension Shift in pgvector

Transitioning from OpenAI (1536) to nomic-embed-text (768) requires schema surgery. To avoid search blackouts during re-indexing, use a temporary staging column.

Generate a migration to add the local-compatible column:

Schema::table('documents', function (Blueprint $table) {    $table->vector('embedding_v2', 768)->nullable();});

Update the Document model to map both columns using the casts() method:

protected function casts(): array{    return [        'embedding' => 'vector:1536', // Legacy        'embedding_v2' => 'vector:768', // New    ];}

This allows your app to read old embeddings while background jobs populate new ones. Point RAG queries to embedding_v2 only after the batch job reaches 100%.

Step 4: Implementing the Python Sidecar for Bulk Ingestion

Processing massive datasets via standard HTTP loops to Ollama is slow. The "sidecar" pattern allows you to hit hardware directly using the sentence-transformers library.

Bypassing HTTP Overhead

A Python service running all-MiniLM-L6-v2 is ideal for speed in CPU environments, producing punchy 384-dimension vectors. This creates a tight sentence-transformers php integration.

FastAPI as the Bridge

Run a worker to act as a high-throughput embedding factory:

from fastapi import FastAPI, Bodyfrom sentence_transformers import SentenceTransformerapp = FastAPI()model = SentenceTransformer('all-MiniLM-L6-v2')@app.post("/embed")async def embed(texts: list[str] = Body(...)):    embeddings = model.encode(texts).tolist()    return {"vectors": embeddings}

Configure the Laravel HTTP client for persistent connections to reduce overhead during bulk jobs:

use Illuminate\Support\Facades\Http;$response = Http::withOptions([    'curl' => [        CURLOPT_TCP_KEEPALIVE => 1,        CURLOPT_FORBID_REUSE => false,    ],])->post('http://python-sidecar:8000/embed', [    'texts' => $chunkOfParagraphs,]);$vectors = $response->json('vectors');

Step 5: Orchestrating the Re-Indexing Job with Laravel Batches

Use Bus::batch to manage the heavy lifting of re-indexing your entire library without crashing VRAM.

Managing Throughput

Balance throughput by matching worker count to hardware capacity. Consumer GPUs typically handle 4 to 8 concurrent workers well.

$batch = Bus::batch([])    ->name('Local Re-Indexing')    ->finally(function (Batch $batch) {        Log::info('Vector migration complete.');    })->dispatch();Document::chunk(500, function ($documents) use ($batch) {    $batch->add(new ReindexVectorJob($documents));});

Implement a retry strategy to handle model "cold starts" during the re-indexing process:

public $tries = 3;public $backoff = [10, 30, 60];public function handle(){    $response = Http::timeout(120)->post('http://localhost:8000/embed', [        'text' => $this->documents->pluck('content'),    ]);    if ($response->failed()) {        throw new \Exception('Local model busy.');    }}

Step 6: Evaluating Retrieval Quality vs. OpenAI

Verify accuracy before decommissioning OpenAI. Use a Cosine Similarity sanity check to compare the 1536-dimension and 768-dimension vectors for relative consistency.

Perform head-to-head testing with "Golden Queries":

$queries = ['How do I reset my API key?', 'What is our refund policy?'];foreach ($queries as $query) {    $results = Document::query()        ->nearestNeighbors('embedding', AI::vector($query), 5)        ->get();    dump($results->pluck('id')->toArray());}

If irrelevant chunks appear, increase chunk overlap. Smaller models can be distracted by short fragments. If precision is lacking, consider `mxbai-embed-large` over `nomic-embed-text`.

Troubleshooting & Common Gotchas

Self-hosting requires active performance maintenance. Manage your inference stack with these strategies:

Stop OOM Errors

Prevent Out of Memory crashes by capping concurrent usage in Ollama:

OLLAMA_NUM_PARALLEL=2OLLAMA_MAX_LOADED_MODELS=1

Eliminate First-Hit Latency

Prevent model unloading by passing the keep_alive parameter:

$response = Http::post('http://localhost:11434/api/embeddings', [    'model' => 'nomic-embed-text',    'prompt' => $userInput,    'keep_alive' => -1,]);

Distance Metrics

Ensure your pgvector index matches the model’s intended math (Cosine vs L2). For Cosine similarity:

$table->rawIndex('embedding', 'embeddings_cosine_idx', 'vector_cosine_ops');

Owning the Stack Without Breaking the Bank

Localizing embeddings ensures data sovereignty and eliminates API costs. Laravel 13 simplifies the implementation, but you must respect hardware constraints. High-density indexes like HNSW in Vespa.ai or pgvector require significant RAM during graph construction; tune your max-links-per-node and throttle concurrency to maintain stability.

Self-hosted SLMs provide a privacy-first solution. By bridging Laravel with a Python sidecar and managing vector dimensions, you build a system resilient to external price hikes and rate limits. A RAG system operating entirely within your perimeter is a significant milestone for engineering teams in 2026.

Sources & Further Reading

• ollama.com — Nomic Embed Library
• laravel.com — Laravel AI Documentation
• github.com — pgvector Repository
• www.sbert.net — SBERT Models
• fastapi.tiangolo.com — FastAPI Documentation

Frequently Asked Questions

Why should I switch from OpenAI to a self-hosted embedding model for RAG?

Switching to a self-hosted embedding model for RAG eliminates recurring API costs and significantly enhances data privacy by keeping proprietary information within your private cloud. Local models like nomic-embed-text provide lower latency by removing network round-trips to external providers. Additionally, self-hosting prevents your pipeline from being affected by third-party rate limits or service outages, ensuring total control over your inference infrastructure.

How do I handle the change in vector dimensions when switching models?

Different models use different vector lengths; for example, OpenAI uses 1536 dimensions while many local SLMs use 768 or 1024. To manage this in pgvector, you should create a new column in your database via a migration to accommodate the new dimensionality. Use a background batch job in Laravel to re-index your documents into the new column while keeping the old one active to ensure zero downtime during the transition.

What is the purpose of the Python sidecar pattern in this pipeline?

The Python sidecar pattern uses a lightweight FastAPI service to handle high-throughput embedding generation directly on your hardware. This bypasses the overhead of traditional PHP HTTP loops and allows you to utilize specialized libraries like sentence-transformers. It enables better concurrency and direct access to GPU resources, which is essential for processing large document batches during re-indexing without overloading your primary Laravel application workers.