The Semantic Cache: Reducing LLM Latency and Costs with pgvector

LLM costs are often higher than necessary. While developers focus on inference speed, users frequently ask the same questions in varied ways. Standard key-value stores like Redis fail here because they require literal string matches—a single synonym or misplaced comma breaks the cache. Semantic Caching leverages vector similarity to serve high-quality responses from your own infrastructure, bypassing external APIs for repetitive queries.

Most stacks already include PostgreSQL, making pgvector a natural choice. However, storing every prompt-response pair introduces an "Embedding Tax." Generating a 1536-dimensional vector for every query incurs latency and cost. If these exceed the savings of the LLM call, the optimization becomes a bottleneck. Efficiency depends on your specific model selection and data distribution.

Furthermore, a semantic cache is only as useful as its underlying knowledge base. If RAG documents update, cached answers become liabilities. Managing "Semantic Invalidation" ensures entries expire when source data shifts. Let’s explore how to build a cache that understands intent and knows when to stop serving stale answers.

The Fuzzy Logic of Semantic Caching for LLMs

Traditional caching is binary, relying on exact hashes (MD5/SHA-256) of request bodies. This works for structured data but fails natural language workflows where different phrasing should yield the same cached result. Semantic Caching for LLMs uses "close enough" matching based on vector proximity rather than string identity.

Why String Hashes Fail AI Workflows

In a support bot, "How do I reset my password?" and "Password reset instructions, please" are identical in intent. A traditional cache treats these as two distinct misses, costing you two separate LLM completions. Using pgvector, we index the intent. We store the vector embedding of the prompt and search for "neighbors" within a specific distance threshold (e.g., < 0.1 for cosine similarity). This transforms the cache into a fluid map of meanings.

-- The basic structure for a semantic cache in PostgreSQL
CREATE TABLE llm_cache (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    prompt TEXT NOT NULL,
    prompt_embedding vector(1536), -- For OpenAI text-embedding-3-small
    response TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Querying the cache for a "close enough" match
SELECT response 
FROM llm_cache 
WHERE prompt_embedding <=> $1 < 0.12 
ORDER BY prompt_embedding <=> $1 ASC 
LIMIT 1;

The Latency Budget: Is Your Cache Too Heavy?

Every cache check requires generating an embedding for the incoming prompt. An external provider like OpenAI adds 50ms–150ms of network latency; local models like all-MiniLM-L6-v2 consume CPU cycles. If GPT-4o takes 2 seconds to stream a response, a 100ms embedding step is a massive win. However, if using a fast local model or Groq-hosted Llama-3 that returns in 300ms, that 150ms embedding step eats half your latency budget.

Semantic caching thrives in high-value, repetitive workflows like customer support or documentation search. If your users are creative writers or developers where every prompt is unique, the low hit rate and embedding tax may actually increase average latency. Analyze your data distribution before implementation.

Architecture: Implementing pgvector Similarity Search as a Cache Layer

With pgvector, PostgreSQL performs proximity searches in high-dimensional space. We aren't just looking for strings; we are measuring the angle between vectors to determine similarity.

Schema Design for Prompt Persistence

A specialized table should store raw prompts, full JSON response payloads (to preserve token counts and metadata), and vectors. Normalize prompts—stripping whitespace and lowercasing—before embedding to maintain cache cleanliness.

Schema::create('semantic_cache', function (Blueprint $table) {
    $table->id();
    $table->text('prompt_norm'); // Normalized version of the user query
    $table->jsonb('response_payload'); // The full LLM response
    $table->vector('embedding', 1536); // Match dimensions to your model
    $table->float('hit_count')->default(0);
    $table->timestamp('last_accessed_at')->nullable();
    $table->timestamps();

    // HNSW index for speed as the cache grows
    $table->indexRaw('embedding vector_cosine_ops', 'semantic_cache_embedding_idx');
});

The Laravel 13 AI SDK Integration Pattern

Intercept requests at the SDK or middleware level. When a request arrives: 1. Generate the prompt embedding. 2. Query PostgreSQL for the nearest neighbor. 3. If a match exists within the threshold, return the cached payload and increment hit_count. 4. If no match, proceed to the LLM, then store the new result and vector.

Threshold Tuning: Precision vs. Recall

We use Cosine Distance (<=>) because it measures the direction of meaning rather than magnitude. A distance of 0 is identical; 2 is opposite. Setting the similarity floor is critical: too loose (e.g., 0.80) and you risk serving a "delete account" cache for an "update account" query. A floor of 0.92 to 0.95 is generally the sweet spot for RAG applications.

$match = DB::table('semantic_cache')
    ->select('response_payload')
    ->where('embedding', '<=>', $queryEmbedding)
    ->whereRaw('(1 - (embedding <=> ?)) > 0.92', [$queryEmbedding])
    ->orderBy('embedding <=> ?', [$queryEmbedding])
    ->first();

The calculation is 1 - distance = similarity. If no match exceeds the 0.92 mark, treat it as a cache miss to avoid hallucinations.

Scaling the Cache: Performance and Invalidation

As the dataset reaches hundreds of thousands of rows, sequential scans fail. We must optimize for lookups under 10ms.

Avoiding the Full Table Scan with HNSW

HNSW (Hierarchical Navigable Small Worlds) builds a multi-layered graph that remains fast even as data distributions shift. Tune m (max connections) and ef_construction for balance. Higher m values improve recall at the cost of RAM.

CREATE INDEX ON semantic_cache 
USING hnsw (embedding vector_cosine_ops) 
WITH (m = 16, ef_construction = 64);

Monitor your buffer cache hit ratio; HNSW indexes perform best when kept in memory. Session-level ef_search tuning can further improve accuracy if hit rates drop.

The Semantic Invalidation Problem

If your underlying RAG data updates, your cache becomes a liability. Attach source_tags or version_ids to cache entries. When your vector database updates, use metadata filters to ignore entries older than the last sync.

SELECT response_text 
FROM semantic_cache 
WHERE embedding <=> :query_vector < 0.1 
AND created_at > :last_kb_update 
AND tags @> ARRAY['policy_docs'] 
LIMIT 1;

Cost Reduction Metrics

The break-even point for semantic caching usually occurs at a 5-10% hit rate. With high-end models costing ~$30/1M tokens, every hit for a 2,000-token prompt saves ~$0.06. Embedding costs (e.g., $0.00002) are negligible in comparison.

PostgreSQL vs. Redis: When to Move Beyond pgvector?

PostgreSQL is often already your source of truth. Using pgvector keeps cache hits close to user permissions and tenant constraints. Only migrate to dedicated vector stores like Redis or Upstash if lookups consistently exceed a 50ms p99 threshold or you require sub-5ms responses for global scale.

Postgres allows invalidating cache entries within the same transaction that updates a RAG document, preventing consistency issues. Use EXPLAIN ANALYZE to monitor index usage.

-- Checking if our index is actually being used
EXPLAIN ANALYZE
SELECT content, (embedding <=> '[0.12, 0.34, ...]') AS distance
FROM semantic_cache
WHERE embedding <=> '[0.12, 0.34, ...]' < 0.1
ORDER BY distance LIMIT 1;

Stop Guessing and Start Indexing

Semantic caching with pgvector reclaims the milliseconds required for a snappy user experience while reducing API bills. The primary challenges are managing the embedding tax and ensuring invalidation logic prevents stale data delivery.

Large-scale HNSW indexes consume significant RAM. For example, in the MAHI healthcare platform, we had to throttle concurrency and tune max-links-per-node and neighbors-to-search-at-insert to prevent OOM crashes. Whether using specialized tools or pgvector, these parameters are critical as datasets grow.

If running Postgres, a semantic layer is the logical evolution for your RAG pipeline. It bridges the gap between keyword lookups and fuzzy AI intent. Keep embeddings small, thresholds tight, and synchronize your invalidation with your source truth.

Sources & Further Reading

• github.com/pgvector/pgvector
• laravel.com AI Documentation
• openai.com Pricing
• supabase.com Vector Guide

Frequently Asked Questions

What is the main advantage of semantic caching over traditional hashing?

Traditional caching uses binary hashes like MD5, which require an exact character match to return a result. In AI workflows, users often ask the same question with different phrasing. Semantic caching uses vector proximity via pgvector to identify queries with the same intent. This allows the system to serve a cached response even if the wording differs, significantly increasing cache hit rates and reducing redundant LLM API calls.

What is the Embedding Tax in semantic caching?

The embedding tax refers to the latency and monetary cost incurred when generating a vector for every incoming query. Before checking the cache, the system must transform text into a 1536-dimensional vector. If the time and cost to generate this embedding approach the speed of the LLM itself, the cache becomes a bottleneck. It is most effective when the primary LLM is slower or more expensive than the embedding model.

How do I choose the right similarity threshold for pgvector?

Selection depends on the specific use case and model. For RAG applications, a cosine similarity threshold between 0.92 and 0.95 is typically the sweet spot. Setting it too low risks serving incorrect or irrelevant answers, while setting it too high creates a strict requirement that might miss valid semantic matches. Always calculate similarity as 1 minus the distance returned by pgvector's cosine distance operator.

How does HNSW indexing help scale a semantic cache?

As a cache grows to thousands of entries, sequential scans for nearest neighbors become too slow for production. HNSW (Hierarchical Navigable Small Worlds) creates a graph-based index that allows for high-speed similarity searches even with high-dimensional data. By tuning parameters like m (max connections) and ef_construction, you can balance the trade-off between RAM usage, search speed, and recall accuracy to keep cache lookups under 10ms.