Let’s be honest: most of us have pushed a RAG agent to production because it "felt right" after five manual prompts. We call it a vibe check, but really, it’s just a prayer that the LLM won’t start inventing facts about your company's refund policy. If you’re building AI-native apps with Laravel, relying on vibes is a fast track to a PR disaster. You need a robust RAG Evaluation Framework PHP engineers can actually trust—one that doesn't force you to jump into a Python notebook every time you need to verify your retrieval quality.
The common wisdom says evaluation belongs in a Python environment, buried under layers of messy boilerplate. That’s a myth that slows down shipping. With the native AI capabilities in Laravel 13, we can build the entire "LLM-as-a-judge" loop directly in our application stack. Why manage a sidecar when your PHP workers can handle the faithfulness checks and relevance scoring themselves? It’s about keeping your logic where your data lives.
I'm going to walk you through a blueprint for automating hallucination detection using Pest and a few clever agentic patterns. We’ll look at how to build a 'Golden Set' of ground-truth data without the manual drudgery (yes, we’ll use the LLM to help). By the end, your CI/CD pipeline will be the gatekeeper that catches nonsense before it hits a user’s screen. It's time to stop guessing and start measuring.
Moving Beyond the 'Vibe Check'
Building a RAG-enabled agent in Laravel often feels like magic—until it starts lying to your users with unearned confidence. In the early stages of development, we usually rely on what I call "vibe checks." You fire off a few queries to your controller, skim the response, and if it looks coherent, you ship it. But as your data grows from a handful of Markdown files to a massive collection of documentation in pgvector, this manual oversight crumbles.
Why manual testing fails at scale
Manual inspection is the ultimate bottleneck for shipping production-grade AI. If you're manually verifying every response, you aren't building a system; you’re babysitting a script. When we tweak our embedding model or adjust the top_k results in our search logic, we need to know immediately if we broke something. We need a quantitative way to measure if our retrieval logic—the backbone of our RAG—is actually feeding the model the right data or just noise. High-traffic systems can't afford the luxury of "guessing" that the vector search is still healthy.
The cost of a confident hallucination
Hallucinations in a Laravel agent aren't always a model failure. Most of the time, it's a context failure. If your retrieval pipeline pulls an irrelevant document snippet, the LLM will still try to be helpful (and wrong). It’s the difference between a failed search and a successful lie. We have to distinguish between the two by testing the faithfulness of the response against the specific retrieved context. Without hard numbers, you're just throwing tokens at a wall and hoping they stick.
// Tracking retrieval health before the LLM generates a response
$context = Document::query()
->nearestTo('vector', $queryVector)
->limit(3)
->get();
// If the similarity distance is too high, your RAG is hallucinating-by-proxy
$avgScore = $context->avg('distance');
if ($avgScore > 0.8) {
Log::warning("Retrieval quality dropped below threshold for query: {$userInput}");
// Trigger a fallback or a more conservative prompt
}
Designing a RAG Evaluation Framework in PHP
So, how do we stop guessing and start measuring? It begins by embedding a RAG Evaluation Framework PHP developers can actually maintain without spinning up a sidecar FastAPI service. We need a system that treats our LLM outputs like any other piece of business logic—subject to rigorous, automated scrutiny before a single token reaches the client.
The triad of metrics: Faithfulness, Relevance, and Groundedness
To quantify "good," we borrow three specific pillars from the research community, adapted for our native stack. First is Faithfulness. This measures if the answer is derived strictly from the retrieved chunks. If your model claims a refund policy is 30 days but the context says 14, the model is "hallucinating" (lying), and your framework should flag it immediately.
Next, we look at Answer Relevance. This ensures the agent didn't just summarize a bunch of technical documents while ignoring the user's actual question. It is the "did you listen to me?" check. Finally, Context Precision audits your retrieval health. If your pgvector query returns ten chunks but only the last one was useful, your search rankings are noisy. High precision means your top-k results are actually doing the heavy lifting.
Defining the evaluation pipeline
In a Laravel environment, this pipeline sits directly in your service layer or within a dedicated test suite. Instead of manual spot-checks, we use an "LLM-as-a-judge" to score the interaction based on the retrieved context, the original prompt, and the final response.
// A simplified conceptual check in Laravel 13
public function evaluate(RagResponse $response): EvaluationResult
{
return AI::withModel('gpt-4o')
->asAgent()
->prompt("Compare this Answer against the Context.
Is every claim supported?
Context: {$response->context}
Answer: {$response->text}")
->execute();
}
Think of this layer as a skeptical editor. It doesn’t care how smooth the prose sounds; it only cares if the librarian (your vector search) handed over the right books and if the writer (the LLM) actually read them. By codifying these metrics, you turn "it feels right" into a deployable metric.
The LLM-as-a-Judge Pattern in Laravel
So, how do we actually codify this "judge" without turning our codebase into a spaghetti of nested API calls? We treat the evaluation as a separate, higher-order concern. By leveraging the Laravel 13 AI SDK, we can define a dedicated agent whose only job is to be a pedantic auditor. This isn't a conversational bot; it’s a cold, calculating validator that checks if your worker agent is making things up or sticking to the facts.
Configuring a 'Critic' agent
In Laravel, we can define our evaluator as a specialized agent class. The secret sauce here is the system prompt. You want to strip away the fluff and force the model into a corner where it can only provide objective analysis. By using the withOutputSchema method or a strictly defined JSON response, we ensure the results can be piped directly into a database or a Pest test suite without messy regex parsing.
use Laravel\AI\Agent;
class RagCritic extends Agent
{
public function systemPrompt(): string
{
return "You are a strict factual auditor. Compare the provided Context against the Answer.
Identify any claims in the Answer not supported by the Context.
Output only JSON with keys: 'faithfulness_score' (0.0-1.0), 'hallucinations' (array), and 'verdict'.";
}
}
Separating the Judge from the Worker
It’s tempting to use the same cheap, fast model for both generation and evaluation. Don't do that. You wouldn't ask a junior developer to review their own high-stakes PR, would you? Your worker agent might use a lighter model like GPT-4o-mini or a local Llama 3 instance to keep latency low. However, your "Critic" needs more horsepower.
Using a "reasoning-heavy" model like GPT-4o or Claude 3.5 Sonnet for the judge provides the nuance needed to catch subtle semantic drifts. Since evaluation usually happens asynchronously—perhaps triggered by a dispatched job after the user gets their response—the slightly higher cost and latency of these "smart" models won't hurt the user experience. You're buying reliability, and in a RAG stack, that's the only currency that matters.
Implementing Faithfulness Tests with Pest
So, we’ve got our high-reasoning "judge" model standing by. Now, we need a repeatable way to haul it into our development workflow. Since we’re staying native to the Laravel stack, Pest is our natural habitat for these evaluations; it turns abstract "vibes" into hard, passing green dots in your terminal.
Creating the evaluation test suite
Think of your evaluation suite as a picky editor who refuses to sign off on a story unless every fact has a primary source. We use Pest’s dataset() to loop through our Golden Set—those hand-verified pairs of questions and answers we know are correct. The goal isn't just checking if the output "looks right," but specifically verifying that the agent didn't hallucinate a fact out of thin air. We are looking for a paper trail between the retrieved context and the final answer.
describe('RAG Faithfulness', function () {
dataset('golden_set', [
['question' => 'How do I reset my API key?', 'context' => 'Users can reset keys in the security tab.'],
]);
test('agent response is grounded in retrieved context', function ($question, $context) {
$response = Agent::worker()->ask($question);
$evaluation = Agent::critic()
->withContext($context)
->checkFaithfulness($response);
expect($evaluation->score)->toBeGreaterThan(0.8);
})->with('golden_set');
});
Mocking the retriever for isolated testing
To really pin down where a failure happens, we need to isolate the reasoning from the retrieval. By using the Laravel AI SDK’s tool-calling, we can intercept the retrieval tool and swap the live vector search for a static string from our test data. This ensures we’re testing the "brain" (the LLM) rather than the "memory" (the database).
Keep in mind that even the best judges have bad days. LLMs are non-deterministic, so a single "pass/fail" check can be noisy. To fix this, we often implement a "majority vote" logic within our test. We run the evaluator three times for every test case. If two out of three passes say it’s faithful, we mark it a win. It’s like getting a second and third opinion before you go into surgery; it smooths out the statistical noise and gives you a result you can actually trust before pushing to production.
Building Your 'Golden Set' Without the Drudgery
Setting up a robust evaluation layer requires a "ground truth"—a set of verified answers that act as your North Star. If you try to write these by hand, you’ll burn out before the first deployment. We need a way to automate the creation of this dataset without sacrificing the quality of our checks.
Synthetic data generation in Laravel
We can use the Laravel 13 AI SDK to bootstrap our "Golden Set." Instead of manual entry, take your documentation or the raw chunks sitting in your Vespa.ai index and feed them to a high-reasoning model. Ask it to generate question-context-answer triplets. It’s like creating a study guide for your agent. You want about 50 to 100 of these to start with to get a clear picture of performance.
// Generating triplets from a document chunk
$triplet = AI::chat()
->withContext($chunk->content)
->prompt("Create a complex question and a factual answer based ONLY on the provided context.")
->asJson()
->send();
EvaluationSet::create([
'question' => $triplet['question'],
'context' => $chunk->content,
'expected_answer' => $triplet['answer'],
'version' => 'v1.0.0',
]);
Don't just hide these in a random JSON file. Store them in a dedicated table managed by migrations. This ensures your evaluation data is versioned right alongside your code, making it easy for any developer on the team to run a local health check without hunting for a shared spreadsheet.
Curating real-world failures
The most valuable part of your dataset isn't the easy wins; it’s the "I don't know" scenarios. To prevent the agent from making things up, include negative examples where the question cannot be answered by the provided context. If your agent tries to be a hero and fabricates an answer when the context is blank, your framework should flag it immediately. It’s the digital equivalent of teaching a junior developer that saying "I'm not sure" is better than guessing the database password; it builds a foundation of restraint that stops hallucinations before they reach the user.
Measuring Semantic Similarity and Retrieval Health
So, your agent says "The sky is azure" while your ground truth says "The sky is blue." A standard PHP assertSame() or str_contains() will fail every time, right? We need a smarter way to measure correctness when language is fluid. If we only rely on keyword matching, we’re essentially punishing the LLM for having a vocabulary.
Beyond exact matches
Semantic similarity testing bridges the gap between binary pass/fail results and the messy reality of natural language. Instead of checking for literal characters, we convert both the agent’s output and our "golden" reference into vectors. If they point in the same direction in high-dimensional space, your agent is likely on the right track. This allows our test suite to handle synonyms and stylistic variations without breaking the build every time a model chooses a different adjective.
Vector-based score validation
Since we’re already using Laravel 13 and pgvector for our retrieval layer, we can repurpose that same infrastructure for evaluation. We calculate the cosine distance (the angle between two vectors) to get a numeric "health" score for every response. A score closer to 1.0 means the agent effectively mirrored the ground truth.
// Comparing the agent response against our expected ground truth
$agentVector = AI::embeddings()->embedText($agentResponse);
$score = DB::table('eval_sets')
->selectRaw('1 - (embedding <=> ?) AS similarity', [$agentVector])
->where('id', $goldenSetId)
->value('similarity');
// Asserting the response is semantically "close enough"
$this->assertGreaterThan(0.85, $score, "Response drifted too far from ground truth.");
Keep in mind that if you swap your embedding model—say, moving from OpenAI to a local HuggingFace model—your historical scores will shift. This isn't a bug; it's a reminder that your evaluation framework is tightly coupled to how your "brain" layer perceives meaning. When the vector space changes, your baseline for a "good" match needs a quick recalibration too.
Integrating Evaluation into Your CI/CD
We’ve built the "Critic" agent and wired up the tests; now we need to ensure they actually run. Don't let your evaluation suite sit in your local environment like a trophy. If you’re tweaking prompts late at night, how do you know you haven’t just broken your bot’s ability to cite sources? You wouldn’t ship PHP code without unit tests, so treat your LLM logic with the same rigor—and perhaps a bit more suspicion.
Automated regression testing
The primary goal here is to stop shipping "dumber" agents. By wrapping our Pest evaluation suite into a GitHub Action, we create a quality gate that prevents performance regressions. If a prompt change causes the Faithfulness score to dip below your specific threshold—say, 0.85—the build fails. It’s a simple way to keep the agent's behavior consistent without manual oversight. Since we're using the Laravel 13 AI SDK, this runs right alongside your standard feature tests in a familiar environment.
- name: Run RAG Evaluation
run: php artisan test --group=rag-eval
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
EVAL_THRESHOLD: 0.85
Monitoring production drift
Real users are far more creative than any synthetic dataset. Your "Golden Set" might look perfect on Tuesday, but performance can decay as model providers update their backends. Visualizing Faithfulness scores over time helps you spot this drift before it becomes a support ticket. Set up a feedback loop: whenever a user clicks that "thumbs-down" button in your UI, pipe that specific query and context back into your evaluation pipeline. This turns user frustration into your most valuable data for the next iteration of your prompt engineering.
Moving Beyond the Guesswork
Let's face it: we've spent far too long crossing our fingers and hoping our LLMs don't start making things up. By pulling evaluation directly into your Laravel stack, you're not just saving on infrastructure overhead; you're making quality a first-class citizen in your code. No more spinning up a separate service just to run a few Python-based tests. You have the tools to audit your agents right where they live.
The real win isn't just catching a single hallucination—it's the telemetry you build over time. As your 'Golden Set' grows and your Pest tests get more rigorous, your confidence in shipping complex AI features will start to match your confidence in shipping a standard CRUD app. We're moving away from a 'vibe check' culture and toward a pipeline that actually understands the nuances of your data.
Sources & Further Reading
- laravel.com — https://laravel.com/docs/11.x/ai
- docs.ragas.io — https://docs.ragas.io/en/stable/concepts/metrics/index.html
- github.com — https://github.com/explodinggradients/ragas
- openai.com — https://openai.com/index/introducing-openai-evals/
- www.anthropic.com — https://www.anthropic.com/news/evaluating-ai-systems
Frequently Asked Questions
What is a RAG Evaluation Framework in PHP?
A RAG Evaluation Framework PHP is a systematic approach to measuring the quality of AI agent responses within a Laravel application. Instead of manual spot-checks, it uses automated metrics like faithfulness, relevance, and groundedness to ensure the LLM generates answers strictly based on retrieved context. By implementing this natively in PHP, developers can maintain the entire evaluation pipeline within their existing application stack, ensuring reliability without external Python dependencies.
How does the 'LLM-as-a-Judge' pattern work?
The LLM-as-a-Judge pattern involves using a high-reasoning model, such as GPT-4o or Claude 3.5 Sonnet, to audit the outputs of a worker agent. In Laravel 13, this is implemented by creating a specialized 'Critic' agent with a strict system prompt. This judge analyzes the worker's response against the retrieved context to calculate a faithfulness score, identifying any hallucinations or unsupported claims before they reach the end user.
Why should I use Pest for RAG evaluation?
Using Pest allows PHP developers to turn abstract evaluation metrics into repeatable, automated tests within their CI/CD pipeline. By creating a 'Golden Set' of verified question-and-answer pairs, you can run faithfulness checks as part of your standard test suite. Pest's dataset functionality enables looping through ground-truth data, while its mocking capabilities allow you to isolate the LLM's reasoning from the retrieval process for more precise debugging.
Related Articles
Discussion
Leave a comment
Comments are moderated before appearing.
No comments yet — be the first to share your thoughts.