AI · Travel RAG case study · Part 2 of 5
Evaluation: building a harness that measures RAG quality reliably
The first time I looked at my eval results in a trace viewer, the system scored 30.67% with a 66.67% hallucination rate. The system was fine. The evals were broken. A schema limit on the judge's own output was quietly turning two out of three scores into zeros.
This essay is a deep-dive from my travel RAG case study. It covers how I decided what "good" means for travel answers, how the harness works, and the uncomfortable lesson that an eval is software too. It can be wrong, and it is wrong silently.
1.Planning
What does "good" mean for a travel answer?
"Best tapas in Seville" has no single right answer. An old-town bar, a modern tasting menu and a neighbourhood local could all be correct. So I could not grade answers against one gold answer. Instead I asked what can go wrong, and picked a check for each failure.
| What can go wrong | Example | How I check it |
|---|---|---|
| Retrieval misses | The answer about Kyoto is built from chunks about Osaka | Retrieval recall: did the expected destinations appear in what was retrieved? |
| The answer ignores its sources | Retrieval was right, but the model answered from memory | Citation accuracy: does the answer mention the destinations it retrieved? |
| The answer is wrong or invented | It names a museum that does not exist | LLM judge: is it grounded, and does it meet a written rubric? |
| It plays along with nonsense | "Tell me about the ruins of Zorblax in southern Spain" | Adversarial questions, judged on whether the model refuses to invent |
I kept the three scores separate instead of blending them into one number. A single composite score hides which part broke.
Where evals sit in the build order
I built the harness right after the core pipeline worked: after ingestion, retrieval, the chat API and the map, and before trip planning, multiple providers or observability. Every later change, from chunking tweaks to provider swaps, could then be measured against a baseline instead of judged by feel.
Budget
The target was under €5 per full run. A run is one answer plus one judge call per question; a cheap model as judge keeps that affordable.
2.Implementation
The dataset: 56 questions written by hand
Each line of a JSONL file is one test. JSONL rather than YAML, so a row can be added or fixed without touching the rest of the file.
{"id": "eval-002",
"question": "What is the significance of the Sagrada Família in Barcelona?",
"expected_destinations": ["Spain"],
"rubric": "Should mention Gaudí as the architect, that it is an unfinished Roman Catholic basilica, and note its UNESCO World Heritage status or ongoing construction.",
"difficulty": "easy",
"category": "factual_recall"}
The rubric is the key field. It doesn't fix one right answer; it lists what a good answer must contain. That makes open-ended questions gradeable.
The hard categories matter most. Edge cases ask things like "Is it safe for a solo female traveller to visit Morocco?", where a careless answer can do harm. Adversarial questions ask about places that do not exist, such as "the ancient ruins of Zorblax in southern Spain". The right answer there is to say they don't exist.
The harness
- Retrieval recall is the share of expected destinations that appear among the retrieved chunks. It is pure Python, so it is free and deterministic.
- Citation accuracy checks whether the answer mentions the destinations it retrieved. It is a heuristic: it looks for the longest word of each destination name in the answer text.
- The judge is Claude Haiku. It sees the question, the retrieved context, the answer and the rubric, and returns a structured verdict: a score from 0 to 1,
grounded,rubric_metand a short reasoning. It is told to be strict: an invented place or statistic meansgrounded=falseand a heavy penalty.
class JudgeVerdict(BaseModel):
score: float = Field(ge=0.0, le=1.0)
grounded: bool
rubric_met: bool
reasoning: str = Field(max_length=500) # remember this line
The runner
One command runs everything: ./scripts/run_eval.sh [--limit N] [--category X] [--concurrency N]. Items run in parallel, five at a time by default, behind an asyncio semaphore. Each run gets an eval_run_id, and every result lands in a Langfuse dataset, so two runs can be compared question by question. A GitHub Actions workflow can run the suite nightly; its schedule is paused until the dataset is re-labelled (TD-117). When the second provider arrived, --all-providers ran the same questions against each one.
3.Challenges
The judge that crashed quietly
When I added Langfuse tracing, I ran the harness on three items to check the wiring. The aggregate came back at a 30.67% judge score and a 66.67% hallucination rate, alarming numbers for a portfolio demo. The per-trace view explained them in seconds.
Retrieval recall and citation accuracy were 1.0 on all three traces. Only the judge disagreed, and only on two of them. The cause was the line I flagged above: reasoning: str = Field(max_length=500). Haiku often wrote more than 500 characters of reasoning, the verdict failed Pydantic validation, and a broad except returned a score of 0 with grounded=false. A crash in the measuring tool was recorded as a hallucination by the system.
The bug had been there since the harness was built. Every earlier aggregate had been dragged down by it, with nothing to show it. In October 2026 I raised the cap to 2,000 characters, which stopped the false zeros. Half the fix is still open: a failed judge call is now logged, but it still scores zero. It should retry or raise a loud error instead.
Labels written for one corpus, run against another
The dataset was written for the guidebook corpus, where every chunk belongs to a country book. So the expected destinations are countries, such as ["Spain"]. The public deployment now runs on Wikivoyage, where chunks belong to city or region articles, such as "Barcelona". The recall metric compares names exactly, so a perfect Barcelona retrieval would no longer match "Spain". The dataset needs re-labelling for the new corpus before its recall numbers mean anything. Until then the harness can at least run against the live corpus (USE_PUBLIC_DB), so judge scores reflect what visitors actually get.
A citation check that guesses
Citation accuracy looks for the longest word of each destination name in the answer. It works for "Andalusia". It is shaky for short or generic names, and it can be fooled by an answer that names a place without using its sources. It is a cheap first signal, not proof of grounding.
Judging Claude with Claude
The judge and the main model come from the same family. Models from one family can rate each other's style generously, and a single judge has no second opinion. I accepted this to keep runs cheap; the stronger setups calibrate the judge against a set of human-labelled answers or use a judge from a different family.
Smaller traps
- Silent uploads. Failed Langfuse uploads were swallowed by a catch-all. Fixed: they are now logged (TD-064).
- Silent single runs.
--all-providerswith only one provider configured quietly ran one provider and exited cleanly (TD-054). - Cost. Every run uses the standard API. The Batch API would halve the cost but needs polling, so it waits until there is a measured baseline (TD-026).
4.Final results
The harness works as infrastructure: one command, 56 questions, three scores per question, every run stored in Langfuse and comparable question by question. What it cannot yet give me is a trustworthy headline number, and I would rather say that than publish one.
| What | Status |
|---|---|
| Dataset | 50 hand-written questions in 6 categories, each with a rubric and a difficulty |
| Retrieval recall | Works; labels need updating for the Wikivoyage corpus |
| Citation accuracy | Works as a cheap heuristic |
| LLM judge | Works; cap raised to 2,000 characters (TD-063). A failed call is logged but still scores zero |
| Per-question history | Every run in Langfuse under its own run ID |
| Provider comparison | Same questions across providers in one command |
| Cost guard | Full multi-provider sweeps need an explicit flag |
| Regression gate | Not yet: the nightly schedule is paused and never failed the build (TD-117) |
| Standard RAG metrics | Not yet: faithfulness, answer relevance and context precision and recall (TD-027) |
One earlier, smaller test did give a clear answer. On a 10-query retrieval comparison, the reranker raised precision at 5 from 0.62 to 0.84. That was enough to justify its 200–300 ms of extra latency. It is a small sample, and I treat it as a direction, not a benchmark.
5.Learnings
- An eval is software, and it fails silently. My harness had a bug for months, and every number it produced looked plausible. Test the measuring tool as hard as the system it measures.
- Never turn a failure into a score. A judge that crashes should retry or raise an error. Recording a crash as 0 made a broken tool look like a broken product.
- Per-question beats aggregate. "Average dropped 3%" tells you something changed. "Question 3 dropped from 0.8 to 0.4" tells you where to look. Keep both, and read the per-question view first.
- Separate scores show what broke. With recall, citation and judge kept apart, the bug was obvious: two perfect scores and one zero cannot all be true.
- Comparing versions is easier than judging absolutely. "Is this better than yesterday?" is a question I can answer reliably. "Is this good?" needs calibration I don't have yet.
- Evals belong early. Built before the features, the harness let me change the chunker, the embedder and the reranker and know each time whether quality moved.
- Labels age with the data. When the corpus changed, the expected answers had to change too. An eval set is tied to the data it was written for.
The short version: before you trust what your evals say about the model, check what they say about themselves.