Case study: AI-powered travel RAG system — what I learned from AI engineering

I planned this project as a 16-week build. It went public at mini-travel-rag.vercel.app on 10 June 2026, about three weeks after the first PDF was extracted. It is a RAG travel assistant: you ask a travel question and get a grounded answer with citations, while the places in the answer appear live on a map. Ask for a trip and you get a structured day-by-day itinerary, with weather, that you can download or export to Notion.

I built it to answer one question for myself: what does it take to put a pre-trained model into a product that strangers can use without it breaking, leaking or running up a bill? The model turned out to be the easy part. This case study starts with the plan against what shipped, then covers the build, what broke and what I would carry into the next system.

This case study is the overview of a series. Each part below goes deeper into one side of the system, with its own diagrams, numbers and mistakes. Read this page first for the whole story, then follow whichever part interests you. It is a personal, non-commercial learning project, not affiliated with any company named; see the disclaimer at the end.

1.At a glance: plan, revision, final product

The product that shipped is not the product I first planned. I revised the plan once, after Phase 14, when the centre of gravity moved from infrastructure to product. The final product differs again: some features were cut, some deferred, and some changes were forced by licensing and by services changing under me. The table puts all three side by side, grouped by product area in the order a user meets them (within each group: shipped, then changed, deferred and cut). The rest of this section explains why.

Why three weeks, not sixteen

The plan budgeted a week per phase; the launch came after three weeks in total. Three things closed the gap:

• AI-assisted building. Claude Code implemented each phase against a precise spec with acceptance tests. A phase took hours to a day, not a week. The bottleneck moved from writing code to deciding what to build and checking that it worked.
• A ruthless MVP bar. The launch bar was one loop: ask a question, get a grounded answer with sources, see the places on a map, and turn it into a trip plan you can keep. Every other feature had to beat its effort against that loop, or wait.
• Launch is not the finish line. I launched when the core loop worked and was safe for strangers (rate limit, cost cap, error tracking), not when every phase was ticked. Hardening continued after launch, driven by real use: 35 pull requests in October.

None of this was fixed in advance: software plans are fluid, and each new piece of information (a hotel partner closing sign-ups, a copyright audit, GDPR obligations, a model being retired, a feature costing more than it returned) meant weighing effort against value again and re-prioritising.

Most decisions came down to effort against value: the map and trip plans were built because they are the loop; weather got in after launch because it cost one free API; hotel search, alerts, accounts and images didn’t beat their effort and were cut or deferred, as below. A few changes weren’t choices at all: copyright forced the corpus swap, and a retired model forced the provider change.

What was cut, and why

Hotel search shows how quickly the calculation changes. Amadeus closed new sign-ups mid-sprint. Because the hotel provider sat behind a Protocol, swapping to LiteAPI was a one-line change, but the sandbox data was poor. The trigger to cut was realising I was about to re-theme 34 styling hits in files I wanted to delete.

What was deferred, and why

Deferred items are logged in TECHNICAL_DEBT.md with a trigger for coming back. After launch, fixing wrong answers (wrong pins, wrong destinations, invented sources) always came before new features: a wrong answer costs more trust than a missing feature.

2.Planning

Before writing any code I wrote one document, ARCHITECTURE.md, and made it the single source of truth. It held the goal, the locked stack, a phase-by-phase plan with acceptance criteria and a cost reserve. Every coding session started by reading the relevant phase section. Every phase closed by updating the doc to match what actually shipped.

The goal

The goal was a production-quality AI travel assistant, not a demo. Concretely that meant real authentication, product analytics, LLM tracing, app traces, error tracking and CI/CD, all in the EU. The corpus was 38 Lonely Planet guidebooks, 400 to 900 pages each.

Architecture

The system has four layers:

• Ingestion. Extraction and cleaning, chunking at three granularities, Voyage embeddings and pgvector on Neon Postgres in Frankfurt.
• Retrieval. A query router picks the chunk granularity and the destination filter. Vector search returns the top 50 chunks, and a reranker cuts them to 5.
• Generation. A FastAPI backend streams the answer over SSE. Alongside the tokens it sends sources and place events, which the Mapbox panel pins live.
• Product. A React + Vite frontend with shadcn/ui, structured trip planning validated with Pydantic, download and Notion export.

Tech stack

Most choices followed three rules: keep data in the EU, prefer one service that does two jobs over two services, and start on free tiers that can grow. Two choices changed during the build (embeddings and the second model), and the table says so.

I set six rules before any code and never broke them:

1. Protocol-first. Every external dependency (extractor, embedder, vector store, reranker, LLM provider, hotel provider, weather) gets a Python Protocol before any concrete class exists.
2. Async everywhere in the backend.
3. Route → Service → Model only. No business logic in routes or Pydantic models.
4. One exception hierarchy. Every domain error subclasses LPError.
5. Alembic owns the schema. No hand-run ALTER TABLE.
6. Settings via Pydantic. No os.getenv() and no hard-coded secrets.

Phases

The plan had 16 one-week phases. Each had a deliverable, an ordered implementation list and acceptance checkboxes. The order was deliberate: ingestion first, then retrieval, the API and the UI. The eval harness came next, before any features were layered on top. Multi-provider support came late, on purpose, so Claude-only would work perfectly first. Observability, deployment and launch came last.

Every phase closed with the same ritual. Run pytest, ruff and mypy --strict. Ask for a production-readiness review covering error handling, logging and edge cases. Then log everything flagged but not fixed in TECHNICAL_DEBT.md. That file became the most useful document in the repo.

Model routing

I routed each task to the cheapest model that could do it well. Every call goes through an LLMOrchestrator keyed by task, so no call site names a model.

The trip planner’s fixed instructions and output format (about 4,200 tokens) use prompt caching, so every trip reads them at a tenth of the price. Unit tests use a StubLLMProvider and never call the real API. Every LLM call emits a Langfuse trace.

Upgrading the models

In October 2026 I moved chat and trips from Sonnet 4.6 to Sonnet 5.5, and the small tasks from Haiku 4.5 to Haiku 5.5. On paper it was a price cut: a third cheaper per token for Sonnet, ten times cheaper for Haiku. Two things only showed up in testing. Both new models think before answering by default, which adds cost and delays the first word, so I turned thinking off (Sonnet 5.5 calls that setting between_tools). And the new models count the same text as about 40% more tokens, which cancels most of the lower price.

I kept the test small on purpose: 12 eval questions (two from each category) graded by the same judge for both models, and three runs of one trip case compared with earlier runs on the old model. It cost about $0.75. The judge score held (65% → 66%), made-up facts went from 2 of 12 answers to none, answers came about 27% faster, and trip plans took about 30 s instead of 55–77 s. Cost per trip fell only about 10%, to roughly $0.10. The gain was speed and quality, not price.

The trip consistency check caught one change in behaviour after release: asked for “the 3rd to the 6th of May”, the new model sometimes planned the default three days instead of four. Moving the reminder next to the day count in the prompt fixed it (three passes out of three). A new model can follow the same prompt differently, so the checks that pin down requirements matter more than the overall score.

Budget

I planned a cost reserve of €350 for the first six months: €100–150 for API testing and evals during the build, €25–35 a month after launch, and a €100 buffer.

3.Implementation

I built it phase by phase with Claude Code, one phase per session, following the order in the doc. Below is what each stage actually does.

Ingestion: 38 PDFs to a vector store

I extracted the PDFs with pdfplumber and used PyMuPDF as a fallback, which was needed on 0.4% of pages. A custom cleaner.py strips page numbers, running headers and 1,003 front-matter pages. It also repairs 226 split ligatures and removes reversed sidebar text. Every book is chunked three ways at once, and each chunk links to its parent:

Chunks are embedded with Voyage voyage-3-lite into a vector(512) column with an HNSW index. The embedding code lives in a shared package that ingestion and the backend both import, so documents and queries are embedded by the same model.

Retrieval and generation

A heuristic query router classifies each question. It picks a granularity and uses a destination gazetteer to scope the search. That is how “Granada” resolves to Spain rather than Nicaragua. When a question names a season or month (“things to do in winter in Vienna”), the search adds that season’s months, weather and events, so climate and event passages surface instead of generic sightseeing. Vector search returns the top 50 chunks, Voyage rerank-2 narrows them to 5, and the answer streams over SSE with the sources attached. SSE was chosen over WebSocket because the stream only goes one way and it works cleanly through HTTP/2 proxies.

Map, trips and Notion

• Live map. spaCy extracts place names from the streamed tokens, Haiku catches what spaCy misses, and each place is geocoded and pushed to the map. Geocoding checks 8,734 landmark coordinates taken from Wikivoyage first, then Mapbox with strict matching inside the destination city. A wrong pin is worse than no pin.
• Trip planning. Sonnet returns an Itinerary validated against a Pydantic schema, with one retry that feeds the validation error back to the model. The plan respects what the user says (their own hotel, their dates, places to skip). The total cost is summed from the plan itself rather than guessed, and every day gets its weather. Plans run as background jobs that the browser polls.
• Download and Notion export. One click downloads the plan as Markdown. A pure-function block builder turns it into a Notion page on a public TravelRAG Trips site. Notion caps a request at 100 blocks, so the first call creates the page and later calls append the rest.

Evals, providers and observability

• Eval harness. 56 questions across six categories: factual recall, recommendation, comparison, multi-hop, edge case and adversarial. A Haiku judge returns a structured verdict. I track retrieval recall, citation accuracy and judge score as three separate metrics, not one composite, and results upload to Langfuse. A separate check plans the same trip several times and flags plans that drift.
• Multi-provider. A second model on Groq runs behind the same LLMProvider Protocol as Claude, and a UI selector switches between them. Each provider declares its capabilities, so a missing capability returns HTTP 400 before the stream opens, never an error mid-stream. The planned local Ollama model was deferred (section 1).
• Observability. Langfuse for LLM traces, Sentry for errors, PostHog for product analytics and OpenTelemetry for app traces, shipped as four sub-phases.

Swapping the corpus for launch

Before launch, a copyright audit changed the plan. Lonely Planet content is licensed for purchase, not for redistribution to anonymous web users. I rebuilt the public product on Wikivoyage (CC BY-SA): 22,044 chunks across 114 destinations and 275 articles, for $0.18 of Voyage spend. The Lonely Planet corpus lives in a separate Postgres database and is never deployed. The backend binds exactly one database at start-up, and production only holds the public database’s address.

The chunker and pipeline needed zero changes. Only a new ingestion adapter was added, which parses Wikivoyage wikitext into the same DocumentStructure shape.

Deployment

• Hosting. The backend runs on Fly.io in Frankfurt with one always-on machine (about €10 a month). The frontend is on Vercel, the database on Neon and the rate-limit counters on Upstash Redis.
• CI/CD. GitHub Actions runs ruff, mypy --strict, pytest, eslint, tsc and vitest on every PR. A push to main deploys the backend, gated on green tests, and only the newest commit is ever deployed.
• Launch hardening. A per-IP rate limit of 5 requests a minute that fails closed. A daily cost cap of €10 with an alert at €5. A Mapbox token locked to the production domain. Failures show a generic message, never raw provider errors.

4.Challenges

Almost none of the hard problems were about the model. They fell into five groups: the data, the contracts between components, my own measurement tools, production and real use, and services I don’t control. Each group produced its own lessons (section 6).

Data

Dirty PDFs. My Spain-only tests passed. Japan, Mexico and Iceland each broke in a new way. Vertical cover sidebars came out reversed (“PLAN YOUR TRIP” became “PIRT RUOY NALP”), ligatures split into separate characters, InDesign printer codes leaked into body pages, and front-matter detection failed on regional editions. The worst case was reversed tokens leaking inline into real sentences, because the cleaner only stripped lines that were entirely noise. An audit across all 38 books, 17 minutes of work, found four left; fixing them took five minutes. I spent two full sessions on cleaning before ingesting a single book.

Wrong map pins. Place names failed the same way after launch. Fuzzy geocoding pinned “Porto” on the whole of Portugal and some landmarks in same-named towns hundreds of kilometres away. The fix was stricter matching and better data (the geocoding tiers in section 3), not a smarter model.

Contracts between components

Two embedding models, one silent bug. The spec called for BGE-large running locally. My laptop is a fanless M2, so embedding 5.7M tokens overnight would have throttled it badly, and I switched to Voyage. That exposed a hidden contract: the backend was about to embed queries with BGE while the documents were embedded with Voyage. Two incompatible vector spaces produce bad results with no error. Moving embedding into one shared package fixed it before shipping. Months later I found the same bug hiding in a safety net: the package fell back to BGE whenever Voyage failed, and even trimmed BGE’s vectors to the same size so they would fit. Questions now use Voyage only, and an outage shows an honest “temporarily unavailable”.

A rate limit that wasn’t about my prompt. Groq rejected requests at 35,846 tokens against a 12,000 tokens-per-minute ceiling. Shrinking retrieval from 5 context chunks to 3 paragraph chunks saved about 2,000 tokens, and the request still failed. A token breakdown showed the input was only about 1,025 tokens; the rest was the max_tokens headroom inherited from Claude’s defaults, which Groq counts against the limit before generating anything. Per-task caps (4,000 for chat, 8,000 for trip planning, 512 for utilities) brought the same request down to 5,025 tokens. Capacity is committed, not consumed.

A cache that never hit. Months after launch, Anthropic emailed that my prompt-cache hit rate was low. The code asked for caching on the chat prompt, but at 372 tokens it was below Claude’s minimum cacheable size, so nothing was ever cached; the trip planner, the most expensive call, never asked at all. Its instructions plus the output format came to about 4,200 identical tokens per trip, now cached and read at a tenth of the price. Caching the 17,000 tokens of guide text for retries looked like the bigger win, but measured it lost money: a cache write costs 25% extra on every trip and only pays back on the rare retry. The larger saving came from asking for shorter output, since writing is most of a plan’s cost and time. Together: 19% cheaper and 27% faster per trip plan.

Measurement

Evals that lied. The first Langfuse-traced eval run scored 30.67% on the judge with a 66.67% hallucination rate. The per-trace view showed the truth in seconds: two of three judge calls had failed Pydantic validation. The judge’s reasoning field was capped at 500 characters, Claude wrote more, and a broad except turned each failure into a score of zero. The one clean trace scored 0.92.

Retrieval and citations were perfect on all three traces, which pointed straight at the judge, not the system. The numbers measured my schema. Without tracing it would have shipped.

Observability the browser eats. The backend Sentry smoke test passed in ten minutes. The frontend one took two hours and had nothing to do with the code: uBlock Origin was blocking *.ingest.sentry.io and PostHog, so my own browser dropped my own telemetry, as every visitor with an ad blocker would. The fix is a same-origin tunnel, logged as debt (TD-068). No unit test in jsdom could have found this.

Production and real use

• Neon cold starts. Neon autosuspends idle databases and kills pooled connections, so the first query after a wake failed with 57P01. A connection check on the pool fixed it, leaving about 9 seconds to first token on a cold resume.
• An open cost tap. The rate limiter failed open when Redis was unreachable, and production had no Redis at all: anonymous users could have made unlimited Claude calls. I kept the URL private until fail-closed limiting and the cost cap were live.
• Trip planning under real conditions. Planning failed intermittently on 429s against a 30k input-tokens-per-minute limit, dropped connections and over-long JSON, and mobile Safari dropped the two-minute request outright. Plans now shorten over-long fields instead of failing and run as the polled background jobs described in section 3. Backoff on 429s is still open.
• Requests read wrongly. The planner suggested a hotel from the guide when the user had said where they were staying, and “plan a weekend in Copenhagen” planned a city called “Weekend”. The planner now reads the user’s own message, and time words never become destinations. Each fix came with a test.

External dependencies

The October hardening pass started because the site was down, with nothing changed in my code. Upstash deletes a free Redis database after 14 idle days, and the fail-closed rate limiter, doing its job, refused every request; a daily keep-alive ping now prevents it. The same week I found that Groq had retired the Llama model behind the alternative provider, and the Anthropic credit ran out mid-test, showing raw provider errors to visitors. The model swap was a config change thanks to the provider Protocol.

5.Results

The numbers behind the final product:

6.Learnings

The lessons follow the same groups as the challenges, plus one for product and process.

Data

• Data quality sets the ceiling. Every artifact that slips past ingestion corrupts a chunk, then retrieval, then the answer; a wrong coordinate puts a pin in the wrong country. Test on a representative sample before you scale, not on the one book that happens to be clean.

Contracts between components

• Make implicit contracts explicit. “We both use BGE” is a contract nobody wrote down. Put the shared dependency in one package, and check that no fallback quietly breaks it again.
• Know how the provider bills you. Tokens per minute, max_tokens reservation, prompt-caching thresholds and org-level input limits shaped the design more than model quality did. Reserved capacity, not cost per token, is the unit that bites, and a cache only pays when its content is reused.

Measurement

• Measure the measurement. A 30% judge score was a schema cap; missing errors were an ad blocker. Report per query, not only in aggregate: aggregates tell you what changed, per-query results tell you why.
• Build evals before features. Comparing two versions of the system is easier and more useful than an absolute score. With the harness in place I could change the chunker, the embedder and the reranker and know each time whether things got better.

Production and real use

• Real bugs live in production. Database autosuspend, fail-open defaults, phones that drop long requests and the way people actually phrase questions are invisible to unit tests. Ship early to somewhere private, then test with real devices and real requests.
• Fail closed when money is on the line. Any path an anonymous user can hit that calls a paid model needs a rate limit and a cost cap that deny by default.

External dependencies

• Plan for dependencies that disappear. A free database deleted after two idle weeks, a model retired, credit run out: none of it was in my code. Keep-alives, balance alerts and swappable providers behind a Protocol are part of the product.
• Upgrade models like any other dependency. A newer model is not automatically cheaper or better for your product. Run a small A/B with the same judge on both sides, and compare cost per answer or per trip, not the price per token: a new tokenizer can quietly eat the discount.

Product and process

• Protocols pay for cutting, not only for swapping. The HotelProvider Protocol made the Amadeus → LiteAPI swap one line, and it made removing the whole feature a bounded job.
• Cutting is a product skill. Every feature had to earn its place against the core loop. Featureitis is easy to fall into and hard to see from the inside.
• Write the debt down. 120 technical-debt entries, each with a trigger for coming back, let me defer things honestly instead of silently. A doc that stays true to what shipped is what kept AI-assisted development reliable across many sessions.

The short version: the model is the core component. AI engineering is everything you build around it so that it stays grounded, affordable and observable in production.