Retrieval-augmented generation looks simple in a demo and fragile in production. The gap is almost never the model — it's chunking, retrieval quality, and whether you can tell when the system is wrong.
Start with evals, not embeddings
Before you tune chunk size or swap vector DBs, write a gold set of questions with expected answers or citations. Every change should move that score. Without evals you're optimizing vibes.
- Keep 50–200 real user questions with source docs labeled.
- Score citation correctness separately from answer fluency.
- Gate deploys on eval regression, not just latency.
Chunking that respects structure
Fixed 512-token windows discard headings, tables, and code boundaries. Prefer structure-aware splits: Markdown sections, HTML landmarks, or semantic paragraphs with overlap only where cross-references matter.
Hybrid retrieval beats pure vectors
Combine sparse (BM25 / keyword) with dense embeddings, then re-rank. Product SKUs, error codes, and proper nouns often fail pure semantic search. Hybrid + a small cross-encoder re-ranker is the pattern we ship most often.
Observe the retrieval path
Log query → retrieved IDs → final answer. When users report bad answers, you need to know if retrieval missed or the model ignored good context. That single pipeline log saves weeks of guessing.