Definition. Retrieval-Augmented Generation (RAG) is an architecture pattern where a model's response is grounded in documents retrieved from an external source at query time, rather than relying only on training data.
How does a RAG pipeline actually work?
A RAG system has four stages, and each one introduces its own failure modes: ingestion (documents are chunked and embedded into vectors), retrieval (a query is embedded and matched against those vectors), re-ranking (the initial matches are reordered by relevance, often with a different model than retrieval used), and generation (the language model produces an answer grounded in the retrieved context). Teams that treat RAG as "plug documents into a vector database" skip the second and third stages and get retrieval quality that looks fine in a demo and degrades on real queries.
How to choose a chunking strategy
Chunking determines what gets embedded and, therefore, what can ever be retrieved. Splitting text at a fixed character count is the simplest approach and the most common source of retrieval failures, because it cuts sentences and ideas at arbitrary boundaries. Semantic or structure-aware chunking — splitting at natural boundaries like headings, paragraphs, or logical sections, and keeping a chunk small enough to embed cleanly but large enough to preserve context — consistently retrieves better in practice.
| Strategy | Strength | Risk |
|---|---|---|
| Fixed-size chunking | Simple, fast, predictable size | Splits ideas mid-sentence or mid-table |
| Semantic / structure-aware | Preserves meaning boundaries | More engineering to implement well |
| Sliding window with overlap | Reduces boundary loss | Increases index size and duplicate noise |
Why retrieval quality matters more than model size
A large, capable model given the wrong context will still produce a wrong or hallucinated answer — it can only reason over what it was given. This is why RAG failures are disproportionately retrieval problems: the embedding model doesn't capture domain-specific meaning well, the similarity search returns near-duplicates instead of diverse relevant results, or there's no re-ranking step to filter noise before it reaches the model's context window.
Compare: vector search vs. hybrid search
| Vector (semantic) search | Hybrid (vector + keyword) | |
|---|---|---|
| Best for | Conceptual, paraphrased queries | Mixed queries — names, codes, exact terms, and concepts |
| Weakness | Misses exact-term matches (IDs, acronyms, product codes) | More moving parts to tune and maintain |
| Typical production choice | Simple domains, short time-to-ship | Domains with technical vocabulary or structured identifiers |
How to reduce hallucination in a RAG system
- Re-rank retrieved chunks before they reach the model — initial vector search optimizes for recall, not precision.
- Instruct the model explicitly to answer only from retrieved context, and to state when the context doesn't contain an answer.
- Return citations alongside the answer so incorrect grounding is visible and checkable, not hidden inside fluent prose.
- Evaluate retrieval and generation separately — a good answer from bad context is a lucky guess, not a working system.
Frequently asked questions
What is RAG used for?
Answering questions using information that changes after training, is proprietary, or is too large to fit in a training set — internal docs, catalogs, support histories, and reference material.
How is RAG different from fine-tuning?
Fine-tuning bakes knowledge into model weights and requires retraining to update. RAG retrieves knowledge at query time, so the source can be updated instantly.
Why does my RAG system return irrelevant answers?
Usually a retrieval problem — poor chunking, an embedding model that doesn't fit the domain, or a missing re-ranking step.
Global deployment considerations
Multi-region RAG systems need to decide where vector indexes live relative to where users query from — cross-region retrieval adds latency that compounds with generation latency. For regulated data (GDPR, CCPA, or sector-specific frameworks), the ingestion pipeline needs to track data lineage per-document so a deletion request can actually be honored across every index the document was embedded into, not just the source system.
Talk about a system ↗