Introduction to RAG Architecture
See how retrieval and generation work together to answer questions from a trusted knowledge base.
Why language models need retrieval
A language model stores broad patterns in its parameters, but it may not know your company handbook, a newly published policy, or a customer’s latest account details. Retrieval-Augmented Generation, or RAG, supplies relevant evidence at request time. The model then uses that evidence to compose an answer. This separates general language ability from changing domain knowledge. It also makes updates easier: instead of retraining a model whenever a document changes, an application can re-index the source and retrieve the new version. Retrieval does not make an answer automatically correct, so evidence quality and answer validation still matter.
The request path
A typical request begins when a user submits a question. The application cleans or rewrites the question, creates a query embedding, and searches an indexed collection for relevant chunks. It may apply permissions, metadata filters, keyword search, or reranking before selecting evidence. The selected chunks are inserted into a prompt with instructions about how to use them. A language model generates the response, and the application can attach citations or source links. Logging each stage creates a trace that helps explain whether a poor answer came from query understanding, retrieval, prompt construction, or generation.
question = "What is our parental leave policy?"
chunks = vector_store.search(question, top_k=4, filters={"tenant_id": "acme"})
evidence = "\n\n".join(
f"[{chunk['source']}] {chunk['text']}" for chunk in chunks
)
prompt = f"Answer only from this evidence; say when it is insufficient.\n\n{evidence}\n\nQuestion: {question}"
answer = model.generate(prompt)
print(answer)The main components
A production RAG system usually contains an ingestion pipeline, a document store, an index, an embedding model, a retrieval service, a prompt builder, a language-model provider, and an evaluation or observability layer. Ingestion turns source files into clean chunks and records provenance. The index makes those chunks searchable. Retrieval selects candidate evidence for a particular question. Generation turns evidence into user-facing language. Each component has a separate contract, which makes it possible to replace a vector database or model without rewriting the entire application.
Knowledge is a pipeline
RAG quality starts before a user asks a question. Documents must be collected from approved sources, parsed without losing important structure, normalized, split, enriched with metadata, and indexed. A document version should be identifiable so stale chunks can be removed or superseded. The pipeline should be repeatable and idempotent: running it twice should not create duplicate evidence. Failed files need a visible error state rather than silently disappearing. Treat ingestion as a data product with ownership, freshness expectations, and checks for missing pages, empty text, broken tables, and unsupported formats.
Grounding the answer
The prompt should clearly identify retrieved text as evidence and user input as a separate data source. It can instruct the model to answer only from the supplied evidence, cite the relevant source, and state when the evidence is insufficient. This is useful guidance, but it is not a security boundary. Retrieved documents can contain misleading instructions or prompt-injection text. The application should limit what the model can do, validate outputs, and keep sensitive source material out of responses when the user is not authorized to see it.
Where RAG fails
RAG can fail because the right document was never indexed, a chunk was split at the wrong place, the query used unfamiliar wording, permissions filtered out necessary evidence, or the retriever ranked a plausible but irrelevant passage first. Even perfect retrieval can lead to a wrong answer if the model misunderstands a table, combines conflicting versions, or fills a gap with prior knowledge. A fluent response can hide every one of these failures. Debugging therefore requires inspecting retrieved chunks and scores, not only reading the final answer.
Designing useful citations
Citations should help a reader verify the answer, not merely decorate it. Store a stable document identifier, title, section, page or anchor, source URL, and version with each chunk. The response layer can map cited identifiers to safe links or short source labels. Avoid showing citations for evidence that was not actually used, and do not expose restricted document names through error messages. For high-risk answers, display supporting excerpts or a “sources used” panel. Citation presence is a helpful signal, but citation correctness and coverage require dedicated tests.
A practical first version
Start with one document collection and a narrow question type. Build ingestion, retrieval, answer generation, citations, and a small test set before adding agents, complex query rewriting, or many data sources. Measure retrieval recall and answer faithfulness on representative questions. Add access controls and logging from the first version, because retrofitting them is difficult when data has already spread across prompts and traces. A small, observable RAG workflow with honest abstention is more useful than a broad demo that answers confidently without reliable evidence.