Topic 2: RAG interview questions
Retrieval-augmented generation design, indexing, search quality, grounding, and production operations. Click a question to reveal a focused answer, then use the examples to shape your own response.
query_vector = embed(question)
chunks = vector_store.search(query_vector, limit=5)
context = '\n'.join(chunk.text for chunk in chunks)
answer = generate(question, context)01What is retrieval-augmented generation?
Retrieval-augmented generation, or RAG, retrieves relevant content for a user question and supplies that content to a generative model as context. The model then writes an answer grounded in the retrieved evidence. RAG keeps knowledge outside model weights, so documents can be updated without retraining. It improves access to private or current information, but retrieval errors still limit the final answer.
02What are the main stages of a RAG pipeline?
A typical pipeline ingests documents, extracts text and metadata, splits content into chunks, creates embeddings, and indexes the chunks. At query time it normalizes the question, retrieves candidates using keyword, vector, or hybrid search, optionally reranks them, builds a context within the token budget, and calls the generator. Finally, it validates grounding and format, returns citations, records telemetry, and handles failure or no-result cases.
03How do you choose a chunking strategy?
I choose chunks that preserve a useful unit of meaning, such as a section, procedure, paragraph group, or table row, while staying small enough for precise retrieval. I keep headings and document identifiers as metadata and use modest overlap only where context may cross boundaries. The right size depends on content and query type, so I compare strategies with recall, answer quality, citation, and token-cost evaluations.
04What is the purpose of chunk overlap?
Chunk overlap repeats a small boundary region so a sentence or idea split between adjacent chunks remains available in at least one retrieved result. It can improve recall for cross-boundary questions, but excessive overlap duplicates tokens, increases index size, and can crowd out distinct evidence. I use overlap selectively, measure its effect on retrieval and answer quality, and prefer structure-aware splitting when documents provide reliable headings.
05How do vector embeddings support RAG?
An embedding model maps a chunk and a query into vectors where semantically similar content is intended to be close. A vector index can then retrieve nearest chunks even when they do not share exact words. Quality depends on the embedding model, preprocessing, language, chunking, and distance metric. Embeddings are a retrieval signal, not proof that a chunk answers the question, so reranking and validation remain useful.
06When should RAG use hybrid search?
Hybrid search combines semantic vector retrieval with lexical search such as BM25. It is valuable when questions mix conceptual language with exact identifiers, product codes, names, error messages, or legal wording. Vector search handles paraphrases; lexical search preserves exact matches. The two result sets need normalization and a sensible fusion or reranking method. I would verify the choice on a query set rather than assume one search type wins.
07What does a reranker do in a RAG system?
A reranker examines the query and an initial set of candidate chunks together, then scores how relevant each pair is. This usually gives more precise ordering than inexpensive first-stage vector or lexical search. Because reranking costs extra latency, retrieve a broader candidate set first and rerank only the top candidates. Measure both recall and end-to-end answer quality; a reranker that removes necessary evidence can hurt results.
08How do you evaluate retrieval quality?
Create a labeled query set with the documents or chunks required to answer each question. Measure recall at k, precision at k, mean reciprocal rank, or normalized discounted cumulative gain, depending on whether finding all relevant evidence or ranking the first useful result matters. Also inspect failures by document type and query category. Retrieval metrics should be paired with grounded answer evaluation because good retrieval can still be poorly used.
09How do you prevent irrelevant context from hurting an answer?
Retrieve enough candidates for recall, then filter or rerank them for precision. Apply metadata constraints such as tenant, permissions, product, language, and date before generation. Remove duplicates, cap the number of chunks, preserve source boundaries, and instruct the model to use only relevant evidence. If the evidence is weak or contradictory, return an explicit insufficient-evidence response instead of filling the context with loosely related text.
10How should RAG handle document permissions?
Authorization must happen during retrieval, before content enters the model context. Each chunk should carry enforceable ownership, tenant, group, or access metadata, and the query should be evaluated against the requesting identity. Do not rely on a prompt telling the model to hide restricted text. Test cross-tenant and revoked-access cases, audit retrieval decisions, and ensure cached results are partitioned by the same security boundaries.
11How do you handle document updates and deletions?
Give each source document a stable identifier and version, and store that identity on every chunk. An ingestion process should detect changes, reprocess affected chunks, upsert the new version, and remove or deactivate chunks from older versions. Deletions need an explicit tombstone or deletion workflow, including vector-index cleanup and cache invalidation. Monitor indexing lag so users do not receive stale answers after a confirmed update.
12What metadata should be stored with RAG chunks?
Useful metadata includes document and chunk identifiers, title, section path, source URL, version, timestamps, content type, language, tenant, and access-control attributes. It supports filtering, citations, freshness checks, debugging, and deletion. Metadata should be normalized and validated during ingestion. Avoid storing sensitive values unnecessarily, and ensure the metadata returned to users is safe to expose rather than blindly copying internal fields.
13How can RAG provide reliable citations?
Carry stable source identifiers and human-readable locations from ingestion through retrieval and generation. Ask the model to associate claims with those sources, but validate that cited identifiers actually appeared in the supplied context. Prefer links, titles, and page or section references that users can inspect. A citation proves where supporting evidence was found; it does not prove the answer is correct, so claims still need groundedness checks.
14What is query rewriting in RAG?
Query rewriting transforms a user question into a search-friendly query, sometimes using conversation history to resolve references such as “that policy.” It can generate alternate queries for different terminology or search methods. Rewriting should preserve the user’s intent and security scope, and the original question should remain available for answer generation. Evaluate whether rewriting improves retrieval, because an incorrect rewrite can systematically hide the right evidence.
15How should conversational RAG use chat history?
Use history to resolve context and user intent, but do not automatically send the entire conversation to retrieval and generation. Create a standalone search query when needed, retain the original question, and select only relevant prior turns. Apply access controls to every retrieved result. Summaries can reduce tokens, but they may lose qualifiers, so maintain a way to inspect source turns for high-stakes or ambiguous requests.
16How do you handle a RAG question with no answer in the corpus?
Set a retrieval-quality threshold using evaluated scores, reranker signals, or evidence checks, and allow the system to abstain. The response should say that the available sources do not establish an answer, identify what was searched, and offer a useful next step such as requesting another document. Do not infer that a low-scoring chunk is relevant merely because the model can produce a fluent response from it.
17What are common RAG failure modes?
Common failures include poor text extraction, chunks that split meaning, stale indexes, missing permissions filters, vocabulary mismatch, low recall, duplicate context, and a prompt that does not enforce evidence use. Tables, scanned PDFs, multilingual text, and rapidly changing documents need special handling. Diagnose the stage first by inspecting retrieved chunks and scores, then compare retrieval, context construction, and generation against a labeled test case.
18How would you reduce RAG latency and cost?
I would profile ingestion, embedding, search, reranking, and generation separately. At query time, use an efficient index, apply metadata filters early, retrieve a bounded candidate set, rerank only when it improves quality, cache safe repeated queries, and trim context without losing evidence. Batch ingestion embeddings and process updates incrementally. Streaming can improve perceived latency, while timeouts and fallbacks keep slow providers from blocking the user.
19When is RAG a poor fit?
RAG is a poor fit when the task needs exact transactional state without a reliable source-of-truth tool, when documents are too unstructured to retrieve accurately, or when the desired behavior is a stable transformation rather than knowledge lookup. It also adds complexity for tiny static prompts. In those cases, use a database query, deterministic code, an API tool, or a simpler model workflow, while retaining validation and authorization.
20What production monitoring does a RAG system need?
Monitor ingestion success and lag, document and chunk counts, embedding errors, index freshness, retrieval latency, scores, empty-result rates, reranking behavior, token usage, model latency, citation validity, abstention rates, and user feedback. Trace each response to its query, retrieved source IDs, prompt version, and model version without logging sensitive content unnecessarily. Alerts should distinguish provider outages, ingestion failures, permission anomalies, and quality regressions.