Advanced RAG
Explore reranking, query rewriting, hybrid retrieval, and techniques for difficult knowledge tasks.
When basic retrieval is not enough
Simple top-k vector search works well for direct questions over clean prose, but harder tasks may involve multiple documents, exact identifiers, long conversations, tables, or ambiguous terminology. Advanced RAG adds stages that improve query interpretation, candidate recall, evidence ordering, or answer verification. Each stage increases cost and creates another possible failure. Add one only when an evaluation set shows a specific gap. A complicated pipeline without a measured problem is difficult to operate and may make a reliable baseline worse.
Query rewriting
A rewrite can turn a conversational question such as “What about contractors?” into a standalone query that includes the earlier subject. It can also expand abbreviations or produce several subqueries for a multi-part request. Preserve the original question and constraints, and validate that the rewrite did not add an unsupported assumption. For sensitive searches, use deterministic expansion or user confirmation when ambiguity affects permissions or scope. Log both forms so a missed result can be traced to the original wording or to the rewrite stage.
question = "What exceptions apply to contractor leave?"
vector_hits = vector_store.search(question, top_k=20)
keyword_hits = keyword_index.search(question, top_k=20)
candidate_ids = {hit["id"] for hit in vector_hits + keyword_hits}
candidates = [vector_store.get(item_id) for item_id in candidate_ids]
scores = reranker.rank(
query=question,
documents=[item["text"] for item in candidates],
)
for score, item in sorted(
zip(scores, candidates), key=lambda pair: pair[0], reverse=True
)[:5]:
print(score, item["source"], item["text"])Multi-query and decomposition
A complex question may need separate searches for definitions, dates, exceptions, and supporting evidence. Query decomposition can retrieve each part independently before combining results. The application should retain which subquestion produced each chunk and detect when one part has no support. Do not let a planner drop constraints while splitting the task. Set limits on the number of subqueries and total candidates, because decomposition can multiply latency and token usage. For simple questions, bypassing this stage is often faster and just as accurate.
Reranking evidence
A reranker examines the question and candidate passages together and predicts which passages are most useful. It can resolve cases where embedding similarity favors a broad topical match over a precise answer. Retrieve a wider candidate set first, then rerank only that bounded set. Measure whether the reranker improves evidence recall, answer quality, and tail latency on your data. Preserve the initial and final rankings in logs. A reranker cannot recover a document that was never retrieved, and it may introduce its own language, domain, or bias failures.
Parent and hierarchical retrieval
Small chunks are precise, but they may lack surrounding definitions or exceptions. Parent retrieval stores small searchable units while returning a larger parent section or document window after a match. Hierarchical systems can first identify a document, then a section, then a paragraph. These approaches improve context when structure is strong, but they can return too much irrelevant text. Cap the expanded window, preserve section labels, and evaluate whether the added context improves answer completeness without crowding out stronger evidence from other sources.
Handling tables and relationships
Questions about tables, timelines, product variants, or linked policies require more than plain semantic similarity. Store structured rows or fields when possible, preserve headers with every value, and consider deterministic filters or calculations before generation. For a question spanning documents, retain entity and document relationships so retrieval can follow an approved graph rather than guessing from text alone. Let the model explain the result after trusted computation. This reduces errors where a model reads a visually arranged table as if it were ordinary prose.
Conflict and answerability
Advanced systems should detect when sources disagree, are outside their effective dates, or fail to cover the question. Rank authoritative and current sources using trusted metadata, but do not hide a meaningful conflict. The answer can explain which source is newer, cite both, and ask the user to confirm the applicable scope. If evidence is absent, produce a clear abstention with a useful next step. Confidence should reflect evidence quality and coverage, not merely the model’s fluent wording or a high similarity score.
Evaluating advanced designs
Compare each advanced technique with a simpler baseline on the same fixed questions and source versions. Track retrieval recall, reciprocal rank, evidence completeness, citation precision, answer faithfulness, answer correctness, abstention quality, latency, and cost. Include adversarial queries, stale documents, access-control boundaries, and no-answer cases. Review failures by stage and keep examples in regression tests. The best design is the one that improves the product’s measured outcomes within its reliability and budget limits, not the one with the most retrieval components.