Getting Retrieval and Tool Results Into Context Well
Learn how to format, filter, and rank external information before it enters a model's context so it actually improves answers.
More retrieved chunks isn't more accurate
Dumping the top 20 search results into context often produces a worse answer than a well-chosen top 5, because irrelevant chunks dilute attention and can introduce contradictory information. Tune retrieval count and ranking against your own evaluation set rather than assuming more context always helps.
Formatting matters as much as content
How retrieved text and tool outputs are presented — clear source labels, consistent structure, explicit 'this is untrusted external content' framing — affects whether the model uses it correctly and whether it can cite it. A wall of unlabeled text dumped into the prompt is harder for the model to attribute and ground an answer in than the same content with clear boundaries and metadata.
Filtering and re-ranking before injection
Treat retrieval as a two-stage problem: a fast, broad first pass followed by a more careful re-ranking or relevance filter before anything is added to context. This second stage is where you catch near-duplicate chunks, outdated versions of a document, and results that matched on surface similarity but don't actually answer the question.
Tool-result compression
A tool call to a database or API can return far more data than is relevant to the current step — a full customer record when only the shipping address is needed. Summarize or extract before injecting large tool outputs into context, the same way you would trim retrieved documents, rather than passing the raw payload straight into the model's window.
Freshness and conflict handling
When retrieved or tool-provided information conflicts with the model's training knowledge or with earlier context, make the authority explicit: instruct the model to prefer retrieved evidence for facts it names as current, and design retrieval to surface the most recent version of a document rather than an outdated duplicate that happens to rank well.
Practical exercise
Take a RAG or tool-augmented feature you have access to and log the raw context assembled for one real query. Identify the least relevant 30% of what was included, remove or compress it, and re-run the same query. Note whether the answer quality, latency, or cost changed, and use that signal to set a real retrieval budget.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.