What is agentic RAG?
Agentic RAG is retrieval-augmented generation where an AI agent controls the retrieval, rather than a fixed pipeline. Instead of a single "search, then answer" step, the agent decides whether it needs to look something up at all, what to search for, which source to use, and whether the results are good enough to answer from, or whether it should search again with a better query.
The contrast is with classic RAG, which is a straight line: take the question, fetch the most similar chunks from a vector database, paste them in front of the model, and generate one answer. That works well when the answer lives in one place and the question is clear. It falls apart when the question is vague, spans several documents, or needs a follow-up lookup that the first search could not have known about.
Agentic RAG turns that straight line into a loop. The agent treats retrieval as a tool it can call as many times as it needs, the same way an agentic workflow treats any other tool. It can reformulate a weak query, search a second source, compare what it found, and only then commit to an answer.
In plain words
Classic RAG is like handing someone the first three pages a search turned up and asking them to answer from those, no matter what. Agentic RAG is like letting a good researcher work: they read the question, decide what to look for, skim the results, realise the first search missed the point, search again with better words, and only answer once they actually have what they need.
How it works
In an agentic RAG system the retrieval step is no longer hard-coded. The agent runs a short decision loop:
- Decide whether to retrieve. For a simple question the agent may answer directly and skip the lookup entirely. Retrieval costs time and tokens, so not every query earns it.
- Plan the query. The agent rewrites the user's question into one or more focused search queries, splitting a multi-part question into separate lookups instead of one muddy search.
- Choose the source. Real systems have more than one place to look: a docs index, a database, a web search, an internal API. The agent picks the right tool for the question rather than always hitting the same index.
- Judge the results. The agent checks whether what came back answers the question at all. If the chunks are off-topic or thin, it searches again with a different query instead of answering from weak context.
- Answer or iterate. Once the retrieved context is good enough, the agent generates the answer. If not, it loops, with a hard cap on how many times.
Coordinating those steps, especially across several sources or specialised sub-agents, is a question of AI agent orchestration. And deciding what actually ends up in the model's context window on each pass is the core of context engineering: agentic RAG is one of the most direct ways to get the right information in front of the model at the right moment.
Why it matters
- It handles questions a single search can't. Multi-hop questions ("which of our customers in regulated industries renewed last quarter") need more than one lookup. A fixed pipeline retrieves once and guesses; an agent retrieves, reads, and retrieves again.
- It retrieves less garbage. By judging results before answering, the agent avoids the classic RAG failure where irrelevant chunks get pasted in and the model dutifully answers from them. Better context in means fewer confident wrong answers out.
- It can say "I don't know." Because retrieval is a decision, not a reflex, the agent can recognise when no source has the answer and stop, instead of inventing one from whatever the search happened to return.
- It uses the right tool per question. One question wants the product docs, the next wants live data from an API. An agent routes each query to the source that can answer it.
Common pitfalls
- Cost and latency multiply. Every retrieval-and-judge loop is extra model calls plus extra searches. An agent that searches three times to answer one question is three times the latency and cost of classic RAG. Add loops only where one-shot retrieval genuinely fails.
- Loops that never settle. An agent told to keep searching until satisfied can re-query forever on a question with no good answer. Always cap the number of retrieval rounds and the token budget.
- Garbage retrieval still poisons the answer. Agentic control does not fix a bad index, stale documents, or wrong chunking. If the underlying data is weak, the agent just searches weak data more thoughtfully. Retrieval quality still starts with the data.
- The judging step can be wrong too. The agent decides whether results are "good enough" using the same kind of model that can be confidently mistaken. A lenient judge accepts thin context; a strict one loops forever. This needs tuning, not blind trust.
- You can't fix what you can't see. With retrieval happening a variable number of times across variable sources, a wrong answer is hard to trace without logging every query, every source, and every judgement. This is why LLM observability matters as much as the retrieval design itself.
Related terms
- What is RAG (retrieval-augmented generation)? - The foundation agentic RAG builds on: ground a model's answer in retrieved documents.
- What is agentic AI? - The broader idea of AI that decides and acts toward a goal; agentic RAG applies that to retrieval.
- What are agentic workflows? - The multi-step, tool-using loop pattern that agentic RAG is one instance of.
- What is a vector database? - Where the documents agentic RAG retrieves from usually live.
- Context engineering - The wider discipline of getting the right information into the model's context, which agentic RAG serves.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.