Agentic RAG: retrieval with a decision loop

Note.The taxonomy here draws on the published survey of agentic RAG and on the workflow-versus-agent framing in Anthropic’s engineering writing. The analysis of when the loop pays and where it fails is ours, and the code is illustrative rather than a library.

What agentic RAG is

The name is doing a lot of work and it helps to unpack it. “RAG” is the base: retrieve some text, put it in the context, generate an answer grounded in it. “Agentic” changes one thing — who decides how the retrieval goes. In classic RAG that is a fixed pipeline you wrote. In agentic RAG it is the model, at runtime.

Versus classic RAG

The retrieval loop

answer = null
rounds = 0
gathered = []

while answer is null and rounds < MAX_ROUNDS:
    query   = model.formulate_query(question, gathered)   # what to search for now
    results = retrieve(query)                              # possibly a chosen source
    gathered += results

    verdict = model.evaluate(question, gathered)          # ← the agentic step
    if verdict.sufficient:
        answer = model.answer(question, gathered)
    elif verdict.no_new_information:
        break                                             # stop; the well is dry
    rounds += 1

# Always return SOMETHING, with a confidence signal — never loop forever.
return answer or model.best_effort(question, gathered, uncertain=True)

The no_new_information exit is the one people forget, and it is what separates a loop that converges from one that thrashes. A model told only to stop when satisfied will, on a question the corpus cannot answer, keep reformulating forever — each round returning the same unhelpful chunks, each round costing a model call. Detecting that a round added nothing is the difference between graceful failure and an unbounded bill.

When the loop earns its cost

The honest test before adopting it: run classic RAG on a representative sample and measure how often one retrieval was enough. If the answer is “usually”, the loop is solving a problem you do not have, at a cost you will notice. Agentic RAG is an answer to a demonstrated single-retrieval failure rate, not a default upgrade.

Failures compound instead of terminating

This is the failure mode specific to putting a loop around retrieval, and it is worth understanding because it is counter-intuitive: the mechanism added to improve robustness introduces a way to get systematically worse.

Q: "Why did invoice 4021 bounce?"

Round 1: query "invoice 4021 bounce"
         → retrieves a doc about EMAIL bounces (wrong 'bounce')
Round 2: model now believes this is about email deliverability
         → query "invoice 4021 email delivery failure"
         → retrieves more email-infrastructure docs
Round 3: confidently answers about SPF records
         → the actual cause (a failed PAYMENT) was never searched for

Classic RAG would have retrieved the payment record in round 1's
top-k and, even if it also surfaced the email doc, generated once
from both. The loop's 'improvement' walked it away from the answer.

The mitigations are specific. Keep the original question in context every round so the model can notice drift. Retrieve a fresh independent result set each round rather than only refining the last query. And have the evaluation step check results against the question, not against the previous results — otherwise it grades consistency, which a wrong path has plenty of.

The cost multiplier

The arithmetic compounds with the context-growth effect. Each round adds its retrieved chunks to the window, so the model calls get more expensive as the loop continues — the third evaluation reasons over everything the first two gathered.

Latency is the cost users feel. Retrieval rounds are serial by nature — you cannot formulate round two’s query until round one’s results are in — so a loop that helps accuracy can hurt the experience enough to matter. For a voice agent this is often disqualifying; for an asynchronous research task it is fine. The right choice depends on whether anyone is waiting.

It cannot fix bad chunking

This is the most important limitation to state because it is the most common misdiagnosis. A team sees poor answers, concludes the retrieval needs to be smarter, and reaches for agentic RAG — when the real problem is upstream, in how documents were split, and no amount of iteration recovers an answer from chunks that never represented it.

The diagnostic is the same one that applies to retrieval generally: take twenty real questions and read the retrieved chunks by hand. If the right passage is present but ranked low, agentic RAG’s reformulation might help. If the right passage is absent from the index entirely, the loop cannot conjure it, and your effort belongs upstream — see choosing the data layer for why recall is a chunking problem first.

Bounding the loop

The fallback deserves emphasis because it changes what failure looks like. Without it, a loop that cannot find the answer either spins or invents one. With it, the loop terminates and says “here is what I found, and I am not confident” — which is a response a downstream system can route to a human. That is the difference between a research tool and a liability.

The through-line: agentic RAG is a real improvement for a real class of questions, and a costly ceremony for the rest. Adopt it because you measured a single-retrieval failure rate you cannot live with, bound it so its failure mode is graceful, and fix your chunking first — because the loop is an amplifier, and an amplifier on a weak signal just gives you louder noise. The broader question of when a loop belongs anywhere is workflow versus agent, and retrieval is one more place to ask it.