Classic RAG retrieves once and answers. Agentic RAG gives a language model retrieval as a tool it can use repeatedly.
How It Works
The model receives the question and tools such as search_documents, search_tickets or query_database. It decides which to call, reads the results, and searches again with refined queries until it has enough to answer.
When It Helps
- Questions needing several pieces of information from different sources.
- Research-style questions where the path isn't known in advance.
- Follow-up searches prompted by what earlier results reveal.
- Choosing between different knowledge sources.
Costs and Risks
- More latency and tokens per question.
- Less predictable behaviour.
- Risk of looping or over-searching.
- Larger surface for prompt injection from retrieved content.
Making It Work
- Give tools clear descriptions and concise outputs.
- Cap the number of steps.
- Ask the model to explain what it's looking for.
- Log every tool call for debugging.
- Evaluate with multi-step questions and compare against simple RAG.
Start Simple
Many "complex" questions are answered fine by one good retrieval step with query rewriting. Use agentic RAG where evaluation shows single-step retrieval falls short.