Build a custom RAG agent with LangGraph

Build a retrieval (opens in a new tab) agent with LangGraph that decides when to search a vector store versus answering the user directly.

LangChain offers built-in agent (opens in a new tab) implementations built on LangGraph (opens in a new tab) primitives. When you need deeper customization, implement the agent directly in LangGraph. This tutorial walks through one retrieval-agent pattern.

In this tutorial you will:

1. Fetch and preprocess documents for retrieval.
2. Index those documents for semantic search and create a retriever tool for the agent.
3. Build an agentic RAG system that can decide when to use the retriever tool.

Concepts

This tutorial covers the following concepts:

• Retrieval (opens in a new tab) using
  • document loaders (opens in a new tab),
  • text splitters (opens in a new tab), embeddings (opens in a new tab), and
  • vector stores (opens in a new tab)
• The LangGraph Graph API (opens in a new tab), including state, nodes, edges, and conditional edges.

Setup

Install the required packages and set your API keys:

pip install -U langgraph langchain langchain-openai langchain-text-splitters beautifulsoup4 requests

Set up LangSmith

RAG applications run retrieval and generation in sequence. When you run the examples in this tutorial, LangSmith logs a trace for each query so you can inspect retrieval, tool calls, and model responses. After you sign up for LangSmith (opens in a new tab), set your environment variables to start logging traces:

export LANGSMITH_TRACING="true"
export LANGSMITH_API_KEY="..."

Or, set them in Python:

 
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = getpass.getpass()

If you are building a production agent, we also recommend you set up LangSmith Engine which monitors your traces, detects issues, and proposes fixes.

Preprocess documents

Use three posts from Lilian Weng's blog (opens in a new tab). Fetch page content with a minimal helper built on requests and BeautifulSoup.

Split the fetched documents into smaller chunks for indexing into the vector store:

Create a retriever tool

Index the split documents into a vector store for semantic search.

Use an in-memory vector store and OpenAI embeddings:

Create a retriever tool using the @tool decorator:

Generate a query or respond

With the retriever tool ready, start building the agent as a LangGraph graph. In the Graph API (opens in a new tab), a graph is made of:

• State (opens in a new tab): Shared data that nodes read and update. This tutorial uses MessagesState (opens in a new tab), which stores a messages list of chat messages (opens in a new tab).

• Nodes (opens in a new tab): Functions that take the current state, run a step (for example, call a model or a tool), and return state updates.

• Edges (opens in a new tab): Connections that define which node runs next, including conditional edges (opens in a new tab) that branch based on the state.

The first node is the agent decision point. Given the conversation so far, the model either answers the user directly or calls the retriever tool when the question needs blog context. That choice is what makes the system agentic rather than a fixed retrieve-then-generate pipeline: retrieval runs only when the model requests it.

Build a generate_query_or_respond node that calls the model on the current messages and binds the retriever_tool with .bind_tools:

Output:

================================== Ai Message ==================================

Hello! How can I help you today?

Ask a question that requires semantic search:

Output:

================================== Ai Message ==================================
Tool Calls:
retrieve_blog_posts (call_tYQxgfIlnQUDMdtAhdbXNwIM)
Call ID: call_tYQxgfIlnQUDMdtAhdbXNwIM
Args:
    query: types of reward hacking

Grade documents

A normal edge always sends the graph to the same next node. A conditional edge (opens in a new tab) chooses the next node at runtime by running a function over the current state. After retrieval, use that pattern to grade whether the documents are relevant: continue to answer generation if they are, or rewrite the question and try again if they are not.

Add a grade_documents routing function that uses a model with a structured output schema GradeDocuments. It returns the name of the next node based on the grading decision (generate_answer or rewrite_question):

Run this with irrelevant documents in the tool response:

Confirm that relevant documents are classified as such:

Rewrite the question

If the grader marks the retrieved documents as irrelevant, the graph should not answer from that context. Instead, rewrite the original user question into a clearer search query, then send control back to the generate-query-or-respond node so the agent can retrieve again. This retry loop is how the agent recovers from a weak first retrieval instead of stopping or hallucinating an answer.

Build the rewrite_question node to improve the original user question when retrieval misses:

Output:

What are the different types of reward hacking described by Lilian Weng, and how does she explain them?

Generate an answer

When the grader accepts the retrieved documents, the graph moves to answer generation. This node is the classic RAG step: combine the original user question with the tool message that holds the retrieved context, then ask the model to produce a grounded reply. Keep the prompt tight so the model answers from the provided context instead of inventing details.

Build the generate_answer node to produce the final reply from the question and retrieved context:

Output:

================================== Ai Message ==================================

Lilian Weng categorizes reward hacking into two types: environment or goal misspecification, and reward tampering. She considers reward hacking as a broad concept that includes both of these categories. Reward hacking occurs when an agent exploits flaws or ambiguities in the reward function to achieve high rewards without performing the intended behaviors.

Assemble the graph

Assemble the nodes and edges into a complete graph:

• Start with generate_query_or_respond and determine whether to call retriever_tool.
• Route to the next step based on whether the model made tool calls:
  • If generate_query_or_respond returned tool_calls, call retriever_tool to retrieve context.
  • Otherwise, respond directly to the user.
• Grade retrieved document content for relevance to the question (grade_documents) and route to the next step:
  • If not relevant, rewrite the question using rewrite_question and then call generate_query_or_respond again.
  • If relevant, proceed to generate_answer and generate the final response using the @[ToolMessage] with the retrieved document context.

Visualize the graph:

src="/oss/images/agentic-rag-output.png" alt="Agentic RAG graph" style={{ height: "800px" }} />

Run the agentic RAG

Test the complete graph by running it with a question:

See also

• Retrieval (opens in a new tab)
• Graph API (opens in a new tab)
• Agents (opens in a new tab)
• Build a RAG agent (opens in a new tab)
• Build a semantic search engine (opens in a new tab)