You’ll notice in the following code a great deal of similarity to what we’ve seen before. The goal is to create a code “template” that we can follow as we create any agent; departing only for the agent’s special requirements and abilities.

As usual, we start with the factory method:

def create_author_chain():
    """Creates the author chain."""
    def author_invoke(state):
        research = state.get("research_findings", [])
        research_text = "\n\n".join(research) if research else "No research available."

        prompt = author_prompt_template.format(
            main_task=state.get("main_task", ""),
            research_findings=research_text,
            draft=state.get("draft", ""),
            review_notes=state.get("review_notes", "")
        )

        try:
            response = llm.invoke(prompt)
            content = response.content if hasattr(response, 'content') else str(response)
            return content if content else "Draft in progress..."
        except Exception as e:
            print(f"Author error: {e}")
            return "Error generating draft. Please try again."

    return author_invoke

# Creating a callable object
author_chain = create_author_chain()

We start by getting the research findings as a collection. We join all the entries into a single text string, or if there are no findings, we create the string “No research available.”

A local variable, prompt, is created from the author_prompt_template and passed to the invoke method of the LLM. The local variable content will contain the metadata or just the response from the LLM if there is no metadata.

The structure, so far, is identical to what we’ve seen before.

Next, we create the author_prompt_template used above,

author_prompt_template = """
You are a professional blogger.

Main Task: {main_task}

Research Findings:
{research_findings}

Current Draft: {draft}

Review Notes: {review_notes}

Instructions:
- If this is the first draft (no current draft), create a comprehensive post based on the findings
- If there is a current draft and review notes, revise the draft to address all feedback
- Use professional tone
- Make the post consise (aim for 250-500 words)

Write the complete post now:
"""

This is pretty self-explanatory. The interpolation variables (e.g., {draft}) will be filled in when the create_author_chain runs.

Finally, we create the author_node,

def author_node(state: ResearchState) -> dict:
    """Author node that creates or revises draft."""
    print("\n>>>Author")

    draft = author_chain(state)
    print(f"Draft created: {len(draft)} characters")

    return {
        "draft": draft,
        "revision_number": state.get("revision_number", 0) + 1
    }

Reviewer

The reviewer follows the same pattern, so we might as well look at it here. We start with the factory

def create_reviewer_chain():
    """Creates the reviewer chain."""
    def reviewer_invoke(state):
        draft = state.get("draft", "")
        revision_num = state.get("revision_number", 0)

        if len(draft.strip()) < 100:
            return "APPROVED - Draft is minimal but acceptable."

        if revision_num >= 4:
            return "APPROVED - Maximum revisions reached. The report is satisfactory."

        prompt = reviewer_prompt_template.format(
            main_task=state.get("main_task", ""),
            draft=draft
        )

        try:
            response = llm.invoke(prompt)
            content = response.content if hasattr(response, 'content') else str(response)
            return content if content else "APPROVED"
        except Exception as e:
            print(f"Review error: {e}")
            return "APPROVED - Error in review, proceeding with current draft."

    return reviewer_invoke

# Creating a callable object
reviewer_chain = create_reviewer_chain()

The reviewer will either approve the draft or kick it back directly to the author for revision. The only other part here is creating the node for the reviewer.

def reviewer_node(state: ResearchState) -> dict:
    """Node that reviews the draft."""
    print("\n>>REVIEWER")

    review = reviewer_chain(state)
    print(f"Review: {review[:100]}...")

    is_approved = "APPROVED" in review.upper()

    if is_approved:
        print("✓ Draft APPROVED")
        return {
            "review_notes": "APPROVED",
            "next_step": "END"
        }
    else:
        print("✗ Revisions needed")
        return {
            "review_notes": review,
            "next_step": "author"
        }

In the next post we’ll finally see how the nodes are used in creating a workflow.

researcher_prompt_template = """You are a researcher for a technical blog 
focused on .NET and AI with examples in C# and Python

Research Topic: {task}

Your goal is to find relevant, up-to-date insights for developers. Focus on:
- Key trends, challenges, or innovations
- Real-world use cases
- Supporting data or quotes from credible sources
- Simple explanations
- Short code examples in C# or Python

Summarize your findings concisely.
"""

In this template we start by telling the researcher what role it will play. We then provide a goal and narrow that goal to a series of topics to focus on and how to present that data.

def create_researcher_agent():
    """Creates a researcher agent that uses Tavily search."""

    def researcher_invoke(input_dict):
        """Execute research using Tavily search."""
        query = input_dict.get("input", "")

        try:
            search_response = tavily_tool.invoke({"query": query})
            results = search_response.get('results', [])
            formatted_results = []

            if results:
                for result in results[:3]:
                    title = result.get('title', 'Untitled')
                    url = result.get('url', 'N/A')
                    content = result.get('content', '')
                    formatted_results.append(f">>{title}\nSource: {url}\n{content[:300]}...\n")

                raw_output = "\n".join(formatted_results)
            elif not raw_output:
                raw_output = "No results found"

            # Summarize with LLM
            summary_prompt = f"""Based on these search results about '{query}',
            provide a concise summary of key findings:
            {raw_output}
            """

            summary_response = llm.invoke(summary_prompt)
            summary = summary_response.content if hasattr(summary_response, 'content') else str(summary_response)

            return {
                "output": summary if summary else raw_output,
                "input": query
            }

        except Exception as e:
            print(f"Research error: {e}")
            return {
                "output": f"Research completed on: {query}. Key information has been gathered from web sources.",
                "input": query
            }

    return researcher_invoke

researcher_agent = create_researcher_agent()

Using the same pattern we used with the Blogger we create a factory to create the researcher agent. Note that we flag that we’ll be using Tavily for searching the web.

We take the raw information obtained and feed it to the LLM asking for a concise summary. This line:

summary = summary_response.content if hasattr(summary_response, 'content') else str(summary_response)

…attempts to get the content attribute if it exists. Otherwise, it returns the summary_response.

Don’t be confused by the inner and outer methods. Remember, indentation is critical in Python.

Finally, we’ll create the research node, just as we did with Blogger.

def research_node(state: ResearchState) -> dict:
    """Research node that gathers information."""
    print("\n>>>RESEARCHER")

    sub_task = state.get("current_sub_task", state.get("main_task"))
    print(f"Researching: {sub_task}")

    try:
        result = researcher_agent({"input": sub_task})
        findings = result.get("output", "Research completed")
        print(f"Found: {str(findings)[:100]}...")
    except Exception as e:
        print(f"Research error: {e}")
        findings = f"Research on {sub_task} - information gathered"

    return {
        "research_findings": [findings]
    }

In the next post, we’ll examine the Author agent.

This application is written in Python, in a Colab notebook, using (among other things) LangChain and LangGraph. To follow along you will need to obtain an API key from OpenAI and a key from Tavily.

If you are a C# programmer with little or no Python experience, don’t panic! Python is pretty readable, and I’ll explain any part that is potentially obscure or confusing.

This will be a multi-agent application. The agents we’ll create will be:

• Blogger which will orchestrate the others
• Researcher, which will search the web for relevant information
• Author, which will write drafts of the blog post
• Reviewer, which will evaluate the drafts and suggest improvements

As a general rule, I try to limit the number of agents to 3-5. Any more than that can get terribly complicated with diminishing returns. Your mileage may vary.

Set Up

The program begins by importing the necessary libraries.

!pip install -q openai==1.66.3 \
                langchain==0.3.20 \
                langchain-openai==0.3.9 \
                langchain_experimental==0.3.4 \
                langchain-tavily==0.2.4 \
                tavily-python==0.5.0 \
                langgraph==0.3.21 \
                langgraph-supervisor==0.0.18

You then load your OpenAI API key and Tavily API key from either a configuration file (which is what I do here) or from the environment.

import json
import os
from pprint import pprint

file_name = 'config.json'
with open(file_name, 'r') as file:
    config = json.load(file)
    os.environ['OPENAI_API_KEY'] = config.get("API_KEY")
    os.environ["OPENAI_BASE_URL"] = config.get("OPENAI_API_BASE")
    os.environ["TAVILY_API_KEY"] = config.get("TAVILY_API_KEY")

Next, we set up the model.

from langchain_openai import ChatOpenAI

model_name = 'gpt-4o-mini'

llm = ChatOpenAI(
    model = model_name,
    temperature=0,
    max_tokens=4096
)

I’ve opted for the gpt-4o-mini model, as it’s the most cost-effective option. I’ve set the temperature to 0 to get the most consistent results.

Tavily is a tool for searching the web, so let’s set that up as well

from langchain_tavily import TavilySearch

tavily_tool = TavilySearch(
    max_results=5,
    topic="general",
    include_answer=False,
    include_raw_content=False,
    search_depth="basic"
)

You can find all the options for this on the Tavily website.

I’m going to want a state object so that I can pass it among the agents.

from typing import TypedDict, Annotated, List
from langgraph.graph import StateGraph, END
import operator

class ResearchState(TypedDict):
    """State for the research workflow."""
    main_task: str
    research_findings: Annotated[List[str], operator.add]
    draft: str
    review_notes: str
    revision_number: int
    next_step: str
    current_sub_task: str

Note, the line research_findings: Annotated [List(str), operator.add) is a Python type annotation. In short, an annotated list has metadata, in this case the function operator.add. This operator, in this context, is used to update state, though it can also be used to reduce the prompt.

Blogger

To get started, we’ll create the (non-trivial) blogger agent. This is actually the most powerful and thus the most complex of the agents.

We begin with the prompt template (this is, essentially, the system prompt)

blogger_prompt_template = """You are a blogger managing a blog post creation workflow.

Current Task: {main_task}

Current State:
- Research Findings: {research_findings}
- Blog Draft: {draft}
- Reviewer Feedback: {review_notes}
- Revision Number: {revision_number}

Your goal is to ensure a clear, engaging, and valuable blog post targeted at software developers.

Decide the next step and respond only with a JSON object (no extra text):
{
  "next_step": "researcher" or "author" or "END",
  "task_description": "Brief description of what needs to be done next"
}

Decision Rules:
- If no research exists, choose "researcher"
- If research exists but no draft, choose "author"
- If draft exists and reviewer says "APPROVED", choose "END"
- If draft needs revision, choose "author"
- If revision_number >= 4, choose "END"
"""

Take a moment to read this over; it sets the parameters and goals of the program. The decision rules are critical, they control the flow, and they set a limit on revisions (in this case 4).

Having told the Blogger what we are trying to accomplish and the general tone of the output we’re ready to create the decision tree that constitutes the workflow for the Blogger. This is pretty long, but much of it is self-explanatory.

def create_blogger_chain():
    """Creates the bloger decision chain."""

    def blogger_invoke(state):
        research = state.get("research_findings", [])
        research_text = "\n".join(research) if research else "No research yet."
        revision = state.get("revision_number", 0)
        has_research = len(research) > 0
        has_draft = bool(state.get("draft", "").strip())
        review = state.get("review_notes", "")

        if "APPROVED" in review.upper() and has_draft:
            print("Blogger: Draft approved, ending workflow")
            return {
                "next_step": "END",
                "task_description": "Report approved and complete"
            }

        if not has_research:
            print("Blogger: No research yet, directing to researcher")
            return {
                "next_step": "researcher",
                "task_description": f"Research the topic: {state.get('main_task', '')}"
            }


        if has_research and not has_draft:
            print("Blogger: Have research, creating first draft")
            return {
                "next_step": "author",
                "task_description": "Write the first draft based on research findings"
            }

        if has_draft and not review:
            print("Blogger: Have draft, sending to reviewer")
            return {
                "next_step": "reviewer",
                "task_description": "Prepare draft for review"
            }

        if review and "APPROVED" not in review.upper() and revision <= 4:
            print(f"Blogger: Revision {revision}, sending back to author")
            return {
                "next_step": "author",
                "task_description": "Revise the draft based on review feedback"
            }

        if revision >= 4:
            print("Blogger: Max revisions reached! Ending")
            return {
                "next_step": "END",
                "task_description": "Maximum revisions reached! Finalizing report"
            }

        # LLM as fallback
        prompt = blogger_prompt_template.format(
            main_task=state.get("main_task", ""),
            research_findings=research_text,
            draft=state.get("draft", "No draft yet."),
            review_notes=review if review else "No review yet.",
            revision_number=revision
        )

        try:
            response = llm.invoke(prompt)
            content = response.content if hasattr(response, 'content') else str(response)

            text = content.strip()
            if text.startswith("```"):
                lines = text.split("\n")
                text = "\n".join([l for l in lines if not l.strip().startswith("```")])
            text = text.strip()

            decision = json.loads(text)

            if "next_step" in decision:
                return decision

        except Exception as e:
            print(f"LLM parsing error: {e}")

        # Final fallback 
        print("Blogger: Using final fallback - continuing with author")
        return {
            "next_step": "author",
            "task_description": "Continue with draft creation"
        }

    return blogger_invoke

# Creating a callable object
blogger_chain = create_blogger_chain()

We begin with a nested function.

def create_blogger_chain():
    """Creates the blogger decision chain."""

    def blogger_invoke(state):
        ...

When called, Python executes the code inside it, that is, the inner function. In this case, the inner function does the real work.

We define a second function inside create_blogger_chain. We do this to create a closure, that is the inner function can access variables from the outer function. This is a common construct when working with langchain. The inner function has access to the llm without it being passed every time. So, in short, create_blogger_chain is actually a factory function that constructs and configures a callable function and then returns it.

We next set up our variables based on starting values in the state object. With that done, we’re ready to progress through a series of possible conditions. These follow the rules established in the template.

All that’s left for the blogger is to create a node where the decision will be implemented. We’ll use this node, and the others we’ll create, when we implement the workflow (after we define all the agents).

def blogger_node(state: ResearchState) -> dict:
    """Blogger decides the next step."""
    print("\n>>>Blogger")

    decision = blogger_chain(state)

    next_step = decision.get("next_step", "researcher")
    task_desc = decision.get("task_description", "Continue work")

    print(f"Decision: {next_step}")
    print(f"Task: {task_desc}")

    return {
        "next_step": next_step,
        "current_sub_task": task_desc,
    }

Note The variable decision is assigned as a result of calling the code we just reviewed. With that in hand, we call get on the decision, asking for the next step. If none is returned, we use researcher.

That’s it for blogger. We’ll review the other agents in the next posting.

Draft created: 2653 characters
{‘author’: {‘draft’: ‘# Harnessing Multi-Agent Systems in C# Applications\n’
‘\n’
‘In the evolving landscape of software development, ‘
‘multi-agent systems (MAS) have emerged as a powerful ‘
‘paradigm, particularly in enhancing the functionality of ‘
‘applications. However, the integration of these systems ‘
‘into C# applications comes with its own set of ‘
‘challenges and considerations. This post explores the ‘
‘key aspects of implementing multi-agent systems in C#, ‘
‘drawing from recent research findings.\n’
‘\n’

‘## Understanding Multi-Agent Systems\n’
‘\n’
‘At its core, a multi-agent system consists of multiple ‘
‘autonomous agents that interact with one another to ‘
‘achieve specific goals. These agents can be designed to ‘
‘perform tasks collaboratively, leading to improved ‘
‘efficiency and problem-solving capabilities. However, as ‘
‘highlighted by Elliot One, simply increasing the number ‘
‘of agents does not guarantee better outcomes. In fact, ‘
‘it can complicate the debugging process, making it more ‘
‘difficult to trace failures and understand system ‘
‘behavior. This underscores the importance of thoughtful ‘
‘design and implementation when developing multi-agent ‘
‘systems.\n’
‘\n’

‘## Multi-Agent Architecture in .NET\n’
‘\n’
‘For developers looking to implement multi-agent systems ‘
‘in C#, the Microsoft Agent Framework provides a robust ‘
‘foundation. A recent tutorial video introduces the ‘
‘concept of multi-agent orchestration and workflows, ‘
‘offering insights into how these systems can be ‘
‘structured within .NET applications. This resource is ‘
‘invaluable for developers seeking to grasp the ‘
‘architectural considerations necessary for effective ‘
‘multi-agent implementation.\n’
‘\n’
‘## Practical Implementation\n’
‘\n’
‘Building a multi-agent system requires a structured ‘
‘approach. A Codelabs resource outlines a step-by-step ‘
‘guide for developers interested in practical ‘
‘implementation. This guide covers prerequisites, ‘
‘essential components, and best practices for creating a ‘
‘functional multi-agent system. By following this ‘
‘structured methodology, developers can mitigate common ‘
‘pitfalls and enhance the overall quality of their ‘
‘applications.\n’
‘\n’
‘## Conclusion\n’
‘\n’
‘While multi-agent systems hold significant potential for ‘
‘enhancing C# applications, developers must navigate the ‘
‘complexities associated with their implementation. By ‘
‘leveraging available resources and adhering to best ‘
‘practices, it is possible to create robust multi-agent ‘
‘systems that improve application functionality without ‘
‘succumbing to the common challenges of increased ‘
‘complexity. As the field continues to evolve, staying ‘
‘informed and educated on the latest developments will be ‘
‘crucial for developers aiming to harness the full power ‘
‘of multi-agent systems in their applications.’

————————————————————
>>REVIEWER Review: APPROVED – The draft effectively introduces the concept of multi-agent systems in C# applications an… ✓ Draft APPROVED {‘reviewer’: {‘next_step’: ‘END’, ‘review_notes’: ‘APPROVED’}}
————————————————————
>>>Blogger Blogger: Draft approved, ending workflow Decision: END Task: Report approved and complete {‘blogger’: {‘current_sub_task’: ‘Report approved and complete’, ‘next_step’: ‘END’}}

We take this apart beginning in part 2.

In order to interact with the world, the agent will use tools (such as code that accesses APIs, searches the Internet, etc.). This creates a dynamic cycle:

Think—the LLM reasons and decides what tool to use
ACT—the LLM uses the tool to take action in the world
Observe—the LLM observes the result of the action and adjusts accordingly, refining its plan

The cycle ends when the LLM has its final answer.

Prompts for ReAct

The prompt must specify the available tools and their descriptions. It will instruct the LLM to use the cycle above and provide examples of the cycle. Finally, it will constrain the output into a machine-readable format.

The prompt must be very clear and precise. The basic template is to tell the agent what it is (“you are a medical assistant”), what tools it has access to, how to use the tool, what the returned observation will look like (structured data such as JSON) and how to provide the interim and final answers.

The output after each tool use is machine-readable so that it can be fed back into the cycle or to a tool or internal prompt.

With CoT (Chain of Thought), the LLM was limited to its existing knowledge and could not deviate from its initial planning. With ReAct the LLM can adapt to the observed results and use tools to extend its knowledge. If an error is encountered, the LLM can adjust its plan to avoid the failure (e.g., using a different tool).

We can tell the LLM to output its plan and observations as it goes. This can be enormously helpful for debugging.

When we look at the “think” phase, we return to the use of CoT. It is here that an initial plan is generated and the LLM uses the tools to execute the plan after which the LLM will observe the results and use them to adjust the plan.

Interacting with the tools brings us to the Model Context Protocol (MCP), a topic for an upcoming blog post.

Recent research has shown that LLMs are, in fact, quite good at reasoning and planning if the problem is broken into a series of steps as a result of the right prompts. This reasoning and planning greatly improves the accuracy of the LLM’s output.

Chain of Thought

One of the great breakthroughs in the field of LLMs was the discovery that appending “think step by step” to the prompt had a profound effect on the accuracy of the LLMs output. Prompting the LLM in this way forces the model to decompose the problem and to generate an “inner monologue” of the steps it is taking to solve the problem.

This has two significant advantages:

• It makes the LLM’s reasoning process more transparent
• It allows the model to check its own work as it goes.

For example, we might ask, “If a plane crashes on the border of the north field and the south field, where will the survivors be buried?”

Without the reasoning prompt, it is entirely possible that the LLM would pick one of the fields at random. However, if we tell it to think step-by-step, it will examine the parts of the question and realize that survivors are not buried at all.

Note that today’s LLMs have a degree of Chain-of-Thought built into them and won’t get this wrong.

There are three key aspects to explain why chain-of-thought reasoning works:

• Decomposing the problem into smaller intermediate steps
• CoT offers the model the ability to keep track of its work and to remember intermediate results.
• Typically, more tokens are allocated to the reasoning, and thus the model can “think” longer.

Debugging

Because we’ve asked the LLM to think step-by-step, it can tell us each step in its reasoning, and we can examine those steps to see when the LLM fell of the rails. This takes a process that might otherwise be opaque and makes it transparent, greatly enhancing the debugging process.

Next up: ReAct…

• Performance
• Environment
• Actuators
• Sensors

Performance

Performance defines success for our agent (the objective and measurable criteria for evaluating the agent’s behavior). A good performance measure will evaluate the state of the environment, not the agent’s internal state.

Designing for performance is typically the hardest part of designing an agent. Thinking deeply about what you want to accomplish, before you start coding, is the key to creating a successful agent.

For example, a naïve performance measure for an automated vehicle might be “Get me to my destination.” A more robust performance measure might include:

• Speed
• Comfort
• Fuel consumption
• Following traffic laws
• Safety

Some of these conflict (e.g., speed v safety)

To ensure that performance matches your goals, you will need to tradeoff values. You might end up with something like this:

• Value safety over speed
• Value speed over comfort
• Value fuel consumption over speed
• Value comfort over fuel consumption
• Value traffic laws over comfort or fuel consumption

Even this is a bit simplistic. Another approach is to give weights to the various factors and see how they balance out. As you can see, deep thinking is required to get this right.

Environment

This refers to the “world” the agent lives in—that is, everything external to the agent. In the case above the roads, streets, traffic lights, pedestrians, etc., constitute the environment.

Actuators

These are the things that change the environment. In our self-driving car these are the steering wheel, the brakes, and the accelerator. In a robot these might be legs, arms, and fingers. And, most relevant to most of us, in code these are methods that call APIs, functions that change values, etc.

Sensors

Sensors are how the agent perceives its environment. Like humans, the only way the agent can know about the world it moves in is through its sensors. For our self-driving car this might include

• GPS
• LiDAR
• Cameras
• Tire pressure sensors

The simple search is a good starting point and depends on writing a good prompt, but we can do better.

1. One problem in searching is that we often retrieve redundant data. For example, the basic search might retrieve ten very similar chunks. Maximal Marginal Relevance first builds a set of results that are relevant but also diverse. It then iterates through the relevant documents looking for ones that are as different as possible. As you might expect, this gives you a diverse set of relevant documents.

2. You will remember that we stored metadata with our documents. This gives us a very powerful tool in searching. We can search for documents based on the metadata, and then within those results, we can search semantically.

For example, we might say, “Give me documents that were created in the past month,” and then from that reduced set we can ask for those that provide information on sales. This is faster and more accurate than other search methods and, of course, can be combined with search methods that reduce redundancy.

3. Let the LLM improve your search. The LLM can rewrite your prompt to be more effective, which will get you a better set of results.

4. Interestingly, the order of returned documents matters. After you get your initial results, reorder them with the most important documents at the very beginning or very end (the middle tends to get lost!) You can then search this subset for your most relevant answers.

• Load your source documents, being careful to keep the meta-data
• Split your document into semantically meaningful chunks
• Convert your text to vector representations in an embedding model
• Store your vector representations in a vector database
• Retrieve the data you need

The first step is to load your document. Here you are taking the raw content and its metadata (e.g., creation date, etc.) and loading it into your application. Next, it is time to split the text, which is called chunking. It turns out that this is critical to creating a useful and fast RAG. Chunks are the atomic units used for retrieval, and to do this well, you’ll want to break your large documents into smaller semantically meaningful pieces.

Here the Goldilocks approach is critical. You don’t want your chunk to be so small that it has no context, nor do you want it to be so big that it covers more than one concept.

There are a few strategies for creating hunks. A simple example is to split your document by paragraphs. If that is impossible (the paragraphs are too long, etc.) you split by sentences, and failing that by lines.

If you are lucky enough to have structured data (e.g., headings, hierarchy, outline, etc.), then the splitter can use that structure to create chunks. If you are working with code, you might have the splitter chunk by classes, etc.

The next step is embedding. This is the most conceptually challenging task, though the tools will do the work for you. Conceptually you are creating a multi-dimensional map where the distance between the various chunks corresponds to how similar they are. This enables searching by semantic meaning rather than just keywords.

Notice that “speaking” and “speech” are close to each other, while “dog” and “keyboard” are further apart.

There are a number of embedding tools, some proprietary and some open source. The open-source models may be harder to set up, but they are free and can run locally.

Finally, we need to store the vectors in a vector database. The vectors are indexed, and searching typically uses Approximate Nearest Neighbor, taking advantage of the multi-dimensional model we created above.

That covers the ingestion workflow. Next is retrieval, which I’ll leave for the next blog post.

The key steps in RAG are

• Load the data
• Split the text into smaller chunks to fit within context limits
• Create a Document object
• Embed the document in vectors that represent semantic meaning
• Store the document—typically in vector stores. These are databases designed to store embeddings and provide fast semantic retrieval
• Invoke a retriever to query the back end to return the most relevant Document object
• Create a prompt for the LLM

Let’s walk through the steps shown in the previous post with these in mind.

Loading the document

First, we need to identify and load the documents. In our case, this consists only of a single text file with an excerpt from Romeo and Juliet. In most real-world scenarios you’ll have multiple data sources.

from langchain_community.document_loaders import TextLoader
loader = TextLoader("RomeoAndJuliet.txt", encoding="utf-8")
docs = loader.load()

Notice that we are using the langchain_community document loader to do the text loading. Langchain will be the principal framework we’ll be working with, and it can load many types of data.

Splitting the text

We saw how to chunk that data in the previous post. We begin by using a text splitter to break large text into overlapping chunks using token-based splitting (not characters). In our case, we will set each chunk to about 1,000 tokens with 200 tokens of overlap. The overlap ensures that nothing is lost.

from langchain.text_splitter import RecursiveCharacterTextSplitter 
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name='cl100k_base',
    chunk_size=1000,
    chunk_overlap=200
)
chunks = loader.load_and_split(text_splitter)

The text splitter that we use is the same one OpenAI uses. The cl100k_base is the tokenizer used by many embedding models. The 200 token overlap prevents losing meaning at the boundaries of the chunks and helps embeddings preserve context.

We use a recursive text splitter because it splits text intelligently, splitting by paragraphs when possible, then by sentences if the paragraphs are too big, then by words and finally by characters.

Embedding in a vector store

embedding_model = OpenAIEmbeddings(model="text-embedding-ada-002")

The embedding_model knows how to take text and send it to OpenAI and get back a vector embedding (a list of numbers). Each chunk you pass into Chroma (see below) will be embedded using this model

Our next task is to build the vector store, using the chunks we created above

vectorstore = Chroma.from_documents(
    chunks,
    embedding_model,
    collection_name="RomeoAndJuliet"
)

Here we embed each of the chunks. For each chunk Chroma calls embedding_model.embed_document which produces the vector

For each chunk, Chroma will store the vector embedding, the original text and the metadata such as the source file, etc. This is used for similarity search (see below).

The final value passed in is the collection_name. The vector store is saved under that name.

Getting the retriever

As noted in the previous post, the next step is to create the retriever, which we do from the vector store, telling it that we want the search_type to be similarity and telling it how many of the most relevant chunks to return.

You get back a LangChain Document with the text chunk and the metadata.

Instantiating the LLM

The next section in the previous post is self-explanatory until we instantiate the LLM.

llm = ChatOpenAI(
    model="gpt-4o-mini",                      
    temperature=0,                
    max_tokens=10000,                 
    top_p=0.95,
    frequency_penalty=1.2,
    stop_sequences=['INST']
)

Here we are using the OpenAI gpt-40-mini LLM – a popular and inexpensive LLM for RAG.

We set the temperature, which is a value that determines randomness in the answer. 0 is deterministic and repeatable.

max_tokens sets the upper bounds on how long the model’s response can be.

top_p=0.95 is tricky. This says that the model should sample from the top 95% probability. However, with temperature set to 0, this is meaningless. If you tinker with temperature, however, this can be useful.

frequency_penalty controls how often a token can repeat in the result. We’re using 1.2 which is a strong penalty creating concise, non-repetitive answers.

stop_sequence says to stop generation when the model outputs INST. This just prevents the model from “leaking” into the next instruction.

That’s it! Together with the previous post, you are now fully equipped to implement your RAG. Enjoy!