The simple search is a good starting point and depends on writing a good prompt, but we can do better.

1. One problem in searching is that we often retrieve redundant data. For example, the basic search might retrieve ten very similar chunks. Maximal Marginal Relevance first builds a set of results that are relevant but also diverse. It then iterates through the relevant documents looking for ones that are as different as possible. As you might expect, this gives you a diverse set of relevant documents.

2. You will remember that we stored metadata with our documents. This gives us a very powerful tool in searching. We can search for documents based on the metadata, and then within those results, we can search semantically.

For example, we might say, “Give me documents that were created in the past month,” and then from that reduced set we can ask for those that provide information on sales. This is faster and more accurate than other search methods and, of course, can be combined with search methods that reduce redundancy.

3. Let the LLM improve your search. The LLM can rewrite your prompt to be more effective, which will get you a better set of results.

4. Interestingly, the order of returned documents matters. After you get your initial results, reorder them with the most important documents at the very beginning or very end (the middle tends to get lost!) You can then search this subset for your most relevant answers.

• Load your source documents, being careful to keep the meta-data
• Split your document into semantically meaningful chunks
• Convert your text to vector representations in an embedding model
• Store your vector representations in a vector database
• Retrieve the data you need

The first step is to load your document. Here you are taking the raw content and its metadata (e.g., creation date, etc.) and loading it into your application. Next, it is time to split the text, which is called chunking. It turns out that this is critical to creating a useful and fast RAG. Chunks are the atomic units used for retrieval, and to do this well, you’ll want to break your large documents into smaller semantically meaningful pieces.

Here the Goldilocks approach is critical. You don’t want your chunk to be so small that it has no context, nor do you want it to be so big that it covers more than one concept.

There are a few strategies for creating hunks. A simple example is to split your document by paragraphs. If that is impossible (the paragraphs are too long, etc.) you split by sentences, and failing that by lines.

If you are lucky enough to have structured data (e.g., headings, hierarchy, outline, etc.), then the splitter can use that structure to create chunks. If you are working with code, you might have the splitter chunk by classes, etc.

The next step is embedding. This is the most conceptually challenging task, though the tools will do the work for you. Conceptually you are creating a multi-dimensional map where the distance between the various chunks corresponds to how similar they are. This enables searching by semantic meaning rather than just keywords.

Notice that “speaking” and “speech” are close to each other, while “dog” and “keyboard” are further apart.

There are a number of embedding tools, some proprietary and some open source. The open-source models may be harder to set up, but they are free and can run locally.

Finally, we need to store the vectors in a vector database. The vectors are indexed, and searching typically uses Approximate Nearest Neighbor, taking advantage of the multi-dimensional model we created above.

That covers the ingestion workflow. Next is retrieval, which I’ll leave for the next blog post.

The key steps in RAG are

• Load the data
• Split the text into smaller chunks to fit within context limits
• Create a Document object
• Embed the document in vectors that represent semantic meaning
• Store the document—typically in vector stores. These are databases designed to store embeddings and provide fast semantic retrieval
• Invoke a retriever to query the back end to return the most relevant Document object
• Create a prompt for the LLM

Let’s walk through the steps shown in the previous post with these in mind.

Loading the document

First, we need to identify and load the documents. In our case, this consists only of a single text file with an excerpt from Romeo and Juliet. In most real-world scenarios you’ll have multiple data sources.

from langchain_community.document_loaders import TextLoader
loader = TextLoader("RomeoAndJuliet.txt", encoding="utf-8")
docs = loader.load()

Notice that we are using the langchain_community document loader to do the text loading. Langchain will be the principal framework we’ll be working with, and it can load many types of data.

Splitting the text

We saw how to chunk that data in the previous post. We begin by using a text splitter to break large text into overlapping chunks using token-based splitting (not characters). In our case, we will set each chunk to about 1,000 tokens with 200 tokens of overlap. The overlap ensures that nothing is lost.

from langchain.text_splitter import RecursiveCharacterTextSplitter 
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name='cl100k_base',
    chunk_size=1000,
    chunk_overlap=200
)
chunks = loader.load_and_split(text_splitter)

The text splitter that we use is the same one OpenAI uses. The cl100k_base is the tokenizer used by many embedding models. The 200 token overlap prevents losing meaning at the boundaries of the chunks and helps embeddings preserve context.

We use a recursive text splitter because it splits text intelligently, splitting by paragraphs when possible, then by sentences if the paragraphs are too big, then by words and finally by characters.

Embedding in a vector store

embedding_model = OpenAIEmbeddings(model="text-embedding-ada-002")

The embedding_model knows how to take text and send it to OpenAI and get back a vector embedding (a list of numbers). Each chunk you pass into Chroma (see below) will be embedded using this model

Our next task is to build the vector store, using the chunks we created above

vectorstore = Chroma.from_documents(
    chunks,
    embedding_model,
    collection_name="RomeoAndJuliet"
)

Here we embed each of the chunks. For each chunk Chroma calls embedding_model.embed_document which produces the vector

For each chunk, Chroma will store the vector embedding, the original text and the metadata such as the source file, etc. This is used for similarity search (see below).

The final value passed in is the collection_name. The vector store is saved under that name.

Getting the retriever

As noted in the previous post, the next step is to create the retriever, which we do from the vector store, telling it that we want the search_type to be similarity and telling it how many of the most relevant chunks to return.

You get back a LangChain Document with the text chunk and the metadata.

Instantiating the LLM

The next section in the previous post is self-explanatory until we instantiate the LLM.

llm = ChatOpenAI(
    model="gpt-4o-mini",                      
    temperature=0,                
    max_tokens=10000,                 
    top_p=0.95,
    frequency_penalty=1.2,
    stop_sequences=['INST']
)

Here we are using the OpenAI gpt-40-mini LLM – a popular and inexpensive LLM for RAG.

We set the temperature, which is a value that determines randomness in the answer. 0 is deterministic and repeatable.

max_tokens sets the upper bounds on how long the model’s response can be.

top_p=0.95 is tricky. This says that the model should sample from the top 95% probability. However, with temperature set to 0, this is meaningless. If you tinker with temperature, however, this can be useful.

frequency_penalty controls how often a token can repeat in the result. We’re using 1.2 which is a strong penalty creating concise, non-repetitive answers.

stop_sequence says to stop generation when the model outputs INST. This just prevents the model from “leaking” into the next instruction.

That’s it! Together with the previous post, you are now fully equipped to implement your RAG. Enjoy!

In this blog post we’ll use Retrieval-Augmented Generation (RAG) to extend an LLM that we’ll get from OpenAI. I’ll use a number of features from the libraries we imported with only a cursory explanation and will come back to them in upcoming blog posts to examine them in more depth. But I want to get to RAG right away because it is rapidly becoming central to AI and because it is cool.

LLMs are incredibly expensive to create and train, and it isn’t feasible to train them on everything. Besides that, much data is proprietary. It may be that you want an LLM that handles (to use the canonical case) your HR policies. Clearly no commercial LLM knows about those policies, nor should they. And equally clearly, you’re not going to train an LLM from scratch. What you want to do is to combine your own corpus of data (HR policy papers, etc.) with an existing LLM, and that is exactly what RAG is for.

In this simple example, we’re going to take a scene or two from Romeo and Juliet and feed it to gpt-40-mini; one of many LLMs available for use at minimal cost (we’ll get into how cost is computed in an upcoming post).

The first thing we’ll do after configuring the OPEN_API_KEY will be to get a TextLoader to import the text file with the scenes from Romeo and Juliet

To do that, we’ll use the TextLoader from langchain_community.document_loaders (again, we’ll examine this and the other referenced modules in upcoming blog posts). We do this in three steps:

1. load the import statement
2. Point the TextLoader to our file
3. Load the file

from langchain_community.document_loaders import TextLoader
loader = TextLoader("RomeoAndJuliet.txt", encoding="utf-8")
docs = loader.load()

Next, we need to divide the text into chunks that the LLM can work with. We do that with a RecursiveCharacterTextSplitter from langchain. We’ll use the cl100k_base encoder, and we’ll set the chunk_size to 1000 (that is 1,000 of those mysterious tokens that, e.g., words are divided into). To ensure that nothing is dropped, we set a property, chunk_overlap to 200.

from langchain.text_splitter import RecursiveCharacterTextSplitter 
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name='cl100k_base',
    chunk_size=1000,
    chunk_overlap=200
)
chunks = loader.load_and_split(text_splitter)

In this particular example, we get six chunks,

len(chunks)
6

By now you are getting annoyed that so much is going by that I’m not explaining. As promised, however, all will be clarified in the next blog post. In fact, we’ll go back through this line by line and explain what each step is doing. But for now, let’s continue…

We need an embedding model which we’ll use to create our vector store (the place we hold onto our chunks) As an aside, the other things held in the vector store are the metadata about each chunk and vectors which are numerical embeddings of the chunks. The vectors are actually the key part of this, they are a long list of numbers that represent the semantic meaning of each chunk which can be used, and will be used below, in a similarity search.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
embedding_model = OpenAIEmbeddings(model="text-embedding-ada-002")
vectorstore = Chroma.from_documents(
    chunks,
    embedding_model,
    collection_name="RomeoAndJuliet"
)

Now that we have the vector store, we need a way to conduct the search, for which we need a retriever. When we instantiate it, we’ll tell the retriever to use a similarity search.

retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 10}
)

The kwargs are additional keyword-arguments you can pass in. In this case, we’re telling the retriever to get 10 results.

Now! We are ready to create our user message and our retrieval Query. A retrieval query is the search query you send to your vector store to fetch relevant chunks. The user message is a natural-language question—it’s the optimized text you give to the retriever so it can find the right documents.

userMessage = "Give me every line having the word swear in it" 
retrievalQuery = "the play 'Romeo and Juliet'"

We can now extract the relevant chunks, iterate through them and create a long string of the resulting context chunks.

relevantChunks = retriever.invoke(retrievalQuery)
contextChunks = [d.page_content for d in relevantChunks]
contextString = ". ".join(contextChunks)

We need to give the LLM context to work from. One great way to do that is to assign a role to the LLM (e.g., “you are a human resource assistant”). In this case, we’ll use a reviewer who knows about plays. This is also a good place to provide explicit directions on how you want the LLM to respond.

qna_system_message = """
You are a play reviewer using the RAG to combine the text of the play with your knowledge of plays in general.
You will review RomeoAndJuliet.txt and provide appropriate answers from the context.
The user input will have the context required by you and will begin with the token: ###Context.
The user questions will begin with the token: ###Question.
Please answer only using the context provided and do not mention anything about the context in your answer.
If the answer is not found in the context, respond "I don't know."
"""

We just need a way to tell the LLM how the context and question will appear, for which we create a template.

qna_user_message_template = """
###Context
{context}

###Question
{question}
"""

Let’s create the final userQuery by combining the context with the user message we created above.

userQuery = qna_user_message_template.format(
    context=contextString,
    question=userMessage
)

Finally, we’re ready to create the prompt that we’ll feed to the LLM

prompt = f"""
[INST]{qna_system_message}

{userQuery}
[/INST]
"""

Next, we instantiate our LLM filling in some parameters that, again, we’ll review in an upcoming blog post

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    model="gpt-4o-mini",                      
    temperature=0,                
    max_tokens=10000,                 
    top_p=0.95,
    frequency_penalty=1.2,
    stop_sequences=['INST']
)

And we are now, at last, ready to feed our prompt to the LLM which will incorporate the RAG we created from the Romeo and Juliet text. Remember that we asked it to give us the lines with the word swear in it

response = llm.invoke(prompt)
response.content

ROMEO.  
O, then, dear saint, let lips do what hands do:  
They pray, grant thou, lest faith turn to despair.  

JULIET.  
Saints do not move, though grant for prayers’ sake.  

ROMEO.  
Then move not while my prayer’s effect I take.  
Thus from my lips, by thine my sin is purg’d.

JULIET.  
O swear not by the moon, th’inconstant moon,  
That monthly changes in her circled orb,  
Lest that thy love prove likewise variable.

ROMEO.  
What shall I swear by?

JULIET.   
Do not swear at all.   
Or if thou wilt, swear by thy gracious self,
Which is the god of my idolatry,
And I’ll believe thee.

ROMEO.
If my heart’s dear love,—

One thing to note is that the LLM interpreted “swear” liberally. For example, in the first verse Romeo says “They pray,” which is pretty close to “swear.”

We need a method to ask more questions. Let’s create a method, that takes a user message, chunks it, creates the prompt and invokes the LLM with that prompt

def UseRag(userMessage):
    """
    Args:
    userMessage: Takes a user input for which the response should be retrieved from the vectorDB.
    Returns:
    relevant context as per user query.
    """
    chunks = retriever.invoke(userMessage)
    contextContent = [d.page_content for d in chunks]
    contextString = ". ".join(contextContent)

    prompt = f"""[INST]{qna_system_message}\n
                {'user'}: {qna_user_message_template.format(context=contextString, question=userMessage)}
                [/INST]"""

    # Quering the LLM
    try:
        response = llm.invoke(prompt)

    except Exception as e:
        response = f'Sorry, I encountered the following error: \n {e}'

    return response.content 

To prove that we’re getting our answers from the RAG, let’s ask a question about text that is not in our excerpt but that would be known by anyone (anything?) that is familiar with the play.

print(UseRag("What town does Romeo live in?"))

I don't know.

Finally, let’s have a bit of fun,

UseRag("Write a 10 line poem in the style of 'Romeo and Juliet'")

In shadows deep where whispered secrets lie,  
Two hearts entwined beneath the moonlit sky.  
A glance exchanged, a spark ignites the night,  
Forbidden love that dances out of sight.  

O sweet Juliet, with beauty rare and bright,  
Your name a curse yet brings my soul delight.  
Though feuding kin may seek to tear apart,  
Our love shall bloom within each beating heart.  

For in this world of strife and bitter woe,  
Together we shall rise; our passion's glow.  

I think that is actually pretty good.

OK, that was a lot, and it went by fast. I look forward to going back through it, line by line, and exploring what each line is doing.

The first step is to get an OpenAI key, as described in this previous blog post (scroll to the bottom). Note, these are not free, but we’re talking a few dollars for this entire series of posts.

With that in hand, create a config.json file that looks like this:

{
    "API_KEY": "gl-U2FsdGVkX1/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/MA8peF",
}

Next, open a browser to colab.research.google.com. Create a new notebook.

Click on the folder icon on the left, and when you see a number of folders, drag your config file to that window…

We’ll need some modules, so in the first cell enter:

!pip install langchain==0.3.25 \
                langchain-core==0.3.65 \
                langchain-openai==0.3.24 \
                chromadb==1.3.4 \
                langchain-community==0.3.20 \
                pypdf==5.4.0

A lot of lines will follow as the correct modules are loaded, and then you will be instructed to restart the runtime. Do this once.

Now let’s add that config file. In the next cell enter

import json
import os

# Load the JSON file and extract values
file_name = "config.json"
with open(file_name, 'r') as file:
    config = json.load(file)
    os.environ['OPENAI_API_KEY'] = config.get("API_KEY") # Loading the API Key
 

After you run it (click on the triangle) You will see a green check mark next to the cell (on the left) and the time it took to run your command.

You are all set! In the next blog post we’ll instantiate our first LLM

Podcast
Video

The podcast is here and the video is here.

• Building from first principles based on my course on Agentics at Johns Hopkins
• Building using Microsoft’s new Agent Framework

The advantage of the first is that you understand the underlying mechanisms in more depth; the advantage of the second is that a lot of the plumbing is done for you and you become more productive more quickly.

I will do the .NET work in C#, and probably do all the other work in Python. See my blogpost on why Python.

I will, to a degree, be documenting what I learn as I learn it, without infringing on copyright, of course.

Project 1 – Jupyter

The work for Johns Hopkins is done in a Jupyter notebook. These are very convenient files that contain runnable cells. You put your code snippet in a cell and run it, either receiving a result or extensive error information. To get started, open Visual Studio Code and click New File. In the drop down on top, choose Jupyter Notebook:

Choose your Python environment, and put code into the first “cell.” Then click on the knob on the left and the cell runs and the output is displayed:

You can add markdown, you can even tell it to generate code based on your description of what you want. It is possible to have these cells generate a proper Python program and/or generate

Open AI Endpoint

Whether you are working in the Jupyter notebook or in Agent Framework, you will want an OpenAI Endpoint. To do so go to https://platform.openai.com, sign in, go to API Keys, create a new key. Your endpoint is the base url used for API calls.

Or… if you are working in the Microsoft ecosystem, in the Azure Portal search for Azure OpenAI and create a resource. Then go to Resource Management -> Keys and Endpoint. Copy the Endpoint and Key 1 (your API key) Deploy under Resource Management -> Model Deployments. Your endpoint will look like this:

https://<your-resource-name>.openai.azure.com/

Copilot can set all this up for you.

Next up: setting up your environment.

I divided the slide into two parts: good news and bad. The good news is the promise of enormously increased productivity, which may well lead to a higher standard of living across the board. Further, as AI progresses, we may see accelerated breakthroughs in many fields, most notably medicine and genetics.

The bad news is that along with these changes will almost certainly come massive displacement. As a society, we’ve seen this kind of displacement before. The advent of the tractor displaced well over 50% of farm workers, and ultimately mechanization displaced over 95% of agricultural labor!

The key difference, however, is that in the past technology displaced the least skilled workers; AI is set to affect many of the most educated and skilled. I’m not sure that society knows how to handle this. Reskilling has a poor performance history, and sending highly paid college-educated programmers back to two-year training programs in robotics does not seem auspicious.

In the very short term, AI presents some real challenges, not least is the environmental impact of massive energy-consuming data centers. Add to that the potential economic impact when the AI boom goes bust and there is plenty to worry about, but here is one more…

Right now we are learning to adjust to AI, but research continues on AGI (Artificial General Intelligence). The next step after that is Artificial Super Intelligence: when AI becomes meaningfully smarter than humans. No one knows quite what to expect when that happens, but I commend to you the book If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. These are not crackpot writers — they are recognized experts in AI and their thesis is chilling.

In coming postings I will divide my attention between two aspects of AI:

• Using AI in developing your application
• Creating applications that incorporate AI

Along the way I will talk about AI in the .NET world (specifically the Azure workflow tools) and outside of .NET (time to brush up on Python).

Next up: Assisted Coding…

MCP (Model, Context, Protocol) is an open standard that lets AI models (read copilot, ChatGPT, etc.) connect to external tools. These tools can be databases, code you write, code you connect to, other AIs, etc. etc. It is the universal API for Agents.

Here’s a link to the code.