Note: Microsoft strongly suggests using secrets and not putting these values in config.json

What is config.json?

The config.json file is a widely used configuration file format in many programming environments, particularly in JavaScript and .NET applications. It serves as a simple way to store application settings, such as API endpoints, feature flags, and other non-sensitive configurations. The structure of a config.json file is straightforward, making it easy for developers to read and modify.

Example of config.json

{
  "ApiUrl": "https://api.example.com",
  "FeatureFlag": true
}

Advantages of Using config.json

1. Simplicity and Accessibility: One of the primary advantages of using config.json is its simplicity. Developers can easily read and modify the file, making it an excellent choice for local development and testing environments. This ease of access allows for rapid iteration and debugging.
2. Version Control: Configuration files can be included in version control systems like Git. This feature is beneficial for tracking changes over time, allowing teams to collaborate effectively and maintain a history of configuration changes.
   
   Note, if you have secret values (such as keys) you do not want them in version control. One solution is to add them to your .gitignore file. A better solution is not to have them in config.json in the first place.
   
3. No Additional Setup Required: Unlike secrets management systems, which may require additional setup and configuration, using config.json typically involves minimal overhead. Developers can start using it right away without needing to integrate with external services.

Disadvantages of Using config.json

1. Security Risks: The most significant drawback of using config.json is its lack of security for sensitive data. If sensitive information, such as passwords or API keys, is stored in this file, it can be easily accessed by anyone with access to the codebase. This poses a substantial risk, especially in production environments.
2. Accidental Exposure: Including config.json in version control can lead to accidental exposure of sensitive data. Developers must be diligent about ensuring that sensitive information is excluded from version control, which can be challenging.
3. Limited to Non-Sensitive Data: While config.json is suitable for general configuration, it is not designed for managing sensitive information securely. Developers must find alternative methods for handling sensitive data, which can complicate the development process.

Secrets Management in Microsoft Agent Framework

What is Secrets Management?

The Microsoft Agent Framework provides a robust secrets management system designed to securely manage sensitive information such as credentials, API keys, and other secrets. This system is particularly useful for applications deployed in production environments where security is paramount.

Example of Secrets Management

Using Azure Key Vault, developers can securely store and retrieve secrets. Here’s a simple example of how to access a secret using the Azure SDK:

var secretClient = new SecretClient(new Uri("https://<your-key-vault-name>.vault.azure.net/"), new DefaultAzureCredential());
KeyVaultSecret secret = await secretClient.GetSecretAsync("MySecret");
string secretValue = secret.Value;

Advantages of Using Secrets Management

1. Enhanced Security: The primary advantage of using a secrets management system is its built-in security features. Secrets are encrypted and access-controlled, ensuring that sensitive data is not exposed in the codebase. This level of security is essential for protecting sensitive information in production environments.
2. Centralized Management: Secrets management systems like Azure Key Vault allow for centralized management of secrets across multiple applications. This centralization simplifies the process of updating and rotating secrets, reducing the risk of outdated or compromised credentials.
3. Integration with Azure Services: The Microsoft Agent Framework’s secrets management seamlessly integrates with other Azure services, providing a cohesive environment for managing application secrets. This integration enhances the overall security posture of applications deployed in the Azure ecosystem.

Disadvantages of Using Secrets Management

1. Complexity: Implementing a secrets management system can introduce additional complexity in setup and management compared to using a simple configuration file. Developers must familiarize themselves with the secrets management system and its APIs, which may require additional time and resources.
2. Cost: Depending on the chosen secrets management solution, there may be associated costs. For example, using Azure Key Vault incurs charges based on the number of operations performed and the amount of data stored. Organizations must weigh these costs against the benefits of enhanced security.
3. Learning Curve: For teams unfamiliar with secrets management practices, there may be a learning curve involved in adopting a new system. Training and documentation may be necessary to ensure that all team members understand how to use the system effectively.

Key Trade-offs

When deciding between config.json and secrets management in the Microsoft Agent Framework, developers must consider several key trade-offs:

1. Security vs. Convenience: Using config.json is convenient for non-sensitive configurations but poses security risks for sensitive data. In contrast, the Microsoft Agent Framework’s secrets management is secure but may require more setup and management effort.
2. Development vs. Production: config.json is often more suitable for development environments, where rapid iteration is essential. However, for production environments, where security is a priority, leveraging secrets management is advisable.
3. Version Control: Configuration files can be versioned easily, allowing for tracking changes over time. However, secrets should never be included in version control to prevent accidental exposure, necessitating a different approach for managing sensitive data.

Conclusion

Choosing between config.json and secrets management in the Microsoft Agent Framework ultimately depends on the specific needs of your application. For general configuration values that do not involve sensitive information, config.json remains a practical choice, provided that developers are diligent about handling sensitive data appropriately. However, for applications that require the management of sensitive information, leveraging the secrets management capabilities of the Microsoft Agent Framework is advisable to ensure security and compliance.

In summary, understanding the differences and trade-offs between these two approaches is crucial for developers aiming to build secure and efficient applications. By carefully considering the specific requirements of your project, you can make an informed decision that balances convenience, security, and maintainability.

Bzzzz Still your turn.

This is the approach I use in the demonstration program to log the invocation of the Tavily search tool.

What is Middleware?

To review, middleware is a software layer that acts as an intermediary between different software applications or components. In the context of the Microsoft Agent Framework, middleware enhances the interaction between agents and their underlying systems, allowing developers to intercept, modify, and log messages and operations. This capability is crucial for debugging, monitoring, and extending the functionality of agents.

ChatClient Middleware

Purpose

ChatClient Middleware is specifically designed to intercept calls made to an IChatClient implementation. This middleware is particularly useful for logging, modifying, or inspecting the raw messages exchanged between the agent and the underlying language model (LLM). By utilizing ChatClient Middleware, developers can gain insights into the communication flow, which is essential for debugging and improving the agent’s performance.

Use Case

A common use case for ChatClient Middleware is logging all messages sent and received by the agent. This can help developers understand how the agent interacts with users and the LLM, allowing for better optimization of responses and overall user experience.

Example

Here’s a simple example of how to implement ChatClient Middleware in C#:

var chatClient = new AIProjectClient(new Uri("your-uri"), new DefaultAzureCredential())
    .GetProjectOpenAIClient()
    .GetProjectResponsesClient()
    .AsIChatClient(deploymentName);

var middlewareEnabledChatClient = chatClient
    .AsBuilder()
    .Use(getResponseFunc: CustomChatClientMiddleware, getStreamingResponseFunc: null)
    .Build();

In this example, CustomChatClientMiddleware would be a function that processes the messages before they are sent to or after they are received from the LLM. This middleware can log the messages, modify them, or even implement additional logic based on the content of the messages.

Agent Middleware

Purpose

Agent Middleware operates at a higher level than ChatClient Middleware. It allows for the interception of all agent runs, enabling developers to inspect and modify the input and output of the agent’s operations. This middleware is essential for managing the overall behavior of the agent, including session management, identity tracking, and token budget management.

Use Case

A typical use case for Agent Middleware is collecting information about the agent’s session or identity. For instance, if an agent needs to maintain context across multiple interactions or manage its resource usage effectively, Agent Middleware would be the appropriate choice.

Example

Here’s how you might implement Agent Middleware in C#:

var agent = new ChatClientAgent(middlewareEnabledChatClient, instructions: "You are a helpful assistant.");

In this example, the ChatClientAgent is initialized with the middleware-enabled chat client. The agent can now leverage the capabilities of both ChatClient and Agent Middleware to enhance its functionality.

Function-Invocation Middleware for a Single Agent

Function-invocation middleware can be applied to both ChatClient and Agent Middleware. This type of middleware allows for the interception of function calls executed by the agent, enabling developers to inspect and modify inputs and outputs. This capability is particularly useful for logging, debugging, and implementing additional logic based on the agent’s operations.

Example of Function-Invocation Middleware in C

Here’s a simple example of how to implement function-invocation middleware for a single agent:

public class FunctionInvocationMiddleware
{
    public async Task InvokeAsync(AgentRunContext context, Func<AgentRunContext, Task> next)
    {
        // Before the function call
        Console.WriteLine($"Before function call: {context.FunctionName}");

        // Call the next middleware in the pipeline
        await next(context);

        // After the function call
        Console.WriteLine($"After function call: {context.Result}");
    }
}

// Usage
var agent = new ChatClientAgent(middlewareEnabledChatClient, instructions: "You are a helpful assistant.")
    .AsBuilder()
    .Use(FunctionInvocationMiddleware.InvokeAsync)
    .Build();

In this example, the FunctionInvocationMiddleware class defines an InvokeAsync method that logs the function name before and after the function call. This middleware can be used to track the execution flow of the agent’s operations, providing valuable insights into its behavior.

In the demonstration program we modify the creation of the ResearcherAgent to ad this bit of code:

.Use(async (agent, context, next, cancellationToken) =>
        {
            if (context.Function.Name == tavilyTool.Name)
            {
                _logger.LogInformation(
                    "Researcher invoking Tavily tool '{Tool}' with arguments {Arguments}",
                    context.Function.Name,
                    context.Arguments);
            }

            return await next(context, cancellationToken);
        })

That’s the only change necessary to have the middleware capture the calls to Tavily tool by the Researcher agent. None of the other agents change.

The trace from this change looks like this:

[trace] → execute_tool tavily_search
info: BlogWriter.ResearcherAgent[0]
      Researcher invoking Tavily tool 'tavily_search' with arguments [query, ChatClient middleware vs Agent middleware Microsoft Agent Framework]
[trace] ← execute_tool tavily_search (1796 ms)

Summary

In summary, the Microsoft Agent Framework offers two distinct types of middleware: ChatClient Middleware and Agent Middleware.

• ChatClient Middleware focuses on the interaction with the chat client, allowing for the logging and modification of messages exchanged with the LLM.
• Agent Middleware deals with the overall operations of the agent, enabling developers to manage session information, identity, and resource usage.

Additionally, function-invocation middleware can be implemented to intercept function calls, allowing for detailed control over the agent’s behavior.

Memory Architecture

1. Short-Term Memory: This component tracks ongoing interactions and is typically volatile, meaning it is cleared after the session ends. It allows the agent to respond to immediate queries and maintain a fluid conversation.
2. Long-Term Memory: In contrast, long-term memory retains information across sessions, enabling the agent to remember details that can enhance future interactions. This memory is crucial for building a relationship with users, as it allows the agent to provide continuity and context in conversations.

Context Providers

In MAF, long-term memory is managed through ContextProviders. These components allow agents to access relevant past interactions, user preferences, and other contextual information. By leveraging ContextProviders, agents can provide more relevant responses based on historical data, making interactions feel more natural and engaging.

Integration with Databases

To effectively manage long-term memory, AI agents require a reliable storage solution. Databases play a critical role in this process. Popular databases such as Neo4j and Azure Cosmos DB are commonly used to store long-term memory.

• Neo4j: This graph database allows for the storage of entities extracted from conversations. It enables the classification and linking of these entities back to the original messages, facilitating complex queries and relationships.
• Azure Cosmos DB: This database offers a unified solution for memory systems, providing speed and scalability essential for AI agents. Its multi-model capabilities allow for the storage of various data types, making it a versatile choice for long-term memory management.

Real-World Use Cases

The integration of databases into the Microsoft Agent Framework opens up a plethora of possibilities for AI agents. Here are some real-world use cases that illustrate the benefits of long-term memory:

Personalized Recommendations

With long-term memory an AI agent can remember user preferences and past interactions to suggest tailored options. If a user frequently inquires about travel destinations, the agent can store this information and offer relevant suggestions in future conversations. This not only enhances user satisfaction but also fosters a sense of connection between the user and the agent.

Contextual Awareness

Long-term memory enables agents to maintain context over multiple sessions. For example, if a user mentions a specific project during one interaction, the agent can recall details about that project in subsequent conversations. This continuity enhances the overall user experience, as it allows for more meaningful and relevant interactions. Users are more likely to engage with an agent that remembers their interests and past discussions.

Supporting Data and Quotes

. According to a blog post on Neo4j, “Long-term memory consists of entities which can be automatically extracted from conversations using the LLM and classified with the POLE+O schema.” This highlights the need for a systematic approach to data storage that allows for easy retrieval and classification.

POLE+P schema is a framework used to analyze systems by breaking them into five interacting dimensions:

• P)eople – Human actors
• O)bjects – tangible or digital artifacts
• L)ocation – spatial or contextual setting
• E)vents – actions that trigger change
• O)rganization – the structure or procedural layer that connects everything

In essence, this is the Who, What, Where, When and How.

A Microsoft Community Hub article states, “Long-term memory is typically shared across sessions,” emphasizing the necessity of a database that can persist data beyond individual interactions. This persistence is crucial for building a comprehensive understanding of user preferences and behaviors.

When an agent interacts with a user, it can save important information (like preferences or past conversations) in a database. The next time the user interacts with the agent, it can retrieve this information to provide a more personalized experience. This process is fundamental to creating a seamless and engaging interaction between users and AI agents.

Code Example

To illustrate how an AI agent might store and retrieve user preferences using a database, here’s a simple example in C#:

public class UserPreferences
{
    public string UserId { get; set; }
    public string Preference { get; set; }
}

public class MemoryDatabase
{
    private List<UserPreferences> preferencesStore = new List<UserPreferences>();

    public void SavePreference(string userId, string preference)
    {
        preferencesStore.Add(new UserPreferences { UserId = userId, Preference = preference });
    }

    public List<string> GetPreferences(string userId)
    {
        return preferencesStore.Where(p => p.UserId == userId).Select(p => p.Preference).ToList();
    }
}

// Usage
var memoryDb = new MemoryDatabase();
memoryDb.SavePreference("user123", "likes travel");
var userPreferences = memoryDb.GetPreferences("user123");

In this code snippet, we define a UserPreferences class to represent user preferences and a MemoryDatabase class to manage the storage and retrieval of these preferences. The SavePreference method allows the agent to store user preferences, while the GetPreferences method retrieves them for future interactions.

Conclusion

The integration of databases into the Microsoft Agent Framework is crucial for enabling long-term memory in AI agents. By leveraging structured data storage, agents can provide personalized and contextually aware interactions, significantly enhancing user experience. As AI technology continues to advance, the ability to remember and learn from past interactions will become increasingly important, making the role of databases in AI development more critical than ever. By understanding and implementing these concepts, developers can create more intelligent and responsive AI agents that truly understand and cater to user needs.

Overview of the Frameworks

LangChain

LangChain is a comprehensive framework designed for building applications that utilize large language models. It provides a rich set of tools for creating chains, retrievers, and tool-calling agents, making it particularly suitable for rapid prototyping and a wide array of use cases. The framework is primarily Python-based, which aligns well with the preferences of many data scientists and AI developers.

LangGraph

LangGraph, on the other hand, is a lower-level orchestration framework that focuses on building stateful multi-agent systems. It is tailored for workflows that require loops, persistence, and cyclic reasoning, making it ideal for complex agent architectures. LangGraph is particularly useful in scenarios where explicit state management is crucial, allowing developers to create intricate workflows that can adapt and respond to changing conditions.

These are not incompatible and many (most?) AI applications using one will use the other as well. I started my demonstration program starting here using LangChain and LangGraph and then translated it to C# starting here, and finally migrated it to Microsoft Agent Framework starting here.

Microsoft Agent Framework

The Microsoft Agent Framework is a robust solution designed specifically for .NET environments and Azure integration. It supports multi-agent orchestration and is built for enterprise-grade applications, providing first-class support for C# and Python. This framework is particularly advantageous for organizations that are heavily invested in the Microsoft ecosystem, as it offers seamless integration with Azure services and tools.

Key Differences

1. Ecosystem Fit

• LangChain: This framework is favored by teams that require flexibility and rapid iteration, especially in Python environments. Its extensive library of integrations (over 1,000) allows developers to quickly adapt and extend their applications.
• LangGraph: LangGraph is the choice for applications that necessitate explicit state management and complex workflows. It excels in scenarios where persistent memory and human-in-the-loop control are essential.
• Microsoft Agent Framework: This framework is ideal for .NET developers and organizations that rely on Azure services.

2. Architecture

• LangChain: The architecture of LangChain emphasizes building agent capabilities, focusing on the skills layer. It is designed for rapid development, allowing developers to create and iterate on applications quickly.
• LangGraph: In contrast, LangGraph serves as the control layer, defining how agents think and manage workflows. It employs structured logic, such as state machines and loops, to facilitate complex decision-making processes.
• Microsoft Agent Framework: This framework orchestrates multiple agents into a cohesive system, making it particularly suitable for enterprise applications that require coordination among various agents.

3. Integration and Support

• LangChain: With its vast ecosystem of integrations, LangChain is versatile and can be applied to a wide range of applications, from simple chatbots to complex data processing pipelines.
• LangGraph: Designed for complex, cloud-agnostic workflows, LangGraph excels in scenarios that require checkpointing and debugging, making it a powerful tool for developers working on intricate systems.
• Microsoft Agent Framework: The deep integration with Azure services and the focus on enterprise-level orchestration make the Microsoft Agent Framework a strong choice for organizations looking to leverage cloud capabilities in their applications.

Use Cases

LangChain

LangChain is particularly well-suited for projects that require quick iterations and flexibility. Some common use cases include:

• Retrieval-Augmented Generation (RAG) Pipelines: These pipelines combine retrieval mechanisms with generative models to produce contextually relevant responses.
• Multi-Step Workflows: LangChain can efficiently manage workflows that involve multiple steps, allowing for seamless transitions between tasks.

LangGraph

LangGraph shines in applications that require persistent memory and complex task automation. Typical use cases include:

• Human-in-the-Loop Control: Scenarios where human oversight is necessary, such as in decision-making processes that require validation or input from users.
• Complex Task Automation: Applications that involve intricate workflows, such as automated customer support systems or multi-agent negotiation platforms.

Microsoft Agent Framework

The Microsoft Agent Framework is ideal for enterprise applications, particularly in sectors that demand robust orchestration and compliance. Use cases include:

• Healthcare Applications: Systems that require secure and compliant handling of sensitive patient data, where multiple agents must work together to provide accurate and timely information.
• Financial Services: Applications that need to adhere to strict regulatory requirements while managing complex workflows involving multiple agents.

Example Code Snippets

To illustrate the capabilities of each framework, here are simple code snippets demonstrating their usage.

LangChain (Python)

from langchain import LLMChain, OpenAI

# Create a simple LLM chain
llm = OpenAI(model="text-davinci-003")
chain = LLMChain(llm=llm, prompt="What is the capital of France?")
response = chain.run()
print(response)  # Output: Paris

LangGraph (Python)

from langgraph import StateMachine

# Define a simple state machine
sm = StateMachine()
sm.add_state("start", on_enter=lambda: print("Starting..."))
sm.add_state("end", on_enter=lambda: print("Ending..."))
sm.add_transition("start", "end")
sm.run("start")  # Output: Starting...

Microsoft Agent Framework (C#)

using Microsoft.AgentFramework;

// Create a simple agent
var agent = new Agent("MyAgent");
agent.OnMessageReceived += (sender, message) => {
    Console.WriteLine($"Received: {message.Content}");
};
agent.Start();

Conclusion

Choosing between LangChain, LangGraph, and the Microsoft Agent Framework ultimately depends on your specific needs, including the programming environment, the complexity of the agent workflows, and the level of integration required with cloud services. Each framework has its strengths, making them suitable for different types of AI applications.

• LangChain is ideal for rapid development and flexibility in Python environments.
• LangGraph excels in scenarios requiring complex workflows and state management.
• Microsoft Agent Framework is the best choice for enterprise applications, particularly for organizations leveraging the Microsoft ecosystem.

That said, I suspect that the driving factor will be which environment the developer (team?) is comfortable with. .NET developers will be driven to Microsoft Agent Framework and most others will use LangChain/LangGraph or another Python framework.

What is OpenTelemetry?

OpenTelemetry is a set of APIs, libraries, agents, and instrumentation that enables developers to collect telemetry data from their applications. This data can include traces, metrics, and logs, which are essential for monitoring application performance and diagnosing issues. OpenTelemetry is designed to be vendor-agnostic, allowing developers to send their telemetry data to various backends for analysis.

Key Trends and Innovations in OpenTelemetry and MAF

1. Built-in Support for OpenTelemetry

One of the most significant advancements in the Microsoft Agent Framework is its built-in support for OpenTelemetry. This integration allows developers to automatically emit spans and metrics without the need for manual wrapping of every agent call. This feature simplifies the process of adding observability to applications, enabling developers to focus on building functionality rather than instrumentation.

2. Semantic Conventions

OpenTelemetry employs semantic conventions that help structure telemetry data in a meaningful way. In the context of AI agents, the OpenTelemetry GenAI semantic conventions ensure that the telemetry data collected is relevant and useful for analysis. This structured approach aids developers in understanding the performance and behavior of their agents more effectively.

3. Support for Multi-Agent Systems

The Microsoft Agent Framework is designed to support multi-agent architectures, allowing for complex interactions and behaviors to be monitored effectively. This capability is particularly beneficial in scenarios where multiple agents work together to complete tasks, as it provides a comprehensive view of the system’s performance.

By integrating OpenTelemetry, developers can monitor agent performance and interactions, gaining insights into usage patterns, error rates, and overall system efficiency. This data can be invaluable for optimizing the application and enhancing user experience.

Challenges in Integration

While the integration of OpenTelemetry with the Microsoft Agent Framework offers numerous benefits, there are challenges that developers may encounter. One notable issue is the missing Activity Events. Reports indicate that while spans and metrics are emitted correctly, some expected Activity Events, such as gen_ai.client.inference.operation.details, are not being generated. This limitation can restrict the granularity of observability, making it difficult to gain a complete understanding of agent interactions and performance.

Code Example: Setting Up OpenTelemetry in a Microsoft Agent Framework Application

To illustrate how to set up OpenTelemetry in a Microsoft Agent Framework application, consider the following code example:

using Microsoft.Extensions.AI;

// Create OpenAI client
var endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT");
var apiKey = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY");
var deploymentName = "gpt-4o-mini";

using var client = new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey))
    .GetChatClient(deploymentName)
    .AsIChatClient()
    .AsBuilder()
    .UseOpenTelemetry(sourceName: "MyAgent", configure: (cfg) => cfg.EnableSensitiveData = true)
    .Build();

var thread = client.GetNewThread();
logger.LogInformation("Agent created successfully with ID: {AgentId}", client.Id);

Explanation of the Code

1. Environment Variables: The code begins by retrieving the Azure OpenAI endpoint and API key from environment variables. This approach ensures that sensitive information is not hard-coded into the application.
2. Creating the OpenAI Client: An instance of AzureOpenAIClient is created using the retrieved endpoint and API key. This client is responsible for interacting with the OpenAI service.
3. Using OpenTelemetry: The UseOpenTelemetry method is called on the chat client builder. This method configures OpenTelemetry for the agent, allowing it to emit telemetry data. The sourceName parameter is set to “MyAgent,” and sensitive data emission is enabled.
4. Creating a New Thread: Finally, a new thread is created for the chat client, and a log message is generated to confirm the successful creation of the agent.

You can use OpenTelemetry on each agent, or you can instrument the model calls. To do the latter you only need to update Program.cs

IChatClient llm = openAIClient
    .GetChatClient(modelName)
    .AsIChatClient()
    .AsBuilder()
    .UseFunctionInvocation()
    .UseOpenTelemetry(sourceName: "BlogWriter.Agents")
    .Use(inner => new TokenCapChatClient(inner, maxTotalTokens: 10000))
    .Build();

Per agent allows you to capture which agent ran and agent-level timing. Per model call allows you to track model name, token usage, tool calls and per-round-trip latency.

Conclusion

Integrating OpenTelemetry with the Microsoft Agent Framework significantly enhances the observability of AI agents, providing developers with valuable insights into their performance and interactions. The built-in support for OpenTelemetry, along with semantic conventions and multi-agent system capabilities, makes it a robust choice for monitoring complex applications.

RAG – A quick example

RAG in detail

Deeper into RAG

The R in RAG

PEAS for Agentic AI

AI Reasoning and Planning

REACT and Agents in AI

Creating a multi-agent system (Python & LangChain/LangGraph) part 1 of 6

Migrating Agentic Code Python -> C# Part 1 of 6

Dependency Injection & Microsoft Agentic Framework

Migrating C# to Microsoft Agentic Framework

Transparency in Agentics

Ensuring Agent Safety

Logging & OpenTelemetry in Microsoft Agentic Framework

Activity Source in Microsoft Agentic Framework

Middleware in Microsoft Agentic Framework

Limit token usage in Microsoft Agentic Framework

Using OpenTelemetry in Microsoft Agent Framework

LangChain/LangGraph vs. Microsoft Agent Framework

Long term memory in Microsoft Agent Framework

Agent vs ChatClient Middleware

Configuration Management (secrets)

Note: many of these blog posts had initial research and drafts done by the Blog Writer multi-agent application that serves as a demo for these articles. All were then edited by me.

Overview of ChatClientAgentRunOptions

At its core, ChatClientAgentRunOptions is designed to manage various parameters for chat interactions within the Microsoft Agent Framework. One of its most significant features is the ability to limit the number of tokens generated in a single response through the max_tokens parameter. This capability is essential for controlling costs, as excessive token consumption can lead to budget overruns, especially in applications with high user engagement.

Key Features

1. Token Management: By specifying a maximum number of tokens that can be generated in a single response, developers can prevent a single interaction from consuming an excessive amount of tokens. This is particularly important in scenarios where the chat agent might otherwise generate verbose or unnecessary responses.
2. Customizable Options: Beyond token management, ChatClientAgentRunOptions allows developers to fine-tune the behavior of the chat agent through various customizable options. Parameters such as temperature, frequency_penalty, and presence_penalty can be adjusted alongside max_tokens to create a more tailored user experience. For instance, a higher temperature might lead to more creative responses, while penalties can help reduce repetitive outputs.
3. Integration with Metrics: The Microsoft Agent Framework provides valuable metrics on token usage, enabling developers to monitor both input and output tokens, estimated costs, and latency. This data is crucial for optimizing performance and ensuring that the application remains within budgetary limits.

Real-World Use Case

To illustrate the practical application of ChatClientAgentRunOptions, consider a developer creating a chat agent that provides weather information. By implementing a max_tokens limit, the developer can ensure that the agent does not generate overly verbose responses that could inflate costs. Here’s a simple example of how this might be implemented in Python:

from agent_framework.openai import OpenAIChatClient, OpenAIChatOptions

# Set default options at construction time
agent = OpenAIChatClient().as_agent(
    instructions="You are a helpful assistant",
    default_options={
        "temperature": 0.7,
        "max_tokens": 150  # Limit output tokens to manage costs
    }
)

result = await agent.run("What is the weather like in Amsterdam?")
print(result)

In this example, the developer has set a max_tokens limit of 150. This means that regardless of the complexity of the user’s query, the agent’s response will be capped at 150 tokens, effectively managing costs while still providing valuable information.

Challenges in Token Management

While the capabilities of ChatClientAgentRunOptions are robust, developers must remain vigilant about token usage, particularly in applications with extensive conversation histories. Here are some challenges they may encounter:

• Token Overuse: If developers do not actively manage token consumption, the cumulative token count can exceed budget limits. This is especially true in applications where multiple users interact with the agent simultaneously. Developers should implement strategies to monitor and control token usage effectively.
• Context Management: Maintaining relevant context in conversations without sending excessive historical data is crucial. Developers are encouraged to trim or limit the stored message history to optimize token usage. This can involve implementing strategies to summarize past interactions or selectively retaining only the most relevant messages.

Conclusion

The ChatClientAgentRunOptions in the Microsoft Agent Framework provides a robust mechanism for managing output tokens, thereby helping developers control costs associated with AI interactions. By leveraging this feature, developers can create efficient and cost-effective chat applications that deliver value to users without exceeding budgetary constraints. As the demand for AI-driven solutions continues to grow, understanding and implementing effective token management strategies will be crucial for developers looking to optimize their applications and maintain financial sustainability.

In summary, the integration of ChatClientAgentRunOptions into your development process can significantly enhance your ability to manage costs while providing a high-quality user experience. By setting appropriate limits on token usage and continuously monitoring performance metrics, developers can navigate the complexities of AI interactions with confidence.

What is Middleware in the Microsoft Agent Framework?

Middleware serves as an intermediary layer that processes requests and responses between users and AI agents. It acts as a bridge, allowing developers to intercept, inspect, and modify the data flowing through the system. This capability is vital for implementing additional logic, such as validation, logging, and, importantly, token management.

In the context of the Microsoft Agent Framework, middleware can be utilized to enhance the functionality of AI agents by providing a structured way to handle requests and responses. This not only improves the overall efficiency of the system but also allows for greater control over how the AI interacts with users.

Why Use Middleware?

1. Control Costs

One of the primary reasons to implement middleware for capping token usage is cost management. Many AI services, including those provided by Microsoft, charge based on the number of tokens processed during interactions. By using middleware to monitor and limit token usage, organizations can prevent unexpected spikes in expenses. This is particularly important for businesses that rely heavily on AI for customer service, content generation, or data analysis.

2. Enhance Security

Security is a paramount concern when dealing with AI agents, especially in environments that handle sensitive data. Middleware can validate requests before they reach the agent, ensuring that only legitimate requests are processed. This validation step is crucial for preventing malicious attacks or unintended data exposure, thereby enhancing the overall security posture of the application.

3. Improve Performance

Middleware can also play a significant role in optimizing the performance of AI agents. By intercepting and modifying requests and responses, middleware can streamline the data flow, reducing latency and improving response times. This ensures that the AI agent operates efficiently, providing users with a seamless experience.

4. Custom Logic Implementation

Another advantage of middleware is the ability to implement custom logic tailored to specific business needs. For instance, organizations can modify the input or output of the AI agent based on predefined business rules or user requirements. This flexibility allows for a more personalized interaction, enhancing user satisfaction and engagement.

Types of Middleware in the Microsoft Agent Framework

The Microsoft Agent Framework offers several types of middleware, each serving a unique purpose:

1. Agent Run Middleware

This type of middleware intercepts all agent runs, allowing developers to inspect and modify both input and output. It is particularly useful for implementing global logic that applies to all interactions with the agent.

2. Function Calling Middleware

Function calling middleware intercepts function calls made by the agent, enabling similar inspection and modification capabilities. This is beneficial for managing specific functions that may require additional validation or processing.

3. Streaming Middleware

Designed specifically for handling streaming data, streaming middleware allows for real-time modifications. This is essential for applications that require continuous data flow, such as live chatbots or real-time analytics.

Example of Middleware Implementation

To illustrate how middleware can be implemented to cap token usage, consider the following example in C#:

public class TokenUsageMiddleware
{
    private readonly RequestDelegate _next;
    private const int MAX_TOKENS = 1000; // Define your maximum token limit

    public TokenUsageMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Inspect the incoming request
        var tokenCount = CountTokens(context.Request.Body);

        // Cap the token usage
        if (tokenCount > MAX_TOKENS)
        {
            context.Response.StatusCode = StatusCodes.Status400BadRequest;
            await context.Response.WriteAsync("Token limit exceeded.");
            return;
        }

        // Call the next middleware in the pipeline
        await _next(context);
    }

    private int CountTokens(Stream requestBody)
    {
        // Logic to count tokens in the request body
        // This is a placeholder for actual token counting logic
        return 0; // Replace with actual token counting implementation
    }
}

In this example, the TokenUsageMiddleware class intercepts incoming requests, counts the tokens, and checks if the count exceeds a predefined limit. If the limit is exceeded, it returns a 400 Bad Request response, effectively capping token usage.

Additional Real-World Use Cases

Cost Management

Organizations utilizing AI services, such as OpenAI’s GPT models, can implement middleware to monitor and cap token usage. This ensures that they remain within budget and avoid unexpected costs associated with excessive API calls.

Data Validation

Middleware can also be employed to validate user inputs before they reach the AI agent. This prevents invalid or harmful requests from being processed, safeguarding the integrity of the system and enhancing user experience.

Conclusion

Middleware in the Microsoft Agent Framework is a powerful tool for managing interactions with AI agents. By implementing middleware to cap token usage, organizations can effectively control costs, enhance security, and improve performance. This makes middleware an essential component of modern AI applications, enabling businesses to leverage the full potential of AI while maintaining control over their resources.

My blog post application code has been updated to cap token usage here.

The series actually begins with a Python application using LangChain, etc. We then port that to C# and from there we port it to Microsoft Agent Framework, here. From there, in subsequent blog posts we add features.

Understanding the Microsoft Agent Framework and OpenTelemetry

The Microsoft Agent Framework is designed to facilitate the development of intelligent agents that can interact with users and other systems. With the integration of OpenTelemetry, MAF now offers a robust solution for tracking and analyzing agent behavior, performance, and interactions. OpenTelemetry is a set of APIs, libraries, and tools that enable developers to collect and export telemetry data, such as logs and metrics, from their applications. This integration allows for a more comprehensive understanding of how AI agents operate in real-world scenarios.

Key Trends and Innovations

1. Unified Observability

One of the standout features of the MAF’s integration with OpenTelemetry is the support for OpenTelemetry conventions. This allows for a unified timeline of tasks across different agents and tools, which is crucial for understanding the interactions and performance of AI agents in production environments. By providing a coherent view of agent activities, developers can more easily identify patterns and correlations that may impact performance.

2. Instrumentation Layer

The MAF includes an automatic instrumentation layer that tracks various metrics, such as token consumption, interaction durations, and tool usage. This layer is essential for diagnosing issues and optimizing agent performance. By automatically capturing these metrics, developers can focus on building and improving their agents rather than spending time on manual logging and monitoring.

3. Integration with Azure

Another significant advantage of the MAF’s integration with OpenTelemetry is its ability to ship traces to Azure AI Foundry. This centralized monitoring and analysis capability allows teams to gain insights into agent behavior across different environments. By leveraging Azure’s powerful analytics tools, developers can visualize performance metrics and identify areas for improvement.

Real-World Use Cases

The integration of OpenTelemetry into the Microsoft Agent Framework opens up a range of practical applications for developers:

• Debugging and Evaluation: Developers can utilize OpenTelemetry to trace agent interactions, making it easier to identify bottlenecks or failures in real-time. This capability is invaluable for ensuring that agents perform as expected and for quickly addressing any issues that arise.
• Performance Monitoring: By tracking metrics such as latency and interaction counts, teams can optimize their agents for better user experiences. Understanding how agents perform under different conditions allows developers to make informed decisions about enhancements and adjustments.

Supporting Data and Quotes

The importance of OpenTelemetry in enhancing observability is underscored by insights from industry experts. According to a Microsoft Tech Community blog post, “OpenTelemetry for traces/spans + attributes (agent, tool, retrieval, latency, tokens)” enables detailed monitoring of agent activities. This level of granularity is essential for developers looking to improve their systems.

As of now, the framework is in public preview (version 1.0.0-rc1), with stable OpenTelemetry contracts guiding its development. This means that developers can start experimenting with these features and provide feedback to shape the future of the framework.

A Simple Explanation of OpenTelemetry

OpenTelemetry can be thought of as a toolkit that helps developers collect and export telemetry data from their applications. In the context of the Microsoft Agent Framework, it allows developers to track how their AI agents are performing, making it easier to spot issues and improve functionality. This is particularly important in complex systems where multiple agents interact with each other and with users.

Code Example: Using OpenTelemetry with the Microsoft Agent Framework

To illustrate how developers can leverage OpenTelemetry within the Microsoft Agent Framework, consider the following C# code example. This snippet demonstrates how to start an activity for an agent session and log interactions using OpenTelemetry:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using Microsoft.Extensions.Logging;

public class Agent
{
    private static readonly ActivitySource activitySource = new ActivitySource("Microsoft.Extensions.AI");

    public void StartSession(string sessionId, ILogger logger)
    {
        logger.LogInformation("Starting agent session with ID: {SessionId}", sessionId);
        using (var activity = activitySource.StartActivity("Agent Session"))
        {
            int interactionCount = 0;
            while (true)
            {
                Console.Write("You (or 'exit' to quit): ");
                var input = Console.ReadLine();
                if (string.IsNullOrWhiteSpace(input) || input.Equals("exit", StringComparison.OrdinalIgnoreCase))
                {
                    logger.LogInformation("User requested to exit the session");
                    break;
                }
                interactionCount++;
                logger.LogInformation("Processing interaction #{InteractionCount}", interactionCount);
                using (var interactionActivity = activitySource.StartActivity("Agent Interaction"))
                {
                    // Simulate processing the input
                }
            }
        }
    }
}

In this example, the StartSession method initiates an agent session and logs each interaction. Each interaction is tracked as a child span, allowing for detailed performance analysis. This approach not only enhances observability but also provides developers with the tools they need to optimize their agents effectively.

Conclusion

The integration of OpenTelemetry into the Microsoft Agent Framework represents a significant advancement in the monitoring and debugging of AI agents. By leveraging these tools, developers can gain valuable insights into their systems, leading to improved performance and user satisfaction.