Keep it Simple

Go Beyond the Azure Portal 🏃‍♂️ Unlocking the Power of Azure REST APIs

2025-04-20T10:30:00.001+05:30

When we think about managing resources in Azure, the first tools that usually come to mind are the Azure Portal, Azure CLI, PowerShell, or perhaps even ARM templates, Bicep, and Terraform. These tools are super useful, user-friendly and powerful, no doubt — but here’s something that often goes unnoticed..

👉 Everything you do in Azure — every button clicked, every script executed — ultimately calls an Azure REST API under the hood.

Yes, every single action.

And once you realize this, a whole new world opens up.

🚀 Why should you care about Azure Rest APIs?

At first glance, working directly with REST APIs might seem like an over-complication. After all, the portal and CLI abstract away the hard parts, right?

But what if you’re in a scenario where:

You need to automate a process that the CLI doesn’t natively support
You’re building a custom dashboard or internal tool for Azure management and operations
You want tighter control over what’s happening under the hood
Your company’s higher management at CXO levels do not have time to go into the Azure portal or not that technical to run scripts to find out information.

In such cases, going directly to the source — the Azure REST APIs — can be incredibly powerful.

🧠 The hidden power of talking to Azure natively

Throughout my cloud career, I’ve leaned on REST APIs in many mission-critical situations.

🔄 Automating various workflows and sometimes creating a small repeatable landing zone.
📊 Querying hundreds of resources across multiple subscriptions in parallel
🔐 Giving the people in authority a dashboard/tool which they can carry along with them and check stuff on the fly
🛠️ Building internal tools that allow teams to act on Azure without accessing the portal

And you know what’s fascinating? These APIs are stable, well-documented, and consistent once you get a hang of them.

🛠️ Getting Started: It's easier than you think

To interact with Azure REST APIs, you’ll need three main components:

An access token: Usually acquired through Azure AD using client credentials or user login.
The right endpoint: Every service has its own base URL and API version. Example: https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups?api-version=2021-04-01
A tool to call the API: You can use curl, Postman, or programmatically use any language with HTTP capabilities (Python, PowerShell, C#, etc.).

🔗 Here’s the official documentation to start with: Azure REST API Reference

🧩Real-World example: List resource group across subscriptions

While the CLI can do this one subscription at a time, using REST APIs with the right bearer token and a script, you can hit multiple subscriptions in parallel. This is a life-saver for large enterprises where scale and speed matter.

Imagine being able to:

Traverse hundreds of subscriptions
Apply filters and tags dynamically
Export metadata directly into your CMDB or data lake

All without touching the portal.

🔓 Once you master it, Your Are Unstoppable

Working with Azure REST APIs does come with a learning curve. Understanding headers, tokens, scopes, API versions, and request bodies can take some practice. But once you master it:

You can build your own CLI.
You can automate things that no one else in your team has thought possible.
You stop waiting for tools to support features — you build them yourself.

And perhaps most importantly — you start thinking like Azure itself.

🎯 Final Thoughts

This is one of the most underutilized and unexplored areas I’ve seen in the Azure world. Everyone’s using the portal, scripts, and pipelines — but very few are tapping into the raw power of REST APIs that make all of it work.

If you’re a cloud architect, developer, or SRE looking to take your Azure game to the next level, start exploring this world. It’s deep, it’s powerful, and it’s incredibly satisfying.

✨ The sky’s the limit — and now, you’re flying without training wheels.

Have you used Azure REST APIs in your work? Got any interesting use cases or stories? Drop a comment or let’s connect — I’d love to hear how others are using this hidden superpower!

Securing Your LLM Applications: Detecting PII in Prompts with LLM-Guard ⚔️

2024-08-16T14:43:00.003+05:30

In the era of Generative AI, where Large Language Models (LLMs) are increasingly woven into the fabric of modern applications, the need for safeguarding sensitive information has never been more critical. As organizations integrate LLMs into their workflows, the challenge of detecting and anonymizing Personally Identifiable Information (PII) in prompts — especially those processed in-memory — becomes paramount. Enter LLM Guard, your AI-powered sentinel in the battle against data exposure.

Why PII Matters: The Stakes Are High

PII is the linchpin of an individual’s digital identity. It’s not just a collection of data points; it’s a digital fingerprint that, when mishandled, can lead to dire consequences. Whether it’s GDPR or HIPAA, global regulations demand stringent measures for PII protection. But what happens when this sensitive information makes its way into LLM prompts? If left unchecked, it could inadvertently proliferate across storage points, model training datasets, and third-party services, amplifying the risk of data breaches and privacy violations.

Anatomy of a Privacy Breach: The Attack Surface

Consider this scenario: A company uses an LLM to automate customer service. The prompts sent to the LLM contain user queries, some of which include PII like names, addresses, or credit card numbers. If the model provider stores these prompts, either for improving the model or for other purposes, the PII is now exposed to risks outside of the company’s control. This scenario underscores the importance of ensuring that any PII in prompts is detected and anonymized before it ever reaches the LLM.

The Role of Anonymize Scanner in LLM Guard

The Anonymize Scanner within LLM Guard acts as your digital guardian, scrutinizing prompts in real-time and ensuring they remain free from PII. Here’s how it works:

PII Detection: The scanner identifies PII entities across various categories, including credit card numbers, personal names, phone numbers, URLs, email addresses, IP addresses, UUIDs, social security numbers, crypto wallet addresses, and IBAN codes.
Anonymization: Once detected, the scanner can anonymize or redact this information, ensuring that the LLM only processes sanitized data.
In-Memory Operation: Unlike traditional methods that might focus on data at rest or in transit, LLM Guard operates directly on prompts in RAM, offering real-time protection without the latency of I/O operations.

Understanding PII Entities: A Closer Look

Let’s break down the types of PII the Anonymize Scanner targets:

Credit Cards: The scanner is adept at identifying credit card formats, including specific patterns like those for Visa or American Express.
Names: It recognizes full names, including first, middle, and last names, ensuring that any personally identifiable name data is flagged.
Phone Numbers: From a simple 10-digit number to complex international formats, the scanner is equipped to detect various phone number patterns.
URLs & Email Addresses: Whether it’s a URL pointing to a personal site or an email address in a customer query, the scanner can identify and anonymize these elements.
IP Addresses: Both IPv4 and IPv6 addresses are within the scanner’s purview, crucial for applications dealing with network data.
UUIDs & Social Security Numbers: Unique identifiers and social security numbers are some of the most sensitive data types, and the scanner is finely tuned to detect these.
Crypto Wallets & IBANs: As financial transactions become more digital, detecting and anonymizing crypto wallet addresses and IBAN codes is vital to prevent fraud and ensure compliance.

Detecting PII in Prompts Using LLM Guard

To illustrate the power of LLM Guard, let’s walk through a sample yet practical example. Below is a Python snippet that demonstrates how to use the Anonymize scanner to detect and redact PII in a prompt before it is processed by an LLM:

# Install the necessary package
pip install llm_guard

# Importing required modules
from llm_guard import scan_prompt
from llm_guard.input_scanners import Anonymize

# Define a prompt that contains PII
prompt = """Make an SQL insert statement to add a new user to our database.
Name is John Doe.
Email is test@test.com.
Phone number is 555-123-4567."""

# Scan the prompt for PII and sanitize it
sanitized_prompt, results_valid, results_score = scan_prompt(input_scanners, prompt)

# Check if the prompt contains any invalid PII data
if any(results_valid.values()) is False:
    print(f"Prompt {prompt} is not valid, scores: {results_score}")
    exit(1)

# Output the sanitized prompt
print(f"Prompt: {sanitized_prompt}")

Output:

Explanation:

The scan_prompt function utilizes the Anonymize scanner to inspect the prompt for any PII entities.
The scanner detects sensitive information such as names, email addresses, and phone numbers, replacing them with placeholders like [REDACTED_PERSON_1], [REDACTED_EMAIL_ADDRESS_1], and [REDACTED_PHONE_NUMBER_1].
The sanitized prompt is then printed, showcasing how the PII has been effectively anonymized.

This real-time detection and redaction process is crucial for maintaining the privacy and security of your data when interfacing with LLMs.

Fortifying Your LLM Applications

Integrating LLM Guard into your application stack is more than just a compliance checkbox; it’s a proactive stance in the ongoing battle for data privacy. By ensuring that PII is detected and anonymized in-memory, before it even has a chance to be processed by an LLM, you’re not only protecting your users but also fortifying your applications against potential breaches.

In a world where AI is becoming omnipresent, LLM Guard offers a robust solution to a critical problem. Don’t wait for a breach to prioritize PII protection — make it a cornerstone of your LLM deployment strategy.

Stay tuned for more on LLM-Guard guardrail features. 🙂

✨Building a Smart Image Parser with .NET, Semantic Kernel, and GPT-4o 🧿

2024-07-03T17:35:00.004+05:30

Hey there, tech enthusiasts! Today, we’re diving into the magical world of AI, .NET, and the Semantic Kernel. Imagine a world where you provide an image URL, and in just a few lines of code, your program analyzes every tiny detail in that image. Sounds cool, right? Well, buckle up, because we’re about to embark on a fun journey to create exactly that!

The Magic Ingredients

Here’s what we’ll be using:

.NET (because we love a sturdy framework)
Semantic Kernel (because semantics matter, folks)
GPT-4o (the brain behind the operation)

Without further ado, let’s jump into the code snippets and see how we can make this happen.

The Controller — Where the Magic Begins

First, we need to set up our controller. This is where all the action happens. Let’s take a look at the code:

using Microsoft.AspNetCore.Mvc;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;

namespace AzureSemanticKernel.AI.Controllers
{
    public class OpenAiController : Controller
    {
        private string open_ai_key = Environment.GetEnvironmentVariable("OPEN_AI_KEY");
        private string open_ai_org = Environment.GetEnvironmentVariable("OPEN_AI_ORG_ID");

        [HttpPost]
        public async Task<string> GetGpt4oImageResponse(string imgUrl)
        {
            try
            {
                var kernelBuilder = Kernel.CreateBuilder();

                kernelBuilder.AddOpenAIChatCompletion("gpt-4o", open_ai_key, open_ai_org);

                var kernel = kernelBuilder.Build();

                var chat = kernel.GetRequiredService<IChatCompletionService>();

                var history = new ChatHistory();
                history.AddSystemMessage("You are a friendly and helpful assistant that responds to questions directly");

                var message = new ChatMessageContentItemCollection
                {
                    new TextContent("Can you do a detail analysis and tell me all the minute details that present in this image?"),
                    new ImageContent(new Uri(imgUrl))
                };

                history.AddUserMessage(message);

                var result = await chat.GetChatMessageContentAsync(history);

                return result.Content;
            }
            catch (Exception ex)
            {
                return ex.Message;
            }
        }
    }
}

Breaking Down the Magic

Let’s dissect this code and understand what each part does.

1. Namespace and Using Statements:

Just the usual suspects. Nothing to see here. Add those packages and move on!

2. Controller Setup:

We create an OpenAiController and define our API key and organization ID from environment variables. Remember, hard-coding keys is like leaving your front door open with a "Please Rob Me" sign.

3. GetGpt4oImageResponse Method:

This is where the fun begins! We define an asynchronous method to process the image URL.
Kernel Creation: We create a kernel builder and add our GPT-4o chat completion service. Think of the kernel as the magical cauldron where all the ingredients mix together.
Chat Service: We get the chat service from the kernel. This service is like our friendly neighborhood barista who knows exactly how we like our coffee.
Chat History: We initialize the chat history and add a system message to set the tone for our AI assistant. We want our assistant to be friendly and helpful, just like your favorite support agent.
User Message: We create a message collection with a text prompt and the image URL. This is like giving the barista your order — “I want a detailed analysis with extra foam, please!”
Response Handling: We send the message to the chat service and wait for the response. If all goes well, we return the content. If something goes wrong, we catch the exception and return the error message (because nobody likes an unhandled exception ruining the party).

Configuration — The Secret Sauce

Finally, let’s not forget the configuration setup in Program.cs:

using DotNetEnv;

Env.Load();

This simple line ensures our environment variables are loaded, so our API keys and organization IDs are safely tucked away. It’s like having a secret recipe locked in a vault.

Wrapping Up

And there you have it! In just a few lines of code, we’ve created a powerful image parser that uses .NET, Semantic Kernel, and GPT-4o to analyze images and return detailed descriptions. Now you can impress your friends, family, and maybe even your boss with your new found AI prowess.

Remember, the key to keeping your code exciting is to blend functionality with a dash of humor and a sprinkle of creativity. Happy coding, and may your bugs be few and your features be many! 🎉

✨Building an Engaging Chat Program with Azure OpenAI and .NET Semantic Kernel 🧿

2024-06-28T17:50:00.002+05:30

In the ever-evolving landscape of artificial intelligence and cloud computing, creating interactive and intelligent applications has become more accessible and powerful. Today, we’ll dive into the fascinating world of integrating Azure OpenAI with .NET Semantic Kernel to build a simple yet engaging chat program. This blog post will walk you through the key components of the application, highlighting the elegance and efficiency of the integration.

Setting the Stage: Loading Configuration

The first step in our journey is setting up the environment configuration. Using the DotNetEnv library, we can effortlessly load environment variables from a .env file. This approach keeps our sensitive information secure and makes the configuration process seamless.

using DotNetEnv;

Env.Load();

This line of code ensures that our application can access the necessary Azure OpenAI credentials and other configuration details stored in the .env file.

Initializing the Kernel: The Common Class

The heart of our chat program lies in the initialization of the Semantic Kernel. The Common class is responsible for setting up the kernel using the credentials and model details from the environment variables.

using Microsoft.SemanticKernel;

namespace AzureSemanticKernel.AI
{
    public static class Common
    {
        private static string key = Environment.GetEnvironmentVariable("AZURE_OPENAI_KEY");
        private static string model = Environment.GetEnvironmentVariable("AZURE_OPENAI_MODEL");
        private static string endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT");

        public static Kernel initializeKernel()
        {
            var kernelBuilder = Kernel.CreateBuilder();
            kernelBuilder.Services.AddAzureOpenAIChatCompletion(model, endpoint, key);
            var kernel = kernelBuilder.Build();

            return kernel;
        }
    }
}

In this class, we retrieve the API key, model, and endpoint from the environment variables and use them to configure the kernel. The initializeKernel method creates and returns a fully initialized kernel ready to handle chat requests.

Bringing It All Together: The Main Controller

Now that we have our configuration and kernel setup, let’s look at how we can create a simple yet powerful chat controller. The SimpleController class is where the magic happens. It receives user messages, processes them using the Semantic Kernel, and returns the responses.

using Microsoft.AspNetCore.Mvc;
using Microsoft.SemanticKernel;

namespace AzureSemanticKernel.AI.Controllers
{
    public class SimpleController : Controller
    {        
        [HttpPost]
        public async Task<string> GetChatKernelResponse(string userMessage)
        {
            try
            {
                var kernel = Common.initializeKernel();

                var result = await kernel.InvokePromptAsync(userMessage);

                return result.ToString();
            }
            catch (Exception ex)
            {
                return ex.Message;
            }
        }
    }
}

The GetChatKernelResponse action handles POST requests with user messages. It initializes the kernel, invokes the chat completion prompt, and returns the response. Error handling ensures that any exceptions are caught and their messages are returned to the user.

Conclusion

By leveraging Azure OpenAI and .NET Semantic Kernel, we’ve created a simple yet robust chat program with just few lines of code. This integration demonstrates the power and flexibility of modern AI and cloud technologies. Whether you’re a seasoned developer or just starting, this project showcases how easily you can build engaging and intelligent applications.

As AI continues to evolve, the possibilities for creating interactive and responsive applications are endless. Dive into the code, experiment with different models and prompts, and unleash the full potential of Azure OpenAI and .NET Semantic Kernel in your projects. Happy coding!

🎉 Automating Disaster Recovery for Azure Service Bus: A Seamless Solution ✨

2024-06-24T10:52:00.010+05:30

Disaster recovery is a critical component of any robust IT infrastructure. For Azure Service Bus, a highly reliable cloud messaging service, automating the failover process is crucial for minimizing downtime and ensuring business continuity. While the Azure portal provides an easy way to initiate a failover with a click of a button, relying on manual intervention is not ideal for enterprise-grade solutions. Automation is the key.

In this blog post, we’ll explore an automated approach to handling disaster recovery for Azure Service Bus using a PowerShell script. This script seamlessly initiates the failover process and manages the underlying tasks, making it easier for infrastructure architects to ensure continuity without manual intervention.

Why Automate Service Bus Failover?

Azure Service Bus offers geo-disaster recovery capabilities that ensure high availability and data protection. However, initiating a failover manually can be risky and not recommended in production scenarios. Automation provides several benefits:

Consistency: Automated scripts ensure that the failover process is executed the same way every time, reducing the risk of human error.
Speed: Automation can trigger failover processes instantly, minimizing downtime.
Efficiency: Automated processes can handle multiple tasks simultaneously, such as reconfiguring namespaces and cleaning up resources.

Understanding the Burn Down of the Primary Namespace

One critical aspect of Azure Service Bus failover is that after the failover, the primary namespace is essentially “burned down.” This means that the primary namespace becomes inactive, and all its entities, such as queues and topics, need to be cleaned up. This cleanup process ensures that the primary namespace is ready to be reconfigured or decommissioned.

The PowerShell Script

Let’s dive into the PowerShell script that automates the Azure Service Bus failover process. This script ensures that the primary and secondary namespaces are provisioned and manages the failover with minimal manual intervention.

Step 1: Parameter Initialization

First, we set the necessary parameters, such as subscription ID, resource group name, primary and secondary namespaces, and the alias name.

$connection = Connect-AzAccount -ErrorAction Stop
Write-Host "Connected to Azure successfully." -ForegroundColor Yellow

#************** Parameters ********************************************************************************************************************
$subscriptionId = ""
$resourceGroupName = ""
$sbusPrimaryNamespace = ""
$sbusSecondaryNamespace = ""
$sbusAliasName = ""
$partnerId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.ServiceBus/ `
             namespaces/$sbusPrimaryNamespace"
#***********************************************************************************************************************************************

Step 2: Provisioning Functions

We define functions to ensure that both namespaces are fully provisioned before and after the failover. These functions poll the provisioning state and wait until it reaches the ‘Succeeded’ state for both normal and Geo-provisioned.

function Wait-ForNamespaceProvisioning {
    param (
        [string]$resourceGroupName,
        [string]$namespaceName
    )
    
    $maxRetries = 30
    $retryCount = 0
    $delay = 60  # Delay in seconds between retries
    
    do {
        $namespace = Get-AzServiceBusNamespace -ResourceGroupName $resourceGroupName -NamespaceName $namespaceName
        if ($namespace.ProvisioningState -eq "Succeeded") {
            Write-Output "Namespace $namespaceName is provisioned."
            return
        }
        
        Write-Output "Namespace $namespaceName is still in provisioning state: $($namespace.ProvisioningState)."
        Start-Sleep -Seconds $delay
        $retryCount++

    } while ($namespace.ProvisioningState -ne "Succeeded" -and $retryCount -lt $maxRetries)
    
    if ($namespace.ProvisioningState -ne "Succeeded") {
        throw "Namespace $namespaceName did not reach 'Succeeded' state within the allotted time."
    }
}

function Wait-ForNamespaceGeoProvisioning {
    param (
        [string]$resourceGroupName,
        [string]$namespaceName
    )
    
    $maxRetries = 30
    $retryCount = 0
    $delay = 60  # Delay in seconds between retries
    
    do {
        $namespace = Get-AzServiceBusGeoDRConfiguration -ResourceGroupName $resourceGroupName `
                     -NamespaceName $namespaceName

        if ($namespace.ProvisioningState -eq "Succeeded") {
            Write-Output "Namespace $namespaceName is geo provisioned."
            return
        }
        
        if($null -eq $namespace){ 
            Write-Output "Namespace $namespaceName is geo provisioned."
            return 
        }
        
        Write-Output "Namespace $namespaceName is still in geo provisioning state: $($namespace.ProvisioningState)."
        Start-Sleep -Seconds $delay
        $retryCount++

    } while ($namespace.ProvisioningState -ne "Succeeded" -and $retryCount -lt $maxRetries)
    
    if ($namespace.ProvisioningState -ne "Succeeded") {
        throw "Namespace $namespaceName did not reach 'Succeeded' state within the allotted time."
    }
}

Step 3: Combined Provisioning Function

This function ensures that both the primary and secondary namespaces are provisioned.

function Wait-ForNamespaceAndGeoProvisining {
    param (
        [string]$resourceGroupName,
        [string]$PrimaryNamespace,
        [string]$SecondaryNamespace
    )

    Wait-ForNamespaceProvisioning -resourceGroupName $resourceGroupName -namespaceName $PrimaryNamespace
    Wait-ForNamespaceProvisioning -resourceGroupName $resourceGroupName -namespaceName $SecondaryNamespace

    Wait-ForNamespaceGeoProvisioning -resourceGroupName $resourceGroupName -namespaceName $PrimaryNamespace
    Wait-ForNamespaceGeoProvisioning -resourceGroupName $resourceGroupName -namespaceName $SecondaryNamespace

    return
}

Step 4: Initiate Failover

This section of the script handles the failover process itself, ensuring both namespaces are ready, initiating the failover, and then performing post-failover cleanup and reconfiguration.

#************** Initiate failover **************************************

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Write-Output "`nFailing Over : Azure Service Bus $sbusPrimaryNamespace ...`n"

Set-AzServiceBusGeoDRConfigurationFailOver `
    -Name $sbusAliasName `
    -ResourceGroupName $resourceGroupName `
    -NamespaceName $sbusSecondaryNamespace `

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Step 5: Cleanup After Failover

Post-failover, we delete all queues in the primary namespace to clean up resources. This is crucial since the primary namespace will be “burned down” after the failover.

Write-Output "`nDeleting all queues in the primary $sbusPrimaryNamespace ..."

$queues = Get-AzServiceBusQueue -ResourceGroupName $resourceGroupName -NamespaceName $sbusPrimaryNamespace
foreach ($queue in $queues) {
    Remove-AzServiceBusQueue `
        -ResourceGroupName $resourceGroupName `
        -NamespaceName $sbusPrimaryNamespace `
        -QueueName $queue.Name

    Write-Host "Deleted queue: $($queue.Name)"
}

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Step 6: Reconfiguration

After cleaning up the primary namespace, the script sets the alias back to the secondary namespace.

Write-Output "`nSetting the alias back after failover ..."

New-AzServiceBusGeoDRConfiguration `
    -Name $sbusAliasName `
    -NamespaceName $sbusSecondaryNamespace `
    -ResourceGroupName $resourceGroupName `
    -PartnerNamespace $partnerId

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Write-Output "`nService Bus failover process completed successfully !!"

#************** DONE ************************************************************

Disconnect-AzAccount

Full codebase:

https://github.com/AtanuGpt/AzureServiceBusDR/blob/main/failover.ps1

Explanation of the Failover Process

Initial Provisioning Check: The script first ensures that both the primary and secondary namespaces are fully provisioned before initiating the failover.
Initiate Failover: The failover process is initiated using Set-AzServiceBusGeoDRConfigurationFailOver, which switches the alias to the secondary namespace.
Cleanup Primary Namespace: After the failover, the script deletes all queues in the primary namespace. This step is critical because the primary namespace is effectively “burned down” or rendered inactive and cleaned up. This involves removing all entities (queues, topics, etc.) in the primary namespace.
Reconfiguration: The alias is reconfigured to point to the secondary namespace, ensuring continued operation.
Final Check: The script performs a final check to ensure both namespaces are in the correct state post-failover.

Conclusion

Automating the disaster recovery process for Azure Service Bus is not just a convenience — it’s a necessity for maintaining high availability and ensuring business continuity. This PowerShell script provides a comprehensive solution, making the failover process seamless and efficient. By adopting automation, you can ensure that your Azure Service Bus environment is always prepared for any eventuality, keeping your services running smoothly even in the face of disruptions.

🚀 Revolutionizing Document Interaction: An AI-Powered PDF Voice-2-Voice Chatbot Using LlamaIndex 🐑, Langchain 🔗 Azure AI Speech 🎤and Google Audio 🔊

2024-06-18T16:12:00.001+05:30

Welcome to the third and last installment of this series on innovative AI-driven chatbots. In our journey from basic text-based interactions to text-2-voice-enabled capabilities, we now introduce a voice-2-voice-enabled PDF chatbot. This advanced system allows users to interact verbally with their PDF documents, significantly enhancing accessibility and usability. Let’s explore how this chatbot works and its implications for users.

Evolution of Chatbots: From Text to Voice

In our initial blog post, we introduced a text-based chatbot capable of processing queries from uploaded documents. This laid the groundwork for seamless interaction with textual information.

http://techiemate.blogspot.com/2024/06/revolutionizing-document-interaction-ai.html

Building on this, our second post showcased text to voice recognition integration, an interactive voice assistant capable of understanding and reading out the answer from the content in your PDFs. This enhancement marked a significant leap towards intuitive user engagement, catering to diverse user preferences and accessibility needs.

http://techiemate.blogspot.com/2024/06/revolutionizing-document-interaction-ai_13.html

Introducing Voice-to-Voice Interaction with PDFs

Today, we introduce our latest innovation: a voice-enabled PDF chatbot capable of both transcribing spoken queries and delivering spoken responses directly from PDF documents. This breakthrough technology bridges traditional document interaction with modern voice-driven interfaces, offering a transformative user experience.

The Technical Backbone: Exploring the Codebase

Let’s delve into the technical components that power our voice-enabled PDF chatbot:

Setting Up Dependencies and Environment

import os
import faiss
import streamlit as st
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage
from llama_index.vector_stores.faiss import FaissVectorStore
import azure.cognitiveservices.speech as speechsdk
import speech_recognition as sr

# Initialize Faiss index for vectorization
d = 1536
faiss_index = faiss.IndexFlatL2(d)
PERSIST_DIR = "./storage"

# Load environment variables
load_dotenv()

The code snippet above sets up necessary dependencies:

Faiss: Utilized for efficient document vectorization, enabling similarity search based on content.
Streamlit: Facilitates the user interface for seamless interaction with the chatbot and document upload functionality.
LangChain: Powers the message handling and communication within the chatbot interface.
LlamaIndex: Manages the storage and retrieval of vectorized document data, optimizing query performance.
Azure Cognitive Services (Speech SDK): Provides capabilities for speech recognition and synthesis, enabling the chatbot to transcribe and respond to spoken queries.
Google Audio : Provides speech into text using Google’s speech recognition API.

Document Handling and Vectorization

def saveUploadedFiles(pdf_docs):
    UPLOAD_DIR = 'uploaded_files'
    try:
        for pdf in pdf_docs:
            file_path = os.path.join(UPLOAD_DIR, pdf.name)
            with open(file_path, "wb") as f:
                f.write(pdf.getbuffer())
        return "Done"
    except:
        return "Error"

def doVectorization():
    try:
        vector_store = FaissVectorStore(faiss_index=faiss_index)
        storage_context = StorageContext.from_defaults(vector_store=vector_store)
        documents = SimpleDirectoryReader("./uploaded_files").load_data()
        index = VectorStoreIndex.from_documents(
            documents,
            storage_context=storage_context
        )
        index.storage_context.persist()
        return "Done"
    except:
        return "Error"

The saveUploadedFiles function saves PDF documents uploaded by users to a designated directory (uploaded_files). The doVectorization function utilizes Faiss to vectorize these documents, making them searchable based on content similarities.

Speech Recognition and Transcription

def transcribe_audio():
    recognizer = sr.Recognizer()
    microphone = sr.Microphone()

    with microphone as source:
        recognizer.adjust_for_ambient_noise(source)
        audio = recognizer.listen(source, timeout=20)
    
    st.write("🔄 Transcribing...")
    
    try:
        text = recognizer.recognize_google(audio)
        return text
    except sr.RequestError:
        return "API unavailable or unresponsive"
    except sr.UnknownValueError:
        return "Unable to recognize speech"

The transcribe_audio function uses the speech_recognition library to capture spoken queries from users via their microphone. It adjusts for ambient noise and listens for up to 20 seconds before transcribing the speech into text using Google's speech recognition API.

Querying and Fetching Data

def fetchData(user_question):
    try:
        vector_store = FaissVectorStore.from_persist_dir("./storage")
        storage_context = StorageContext.from_defaults(
            vector_store=vector_store, persist_dir=PERSIST_DIR
        )
        index = load_index_from_storage(storage_context=storage_context)
        query_engine = index.as_query_engine()
        response = query_engine.query(user_question)
        return str(response)
    except:
        return "Error"

The fetchData function retrieves relevant information from vectorized documents based on user queries. It loads the persisted Faiss index from storage and queries it to find and return the most relevant information matching the user's question.

Defining the Welcome Message

The WelcomeMessage variable contains a multi-line string that introduces users to the voice-enabled PDF chatbot. It encourages them to upload PDF documents and start asking questions:

WelcomeMessage = """
Hello, I am your PDF voice chatbot. Please upload your PDF documents and start asking questions to me. 
I would try my best to answer your questions from the documents.
"""

This message serves as the initial greeting when users interact with the chatbot, providing clear instructions on how to proceed.

Initializing Chat History and Azure Speech SDK Configuration

The code block initializes the chat history and sets up configurations for Azure Speech SDK:

if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
        AIMessage(content=WelcomeMessage)
    ]

AZURE_SPEECH_KEY = os.getenv("AZURE_SPEECH_KEY")
AZURE_SPEECH_REGION = os.getenv("AZURE_SPEECH_REGION")
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")

speech_config = speechsdk.SpeechConfig(subscription=AZURE_SPEECH_KEY, region=AZURE_SPEECH_REGION)
speech_config.speech_synthesis_voice_name = "en-US-AriaNeural"
speech_config.speech_synthesis_language = "en-US"

speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

Chat History Initialization: Checks if chat_history exists in the Streamlit session state. If not, it initializes it with the WelcomeMessage wrapped in an AIMessage object. This ensures that the chat starts with the welcome message displayed to the user.
Azure Speech SDK Configuration: Retrieves Azure Speech API key and region from environment variables (AZURE_SPEECH_KEY and AZURE_SPEECH_REGION). It sets up a SpeechConfig object for speech synthesis (speech_synthesizer). The voice name and language are configured to use the "en-US-AriaNeural" voice for English (US).

Streamlit Integration: User Interface Design

def main():
    load_dotenv()  

    st.set_page_config(
        page_title="Chat with multiple PDFs",
        page_icon=":sparkles:"
    )

    st.header("Chat with single or multiple PDFs :sparkles:")   

    for message in st.session_state.chat_history:
        if isinstance(message, AIMessage):
            with st.chat_message("AI"):
                st.markdown(message.content)
        elif isinstance(message, HumanMessage):
            with st.chat_message("Human"):
                st.markdown(message.content)     

    with st.sidebar:
        st.subheader("Your documents")
        pdf_docs = st.file_uploader(
            "Upload your PDFs here and click on 'Process'", 
            accept_multiple_files=True
        )

        if st.button("Process"):
            with st.spinner("Processing"):
                IsFilesSaved = saveUploadedFiles(pdf_docs)
                if IsFilesSaved == "Done":
                    IsVectorized = doVectorization()
                    if IsVectorized == "Done":
                        st.session_state.isPdfProcessed = "done"
                        st.success("Done!")
                    else:
                        st.error("Error! in vectorization")
                else:
                    st.error("Error! in saving the files")

    if st.button("Start Asking Question"):
        st.write("🎤 Recording started...Ask your  question")
        transcription = transcribe_audio()
        st.write("✅ Recording ended")

        st.session_state.chat_history.append(HumanMessage(content=transcription))

        with st.chat_message("Human"):
            st.markdown(transcription)
        
        with st.chat_message("AI"):
            with st.spinner("Fetching data ..."):
                response = fetchData(transcription)
                st.markdown(response)    
                
        result = speech_synthesizer.speak_text_async(response).get()
        st.session_state.chat_history.append(AIMessage(content=response))
        
    if "WelcomeMessage" not in st.session_state:
        st.session_state.WelcomeMessage = WelcomeMessage
        result = speech_synthesizer.speak_text_async(WelcomeMessage).get()

#============================================================================================================
if __name__ == '__main__':
    main()

User Experience: Seamless Interaction

Imagine uploading a collection of PDF documents — research papers, technical manuals, or reports — and simply speaking your questions aloud. The chatbot not only transcribes your speech but also responds audibly, providing immediate access to relevant information. This seamless interaction is particularly beneficial for users with visual impairments or those multitasking who prefer auditory information consumption.

Demo 💬🎤🤖

Enhancing Accessibility and Efficiency

Our voice-enabled PDF chatbot represents a significant advancement in accessibility technology. By integrating speech recognition, document vectorization, and AI-driven query processing, we empower users to effortlessly interact with complex information. This technology not only enhances accessibility but also boosts efficiency by streamlining the process of retrieving information from documents.

Conclusion: Paving the Way Forward

As we continue to explore the capabilities of AI in enhancing user experiences, the voice-enabled PDF chatbot stands as a testament to innovation in accessibility and usability. Whether you’re a researcher seeking insights from academic papers or a professional referencing technical documents, this technology promises to revolutionize how we interact with information.

Stay tuned as we push the boundaries further, exploring new applications and advancements in AI-driven technology. Stay connected and yes Happy Coding ! 😊

🚀 Revolutionizing Document Interaction: An AI-Powered PDF Text-2-Voice Chatbot Using LlamaIndex 🐑, Langchain 🔗 and Azure AI Speech 🔊

2024-06-13T16:01:00.004+05:30

In this blog post, we’re diving into the creation of an intelligent PDF voice chatbot using cutting-edge technologies like LangChain, LlamaIndex, and Azure AI Speech. This isn’t just another chatbot; it’s an interactive voice assistant capable of understanding and reading out the answer from the content in your PDFs. This project is a step up from my previous blog post where we explored building a text-based PDF chatbot without the voice functionality. If you missed that, be sure to check it out here

Technologies Used

LangChain: For chaining language models and building complex applications.
LlamaIndex: To index and query documents efficiently.
Azure AI Speech: For speech synthesis, giving our bot a human-like voice.
Streamlit: To create a user-friendly web interface.

Let’s jump into the code and see how these technologies come together to create our voice assistant chatbot.

Setting Up the Environment

First, ensure you have all the necessary libraries installed. You can do this by running:

pip install os faiss-cpu streamlit python-dotenv azure-cognitiveservices-speech langchain llama_index

Also, make sure you have your Azure Cognitive Services API keys ready.

The Code

Here’s the complete code for our voice chatbot:

import os
import faiss
import streamlit as st
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage
from llama_index.vector_stores.faiss import FaissVectorStore
import azure.cognitiveservices.speech as speechsdk

d = 1536
faiss_index = faiss.IndexFlatL2(d)
PERSIST_DIR = "./storage"

AZURE_SPEECH_KEY = os.getenv("AZURE_SPEECH_KEY")
AZURE_SPEECH_REGION = os.getenv("AZURE_SPEECH_REGION")
OPENAI_API_KEY=os.getenv("OPENAI_API_KEY")

def saveUploadedFiles(pdf_docs):
    UPLOAD_DIR = 'uploaded_files'
    try:
        for pdf in pdf_docs:
            file_path = os.path.join(UPLOAD_DIR, pdf.name)
            with open(file_path, "wb") as f:
                f.write(pdf.getbuffer())
        return "Done"
    except:
        return "Error"

def doVectorization():    
    try:
        vector_store = FaissVectorStore(faiss_index=faiss_index)
        storage_context = StorageContext.from_defaults(vector_store=vector_store) 
        documents = SimpleDirectoryReader("./uploaded_files").load_data()
        index = VectorStoreIndex.from_documents(
            documents, 
            storage_context=storage_context
        )
        index.storage_context.persist()
        return "Done"
    except:
        return "Error"

def fetchData(user_question):
    try:
        vector_store = FaissVectorStore.from_persist_dir("./storage")
        storage_context = StorageContext.from_defaults(
            vector_store=vector_store, persist_dir=PERSIST_DIR
        )
        index = load_index_from_storage(storage_context=storage_context)
        query_engine = index.as_query_engine()
        response = query_engine.query(user_question)
        return str(response)
    except:
        return "Error"

WelcomeMessage = """
Hello, I am your PDF voice chatbot. Please upload your PDF documents and start asking questions. 
I will try my best to answer your questions based on the documents.
"""

if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
        AIMessage(content=WelcomeMessage)
    ]

AZURE_SPEECH_KEY = os.getenv("AZURE_SPEECH_KEY")
AZURE_SPEECH_REGION = os.getenv("AZURE_SPEECH_REGION")
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")

speech_config = speechsdk.SpeechConfig(subscription=AZURE_SPEECH_KEY, region=AZURE_SPEECH_REGION)
speech_config.speech_synthesis_voice_name = "en-US-AriaNeural"   
speech_config.speech_synthesis_language = "en-US"

speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

def main():
    load_dotenv()  

    st.set_page_config(
        page_title="Chat with multiple PDFs",
        page_icon=":sparkles:"
    )

    st.header("Chat with single or multiple PDFs :sparkles:")   

    for message in st.session_state.chat_history:
        if isinstance(message, AIMessage):
            with st.chat_message("AI"):
                st.markdown(message.content)
        elif isinstance(message, HumanMessage):
            with st.chat_message("Human"):
                st.markdown(message.content)     

    with st.sidebar:
        st.subheader("Your documents")
        pdf_docs = st.file_uploader(
            "Upload your PDFs here and click on 'Process'", 
            accept_multiple_files=True
        )

        if st.button("Process"):
            with st.spinner("Processing"):
                IsFilesSaved = saveUploadedFiles(pdf_docs)
                if IsFilesSaved == "Done":
                    IsVectorized = doVectorization()
                    if IsVectorized == "Done":
                        st.session_state.isPdfProcessed = "done"
                        st.success("Done!")
                    else:
                        st.error("Error! in vectorization")
                else:
                    st.error("Error! in saving the files")
        
    user_question = st.chat_input("Ask a question about your document(s):")

    if user_question is not None and user_question != "":
        st.session_state.chat_history.append(HumanMessage(content=user_question))

        with st.chat_message("Human"):
            st.markdown(user_question)
        
        with st.chat_message("AI"):
            with st.spinner("Fetching data ..."):
                response = fetchData(user_question)
                st.markdown(response)    
                
        result = speech_synthesizer.speak_text_async(response).get()
        st.session_state.chat_history.append(AIMessage(content=response))
    
    if "WelcomeMessage" not in st.session_state:
        st.session_state.WelcomeMessage = WelcomeMessage
        result = speech_synthesizer.speak_text_async(WelcomeMessage).get()

if __name__ == '__main__':
    main()

Breaking Down the Code

Environment Setup

We start by importing necessary libraries and loading environment variables using dotenv. The .env file :

OPENAI_API_KEY = ""
AZURE_OPENAI_API_KEY = ""
AZURE_SPEECH_KEY = ""
AZURE_SPEECH_REGION = ""

File Upload and Vectorization

The saveUploadedFiles function handles file uploads, saving PDFs to a directory. The doVectorization function uses Llama_Index and FAISS to create a searchable index from the uploaded documents.

Fetching Data

The fetchData function retrieves answers to user questions by querying the vector index.

Voice Synthesis

We use Azure AI Speech service to convert text responses into speech. This involves configuring the speechsdk.SpeechConfig and synthesizing speech with speechsdk.SpeechSynthesizer.

Streamlit Interface

Streamlit makes it easy to build a web interface. The sidebar allows users to upload and process PDFs. The main chat interface displays the conversation and handles user inputs.

Running the App

Run the Streamlit app using:

streamlit run your_script_name.py

Upload your PDF documents, ask questions, and listen as the bot responds with both text and voice.

Conclusion

By combining LangChain, LlamaIndex, and Azure AI, we’ve created an intelligent PDF voice chatbot that can make interacting with documents more engaging and accessible. Whether you’re using it for research, studying, or just exploring, this project showcases the potential of modern AI and NLP technologies.

Don’t forget to check out the previous blog post for a text-only version of this chatbot. Happy coding!

🚀 Revolutionizing Document Interaction: An AI-Powered PDF Chatbot Using LlamaIndex 🐑 and Langchain 🔗

2024-06-11T11:47:00.001+05:30

In today’s fast-paced digital world, efficiently accessing and extracting information from documents is more crucial than ever. What if you could chat with your PDFs, asking questions and getting precise answers instantly? Enter our innovative AI-powered PDF chatbot, combining the capabilities of LlamaIndex and Langchain. In this blog post, I’ll take you through the journey of building this exciting project and how it can transform your document interaction experience.

The Vision: Intelligent Document Interaction

Imagine you’re dealing with multiple PDF documents, trying to extract specific information quickly. Traditional methods can be time-consuming and tedious. Our AI chatbot changes the game by allowing you to interact with your documents through natural language queries, providing accurate answers instantly. This project leverages advanced AI technologies, including LlamaIndex for vector storage and Langchain for conversational AI, to make document interaction smarter and more intuitive.

The Technology Stack

To bring this vision to life, we used a robust technology stack:

LlamaIndex: For creating and managing a vector store to efficiently handle document data.
Langchain: For building the conversational AI interface.
FAISS (Facebook AI Similarity Search): For fast and scalable similarity search.
Streamlit: For creating an interactive and user-friendly web interface.
Python: The backbone of our project, tying everything together.

Let’s Dive into the Code

Here’s a detailed walkthrough of the code that powers our AI PDF chatbot.

Setting Up the Environment

First, we need to import the necessary libraries and set up the FAISS index and environment variables.

import os
import faiss
import streamlit as st
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage
from llama_index.vector_stores.faiss import FaissVectorStore

d = 1536
faiss_index = faiss.IndexFlatL2(d)
PERSIST_DIR = "./storage"

Handling File Uploads

We need a function to save uploaded PDF files. This function saves the files to a specified directory.

def saveUploadedFiles(pdf_docs):
    UPLOAD_DIR = 'uploaded_files'
    try:
        for pdf in pdf_docs:
            file_path = os.path.join(UPLOAD_DIR, pdf.name)
            with open(file_path, "wb") as f:
                f.write(pdf.getbuffer())
        return "Done"
    except:
        return "Error"

Vectorizing Documents

The doVectorization function reads the uploaded PDFs, converts them into vectors, and stores them in the FAISS index.

def doVectorization():    
    try:
        vector_store = FaissVectorStore(faiss_index=faiss_index)
        storage_context = StorageContext.from_defaults(vector_store=vector_store) 
        documents = SimpleDirectoryReader("./uploaded_files").load_data()
        index = VectorStoreIndex.from_documents(
            documents, 
            storage_context=storage_context
        )
        index.storage_context.persist()
        return "Done"
    except:
        return "Error"

Fetching Data Based on User Queries

The fetchData function processes user queries and retrieves relevant information from the vectorized documents.

def fetchData(user_question):
    try:
        vector_store = FaissVectorStore.from_persist_dir("./storage")
        storage_context = StorageContext.from_defaults(
            vector_store=vector_store, persist_dir=PERSIST_DIR
        )
        index = load_index_from_storage(storage_context=storage_context)
        query_engine = index.as_query_engine()
        response = query_engine.query(user_question)
        return str(response)
    except:
        return "Error"

Building the Streamlit Interface

The main function sets up the Streamlit interface, handles file uploads, processes documents, and manages user interactions.

if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
        AIMessage(content="Hello, I am a bot. How can I help you?")
    ]

def main():
    load_dotenv()
    os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")

    st.set_page_config(
        page_title="Chat with multiple PDFs",
        page_icon=":sparkles:"
    )

    st.header("Chat with single or multiple PDFs :sparkles:")

    for message in st.session_state.chat_history:
        if isinstance(message, AIMessage):
            with st.chat_message("AI"):
                st.markdown(message.content)
        elif isinstance(message, HumanMessage):
            with st.chat_message("Human"):
                st.markdown(message.content)

    with st.sidebar:
        st.subheader("Your documents")
        pdf_docs = st.file_uploader(
            "Upload your PDFs here and click on 'Process'", 
            accept_multiple_files=True
        )

        if st.button("Process"):
            with st.spinner("Processing"):
                IsFilesSaved = saveUploadedFiles(pdf_docs)
                if IsFilesSaved == "Done":
                    IsVectorized = doVectorization()
                    if IsVectorized == "Done":
                        st.session_state.isPdfProcessed = "done"
                        st.success("Done!")
                    else:
                        st.error("Error! in vectorization")
                else:
                    st.error("Error! in saving the files")

    if "isPdfProcessed" in st.session_state:
        user_question = st.chat_input("Ask a question about your document(s):")

        if user_question is not None and user_question != "":
            st.session_state.chat_history.append(HumanMessage(content=user_question))

            with st.chat_message("Human"):
                st.markdown(user_question)
            
            with st.chat_message("AI"):
                with st.spinner("Fetching data ..."):
                    response = fetchData(user_question)
                    st.markdown(response)    
            
            st.session_state.chat_history.append(AIMessage(content=response))

if __name__ == '__main__':
    main()

Bringing It All Together

With our AI PDF chatbot, you can upload multiple PDFs, process them, and interact with the content effortlessly. The chatbot provides precise answers to your queries, making document interaction more efficient and enjoyable. This project showcases the power of combining LlamaIndex and Langchain to create intelligent, user-friendly AI applications.

Join the AI revolution and transform how you interact with your documents. Happy coding! 🚀

Bringing AI to Your Fashion Shop: Meet ShopBot, Your Friendly Shopping Assistant! 🛒

2024-06-06T10:25:00.002+05:30

Welcome to the future of online shopping! Imagine having a friendly assistant right at your fingertips, ready to help you navigate through a variety of stylish clothing, accessories, and more. Say hello to ShopBot, an AI-powered assistant for your Fashion Shop, designed to enhance your shopping experience and make it more enjoyable than ever.

In this blog post, we’ll try to simulate a shopping store and dive into the exciting world of AI in e-commerce, showcase how ShopBot works, and provide you with all the code snippet to get started with your very own AI shopping assistant.

Why AI in E-Commerce?

Artificial Intelligence has revolutionized many industries, and e-commerce is no exception. Here are a few reasons why integrating AI into your online store can be a game-changer:

Personalized Shopping Experience: AI can tailor recommendations based on customer preferences and browsing history.
24/7 Customer Support: AI chatbots can assist customers at any time, providing instant responses and support.
Efficient Navigation: AI can help customers find products quickly and easily, enhancing their overall shopping experience.

Introducing ShopBot

ShopBot is an AI assistant designed specifically for aFashion Shop. It assists customers in browsing products, providing information, and guiding them through the checkout process. Let’s see how you can create your very own ShopBot using Python, Streamlit and Azure Open AI

Setting Up ShopBot

Here’s a step-by-step guide to creating ShopBot. The code below demonstrates how to use Azure OpenAI, Streamlit, and a predefined product list to build an engaging shopping assistant.

Step 1: Import Required Libraries

First, import the necessary libraries:

import os
from openai import AzureOpenAI
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage
import streamlit as st

Step 2: Load Environment Variables

Load your environment variables to access your Azure OpenAI API keys:

load_dotenv()

Step 3: Define the Product List

Here’s a sample product list. Credit goes to Yennhi95zz’s medium blog post for this comprehensive list:

product_list = '''
# Fashion Shop Product List

## Women's Clothing:
- T-shirt  
- Price: $20  
- Available Sizes: Small, Medium, Large, XL  
- Available Colors: Red, White, Black, Gray, Navy

- Elegant Evening Gown  
- Price: $150  
- Available Sizes: Small, Medium, Large, XL  
- Available Colors: Black, Navy Blue, Burgundy

- Floral Summer Dress  
- Price: $45  
- Available Sizes: Small, Medium, Large  
- Available Colors: Floral Print, Blue, Pink

- Professional Blazer  
- Price: $80  
- Available Sizes: Small, Medium, Large, XL  
- Available Colors: Black, Gray, Navy

## Men's Clothing:
- Classic Suit Set  
- Price: $200  
- Available Sizes: Small, Medium, Large, XL  
- Available Colors: Charcoal Gray, Navy Blue, Black

- Casual Denim Jeans  
- Price: $35  
- Available Sizes: 28, 30, 32, 34  
- Available Colors: Blue Denim, Black

- Polo Shirt Collection  
- Price: $25 each  
- Available Sizes: Small, Medium, Large, XL  
- Available Colors: White, Blue, Red, Green

## Accessories:
- Stylish Sunglasses  
- Price: $20  
- Available Colors: Black, Brown, Tortoise Shell

- Leather Handbag  
- Price: $60  
- Available Colors: Brown, Black, Red

- Classic Wristwatch  
- Price: $50  
- Available Colors: Silver, Gold, Rose Gold

## Footwear:
- High-Heel Ankle Boots  
- Price: $70  
- Available Sizes: 5-10  
- Available Colors: Black, Tan, Burgundy

- Comfortable Sneakers  
- Price: $55  
- Available Sizes: 6-12  
- Available Colors: White, Black, Gray

- Formal Leather Shoes  
- Price: $90  
- Available Sizes: 7-11  
- Available Colors: Brown, Black

## Kids' Collection:
- Cute Cartoon T-shirts  
- Price: $15 each  
- Available Sizes: 2T, 3T, 4T  
- Available Colors: Blue, Pink, Yellow

- Adorable Onesies  
- Price: $25  
- Available Sizes: Newborn, 3M, 6M, 12M  
- Available Colors: Pastel Blue, Pink, Mint Green

- Trendy Kids' Backpacks  
- Price: $30  
- Available Colors: Blue, Red, Purple

## Activewear:
- Yoga Leggings  
- Price: $30  
- Available Sizes: Small, Medium, Large  
- Available Colors: Black, Gray, Teal

- Running Shoes  
- Price: $40  
- Available Sizes: 6-12  
- Available Colors: White, Black, Neon Green

- Quick-Dry Sports T-shirt  
- Price: $20  
- Available Sizes: Small, Medium, Large  
- Available Colors: Red, Blue, Gray

'''

Step 4: Create the Context for ShopBot

Define the context for your AI assistant:

def get_context():
    context = [{'role': 'system',
                'content': f"""
    You are ShopBot, an AI assistant for my online fashion shop - My Fashion Shop. 
    Your role is to assist customers in browsing products, providing information, and guiding them through the checkout process. 
    Be friendly and helpful in your interactions. We offer a variety of products across categories such as Women's Clothing, 
    Men's clothing, Accessories, Kids' Collection, Footwears and Activewear products. 
    Feel free to ask customers about their preferences, recommend products, and inform them about any ongoing promotions.
    The Current Product List is limited as below:

    ```{product_list}```

    Make the shopping experience enjoyable and encourage customers to reach out if they have any questions or need assistance.
    """}]
    return context

Step 5: Define the Function to Get AI Responses

Create a function to get responses from the AI:

def get_completion_from_messages(messages):
    client = AzureOpenAI(
        api_key = os.environ["AZURE_OPENAI_API_KEY"],
        api_version = os.environ["API_VERSION"],
        azure_endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
    )

    chat_completion = client.chat.completions.create(
        model = os.environ["DEPLOYMENT_MODEL"],
        messages = messages
    )

    return chat_completion.choices[0].message.content

Step 6: Initialize Streamlit Session State

Initialize the chat history and context:

if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
      AIMessage(content="Hello! I'm your Ecom Assistant. How can I help you today"),
    ]

if "chat_msg" not in st.session_state:
    st.session_state.chat_msg = get_context()

Step 7: Set Up Streamlit UI

Configure Streamlit page and display the chat interface:

st.set_page_config(page_title="Chat with My Shop", page_icon=":speech_balloon:")

st.title("Ecommerce Chatbot :speech_balloon:")


for message in st.session_state.chat_history:
    if isinstance(message, AIMessage):
        with st.chat_message("AI"):
            st.markdown(message.content)
    elif isinstance(message, HumanMessage):
        with st.chat_message("Human"):
            st.markdown(message.content)

Step 8: Handle User Input

Capture and respond to user queries:

user_query = st.chat_input("Type a message...")
if user_query is not None and user_query.strip() != "":
    st.session_state.chat_msg.append({"role" : "user", "content" : f"{user_query}"})
    st.session_state.chat_history.append(HumanMessage(content=user_query))

    with st.chat_message("Human"):
        st.markdown(user_query)    

    with st.chat_message("AI"):
        with st.spinner("Please wait ..."): 
            response = get_completion_from_messages(st.session_state.chat_msg)
            st.markdown(response)

    st.session_state.chat_msg.append({"role" : "user", "content" : f"{response}"})
    st.session_state.chat_history.append(AIMessage(content=response))

Step 8: Run the application

Run the app:

streamlit run <<yourapp>>.py

Try It Yourself!

That’s it! You’ve now created your very own AI shopping assistant. With ShopBot, your customers can enjoy a personalized and efficient shopping experience, making them feel valued and well-assisted. Whether they need recommendations, product details, or help with the checkout process, ShopBot is here to help.

Feel free to customize the product list and enhance the functionality to suit your store’s unique needs. Happy coding and happy shopping!

Integrating Azure Cognitive Services with OpenAI for interactive Voice-based AI Responses 🤖🎤

2024-05-30T12:52:00.009+05:30

In today's fast-moving digital world, mixing top-notch AI tech is opening up endless chances for cool experiences. When we combine Azure Cognitive Services, which are really good at understanding speech, with OpenAI, which is great at understanding natural language, magic happens. It lets us make AI systems that talk back to us in real time, understanding what we say like it's second nature. Jump into the world where Azure meets OpenAI, and see how it's changing the way we interact with computers.

In this blog post, we’ll explore a Python script that achieves this integration, enabling a fully interactive voice-based AI system.

Overview

The script leverages Azure’s Speech SDK to recognize spoken language and OpenAI’s language model to generate responses. The process involves several key steps:

Loading Environment Variables: Using dotenv to manage API keys securely.
Configuring Azure Speech SDK: Setting up speech recognition and synthesis.
Invoking OpenAI API: Using the recognized speech to generate a response.
Synthesizing AI Response: Converting the AI-generated text back to speech.

Let’s delve into the details.

Prerequisites

Before you start, ensure you have the following:

Azure Cognitive Services subscription with Speech API.
OpenAI API key.
Python environment with necessary libraries installed (azure-cognitiveservices-speech, python-dotenv, langchain_openai).

Code Walkthrough

Below is the complete script with comments to guide you through each section:

import os
from dotenv import load_dotenv
import azure.cognitiveservices.speech as speechsdk
from langchain_openai import OpenAI

# Load environment variables from a .env file
load_dotenv()

# Retrieve API keys from environment variables
AZURE_SPEECH_KEY = os.getenv("AZURE_SPEECH_KEY")
AZURE_SPEECH_REGION = os.getenv("AZURE_SPEECH_REGION")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

# Initialize the OpenAI language model with the API key
llm = OpenAI(api_key=OPENAI_API_KEY)

# Configure Azure Speech SDK for speech recognition
speech_config = speechsdk.SpeechConfig(subscription=AZURE_SPEECH_KEY, region=AZURE_SPEECH_REGION)
speech_config.speech_recognition_language = "en-US"

# Set up the audio input from the default microphone
audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

print("You can speak now. I am listening ...")

# Start speech recognition
speech_recognition_result = speech_recognizer.recognize_once_async().get()

# Check the result of the speech recognition
if speech_recognition_result.reason == speechsdk.ResultReason.RecognizedSpeech:
    output = speech_recognition_result.text
    print(output)  # User question

    # Invoke the OpenAI API with the recognized speech text
    result = llm.invoke(output)
    print(result)  # AI Answer

    # Configure audio output for speech synthesis
    audio_config = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)
    speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=audio_config)
    
    # Synthesize the AI response into speech
    speech_synthesizer_result = speech_synthesizer.speak_text_async(result).get()

elif speech_recognition_result.reason == speechsdk.ResultReason.NoMatch:
    print("No speech could be recognized: {}".format(speech_recognition_result.no_match_details))

elif speech_recognition_result.reason == speechsdk.ResultReason.Canceled:
    cancellation_details = speech_recognition_result.cancellation_details
    print("Speech Recognition canceled: {}".format(cancellation_details.reason))
    if cancellation_details.reason == speechsdk.CancellationReason.Error:
        print("Error details: {}".format(cancellation_details.error_details))

Step-by-Step Breakdown

1. Loading Environment Variables

Using the dotenv package, we load sensitive API keys from a .env file, ensuring they are not hard-coded in the script.

2. Configuring Azure Speech SDK

We set up the Azure Speech SDK with the necessary subscription key and region. This configuration allows the SDK to access Azure’s speech recognition services.

3. Speech Recognition

The SpeechRecognizer object listens for speech input via the default microphone and processes the speech to text asynchronously. Upon recognition, it checks the result and extracts the recognized text.

4. Invoking OpenAI API

The recognized text is then passed to OpenAI’s language model, which generates a relevant response. This integration allows for dynamic interaction, where the AI can understand and respond contextually.

5. Speech Synthesis

Finally, the AI-generated text response is synthesized back into speech using Azure’s SpeechSynthesizer, providing a spoken response through the default speaker.

Error Handling

The script includes basic error handling for different outcomes of the speech recognition process:

RecognizedSpeech: When speech is successfully recognized.
NoMatch: When no speech is recognized.
Canceled: When the recognition process is canceled, potentially due to errors.

Conclusion

Integrating Azure Cognitive Services with OpenAI creates a powerful platform for developing interactive voice applications. This combination leverages the robust speech recognition and synthesis capabilities of Azure with the advanced natural language understanding of OpenAI. Whether for virtual assistants, customer support, or other innovative applications, this integration showcases the potential of modern AI technologies.

Feel free to experiment with the code and adapt it to your specific use case. The possibilities are endless when combining these advanced tools in creative ways.

Creating a Fake Data Generator with Python 🤖📅

2024-05-28T10:54:00.004+05:30

In the world of data science and machine learning, having access to diverse datasets for experimentation is crucial. However, real-world data can be scarce, sensitive, or simply not available. This is where generating fake data becomes invaluable. In this post, I’ll walk you through creating a fake data generator using Python’s Faker library and displaying this data in a well-styled table using pandas.

Python output

HTML Output

What You’ll Need

Before we dive in, ensure you have the following installed:

Python 3.x
pandas library: pip install pandas
Faker library: pip install faker

Step-by-Step Guide

Let’s build a script that generates a set of fake personal data and displays it in a structured format.

1. Import Necessary Libraries

First, import the Faker and pandas libraries:

from faker import Faker
import pandas as pd

2. Initialize Faker

Create an instance of the Faker class:

fake = Faker()

3. Generate Fake Data

Define a dictionary to store the fake data:

data = { 
    'First Name': [],
    'Last Name': [],
    'DOB': [],
    'Email': [],
    'City': [],
    'Country': [],
    'ZipCode': [],
}

Use a loop to populate the dictionary with fake data: say 10 records

for i in range(11):
    data['First Name'].append(fake.first_name())
    data['Last Name'].append(fake.last_name())
    data['DOB'].append(fake.date_of_birth())
    data['Email'].append(fake.email())
    data['City'].append(fake.city())
    data['ZipCode'].append(fake.zipcode())
    data['Country'].append(fake.country())

4. Create a DataFrame

Convert the dictionary to a pandas DataFrame:

df = pd.DataFrame(data)

Print the DataFrame to see the generated data:

print(df)

5. Style the DataFrame (Optional)

To display the DataFrame with borders, define a style and save the styled DataFrame to an HTML file:

styles = [
    {'selector': 'table', 'props': [('border-collapse', 'collapse')]},
    {'selector': 'th, td', 'props': [('border', '1px solid black'), ('padding', '5px')]},
]

styled_df = df.style.set_table_styles(styles)
styled_df.to_html('styled_table.html')

6. Open the Styled Table (Optional)

The styled DataFrame is saved as styled_table.html. You can open this file in a web browser to view the data in a neat, tabular format with borders.

print("Styled table saved as 'styled_table.html'. Open this file in a web browser to view the table with borders.")

Full Code

Here’s the full script:

from faker import Faker
import pandas as pd

fake = Faker()

data = { 
    'First Name': [],
    'Last Name': [],
    'DOB': [],
    'Email': [],
    'City': [],
    'Country': [],
    'ZipCode': [],
}

for i in range(11):
    data['First Name'].append(fake.first_name())
    data['Last Name'].append(fake.last_name())
    data['DOB'].append(fake.date_of_birth())
    data['Email'].append(fake.email())
    data['City'].append(fake.city())
    data['ZipCode'].append(fake.zipcode())
    data['Country'].append(fake.country())

df = pd.DataFrame(data)

print(df)

# Generate a styled DataFrame and view it by saving the DataFrame to an HTML file and then opening that file in a web browser
# Define the table style to add borders
styles = [
    {'selector': 'table', 'props': [('border-collapse', 'collapse')]},
    {'selector': 'th, td', 'props': [('border', '1px solid black'), ('padding', '5px')]},
]

styled_df = df.style.set_table_styles(styles)
styled_df.to_html('styled_table.html')

print("Styled table saved as 'styled_table.html'. Open this file in a web browser to view the table with borders.")

Conclusion

Using Python’s Faker library, you can easily generate realistic fake data for various purposes, from testing to developing machine learning models. By combining it with pandas, you can structure this data in a DataFrame and apply styles to make it more readable and visually appealing. This approach is not only practical but also enhances the presentation of your data, making it easier to analyze and share.

Open the styled_table.html file in your web browser, and you'll see a beautifully formatted table containing the fake data, ready for use in your projects. Happy coding!

Virtual Network IP Addresses planning

2024-02-02T14:23:00.002+05:30

During Virtual network setup, the IP Address planning is very important else may run into over-provisioning or under-provisioning of resources. Be it in cloud or on-premise the concepts remain the same.

Below is the list which will help as a reference while you plan for your Virtual Network and Subnet configuration. 👍

Azure Automation : Periodically delete log files in Virtual Machines

2022-10-19T16:51:00.006+05:30

It's vary common that you are running multiple agents in your virtual machines. Microsoft also deploys several agents behind the scene in your VMs to satisfy various configurations requirements. Now the problem starts when these agents start generating logs files locally and eventually starts eating up the disk space.

You naturally by instinct jumps in to scale up your VMs or starts adding more disks to it inorder to maintain the performance. But this approach would cost your dearly as your Azure bill would shoot up substantially. Why not clean up those logs periodically with a scheduler and make your boss happy 😀

Azure Automation with runbook type PowerShell is there to make you achieve this very comfortably. Create an automation account with a PowerShell runbook. The script is given below. Copy-Paste it >> Test it >> Schedule it. Done! Smart work isn't it 👍

$ResourceGroup = '<<resource group name>>'
$FolderPath = 'rm /<<log folder path>>/*' #E.g. rm /home/user/temp/*

try
{
	Write-Output "Connecting ..."
	$message = Connect-AzAccount -Identity 
	Write-Output "Connection successful"

	#Get all Azure VMs which are in running state and are running Linux
	$myAzureVMs = Get-AzVM -status | Where-Object {$_.PowerState -eq "VM running" -and $_.StorageProfile.OSDisk.OSType -eq "Linux"}
	Write-Output "Total Linux VMs detected : $($myAzureVMs.Name.Count)"
	Write-Output "VM Names : $($myAzureVMs.Name)" 

	# Run the script against all the listed VMs
	if($myAzureVMs.Name.Count -gt 1)
	{
		For($val = 0; $val -le $myAzureVMs.Name.Count-1; $val++)
		{
			Write-Output "Target VM : $($myAzureVMs.Name[$val]), File deletion in progress ..."
			$message = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $myAzureVMs.Name[$val] -CommandId 'RunShellScript' -ScriptString $FolderPath
			Write-Output "Operation completed successfully" 
		}	
	}
	else {
		Write-Output "Target VM : $($myAzureVMs.Name), File deletion in progress ..."
		$message = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $myAzureVMs.Name -CommandId 'RunShellScript' -ScriptString $FolderPath
		Write-Output "Operation completed successfully" 
	}
}
catch
{
	Write-Error -Message $_.Exception
    throw $_.Exception
}

Azure Active Directory Login for Linux Virtual Machines

2022-09-14T17:07:00.015+05:30

To improve the security of Linux virtual machines (VMs) in Azure, now we can integrate the VM with Azure Active Directory (Azure AD) authentication. Azure AD can be used as a core authentication platform to SSH into a Linux VM. This functionality allows organizations to manage security access to VMs with Azure role-based access control (RBAC) and Conditional Access policies.

This article shows how to create and configure a Linux VM and log in with Azure AD by using OpenSSH certificate-based authentication.

1.1. Benefits

Use your Azure AD credentials to log in to Azure Linux Virtual machines. No need to remember any separate login credentials.
No separate Bastion Virtual machine is required.
No public IP/Port exposure for any Virtual machines to the outside world.
Get Open SSH key-based authentication without needing to distribute SSH keys to users or provision SSH public keys on any Azure Linux VMs that you deploy.
Reduce reliance on local administrator accounts, credential theft, and weak credentials.
With RBAC, specify who can log in to a virtual machine as a regular user or with administrator privileges. When users join your team, you can update the Azure RBAC policy for the VM to grant access as appropriate. When employees leave your organization and their user accounts are disabled or removed from Azure AD, they no longer have access to your resources.
With Conditional Access, configure policies to require multifactor authentication or to require that your client device is managed (for example, compliant or hybrid Azure AD joined) before you can use it SSH into Linux VMs.

Note

Azure Active Directory native client for Azure Bastion is officially supported by Microsoft and is now GA. In this article we will use this feature to solve our objective of login to virtual machines with active directory login credentials.

1.2. Installing Azure CLI on Local Laptop/Desktop

Open command prompt. Type az at the root. This will check if azure cli is already installed in your VM and if found you would see something similar below and you can skip this section and move to next section.

If not found then go ahead and install Azure CLI from the below link.

Windows : https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli

Linux : curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

1.3. Configuring AAD login feature on New VMs

Create a new Linux Virtual Machine. Keeping all other configurations of your choice, make sure to check the "Login with Azure AD" feature on under the Management tab. 💡 This is very important! The identity will be automatically get selected to system assigned managed identity.

Once the VM gets provisioned, you must see the installed Azure AD Login extensions under the Extensions + applications menu under Settings blade of the respective VM.

1.4. Enabling AAD login feature on Existing VMs

Go to the Setting blade of the respective VM and click on Extensions + application menu. Click +Add to add a new extension

Select extension "Azure AD based SSH Login" and click Next and then Review and create button. Click Create after the validation passed message is shown.

After the installation complete check the Extensions + application menu again under the Settings blade of the VM. This time it must show the installed extension.

1.5. Terraform Script to enable AAD login (Linux VM)

resource "azurerm_linux_virtual_machine" "example" {
   // blah-blah
   identity {
      type = "SystemAssigned" 
   }
}

resource "azurerm_virtual_machine_extension" "example" {
   name = "AADSSHLoginForLinux_extn"
   virtual_machine_id = azurerm_linux_virtual_machine.example.id
   publisher = "Microsoft.Azure.ActiveDirectory"
   type = "AADSSHLoginForLinux"
   type_handler_version = "1.0"
}

1.6. Enabling RBAC (Configure role assignments)

Any of the following roles must be given to the user accessing the VM.

💡 Assigning any of these roles to a particular user or to a particular group is mandatory inorder to access the VM via Azure AD credentials.

Virtual Machine Administrator Login: Users who have this role assigned can log in to an Azure virtual machine with administrator privileges.
Virtual Machine User Login: Users who have this role assigned can log in to an Azure virtual machine with regular user privileges

From the Access control (IAM) menu of the respective VM; click on +Add and select Add role assignment submenu option. You must have "Owner" or "User Access Administrator" privilege to perform this operation.

Search for any of the above role and proceed. I have selected the "Virtual Machine Administrator Login" role for this article. Click Next

Select an user or group to whom you want the role to get assigned. Click Review + assign

1.7. Configuring Azure Bastion (With Native support, New Bastion Host)

Create a new azure bastion service. Keeping all other configurations of your choice, make sure to check the "Native client support" feature on under the Advanced tab. 💡 This is very important!

1.8. Configuring Azure Bastion (With Native support, Existing Bastion Host)

Go to the Configuration option under the Setting tab and check the "Native client support" checkbox. Hit Apply.

1.9. Terraform Script to enable Native client support for Azure Bastion Host

resource "azurerm_bastion_host" "example" {
   // blah-blah
   tunneling_enabled = true
}

1.10. Login to Application VMs with Azure AD credentials via Bastion Host (Local Laptop/Desktop)

Open Windows Command Prompt or Windows PowerShell IDE in your machine.
Type az login and press enter. (Make sure you are disconnected from G+D VPN, 💡 This is important!). This will open up a browser window as would ask for login credentials. Please enter your azure active directory login credentials.

Upon successful login, it will show all the subscription on which you have access under the tenant. You can execute the following command to list all subscriptions in a readable table format.

az account list -o table

Type the following command to check the current subscription you are logged in.

az account show -o table

If it's the right subscription then move to the next step. Else execute the following command to set your correct subscription.

az account set --subscription "<<subscription_id or subscription_name>>"

Get the resource id of your target VM by running the following query. Copy the return value and keep in notepad. You would need this in the next step

 az vm show --name <<vm name>> --resource-group <<resource group name>> --query "id"

Now the important part which we are waiting for. We are going to login to our target VM via Azure Bastion. Run the following command

 az network bastion ssh \
 --name "<<bastion-host name>>" \
 --resource-group "<<resource group name>>" \
 --target-resource-id "<<vm resource id copied from the previous step>>" \
 --auth-type "AAD"

Voila!... you are now successfully connected to your target VM via Azure Bastion. 😊 The prompt will quickly change to show your [AAD login id]@[target VM name]

Congratulations! for coming this far. Hope this article will help you to further explore more on this topic.

Do share with me about your experience and what you have built upon this foundation. I would love to hear from you.

Azure SQL Diagnostics & Event Hub

2022-02-10T18:30:00.002+05:30

Auditing & Monitoring is one of the strong pillar in any SDLC process and in today's cloud arena it became even more important. In this article we would learn how to enable auditing capability in Azure SQL database and would also witness some LIVE stream of audit data. When it comes to auditing in Azure SQL, the platform gives you three choices for log destination. You can choose as per your requirement

You can enable auditing either at the server level or at the database level. For this article we would go ahead and enable it at the database level. So, go to your Azure SQL database instance, select Auditing from the left pane under Security and enable auditing.

We are going to use Event Hub for this article to stream all telemetry data from Azure SQL. So, go ahead and select that option and provide the required details. Make sure you have created your Event Hub resource and namespace beforehand in the same region where your Azure SQL service is running. Hit Save.

This will go ahead and add a new log category named "SQLSecurityAuditEvents" in the Azure SQL Diagnostic setting. This is the LIVE telemetry information you need to capture for all your SQL compliance and security audits. For example

What are all SQL statements getting executed and on which database, from which IP address those request came and at what datetime, how many rows got affected and much more.....
Successful logins
Failed logins

That's it. You are done with the required configuration.

Now lets run few SQL queries to generate some audit logs. I have used Azure Data Studio to connect and run the queries on my database but you can use any tool of your choice. Make sure you have whitelisted your IP to satisfy the firewall rules.

Now there's no way you can view the logs by directly going into your Event Hub. So, inorder to view the logs we will take help of Azure Service Bus Explorer desktop application. So, open the explorer, connect to your event hub and fire up a listener to witness live feed the logs.

Browse through the events and select one as per your requirement. Copy the EventText and view it in any JSON Format Viewer. Your query and many other valuable information will show up in the logs.

Stop the Listener when done.

In real time scenario you won't be using Service Bus Explorer to view and monitor live events. There will be some Big Data analytic service sitting at the other side of Event Hub which will do all the big data processing, monitoring and reporting on the Event Hub logs data.

Hope this article will give you an understand on how to log and monitor telemetry data from Azure SQL.

3. Azure Arc : Running Azure App Service in Azure Arc Enabled K8s

2021-07-14T14:46:00.008+05:30

In the first article of my Azure Azc series; we have successfully onboarded an Azure Kubernetes Service (AKS) cluster into Azure Arc enabled Kubernetes. If you haven't followed that yet, I highly recommend you to visit that article first and configure your clusters if you want to follow along.

1. Azure Arc : Setting up 'Kubernetes - Azure Arc' with AKS cluster

In this article we would run an Azure App Service - Web App inside our Azure Arc for Kubernetes cluster.

Hang on! Just read the above statement again. Azure Arc is meant for hybrid, multi-cloud capability and running an application on that doesn't mean that we can run it on any cloud behind the scene? Yes, your are right 😊, your Azure App Service can now run anywhere; hardly matters its on-premise or on any other cloud provider's platform. Sounds Interesting! Let's dive.

Lets go inside the resource group (arc-rg-01 in my case) where we have both our Kubernetes service cluster and the Kubernetes - Azure Arc cluster.

We would need a public static IP address to configure our app service which we would see shortly. So, for now just go ahead and create a public ip address in the your managed cluster (MC_) resource group (because I am running my AKS cluster in Azure) before moving forward. I gave the name as "my-webapp-arc-pip" leaving all other setting as default.

Once done... copy the IP address and keep it in notepad.

Search for "Azure Arc" in the search bar (azure portal) and select it from the result

Choose 'Kubernetes clusters' under Infrastructure to find your Azure Arc for K8s cluster.

Click on that, choose "Extensions (preview)" and then click +Add. The extension is in preview at the time of writing this article but its works perfectly. So, go ahead.

Select "Application services extension (Preview)"

Give your instance name, create a custom location and put the public ip address (copied to your notepad) in the static ip textbox. This custom location will show up in your azure region list once we configure the web app. This is again a magic which we will see shortly 😊

Click Next, Next and at the "Run Script" section, it will automatically generate the required PowerShell script which you need to download or copy to clipboard so that you can run it using any powershell platform.

I just copied it to clipboard but don't run it straightway. We need to enable the custom-location feature for our k8s cluster (do not confuse this with arc cluster). This is a very important step often get missed; else the generated powershell script would fail throwing permission denied error. So, let's go ahead and run the below script. Replace the values as per your environment.

az extension add --name customlocation

az connectedk8s enable-features -n "<cluster-name" -g "<resource group>" --features cluster-connect custom-locations

Now run the generated PowerShell script. I ran it using PowerShell ISE editor but you are free to use any of your choice. The script would take a while to finish, so go grab your favorite beverage and relax. I already took mine 🍺
On successful completion of the script, you will see the "microsoft.web.appservice" extension has been installed successfully in your cluster.

Now check the Custom locations under Management to find that the custom location has also been created successfully.

Great! now we are all set to create a app service (web app) and run in on out K8s Arc cluster. So, lets get going😉

Click "App services" under Services and click to create a new web app

Now in the next screen before doing anything, click the region dropdown. Voila! our new custom region shows up. Yeah! its right there.... amazing. Select that.

Fill up other details as required. You might have also noticed by this time that url has also changed on change of custom location as a region. This is enough to tell us that this not an ordinary azure app service which we use to create. I have selected the runtime stack as .Net core but you are free to choose any of your choice. Review and create.

Once complete, check the "App services" under Services to find our service.

Click on that and get the URL. Run it!

A big virtual High Five!!!! We have done it!.

Now our Azure app service is running on Azure Arc for K8s cluster and the underlying cluster can be anywhere.. on-premise, any cloud provider's platform, wherever... We just do not mind 😋

We can keep deploying our code to this app service manually, from VS or by DevOps pipeline and let Azure do all the heavy lifting to host that in a multi cloud platform.

Congratulations! for coming this far. Hope this article will help you to further explore more on this topic.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

2. Azure Arc : Deploying .Net Core App into Azure Arc enabled K8s with Git Ops

2021-07-07T12:03:00.036+05:30

In the first article of my Azure Azc series we have successfully onboarded an Azure Kubernetes Service (AKS) cluster into Azure Arc enabled Kubernetes. If you haven't followed that yet, then I highly recommend to visit that article first as we will continue to build on that cluster.

1. Azure Arc : Setting up 'Kubernetes - Azure Arc' with AKS cluster

We will also make use of Git Repository and Docker Hub registry. For this article, I will be using my public repo and hub but you are free to use yours.

PassHours/Hello-AzureArc (git repo)
atanudochub/helloazurearc (docker hub registry/image)

I have already developed a sample .Net core web app and uploaded it in my git repository. It's a very basic web app to display a welcome screen "Welcome to Azure Arc" but you got the idea 😊

The application is containerized and the image has been uploaded in my docker hub registry.

So, the platform is all set for us to proceed.

Lets go inside the resource group (arc-rg-01 in my case) where we have both our Kubernetes service cluster and the Kubernetes - Azure Arc cluster.

Get into the Kubernetes - Azure Arc cluster and click the "GitOps" option from the left menu. It will be empty as we have configured nothing yet.

Now the interesting part. We would like to deploy our Hello-AzureArc containerized application into our AKS cluster through the Azure Arc for Kubernetes GitOps channel and you would be surprised to see that how easy it is. Microsoft has done a great job in simplifying things here by providing a single command 👉 az k8sconfiguration create. All the heavy duty complex operations are taken care behind the scene for you.

az k8sconfiguration create \
  --name <<your choice>> \
  --cluster-name <<your cluster name>> \
  --resource-group <<your k8s cluster resource group>> \
  --operator-instance-name <<your choice>> \
  --operator-namespace <<your choice>> \
  --repository-url <<your git repo url>> \
  --scope cluster --cluster-type connectedClusters \
  --operator-params="--git-poll-interval 5s --git-readonly"

I have issued the following command in the cloud bash shell of Azure portal. You can also issue the exact same command if you are following along with me else replace the values accordingly. I have scoped the deployment to cluster level but you can also scope it to namespace level as well.

The interesting part to note here in the command is the last line --operator-params. This is the param where you are applying your gitops command. I am giving the command instruction here to poll my git repo (as mentioned in the --repository-url param) every 5 seconds (bit aggressive here for this demo but in real life it would be much higher value, may be every 30 mins), get all the changes and apply in my cluster. A typical continuous deployment scenario.

Upon completion, you will find the configuration (hello-arc in my case) showing up under the GitOps menu. The Operator state will show "Pending" first and then will turn to "Installed" after completion of the deployment process. So, have some patience here.

It signals done but we need to verify. Lets check our AKS cluster now in the following sequence in the Kubernetes resources section.

👉 Namespace. hello-arc is our namespace created successfully

👉 Workloads. Select the "Pods" tab and filter it by "default" namespace as we have not mapped it to any particular namespace and hence default. Three pods are up an running successfully.

Refer hello-azurearc.yml deployment yml in the git repo

👉 Services and ingresses. Our hello-arc-service is up and running having an external load balancer IP address.

Refer hello-azurearc.yml deployment yml in the git repo

Click the External IP and voila! our app is running as expected. Auto magic isn't it 👼 We have just issued a single command and surprise!

So, I believe it's enough to witness the power of Azure Arc for Kubernetes. I have just touched the tip of the iceberg; purposefully avoided going deep as this could unnecessarily confuse readers and make the article long and complex. But you can explore. 👍

The most significant part is that your kubernetes cluster can be running anywhere (on premise or on any cloud); the process is exact the same. You can manage and deploy apps into your clusters running anywhere sitting right on Azure. Cool !

Now if you make any changes in your application; just dockerize it; update the image with a new version number and upload it to the registry. The git operation will fire up automatically; would read that from your deployment yml in the git repo; download the image from your container registry and deploy it in your cluster behind the scene as a part of continuous deployment.

You can also run all your kubectl commands on the cluster and it works just fine

That's it.

Congratulations! for coming this far. Hope this article will help you to further explore more on this topic.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

1. Azure Arc : Setting up 'Kubernetes - Azure Arc' with AKS cluster

2021-07-02T15:12:00.008+05:30

At Ignite 2019, Satya Nadella, CEO of Microsoft, announced an array of new hybrid cloud products and services. The most significant announcement is Azure Arc, the hybrid and multi-cloud platform from Microsoft. With Azure Arc, Microsoft has expanded the support for resources running outside of Azure.

As of today Azure Arc allows you to manage the following resources outside Azure.

Servers - both physical and virtual machines running Windows or Linux.
Kubernetes clusters - supporting multiple Kubernetes distributions.
Azure data services - Azure SQL Managed Instance and PostgreSQL Hyperscale services.
SQL Server - enroll instances from any location.

Microsoft Azure Arc documentation: https://docs.microsoft.com/en-us/azure/azure-arc/overview

Imp: Azure Arc is not available in all the regions as of today. In some regions it's already GA or in Preview or Not Available at all. So make sure you check Azure Product availability page often https://azure.microsoft.com/en-in/global-infrastructure/services/?products=azure-arc&regions=all

I do not have any K8s cluster running anywhere (not on-premise neither in any other cloud) except one running in Azure, so in this article, I would onboard that AKS cluster into Azure Arc. But the process is same for any kubernetes cluster running in any on-premise/cloud platform.

So, without much theory... let's simply dive! 😉

Create an AKS cluster in Azure if you do not have it already. A basic cluster setup would do if in case you want to follow along.

We will issue some CLI commands in this demo. So, it's a good practice to have the latest CLI version installed if not already as old version may not support these commands.

Azure service principal assigned with the “Contributor” role is also required. So, login to azure from your CLI terminal and get that created. It can be any unique service principal name of your choice.

Copy the details from the above command and keep it in notepad. We would need these shortly

Now two namespaces needs to be registered. Microsoft.Kubernetes and Microsoft.KubernetesConfiguration.

The above commands are asynchronous in nature and hence you need to check their progress by the below commands. The Registration Status must show as "Registered" before you proceed forward.

Great!... now you have the stage ready. Microsoft provides an az_connect_aks.sh script which can be downloaded from https://github.com/microsoft/azure_arc/blob/main/azure_arc_k8s_jumpstart/aks/arm_template/scripts/az_connect_aks.sh. So, please do that and proceed with the next step
Replace the appId, password, tenantId with the values you had saved earlier in your notepad. Step 3. For resourceGroup and arcClusterName provide your resource group name and the AKS cluster name respectively

Save the file and let's open the shell web interface (shell.azure.com) to run this script. Make sure the bash mode is selected

On successful upload, issue the ls -l command to find the file in the root directory

Run the file ". ./az_connect_aks.sh". The operation might take a while to complete.

Once done... go to the resource group of you AKS cluster to find one more cluster in there and that's the ARC cluster

You can also check the cluster by going into the Kubernetes - Azure Arc service instance

That's it. You have successfully onboarded a kubernetes cluster in Kubernetes - Azure Arc. Now you can centrally monitor your kubernetes clusters running on any cloud, apply policies on those and do Git Operations sitting right in Azure. Isn't that cool stuff.

Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Mocking data in Azure Logic Apps

2020-12-28T12:12:00.004+05:30

In any distributed system development, various pieces of an application gets developed by multiple teams from different parts of the geography. This is a OK till you hit a roadblock when you cannot proceed until another team finishes their work and provide the data you need. In today's microservices design model this is even more relevant where you need a dependent service to return the data you need so that you can complete your work.

This type is scenario is inevitable and you cannot help but rather have to live with it. So what to do? 😒

Mocking!! Yes you are right. Why not mock the data you need to finish your work and later just do the plumbing work with the actual service when it's up and ready to throw back actual data. This is a great idea where teams can work on their own schedule without waiting and running after each other.

In Azure, when it comes to mocking, developers often fall back on API Management to implement this with the help of APIM polices. But why going for such costly idea where the need is only to mock data when you can use your other friend Logic Apps to achieve the same. Let's check it out 👍

Scenario:

We need to get the employee details from the database service (developing by the back-end team) by passing the employee code. But the back-end team is not yet there and hence we need to mock the employee details.

Let's get started.

Define an http endpoint where we will send the employee code as below

Add a new action and search for the HTTP action

Fill the details as per your requirement. In my case it's a GET call with the following details.

As mentioned the URI is the service that's getting developed by the backend team which is not yet ready. So, in this action we have to inject out mocking logic.

Click the "..." eclipse at the top right section of the action and select "Static result (Preview)" from the context menu. Yeah it's preview but works just fine. Go ahead!

Enable the Static result to get the following window. The default selections are good to go for our mocking purpose

But strangely there's no place to provide the static json response content here. Don't worry! It's there but we need to tweak a bit 😉

Hit the JSON mode icon to switch to the JSON view mode as below

Now here in the outputs section paste your return JSON in a body tag as below. I just need the employee first_name, last_name , "dept" and "rank" from the database service. So, I have mocked that piece as the back-end service is not yet ready and I can continue with my development 😃

Hit Done as return to the workflow. Now you will find a new icon added to the action which denotes that this action has a static result configured

As a last step let's add an "Response" action in the workflow and return back the employee details to the caller.

Save your logic app.

Done! let take a bird's eye view of our logic app workflow. This will help you to match if you are doing it in parallel along with this article

Testing time! Copy the http endpoint from the trigger (first action) and paste it in any rest client to make the call. I have used postman rest client as below. Hit Send and get your static result back.

That's it. You have successfully implemented mocking and what a cool way to do so.

Isn't that great!

Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Inject Inline javascript code in your Azure Logic App workflow

2020-12-14T17:24:00.006+05:30

JavaScript is web developers all time favourite. Writing small snippets of JS code into your application to implement few cool functionality is very tempting and developers; I know one cannot think of any web application which do not have any JS flavor in some form or the other 😀

Wouldn't it be great if we can use our JavaScript knowledge and inject a JS snippet into an Azure Logic App workflow and pull a magic out of it. I know you are thinking "Yessss! Why Not", so lets see how it an be done.

[Please note that is article expects you to have the basic understanding of Azure Logic Apps and Functions. You can check on Microsoft Azure documentation for such]

Our scenario for this article:

We need an Http endpoint which will accept a temperature and will do the Celsius or Fahrenheit conversion accordingly. If we send the temperature as Celsius then it will give us its Fahrenheit value and vice-versa.

In most cases I have seen developers quickly jumps into writing Azure Function to do this kind of simple stuff but after going through this article I hope you would think twice.

Our JSON payload will be as below which we will send while calling the Logic App endpoint. It's just sending the temperature in Celsius and expecting its respective Fahrenheit value.

{
   "temperature":"40",
   "metric":"C"
}

Now the JavaScript snippet to do the conversion is as below. It's just applying the formula and doing the respective Celsius and Fahrenheit conversion after reading the temperature and metric from the http request. If metric is coming as "C" then the result will be it's respective Fahrenheit and vice-versa.

var temp = workflowContext.trigger.outputs.body.temperature; 
var metric = workflowContext.trigger.outputs.body.metric;
var cf; 
if (metric == "C")
    cf = (temp * 9/5) + 32 + " degree Fahrenheit";
else if(metric == "F")
    cf = (temp - 32) * 5/9 + " degree Celsius";
else
    cf = "Invalid input"
return cf;

That's it! really 😲

Now lets create a Logic App and inject this piece of JavaScript snippet. There is an "Inline Code" Operation with an Action "Execute JavaScript Code (preview)" in Logic App workflow which we will use to achieve our result. Yes it's in preview but works just fine. Don't worry... you can safely use it 👍

Below is the full workflow and by looking at it you can pretty well understand what's it doing. So simple!

Save your Logic App and copy the Http Post Url to test it. I will use Postman rest client to test the endpoint but you can use any rest client of your choice.

Test 1: Sending 20 degree Celsius and getting it's respective Fahrenheit value

Test 2: Sending 68 degree Fahrenheit and getting it's respective Celsius value

Cool! the outputs are as expected. With few line of JS snippet you are able to pull out the magic of temperature conversion which would otherwise need an Azure Function.

Isn't that great!

Great Job and Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Implementing Fan-Out/Fan-In Design Pattern using Azure Logic Apps

2020-11-30T17:08:00.003+05:30

Azure Logic Apps are great for building system integrations. It doesn't matter your systems are running on premise or in cloud (any); with this great iPaaS service you can bring them all together in a common platform very quickly.

But did you ever thought that you can also use this service to host multiple web services which can work in parallel to give the desired output.

Think of a situation below:

"You have three web services each destined to perform a special task and return back a result. Now this result from each service individually doesn't makes any sense. You need to collate all the results, do some data massaging on it and return back the end result to the caller.". Very common scenario nowadays. Right?

Yes you can achieve this by

Calling each service one-by-one, store their individual result and then write code to prepare the end result
Azure Durable Functions with Orchestrator controller
Azure API Management with APIM policies.

All are right answers!

This article is not to discourage you from using any of the above but rather intended to give you one more thought process to achieve the same using Azure Logic Apps which is worth a shot. Let's check it out.

[Please note that is article expects you to have the basic understanding of Azure Logic Apps and Functions. You can check on Microsoft Azure documentation for such. 👍]

I will use three very simple web services for this article. The services are nothing but Azure Functions which would simply return an integer value to the caller (in this case the logic app) where all will get summarized before flushing out the end result.

I have used one .Net and two Node.JS HttpTrigger function apps just to give it a multi-language web service flavour. The apps are just returning 10,20 and 30 respectively as their output.

.Net Function app - 01

using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{    
    log.LogInformation("C# HTTP trigger function processed a request.");
    return new OkObjectResult(10);
}

Node.JS Function app - 02

module.exports = async function (context, req) {
    context.log('JavaScript HTTP trigger function processed a request.');    
    var responseMessage = 20;
    context.res = {
        body: parseInt(responseMessage)
    };
}

Node.JS Function app - 03

module.exports = async function (context, req) {
    context.log('JavaScript HTTP trigger function processed a request.');
    var responseMessage = 30;
    context.res = {
        body: parseInt(responseMessage)
    };
}

We are done with our three web services hosted as Azure Function apps. Cool 😊

Now let's put a Logic App in front which will act as an abstraction layer to all the web services and would give us a single http endpoint to consume. The logic app will call all the web services in parallel (Fan-Out), collect all results (Fan-In), do a summation and would flush out the end result. So, in our case the end result would be 60.. right!

Let's take a bird's eye view of our Logic App

The first three variables are declared to store the respective values from the three web services. The fourth and fifth variable are for doing the summation and final result.

Three web services are fanned-out in parallel as there is no dependency between each other. This will boost the speed as parallel processing is always faster. When all the runs are complete then the flow will be fanned-in for performing the data massaging (i.e. adding all the three results). The fourth and fifth variable are used to store the summation and ultimate flushing out the end result as a response to the caller.

Now our Logic App is in place and we are ready to test. Just copy the Logic App Http endpoint from the workflow and call it from Postman (or any other rest client is just fine). In a moment you will find the output of 60 flashing in your screen. Voila! It just worked as expected and we have successfully showcased a very rich design pattern with the help of Azure Logic App.

Great Job and Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Azure Logic App Integration : Transform XML to JSON

2020-09-29T17:34:00.008+05:30

In Enterprise Integration or iPaaS, a scenario that often comes up is to convert a XML payload to Json. You might be receiving XML data from one system but needs to convert that to JSON before pushing it to another system. If both the source and destination systems understands same language then it's OK but that is not always the case. So, there's a challenge.

Azure Logic App has some cool power and we will try to leverage on such in our integration scenario. Here our legacy source system can only send XML data but our destination system only understands JSON.

Prerequisites:

Azure Subscription (Free subscription will also do)
Basic XML, XSLT understanding
Basic JSON understanding
Azure Logic App basic Request/Response understanding

Let's get Started

Create a simple xml document (Vendor.xml). This is the payload our source system will send.

<?xml version="1.0" encoding="UTF-8"?>
<Vendor>
   <VendorNumber>VND1</VendorNumber>
   <VendorName>My Vendor</VendorName>
   <GroupId>10</GroupId>
   <Company>USRT</Company>
   <VendorCity>My City</VendorCity>
   <VendorContactPhone>1234567890</VendorContactPhone>
   <VendorContactEmail>myvendor@vendoremail.com</VendorContactEmail>
   <VendorCountryRgn>USA</VendorCountryRgn>
</Vendor>

But our destination system only understands JSON. Now we need to transform the above XML into JSON keeping all other information intact. So after transformation the data should look like this

{
   "VendorNumber":"VND1",
   "VendorName":"My Vendor",
   "Company":"USRT",
   "VendorCity":"My City",
   "VendorContactPhone":"1234567890",
   "VendorContactEmail":"myvendor@vendoremail.com",
   "VendorCountryRgn":"USA"
}

Looks simple! but not that simple in Logic App. There's a few marketplace connectors available but that won't help much when it comes to customization and also they not free and has licensing and pricing model complexities. So why to go that route when we have a free yet powerful friend in hand which can do this task quite comfortably and also is a first class citizen in Azure Logic App. XSLT Transformation.

Here goes our Vendor.xsl document. Save it in your machine as you would need this later. It's quite simple and self-explanatory. XSLT has immense power and hence you can understand what level of control you will have in handling XML complexities.

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output omit-xml-declaration="yes" method="text" media-type="application/json"/>
    <xsl:template match="/">
        <xsl:text>{ </xsl:text>
        
        <xsl:text>"VendorNumber" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorNumber"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorName" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorName"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"Company" : "</xsl:text>
        <xsl:value-of select="Vendor/Company"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorCity" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorCity"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorContactPhone" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorContactPhone"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorContactEmail" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorContactEmail"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorCountryRgn" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorCountryRgn"/>
        <xsl:text>" </xsl:text>
        
        <xsl:text> }</xsl:text>
    </xsl:template>
</xsl:stylesheet>

We are done with our basic ground work. Now it's time to step into Azure. 😃
Login to Azure portal
We cannot apply this XSLT file straightway on our input XML in Logic App. For that we need to create an "integration accounts". Let's do that. Search for integration in the Azure search bar and select.

Click +Add at the top and fill up the necessary details as per your choice. Create the account.

Once the integration account is created it will take you to the following landing page. There are lots of components available but for the sake of simplicity I won't go into those as it's not required for your case. It's required for typical B2B scenarios and if you are keep to learn then Azure has good documentation around it. Here we only need the Maps component. So click on that.

Click +Add at the top and upload your Vendor.xsl file. This is the XSLT file we have created earlier. Click OK

Great. Now let create the logic app for some real action. Search for Logic App in the Azure search bar and select.

Click +Add at the top and fill up the following details as per your choice. One important point to remember here. Make sure your logic app location is same as your integration account location. This is a requirement. Create your logic app.

Now the most important part. We need to tell the Logic App to use the integration account so that it could pick up the XSLT transformation map from there. Go to the Settings>>Workflow Settings and map your integration account. Click Save

Click Logic App designer under Development Tools and choose "When a HTTP request is received" trigger from the list of common triggers

This will take you to the logic app workflow designer with the above trigger as the first element in the workflow. Do nothing here and click the "Next Step"

Search for "Transform XML" and select Transform XML from the list of actions.

For Contents, select the Body from the Dynamic content list. The is the http request body which we are going to send as an XML payload later.

Choose the XSLT map from the "Map" dropdown. This is coming from your integration account where we have uploaded the Vendor.xsl file.

Now select the Add new parameter dropdown and select "Transform options" and the value as "Apply XSLT output attributes". This is very important else you will get a runtime error "Code": "InvalidXsltContent", "Message": "An error occurred while processing map. 'Token Text in state Start would result in an invalid XML document. Make sure that the ConformanceLevel setting is set to ConformanceLevel.Fragment or ConformanceLevel.Auto if you want to write an XML fragment. ". This is to just to inform the runtime to use the output format as defined in the XSLT file which in our case is application/json. So, make sure you do not miss this step 👍

Now we do not have any real source and destination system for this demo. So, we would return back the transformed JSON to the caller to simulate the XML to JSON conversion. Click on "+New Step", search for "Response" and select the Response Action

Select the "Transformed XML" from the Dynamic content selection window

So, we have 3 steps in the workflow. Save the Logic app. This will generate the HTTP endpoint for us to test. Copy the HTTP POST URL endpoint and open any rest client of your choice. My favorite is Postman

Paste the url endpoint to make a POST call. In the body copy-paste the XML data from the Vendor.xml file we have created previously. Hit Send and Voila!! We are successfully getting back JSON response by sending XML request. Isn't that great stuff😚

Great Job and Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Creating ARM Templates with Azure Resource Group Project

2020-09-07T18:02:00.000+05:30

Automation is the buzzword nowadays and when it comes to Cloud, it becomes more inevitable. Creating various Azure resources and then logically grouping them inside a Resource group is a very common task we perform while working in Azure. We can go ahead and hit the Azure portal and create all the necessary resources by simple clicks and set up the environment in no time and get things running. Cool!

This may be good and acceptable for your development environment. What about other environments such as QA,Staging or even Production. Is it feasible to repeat the same task over and over again in each environment and create the exact same stuff. What about human error. Manual processes are largely error prone and a slight hiccup could lead to a big costly mistake. Moreover you might not have permission to all the environments and the same you might forget the exact resource configurations few months down the line. So, in short there's hell lot of things that can go wrong if things are manual.

Hence Automation. In this article I will walk you through the steps of creating a simple ARM template containing an Azure Storage resource in VS-2017 solution and then deploying the same with the help of PowerShell to Azure Cloud. Hope you all would enjoy the journey!

Please note that I have purposefully left out some complex areas or haven’t deep dived into details at some places. This is just to make this article an easy read and help as a quick guide from where to start in creating and deploying ARM templates

Prerequisites:

Microsoft Visual Studio 2017 with Azure SDK installed.
Azure Subscription (Free subscription of 30 days will also do)
Basic JSON understanding
Windows PowerShell (Command Line or ISE)

Let's get Started

Fire up your Visual Studio 2017
File > New > Project. Click on "Cloud" in the left panel and then select "Azure Resource Group" project type. Give the project name, location as per your choice. Click OK

Select "Blank Template" from the list of templates as of now. We will add resources later. Click OK

A Resource Group project will be created as below.

The main entry point is the azuredeploy.json. This is where we will create all the resources along with its respective configurations. Next is azuredeploy.parameters.json which is basically used to override any parameter that is defined in the azuredeploy.json file. Anything that needs to be customized on a per item basis should be placed here. Deploy-AzureResourceGroup.ps1 is the powershell script require to deploy the resources.

We will only make use the azuredeploy.json in this demo and perform the other two steps a bit differently so to avoid any complexity and help in easy understanding.

Double-Click on the azuredeploy.json file to open the code window.

There are four main sections in the json file.

parameters : Here we will define all the parameters respective to a resource in key-value pair instead of hard-coding in the resource body.

variables : This is optional. These are piece of information which the ARM template can work out on the fly. Specially used to deploy similar resources in different environments.

resources : All resource code blocks will go into this section following the same structure and pattern.

outputs : This is optional. Used when linking other ARM template with the master template or infact creating linked templates. Will not use this in this demo.

Once you opened the code view of the azuredeploy.json file, VS-2017 will automatically open a JSON Outline window for powerful navigation purpose.

If you do not see this windows then goto View > Other Windows > JSON Outline

Now our target is to create a Azure Storage resource here. Right click on resource section inside the JSON Outline window and select "Add New Resource"

Scroll down to find the "Storage Account" resource. Give it a name of your choice. I gave demoStorage but it can be any name. Click "Add"

Visual Studio will add all the necessary boilerplate code required for this resource type and will populate all the required sections of the ARM template with some default values.

Now lets do some minor modifications. The Storage name should be unique like any other Azure resources. Visual Studio has tried to make it unique by its own way but what if I want the storage name to start with "demo" as a prefix. Let do this.

We will add one parameter name "demoStoragePrefix" as below in the parameters section

"demoStoragePrefix": {
        "type": "string",
        "minLength": 1,
        "defaultValue": "demo"

In the variables section lets modify the demoStorageName variable to read the prefix from the parameter we just defined above.

"demoStorageName": "[concat(parameters('demoStoragePrefix'), uniqueString(resourceGroup().id))]"

Now also add an additional parameter name "metadata" in the parameters section. This is optional and most people do overlook but I strongly recommend to add this with every template as it makes the code more readable.

"metadata": {
      "type": "string",
      "defaultValue": "Creation of Storage Account"

Now the new modified json code block looks like this with the above changes in place.

Save the file and solution. We are done with our ARM template. Now lets dive into the deployment part.

Open PowerShell window (Command line or ISE window will do)

Connect to your Azure account. Execute the below command.

Connect-AzureRmAccount

This will prompt for your Azure username and password. Provide the details and login. A successful login with show all your subscriptions. I have only one subscription and hence it shows the details related to that.

Account          : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SubscriptionName : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SubscriptionId   : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
TenantId         : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Environment      : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Next set some deployment variables which we will use later in the steps. Select these steps and execute.

## Define Deployment Variables

   ## Your resource location. In my case I placed it in SE Asia
   $location = 'SouthEast Asia' 
   ## Resource Group Name
   $resourceGroupName = 'ARMdemo-simple-storage'
   ## Resource Group Deployment Name
   $resourceDeploymentName = 'ARMdemo-simple-storage-deployment'
   ## The path of the template file in your local drive. Your location folder path can be different
   $templatePath = $env:SystemDrive + '\' + 'Personal\Projects\DemoAzureRmStorageDeployment'
   ## The template filename. It will be same if you have taken the default filename which Visual Studio provides
   $templateFile = 'azuredeploy.json'
   ## The full template path along with the file
   $template = $templatePath + '\' + $templateFile

Create a Resource group in Azure under which our Storage resource will be placed. Now here you might be thinking that why we are not creating this resource group through the same ARM template. You are absolutely right but I have purposefully kept this separate to make our ARM template look simple. Otherwise we have to make it a bit complex by adding priority in the template regarding which resource should get created first and depends on what. Hope you understand.

### Create Resource Group
 New-AzureRmResourceGroup `
    -Name $resourceGroupName `
    -Location $location `
    -Verbose -Force

Execute the above script. It will create the resource group name "ARMdemo-simple-storage" in "SouthEast Asia" region. These are coming from the deployment variables we set before. Your name and region can be different based on what you have set in the variables.

Now the most important step. Deploying our ARM template with Powershell. Execute the following command. The command "New-AzureRmResourceGroupDeployment" will pick the ARM template from the location of our local drive and would deploy the resource under the resource group created above. The parameters passed are very much self explanatory

### Deploy Resources

New-AzureRmResourceGroupDeployment -Name $resourceDeploymentName -ResourceGroupName $resourceGroupName -TemplateFile $template -Verbose -Force

Wait for sometime for the deployment to finish. The command will also validate the ARM template before deployment and if looks fine will display "-Template is valid" else will throw an error message with description of what and where it went wrong.
DONE with our automation.

First lets check the resource group. Our "ARMdemo-simple-storage" is there as expected.

Let dive into it. Yes our storage account (provisioned thorough the ARM template) is showing up under the resource group in location : Southeast Asia.

One thing to notice here is the unique storage name. The prefix is "demo" which we had set in the ARM template and the rest is appended from the resourceGroup id. Recall the below variable we set in the template. Amazing!

"variables": {
     "demoStorageName": "[concat(parameters('demoStoragePrefix'), uniqueString(resourceGroup().id))]"
}

Click on the storage and you can view more details.

Great Stuff and Well Done! No more manual steps anymore. Going forward any redundant work you do, try to automate it. It will be a blessing.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

Azure Logic App : A must have weapon in your armoury

2020-07-06T17:45:00.009+05:30

I often get involved in discussions where people talks about automation and orchestrating stuff so that manual operations can be mitigated. This discussion generally leads towards implementing DevOps tools and practices and to setup a sound CI/CD pipeline, etc. But, I rarely come across a situation where people brings Logic App in this discussion. It a great service and should be a part of any automation discussion.

Take an example (Order handling in an arbitrary eCommerce scenario). The situation is like

Receiving the order in the system
Save the order details in a database
Generate a service ticket (ITSM) so that the order can be handled
Generate a SMS and send it to the customer confirming the order

A very common workflow..right! What would you prefer here? A DevOps Automation or an Workflow Automation? Obviously the second one, and here in this article we will implement the above scenario with the help of a Logic App workflow and would try to feel the power of this awesome service.

Please note that for simplicity purpose and to make this article an easy read we will go over a waterfall workflow model with sequential steps.

Let's revisit out scenario once again and this time we will put the integration pointers.

Receiving the order in the system 👉 Postman REST API client
Save the order details in a database 👉 Azure Table Storage
Generate a service ticket (ITSM) so that the order can be handled 👉 FreshDesk (free account will do)
Generate a SMS and send it to the customer confirming the order 👉 Twillio (free account will do)

Make sure you have the above settings done before moving forward.

OK, now we are all set. Let's dive.

First let's define the order information. We will use a simple JSON payload here for this demo just to simulate an order. Please replace the phone number with yours (with proper +Country code) 📱

{
   "orderNo": "1",
   "orderItem" : "Headphone",
   "orderCustName": "My Good Customer",
   "orderCustMobile" : "+91XXXXXXXXXX",
   "orderCity": "Bangalore",
   "orderState": "Karnataka"
}

Select your subscription, fill up the necessary details as per your choice, review and create the logic app.

Select "When a HTTP request is received" trigger in the designer

Click the link "Use sample payload to generate schema" hyperlink at the bottom and paste the order JSON payload to generate the JSON schema. Save the logic app to generate the HTTP Post Url.

Click "+ New Step" to add an Azure Storage table data insert step. Make sure you have already created a blank table in Azure Storage account so that you can insert data in that table directly using this step. You can create that table from Logic app as well, but it's upto you. Here I have already created that table inorder to reduce an extra step

Search for Azure Table Storage and select "Insert Entity" action

Give a name for your table storage connection, select your storage account and hit "Create"

Select your table name from the Table dropdown. Paste the JSON payload here and replace the values with the parameters from the dynamic content selection box. As we are saving data in Azure Storage Table so make sure to modify your payload with PartitionKey and RowKey parameters to make an unique combination for each row item. I have used the orderNo and orderCustMobile fields but you can make your wish here 😐

Click "+ New Step" to add a FreshDesk service ticket action.

Provide your FreshDesk connection parameters and hit Create

Fill up the ticket details to be logged. You can decorate it as per your requirement. For the Email dropdown choose any dummy name as it doesn't matter for this demo. In real life scenario you will be having your user list. Here our purpose is to log a service ticket.

Click "+ New Step" to add a Twilio SMS action. Select "Send Text Message (SMS)" action

Enter the Connection Name of your choice, Twilio Account id (the ACCOUNT SID value in the Twilio portal) and the Twilio Access Token (the AUTH TOKEN value in the Twilio portal) and hit "Create"

Select the From Phone number which will be automatically populated from your Twilio account. Decorate your SMS message as you wish.

Click "Save" at the top to save your logic app. This step is very important 😀

Show Time !!!!!!!

Copy the Http POST Url from the first step of your logic app and open postman. We are about to make a rest call.

Paste the Url at the address bar, select JSON (application/json) as content type and put your JSON payload in the body as raw input. Hit Send to get a 202 Accepted response back.

You have successfully made a rest call. Now three activities should happen. We must receive a record in table storage, service ticket logged in FreshDesk and a SMS in our mobile phone.

Open Azure Storage explorer and Aha! we have a new record in table

Open your FreshDesk ticketing dashboard and Voila! we have new service ticket 😲

Click the ticket and drill down to its details. Yep it showing the order details just as we configured it to be 👍

Awesome!... now as a last step lets check the mobile phone only to find the SMS sitting in our inbox. Now how cool is that !!!!!!!! 👏👏👏

We have successfully achieved our goal and the power of logic app simply drives us crazy. We have achieved a complicated, cost-effective and integrated solution using the serverless flavour of Azure with NO CODE. Good job!

Congratulations! for coming this far. Hope this article will help you to further explore this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it up to any level and integrate. I would love to hear from you.

Azure Security : Monitor KeyVault events to generate instant notification

2020-07-02T16:27:00.007+05:30

Security is everybody's responsibility. How many times you have heard this bold one liner? I am sure it's too many times during any design discussion or working with the security folks of any organization. 😐

So, when the talks starts rolling towards managing secrets and keys, one service of Azure you just cannot ignore is Azure KeyVault. You can keep all your secret parameters in this highly recommended HSM-protected managed vault and can securely use them from other services without the fear of those getting exposed. Great, you have implemented a fine level of security and a big round of applause 👏👏 in the room.

But.. ummm ...ok.. Now what about someone having access to the vault; intentionally (or accidentally may be a proper word) changes the value of any secret behind the scene without your knowledge. All the dependent downstream systems suddenly stops functioning and becomes unavailable straightaway. Isn't this qualifies for a security breach? Agree! 😔

I love a famous saying "It's not about the change but not having any knowledge of that change leads to road block". Let's take this as a thumb-rule and implement a notification system in place when someone adds/updates a secret in the KeyVault with a serverless approach throughout.

This article may also help you in your AZ-500 : Microsoft Azure Security Technologies preparation and lab. Let's dive! 👍

Prerequisites:

Azure Subscription (Free subscription of 30 days will also do)
SendGrid API Key (Follow this to create a free SendGrid account if you do not have it already and get the API Key)
Basic knowledge of Azure services

We will use Azure Logic Apps along with Event Grid trigger to catch any KeyVault change event and would fire an instant email notification to the administrators.

Login to Azure (If you do not have any subscription already then create one else login to your existing one)
Create a resource group and then create a KeyVault in that resource group. Follow this link if you are new to KeyVault. Just create the vault and do not add any secrets now.
Create an Azure Logic App (you can give any name of your choice; I have given kv-change-notification)

Go to your Logic App and in the designer select the Event Grid trigger as below

Select your tenant and hit Sign In. This will ask you for your Azure credential and will create your Event Grid connector.

Click "Continue" to attach the event. Fill the details as required to connect with your KeyVault created earlier.

Select your Subscription,

Resource Type as "Microsoft.KeyVault.vaults",

Resource Name is your KeyVault name (In my case I have give my vault name as MyVault0001),

Event Type Item - 1 as "Micsrosoft.KeyVault.SecretNewVersionCreated"

Click "+ New Step" to choose the event to fire if the above step holds true. Search for SendGrid and select the send email action

Provide a connection name and paste your SendGrid API . Hit Create. This will validate and would create your SendGrid connector.

If the above validation is successful, the next window would ask you to provide all the mailing details. Fill it up as per the requirement. Make sure the From and To email addresses really exists.

Hit Save at the top to save the logic app workflow. This is important.

Testing Time! 🙆

Go to your KeyVault resource and enter a new secret.

Check your mail. Voila! You have new mail 📧

Now go ahead and modify the value of the secret in the vault and get notified once again. Great job!

So, with just few clicks you have developed a robust solution (that too serverless) which surely going to make the security admins very happy. Applause 👏 once again.

Congratulations! for coming this far. Hope this article will help you to further explore this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it up to any level and integrate. I would love to hear from you.