While I was playing around with the Azure Resource Manager Copy Operation, I started thinking about what I could do with it. The first wild idea that popped into my head was, to use it to deploy multi-geo environments from a single ARM Template.

Alright, some of you might think that it’s not such great idea, and I can appreciate that. But I’m just too curious, so let’s give this a chance.

The Scenario

Imagine a service that needs to be deployed to multiple Data Centers. The Virtual Machine configuration for each environment is identical and generally requires to be deployed to at least five Microsoft Azure Data Centers.

The following Azure Resource Manager (ARM) Template, deploys CentOS 7.1 Virtual Machines to East US, West US, West Europe, East Asia and Southeast Asia Azure Data Centers. To accomplish this, I defined an array of Azure Regions, that is then used to populate the location property for each Resource.

For this to work, I had to create copies the Virtual Network (VNet), the Storage Account, the Public IP, the Network Interface (NIC) and the Virtual Machine in each region. Building on the ARM Template from my earlier post, I decided to use the vmCount as the control value for the Copy Operation applied of each Resource. This strategy worked out because the Template creates one Virtual Machine per region.

This ARM Template should be considered an experiment, because the Resource Group resides in a particular Azure Region and it contains Resources from across the world. As of writing this post, I can tell you that this is pretty cool, and that I’m curious to find out more about the long-term maintainability of this scenario.

Taking this a few steps further, the Custom Script for Linux Virtual Machine Extension could be used to download, unpack, install and configure a service. Then an Azure Traffic Manager instance could be created and configured to load balance traffic across the multi-geo environment.

Building the ARM Template

This step required some gymnastics, because I wasn’t familiar with the Resouce Dependencies for some of the Resources in the Template. After a couple of minutes, it was clear that I needed to add Copy Operations to all of the Resources. Using the copyIndex() and concat() functions, I named the Resources. Then I made sure that everything was referenced properly.

DeploymentTemplate.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "User name for the Virtual Machine."
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "Password for the Virtual Machine."
            }
        },
        "dnsPrefixNameForPublicIP": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
            }
        },
        "OSVersion": {
            "type": "string",
            "defaultValue": "7.1",
            "allowedValues": [
                "7.1"
            ]
        },
        "vmCount": {
            "type": "int",
            "defaultValue": 1
        },
        "virtualNetworkName": {
            "type": "string",
            "metadata": {
                "description": "Virtual Network Name"
            }
        },
        "vmNamePrefix": {
            "type": "string",
            "metadata": {
                "description": "Virtual Machine Name Prefix"
            }
        },
        "vmSize": {
            "type": "string",
            "defaultValue": "Standard_A1",
            "metadata": {
                "description": "Virtual Machine Size"
            }
        },
        "locations": {
            "type": "array",
            "defaultValue": [
                "East Asia",
                "Southeast Asia",
                "East US",
                "West US",
                "West Europe"
            ],
            "metadata": {
                "description": "Locations"
            }
        }
    },
    "variables": {
        "addressPrefix": "10.0.0.0/16",
        "dataDiskSize": "1023",
        "imageOffer": "CentOS",
        "imagePublisher": "OpenLogic",
        "nicName": "NIC",
        "OSDiskName": "osdisk",
        "publicIPAddressType": "Dynamic",
        "storageAccountType": "Standard_LRS",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "vmStorageAccountContainerName": "vhds"
    },
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[concat(parameters('newStorageAccountName'),copyIndex())]",
            "apiVersion": "2015-05-01-preview",
            "location": "[parameters('locations')[copyIndex()]]",
            "tags": {
                "displayName": "StorageAccount"
            },
            "properties": {
                "accountType": "[variables('storageAccountType')]"
            },
            "copy": {
                "name": "storageAccountCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[concat(parameters('dnsPrefixNameForPublicIP'),copyIndex())]",
            "location": "[parameters('locations')[copyIndex()]]",
            "tags": {
                "displayName": "PublicIPAddress"
            },
            "properties": {
                "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
                "dnsSettings": {
                    "domainNameLabel": "[concat(parameters('dnsPrefixNameForPublicIP'),copyIndex())]"
                }
            },
            "copy": {
                "name": "publicIpCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[concat(parameters('virtualNetworkName'),copyIndex())]",
            "location": "[parameters('locations')[copyIndex()]]",
            "tags": {
                "displayName": "VirtualNetwork"
            },
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            },
            "copy": {
                "name": "vnetCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/networkInterfaces",
            "name": "[concat(variables('nicName'),copyIndex())]",
            "location": "[parameters('locations')[copyIndex()]]",
            "tags": {
                "displayName": "NetworkInterface"
            },
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/', parameters('dnsPrefixNameForPublicIP'), copyIndex())]",
                "[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'), copyIndex())]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "[concat('ipconfig',copyIndex())]",
                        "properties": {
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('dnsPrefixNameForPublicIP'),copyIndex()))]"
                            },
                            "subnet": {
                                "id": "[concat(resourceId('Microsoft.Network/virtualNetworks',parameters('virtualNetworkName')), copyIndex(),'/subnets/',variables('subnetName'))]"
                            }
                        }
                    }
                ]
            },
            "copy": {
                "name": "nicCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "copy": {
                "name": "nodeCopy",
                "count": "[parameters('vmCount')]"
            },
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'), copyIndex())]",
                "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), copyIndex())]"
            ],
            "location": "[parameters('locations')[copyIndex()]]",
            "name": "[concat(parameters('vmNamePrefix'),copyIndex())]",
            "properties": {
                "hardwareProfile": {
                    "vmSize": "[parameters('vmSize')]"
                },
                "osProfile": {
                    "computername": "[concat(parameters('vmNamePrefix'), copyIndex())]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "[variables('imagePublisher')]",
                        "offer": "[variables('imageOffer')]",
                        "sku": "[parameters('OSVersion')]",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "osdisk",
                        "vhd": {
                            "uri": "[concat('http://',parameters('newStorageAccountName'),copyIndex(),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',concat(variables('OSDiskName')),'.vhd')]"
                        },
                        "caching": "ReadWrite",
                        "createOption": "FromImage"
                    },
                    "dataDisks": [
                        {
                            "name": "datadisk1",
                            "diskSizeGB": "[variables('dataDiskSize')]",
                            "lun": 0,
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),copyIndex(),'.blob.core.windows.net/vhds/',concat(parameters('vmNamePrefix')),'dataDisk1' ,'.vhd')]"
                            },
                            "caching": "None",
                            "createOption": "Empty"
                        }
                    ]
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('nicName') ,copyIndex()))]"
                        }
                    ]
                }
            },
            "tags": {
                "displayName": "VirtualMachine"
            },
            "type": "Microsoft.Compute/virtualMachines"
        }
    ]
}

Publishing the ARM Template

Prepare the Template Parameter File a.k.a DeploymentTemplate.param.dev.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "value": "briseboiscentos"
        },
        "adminUsername": {
            "value": "briseboiscentos"
        },
        "dnsPrefixNameForPublicIP": {
            "value": "centosdns"
        },
        "virtualNetworkName": {
            "value": "centosvnet"
        },
        "vmNamePrefix": {
            "value": "briseboiscentos"
        },
        "vmSize": {
            "value": "Standard_A1"
        },
        "vmCount": {
            "value": 5
        },
        "locations": {
            "value": [
                "East Asia",
                "Southeast Asia",
                "East US",
                "West US",
                "West Europe"
            ]
        }
    }
}

Then deploy the Azure Resource Manager Template and Parameters using PowerShell.

New-AzureResourceGroup -Name 'centosbrisebois' `
                       -Location 'eastus' `
                       -TemplateFile 'DeploymentTemplate.json' `
                       -TemplateParameterFile 'DeploymentTemplate.param.dev.json' `
                       -Force -Verbose
# New-AzureResourceGroup VERBOSE Log

VERBOSE: 12:26:05 AM - Created resource group 'centosbrisebois' in location 'eastus'
VERBOSE: 12:26:06 AM - Template is valid.
VERBOSE: 12:26:07 AM - Create template deployment 'DeploymentTemplate'.
VERBOSE: 12:26:12 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos3' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet1' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet0' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet3' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet4' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos1' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos0' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet2' provisioning status is running
VERBOSE: 12:26:12 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos2' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns0' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos4' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns1' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns2' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns3' provisioning status is running
VERBOSE: 12:26:14 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns4' provisioning status is running
VERBOSE: 12:26:24 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns2' provisioning status is succeeded
VERBOSE: 12:26:24 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet3' provisioning status is succeeded
VERBOSE: 12:26:24 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet4' provisioning status is succeeded
VERBOSE: 12:26:24 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet2' provisioning status is succeeded
VERBOSE: 12:26:26 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns3' provisioning status is succeeded
VERBOSE: 12:26:26 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns4' provisioning status is succeeded
VERBOSE: 12:26:26 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet1' provisioning status is succeeded
VERBOSE: 12:26:26 AM - Resource Microsoft.Network/virtualNetworks 'centosvnet0' provisioning status is succeeded
VERBOSE: 12:26:28 AM - Resource Microsoft.Network/networkInterfaces 'NIC4' provisioning status is succeeded
VERBOSE: 12:26:28 AM - Resource Microsoft.Network/networkInterfaces 'NIC2' provisioning status is succeeded
VERBOSE: 12:26:28 AM - Resource Microsoft.Network/networkInterfaces 'NIC3' provisioning status is succeeded
VERBOSE: 12:26:28 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns0' provisioning status is succeeded
VERBOSE: 12:26:28 AM - Resource Microsoft.Network/publicIPAddresses 'centosdns1' provisioning status is succeeded
VERBOSE: 12:26:31 AM - Resource Microsoft.Network/networkInterfaces 'NIC1' provisioning status is succeeded
VERBOSE: 12:26:31 AM - Resource Microsoft.Network/networkInterfaces 'NIC0' provisioning status is succeeded
VERBOSE: 12:26:38 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos3' provisioning status is succeeded
VERBOSE: 12:26:40 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos3' provisioning status is running
VERBOSE: 12:27:04 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos2' provisioning status is running
VERBOSE: 12:27:04 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos1' provisioning status is succeeded
VERBOSE: 12:27:04 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos2' provisioning status is succeeded
VERBOSE: 12:27:09 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos1' provisioning status is running
VERBOSE: 12:27:09 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos4' provisioning status is succeeded
VERBOSE: 12:27:11 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos4' provisioning status is running
VERBOSE: 12:28:35 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos3' provisioning status is succeeded
VERBOSE: 12:28:57 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos2' provisioning status is succeeded
VERBOSE: 12:29:09 AM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos0' provisioning status is succeeded
VERBOSE: 12:29:11 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos0' provisioning status is running
VERBOSE: 12:29:20 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos4' provisioning status is succeeded
VERBOSE: 12:31:10 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos0' provisioning status is succeeded
VERBOSE: 12:31:48 AM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos1' provisioning status is succeeded

This deployment took about 5 minutes to complete across five Microsoft Azure Data Centers. I’m amazed by the fact that I was able to do this with a single Virtual Machine definition. This obviously comes with a price. As you may have noticed, the Azure Resource Manager Template can get complicated.

This experiment was quite interesting and will definitely fuel more experiments with the Azure Resource Manager. Please use the comment section below ask questions, share your experiences and to discuss best practices.

Helpful Azure Resource Manager Documentation

• Authoring Azure Resource Manager Templates
• Deploy an application with Azure Resource Manager Template
• Troubleshooting Resource Group Deployments in Azure
• Azure Resource Manager Template Functions
• Advanced Template Operations

Deploying 20 CentOS VMs in 4 Minutes!

I recently started to toy around with scenarios that required me to deploy multiple duplicates of the same CentOS Virtual Machine configuration. Working on this scenario got me curious. So I decided to build a template that would allow me to deploy 20 CentOS Virtual Machines each with one 1TB data disk and one public IP addresses.

To my surprise, deploying these 20 Standard A1 CentOS Virtual Machines on Microsoft Azure took 4 minutes!

Building the ARM Template

Let’s start by taking a CentOS ARM Template from a previous post. It will be our starting point for this exercise. Now, let’s removed the extra data disk and removed the Custom Script for Linux Virtual Machine Extension.

To duplicate a resource, we must use the copy operation. It enables us to use an index number or to iterate through an array of values that can be used when deploying a resource.

"copy": {
          "name": "nodeCopy",
          "count": "[parameters('vmCount')]"
}

In this specific scenario, we want all our Virtual Machines to belong to the same Virtual Network and Subnet. Therefore, we need to duplicate each Virtual Machine, their Network Interface Cards (NIC) and their Public IP Addresses.

The following template, demonstrates the use of copyIndex() and concat(), to generate predictable identifiers for each copy.

DeploymentTemplate.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "User name for the Virtual Machine."
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "Password for the Virtual Machine."
            }
        },
        "dnsPrefixNameForPublicIP": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
            }
        },
        "OSVersion": {
            "type": "string",
            "defaultValue": "7.1",
            "allowedValues": [
                "7.1"
            ]
        },
        "vmCount": {
            "type": "int",
            "defaultValue": 1
        },
        "virtualNetworkName": {
            "type": "string",
            "metadata": {
                "description": "Virtual Network Name"
            }
        },
        "vmNamePrefix": {
            "type": "string",
            "metadata": {
                "description": "Virtual Machine Name Prefix"
            }
        },
        "vmSize": {
            "type": "string",
            "defaultValue": "Standard_A1",
            "metadata": {
                "description": "Virtual Machine Size"
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "West US",
            "metadata": {
                "description": "Location"
            }
        }
    },
    "variables": {
        "addressPrefix": "10.0.0.0/16",
        "dataDiskSize": "1023",
        "imageOffer": "CentOS",
        "imagePublisher": "OpenLogic",
        "nicName": "NIC",
        "OSDiskName": "osdisk",
        "publicIPAddressType": "Dynamic",
        "storageAccountType": "Standard_LRS",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]",
        "vmStorageAccountContainerName": "vhds",
        "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',parameters('virtualNetworkName'))]"
    },
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('newStorageAccountName')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "StorageAccount"
            },
            "properties": {
                "accountType": "[variables('storageAccountType')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[concat(parameters('dnsPrefixNameForPublicIP'),copyIndex())]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "PublicIPAddress"
            },
            "properties": {
                "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
                "dnsSettings": {
                    "domainNameLabel": "[concat(parameters('dnsPrefixNameForPublicIP'),copyIndex())]"
                }
            },
            "copy": {
                "name": "publicIpCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[parameters('virtualNetworkName')]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "VirtualNetwork"
            },
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/networkInterfaces",
            "name": "[concat(variables('nicName'),copyIndex())]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "NetworkInterface"
            },
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/', concat(parameters('dnsPrefixNameForPublicIP'),copyIndex()))]",
                "[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "[concat('ipconfig', copyIndex())]",
                        "properties": {
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('dnsPrefixNameForPublicIP'),copyIndex()))]"
                            },
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            }
                        }
                    }
                ]
            },
            "copy": {
                "name": "nicCopy",
                "count": "[parameters('vmCount')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "copy": {
                "name": "nodeCopy",
                "count": "[parameters('vmCount')]"
            },
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
                "[concat('Microsoft.Network/networkInterfaces/', concat(variables('nicName'),copyIndex()))]"
            ],
            "location": "[parameters('location')]",
            "name": "[concat(parameters('vmNamePrefix'),copyIndex())]",
            "properties": {
                "hardwareProfile": {
                    "vmSize": "[parameters('vmSize')]"
                },
                "osProfile": {
                    "computername": "[concat(parameters('vmNamePrefix'),copyIndex())]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "[variables('imagePublisher')]",
                        "offer": "[variables('imageOffer')]",
                        "sku": "[parameters('OSVersion')]",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "osdisk",
                        "vhd": {
                            "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',concat(variables('OSDiskName'),copyIndex()),'.vhd')]"
                        },
                        "caching": "ReadWrite",
                        "createOption": "FromImage"
                    },
                    "dataDisks": [
                        {
                            "name": "datadisk1",
                            "diskSizeGB": "[variables('dataDiskSize')]",
                            "lun": 0,
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/vhds/',concat(parameters('vmNamePrefix'),copyIndex()),'dataDisk1' ,'.vhd')]"
                            },
                            "caching": "None",
                            "createOption": "Empty"
                        }
                    ]
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('nicName'),copyIndex()))]"
                        }
                    ]
                }
            },
            "tags": {
                "displayName": "VirtualMachine"
            },
            "type": "Microsoft.Compute/virtualMachines"
        }
    ]
}

Publishing the ARM Template

Prepare the Template Parameter File a.k.a DeploymentTemplate.param.dev.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "value": "briseboiscentos"
        },
        "adminUsername": {
            "value": "briseboiscentos"
        },
        "dnsPrefixNameForPublicIP": {
            "value": "centosdns"
        },
        "virtualNetworkName": {
            "value": "centosvnet"
        },
        "vmNamePrefix": {
            "value": "briseboiscentos"
        },
        "vmSize": {
            "value": "Standard_A1"
        },
        "vmCount": {
            "value": 20
        },
        "location": {
            "value": "East US"
        }
    }
}

Then deploy the Azure Resource Manager Template and Parameters using PowerShell.

Switch-AzureMode AzureResourceManager

New-AzureResourceGroup -Name 'centosbrisebois' `
                       -Location 'eastus' `
                       -TemplateFile 'DeploymentTemplate.json' `
                       -TemplateParameterFile 'DeploymentTemplate.param.dev.json' `
                       -Force -Verbose 

# New-AzureResourceGroup VERBOSE Log

VERBOSE: 7:17:09 PM - Created resource group 'centosbrisebois' in location 'eastus'
VERBOSE: 7:17:09 PM - Template is valid.
VERBOSE: 7:17:11 PM - Create template deployment 'DeploymentTemplate'.
VERBOSE: 7:17:16 PM - Resource Microsoft.Network/virtualNetworks 'centosvnet' provisioning status is running
VERBOSE: 7:17:16 PM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns16' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns17' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns9' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns6' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns1' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns0' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns14' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns5' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns2' provisioning status is running
VERBOSE: 7:17:21 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns12' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns15' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns11' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns18' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns19' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns8' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns3' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns10' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns7' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns13' provisioning status is running
VERBOSE: 7:17:24 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns4' provisioning status is running
VERBOSE: 7:17:28 PM - Resource Microsoft.Network/virtualNetworks 'centosvnet' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns15' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns7' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns4' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns16' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns1' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns14' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns5' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns2' provisioning status is succeeded
VERBOSE: 7:17:33 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns12' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC14' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC18' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC9' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC8' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC4' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC13' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC12' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC15' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC5' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC1' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC7' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC16' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/networkInterfaces 'NIC2' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns18' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns19' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns8' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns10' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns13' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns9' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns6' provisioning status is succeeded
VERBOSE: 7:17:36 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns0' provisioning status is succeeded
VERBOSE: 7:17:38 PM - Resource Microsoft.Network/networkInterfaces 'NIC10' provisioning status is succeeded
VERBOSE: 7:17:38 PM - Resource Microsoft.Network/networkInterfaces 'NIC19' provisioning status is succeeded
VERBOSE: 7:17:43 PM - Resource Microsoft.Storage/storageAccounts 'briseboiscentos' provisioning status is succeeded
VERBOSE: 7:17:46 PM - Resource Microsoft.Network/networkInterfaces 'NIC0' provisioning status is succeeded
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos10' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos1' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos19' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos2' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos12' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos7' provisioning status is running
VERBOSE: 7:17:46 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos16' provisioning status is running
VERBOSE: 7:17:48 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos6' provisioning status is running
VERBOSE: 7:17:48 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos0' provisioning status is running
VERBOSE: 7:17:48 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos15' provisioning status is running
VERBOSE: 7:17:48 PM - Resource Microsoft.Network/networkInterfaces 'NIC6' provisioning status is succeeded
VERBOSE: 7:17:55 PM - Resource Microsoft.Network/networkInterfaces 'NIC11' provisioning status is succeeded
VERBOSE: 7:17:55 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns11' provisioning status is succeeded
VERBOSE: 7:18:03 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos11' provisioning status is running
VERBOSE: 7:18:05 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos4' provisioning status is running
VERBOSE: 7:18:05 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos14' provisioning status is running
VERBOSE: 7:18:08 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos13' provisioning status is running
VERBOSE: 7:18:08 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos18' provisioning status is running
VERBOSE: 7:18:08 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos9' provisioning status is running
VERBOSE: 7:18:08 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos8' provisioning status is running
VERBOSE: 7:18:10 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos5' provisioning status is running
VERBOSE: 7:18:13 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns17' provisioning status is succeeded
VERBOSE: 7:18:15 PM - Resource Microsoft.Network/networkInterfaces 'NIC3' provisioning status is succeeded
VERBOSE: 7:18:15 PM - Resource Microsoft.Network/networkInterfaces 'NIC17' provisioning status is succeeded
VERBOSE: 7:18:15 PM - Resource Microsoft.Network/publicIPAddresses 'centosdns3' provisioning status is succeeded
VERBOSE: 7:18:18 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos17' provisioning status is running
VERBOSE: 7:18:18 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos3' provisioning status is running
VERBOSE: 7:19:39 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos12' provisioning status is succeeded
VERBOSE: 7:19:44 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos6' provisioning status is succeeded
VERBOSE: 7:19:47 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos0' provisioning status is succeeded
VERBOSE: 7:19:54 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos19' provisioning status is succeeded
VERBOSE: 7:19:54 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos16' provisioning status is succeeded
VERBOSE: 7:19:57 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos10' provisioning status is succeeded
VERBOSE: 7:19:57 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos2' provisioning status is succeeded
VERBOSE: 7:19:59 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos8' provisioning status is succeeded
VERBOSE: 7:20:02 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos5' provisioning status is succeeded
VERBOSE: 7:20:12 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos15' provisioning status is succeeded
VERBOSE: 7:20:12 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos7' provisioning status is succeeded
VERBOSE: 7:20:14 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos14' provisioning status is succeeded
VERBOSE: 7:20:17 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos13' provisioning status is succeeded
VERBOSE: 7:20:17 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos18' provisioning status is succeeded
VERBOSE: 7:20:17 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos9' provisioning status is succeeded
VERBOSE: 7:20:17 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos4' provisioning status is succeeded
VERBOSE: 7:20:17 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos11' provisioning status is succeeded
VERBOSE: 7:20:29 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos17' provisioning status is succeeded
VERBOSE: 7:21:04 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos3' provisioning status is succeeded
VERBOSE: 7:21:19 PM - Resource Microsoft.Compute/virtualMachines 'briseboiscentos1' provisioning status is succeeded

Voila! 4 minutes later, we are now ready to start working with our 20 new CentOS Virtual Machines.

Additional Thoughts

In some circumstances, using a copy index (0,1,2,3…) to name resources isn’t practical. When this is the case, it is possible to use the copy index to read from an array of values. In the following example, we are creating three Azure Storage Accounts using by concatenating elements from an array of names.

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "storageAccountNames": {
            "type": "array",
            "defaultValue": [
                "archive",
                "data",
                "vhds"
            ]
        },
        "count": {
            "type": "int",
            "defaultValue": 3
        }
    },
    "resources": [
        {
           "type": "Microsoft.Storage/storageAccounts",
            "name": "[concat('brisebois',parameters('storageAccountNames')[copyIndex()])]",
            "apiVersion": "2015-05-01-preview",
            "location": "[resourceGroup().location]",
            "tags": {
                "displayName": "[concat('brisebois',parameters('storageAccountNames')[copyIndex()])]"
            },
            "properties": {
                "accountType": "Standard_LRS"
            },
            "copy": {
                "name": "storageAccountCopy",
                "count": "[parameters('count')]"
            }
        }
    ]
}

ARM Resources

• Authoring Azure Resource Manager Templates
• Deploy an application with Azure Resource Manager Template
• Troubleshooting Resource Group Deployments in Azure
• Azure Resource Manager Template Functions
• Advanced Template Operations

Lock it Down!

Accidents happen. Resource Locks help prevent them.

The Resource Lock level is applied at the resource group or resource scope. These can be set by the administrators and current values include CanNotDelete and ReadOnly.

Using a modified version of the ARM Template from a post on creating a CentOS Virtual Machine, let’s provision a VM and protect it from accidental deletion.

As a best practice, we should consider implementing Resource Locks for mission critical resources.

The Azure Resource Manager (ARM) Template

To make things easy to follow, both data disks and the Custom Script for Linux Virtual Machine extension were removed from the ARM Template.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "User name for the Virtual Machine."
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "Password for the Virtual Machine."
            }
        },
        "dnsNameForPublicIP": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
            }
        },
        "OSVersion": {
            "type": "string",
            "defaultValue": "7.1",
            "allowedValues": [
                "7.1"
            ]
        }
    },
    "variables": {
        "addressPrefix": "10.0.0.0/16",
        "dataDiskSize": "100",
        "imageOffer": "CentOS",
        "imagePublisher": "OpenLogic",
        "location": "West US",
        "nicName": "myVMNic",
        "OSDiskName": "osdisk",
        "publicIPAddressType": "Dynamic",
        "scriptUrl": "https://briseboispackages.blob.core.windows.net/linux/centos-vm-disk-utils-0.1.sh",
        "storageAccountType": "Standard_LRS",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]",
        "virtualNetworkName": "MyVNET",
        "vmName": "msbriseboislinux",
        "vmSize": "Standard_A3",
        "vmStorageAccountContainerName": "vhds",
        "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]"
    },
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('newStorageAccountName')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "StorageAccount"
            },
            "properties": {
                "accountType": "[variables('storageAccountType')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[parameters('dnsNameForPublicIP')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "PublicIPAddress"
            },
            "properties": {
                "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
                "dnsSettings": {
                    "domainNameLabel": "[parameters('dnsNameForPublicIP')]"
                }
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[variables('virtualNetworkName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "VirtualNetwork"
            },
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/networkInterfaces",
            "name": "[variables('nicName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "NetworkInterface"
            },
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/', parameters('dnsNameForPublicIP'))]",
                "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',parameters('dnsNameForPublicIP'))]"
                            },
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            }
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Compute/virtualMachines",
            "name": "[variables('vmName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "VirtualMachine"
            },
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
                "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
            ],
            "properties": {
                "hardwareProfile": {
                    "vmSize": "[variables('vmSize')]"
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
                        }
                    ]
                },
                "osProfile": {
                    "computername": "[variables('vmName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "[variables('imagePublisher')]",
                        "offer": "[variables('imageOffer')]",
                        "sku": "[parameters('OSVersion')]",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "osdisk",
                        "vhd": {
                            "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
                        },
                        "caching": "ReadWrite",
                        "createOption": "FromImage"
                    }
                }
            }
        }
    ]
}

Template Parameters

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "value": "mslinuxbrisebois"
        },
        "adminUsername": {
            "value": "brisebois"
        },
        "dnsNameForPublicIP": {
            "value": "briseboisdns"
        }
    }
}

Deploying the Template to Azure

To deploy the ARM Template, navigate to the folder that contains both the template and parameter files. Then call the New-AzureResourceGroup cmdlet.

Switch-AzureMode AzureResourceManager

New-AzureResourceGroup -Name 'mslinuxbrisebois' `
                       -Location 'westus' `
                       -TemplateFile 'DeploymentTemplate.json' `
                       -TemplateParameterFile 'DeploymentTemplate.param.dev.json' `
                       -Verbose

Creating a Resource Lock

Resource Locks can be applied on Resource Groups and on individual Resources like VMs and Storage Accounts. In this example, we are locking a Virtual Machine so that it cannot be deleted by accident.

New-AzureResourceLock -LockLevel CanNotDelete `
                      -LockNotes 'Critial Virtual Machine, do not delete.' `
                      -LockName 'msbriseboislinuxlock' `
                      -ResourceName 'msbriseboislinux' `
                      -ResourceType 'Microsoft.Compute/virtualMachines' `
                      -ResourceGroupName 'mslinuxbrisebois' `
                      -Verbose

Deleting the Resource Group

To show what happens when we try to delete a Resource Group that contains a Resource Lock, we can execute the Remove-AzureResourceGroup cmdlet.

Remove-AzureResourceGroup -Name 'mslinuxbrisebois'

The cmdlet should fail with an error similar to the following.

Remove-AzureResourceGroup : ScopeLocked: The scope
'/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourcegroups/mslinuxbrisebois' cannot perform delete operation because
following scope(s) are locked: '/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/mslinuxbrisebois/providers/Mi
crosoft.Compute/virtualMachines/msbriseboislinux'. Please remove the lock and try again.
At line:1 char:1
+ Remove-AzureResourceGroup -Name 'mslinuxbrisebois'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Remove-AzureResourceGroup], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.RemoveAzureResourceGroupCommand

Removing the Resource Lock & Deleting the Resource Group

Deleting a Resource Group that contains Resources Lock requires us to start by removing all Locks. Then we can use the Remove-AzureResourceGroup cmdlet to delete the Resource Group.

# remove the lock before the resource group
Get-AzureResourceLock -ResourceGroupName 'mslinuxbrisebois' `
                      -ResourceName 'msbriseboislinux' `
                      -ResourceType 'Microsoft.Compute/virtualMachines' `
| Remove-AzureResourceLock

Remove-AzureResourceGroup -Name 'mslinuxbrisebois'

Creating a CentOS VM Using ARM

Docker and Open Source projects are getting lots of attention, so I decided that it was time for me to build a Linux Virtual Machine on Microsoft Azure. This post is all about creating an Azure Resource Manager Template for a CentOS Virtual Machine with two stripped Data Disks. This template should be used as a starting point and may require some tweaking to meet your needs. Feel free to share your thoughts by using the comment section.

Finding a CentOS VM Image

Using our favorite PowerShell tools, we can discover details about pre-built CentOS 7.1 Azure Virtual Images.

Switch-AzureMode -Name AzureResourceManager

$location = 'westus'

Get-AzureVMImagePublisher -Location $location

$publisherName = 'OpenLogic'

Get-AzureVMImageOffer -Location $location `
                      -PublisherName $publisherName
$offer = 'CentOS'

Get-AzureVMImageSku -Location $location `
                    -PublisherName $publisherName `
                    -Offer $offer `
    | Select-Object -Property 'Skus'

$sku = '7.1'
                                 
Get-AzureVMImage -Location $location `
                 -PublisherName $publisherName `
                 -Offer $offer `
                 -Skus $sku

$verison = '7.1.201504'

Get-AzureVMImageDetail -Location $location `
                       -PublisherName $publisherName `
                       -Offer $offer `
                       -Skus $sku `
                       -Version $verison

Building the ARM Template

Starting from the Ubuntu Template in Visual Studio we can use the CentOS Virtual Image information, found in the previous section of this post, to create a CentOS 7.1 Standard_A3 Virtual Machine. To regroup the data disks as a RAID0 volume we can use the Create Ubuntu vm data disk raid0 template as a guide. Then we can make small changes to reach our goal.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "User name for the Virtual Machine."
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "Password for the Virtual Machine."
            }
        },
        "dnsNameForPublicIP": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
            }
        },
        "OSVersion": {
            "type": "string",
            "defaultValue": "7.1",
            "allowedValues": [
                "7.1"
            ]
        }
    },
    "variables": {
        "addressPrefix": "10.0.0.0/16",
        "dataDiskSize": "100",
        "imageOffer": "CentOS",
        "imagePublisher": "OpenLogic",
        "location": "West US",
        "nicName": "myVMNic",
        "OSDiskName": "osdisk",
        "publicIPAddressType": "Dynamic",
        "scriptUrl": "https://briseboispackages.blob.core.windows.net/linux/centos-vm-disk-utils-0.1.sh",
        "storageAccountType": "Standard_LRS",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]",
        "virtualNetworkName": "MyVNET",
        "vmName": "msbriseboislinux",
        "vmSize": "Standard_A3",
        "vmStorageAccountContainerName": "vhds",
        "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]"
    },
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('newStorageAccountName')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "StorageAccount"
            },
            "properties": {
                "accountType": "[variables('storageAccountType')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[parameters('dnsNameForPublicIP')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "PublicIPAddress"
            },
            "properties": {
                "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
                "dnsSettings": {
                    "domainNameLabel": "[parameters('dnsNameForPublicIP')]"
                }
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[variables('virtualNetworkName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "VirtualNetwork"
            },
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/networkInterfaces",
            "name": "[variables('nicName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "NetworkInterface"
            },
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/', parameters('dnsNameForPublicIP'))]",
                "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',parameters('dnsNameForPublicIP'))]"
                            },
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            }
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Compute/virtualMachines",
            "name": "[variables('vmName')]",
            "location": "[variables('location')]",
            "tags": {
                "displayName": "VirtualMachine"
            },
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
                "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
            ],
            "properties": {
                "hardwareProfile": {
                    "vmSize": "[variables('vmSize')]"
                },
                "osProfile": {
                    "computername": "[variables('vmName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "[variables('imagePublisher')]",
                        "offer": "[variables('imageOffer')]",
                        "sku": "[parameters('OSVersion')]",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "osdisk",
                        "vhd": {
                            "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
                        },
                        "caching": "ReadWrite",
                        "createOption": "FromImage"
                    },
                    "dataDisks": [
                        {
                            "name": "datadisk1",
                            "diskSizeGB": "[variables('dataDiskSize')]",
                            "lun": 0,
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/vhds/',variables('vmName'),'dataDisk1' ,'.vhd')]"
                            },
                            "caching": "None",
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk2",
                            "diskSizeGB": "[variables('dataDiskSize')]",
                            "lun": 1,
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/vhds/',variables('vmName') ,'dataDisk2','.vhd')]"
                            },
                            "caching": "None",
                            "createOption": "Empty"
                        }
                    ]
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
                        }
                    ]
                }
            }
        },
        {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "name": "[concat(variables('vmName'), '/azureVmUtils')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[variables('location')]",
            "dependsOn": [
                "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
            ],
            "properties": {
                "publisher": "Microsoft.OSTCExtensions",
                "type": "CustomScriptForLinux",
                "typeHandlerVersion": "1.2",
                "settings": {
                    "fileUris": [
                        "[variables('scriptUrl')]"
                    ],
                    "commandToExecute": "bash vm-disk-utils-0.1.sh -s"
                }
            }
        }
    ]
}

Publishing the ARM Template

Prepare the Template Parameter File

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "newStorageAccountName": {
            "value": "mslinuxbrisebois"
        },
        "adminUsername": {
            "value": "brisebois"
        },
        "dnsNameForPublicIP": {
            "value": "briseboisdns"
        }
    }
}

Then deploy the Azure Resource Manager Template and Parameters using PowerShell.

Set-AzureSubscription -SubscriptionName (Get-AzureSubscription -Current).SubscriptionName `
                      -CurrentStorageAccountName 'briseboispackages'

Switch-AzureMode AzureResourceManager

New-AzureResourceGroup -Name 'mslinuxbrisebois' `
                       -Location 'westus' `
                       -TemplateFile 'LinuxVirtualMachine.json' `
                       -TemplateParameterFile 'LinuxVirtualMachine.param.dev.json' `
                       -Force -Verbose

# New-AzureResourceGroup VERBOSE Log

VERBOSE: 10:15:18 AM - Created resource group 'mslinuxbrisebois' in location 'westus'
VERBOSE: 10:15:19 AM - Template is valid.
VERBOSE: 10:15:21 AM - Create template deployment 'LinuxVirtualMachine'.
VERBOSE: 10:15:24 AM - Resource Microsoft.Storage/storageAccounts 'mslinuxbrisebois1' provisioning status is running
VERBOSE: 10:15:34 AM - Resource Microsoft.Network/publicIPAddresses 'briseboisdns' provisioning status is running
VERBOSE: 10:15:36 AM - Resource Microsoft.Network/virtualNetworks 'MyVNET' provisioning status is running
VERBOSE: 10:15:46 AM - Resource Microsoft.Network/virtualNetworks 'MyVNET' provisioning status is succeeded
VERBOSE: 10:15:49 AM - Resource Microsoft.Network/publicIPAddresses 'briseboisdns' provisioning status is succeeded
VERBOSE: 10:15:51 AM - Resource Microsoft.Network/networkInterfaces 'myVMNic' provisioning status is succeeded
VERBOSE: 10:15:58 AM - Resource Microsoft.Storage/storageAccounts 'mslinuxbrisebois1' provisioning status is succeeded
VERBOSE: 10:16:03 AM - Resource Microsoft.Compute/virtualMachines 'msbriseboislinux1' provisioning status is running
VERBOSE: 10:18:16 AM - Resource Microsoft.Compute/virtualMachines 'msbriseboislinux1' provisioning status is succeeded
VERBOSE: 10:18:21 AM - Resource Microsoft.Compute/virtualMachines/extensions 'msbriseboislinux1/azureVmUtils' provisioning status is running
VERBOSE: 10:20:55 AM - Resource Microsoft.Compute/virtualMachines/extensions 'msbriseboislinux1/azureVmUtils' provisioning status is succeeded

This template will use the following script inspired by vm-disk-utils-0.1.sh for Ubuntu to create a RAID0 out of all the unformated data disks. A small modification has been applied for this to work on CentOS.

#!/bin/bash

# The MIT License (MIT)
#
# Copyright (c) 2015 Microsoft Azure
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#
# Script Name: centos-vm-disk-utils.sh
# Author: Trent Swanson - Full Scale 180 Inc github:(trentmswanson)
# Version: 0.1
# Last Modified By: Alexandre Brisebois
# Description:
#  This script automates the partitioning and formatting of data disks
#  Data disks can be partitioned and formatted as seperate disks or in a RAID0 configuration
#  The scrtip will scan for unpartitioined and unformatted data disks and partition, format, and add fstab entries
# Parameters :
#  1 - b: The base directory for mount points (default: /datadisks)
#  2 - s  Create a striped RAID0 Array (No redundancy)
#  3 - h  Help 
# Note : 
# This script has only been tested on CentOS 7.1 and must be root

help()
{
    echo "Usage: $(basename $0) [-b data_base] [-h] [-s]"
    echo ""
    echo "Options:"
    echo "   -b         base directory for mount points (default: /datadisks)"
    echo "   -h         this help message"
    echo "   -s         create a striped RAID array (no redundancy)"
}

log()
{
    # Un-comment the following if you would like to enable logging to a service
    #curl -X POST -H "content-type:text/plain" --data-binary "${HOSTNAME} - $1" https://logs-01.loggly.com/inputs/<key>/tag/es-extension,${HOSTNAME}
    echo "$1"
}

if [ "${UID}" -ne 0 ];
then
    log "Script executed without root permissions"
    echo "You must be root to run this program." >&2
    exit 3
fi

#A set of disks to ignore from partitioning and formatting
BLACKLIST="/dev/sda|/dev/sdb"

# Base path for data disk mount points
DATA_BASE="/datadisks"

while getopts b:sh optname; do
    log "Option $optname set with value ${OPTARG}"
  case ${optname} in
    b)  #set clsuter name
      DATA_BASE=${OPTARG}
      ;;
    s) #Partition and format data disks as raid set
      RAID_CONFIGURATION=1
      ;;
    h)  #show help
      help
      exit 2
      ;;
    \?) #unrecognized option - show help
      echo -e \\n"Option -${BOLD}$OPTARG${NORM} not allowed."
      help
      exit 2
      ;;
  esac
done

get_next_md_device() {
    shopt -s extglob
    LAST_DEVICE=$(ls -1 /dev/md+([0-9]) 2>/dev/null|sort -n|tail -n1)
    if [ -z "${LAST_DEVICE}" ]; then
        NEXT=/dev/md0
    else
        NUMBER=$((${LAST_DEVICE/\/dev\/md/}))
        NEXT=/dev/md${NUMBER}
    fi
    echo ${NEXT}
}

is_partitioned() {
    OUTPUT=$(partx -s ${1} 2>&1)
    egrep "partition table does not contains usable partitions|failed to read partition table" <<< "${OUTPUT}" >/dev/null 2>&1
    if [ ${?} -eq 0 ]; then
        return 1
    else
        return 0
    fi    
}

has_filesystem() {
    DEVICE=${1}
    OUTPUT=$(file -L -s ${DEVICE})
    grep filesystem <<< "${OUTPUT}" > /dev/null 2>&1
    return ${?}
}

scan_for_new_disks() {
    # Looks for unpartitioned disks
    declare -a RET
    DEVS=($(ls -1 /dev/sd*|egrep -v "${BLACKLIST}"|egrep -v "[0-9]$"))
    for DEV in "${DEVS[@]}";
    do
        # The disk will be considered a candidate for partitioning
        # and formatting if it does not have a sd?1 entry or
        # if it does have an sd?1 entry and does not contain a filesystem
        is_partitioned "${DEV}"
        if [ ${?} -eq 0 ];
        then
            has_filesystem "${DEV}1"
            if [ ${?} -ne 0 ];
            then
                RET+=" ${DEV}"
            fi
        else
            RET+=" ${DEV}"
        fi
    done
    echo "${RET}"
}

get_next_mountpoint() {
    DIRS=$(ls -1d ${DATA_BASE}/disk* 2>/dev/null| sort --version-sort)
    MAX=$(echo "${DIRS}"|tail -n 1 | tr -d "[a-zA-Z/]")
    if [ -z "${MAX}" ];
    then
        echo "${DATA_BASE}/disk1"
        return
    fi
    IDX=1
    while [ "${IDX}" -lt "${MAX}" ];
    do
        NEXT_DIR="${DATA_BASE}/disk${IDX}"
        if [ ! -d "${NEXT_DIR}" ];
        then
            echo "${NEXT_DIR}"
            return
        fi
        IDX=$(( ${IDX} + 1 ))
    done
    IDX=$(( ${MAX} + 1))
    echo "${DATA_BASE}/disk${IDX}"
}

add_to_fstab() {
    UUID=${1}
    MOUNTPOINT=${2}
    grep "${UUID}" /etc/fstab >/dev/null 2>&1
    if [ ${?} -eq 0 ];
    then
        echo "Not adding ${UUID} to fstab again (it's already there!)"
    else
        LINE="UUID=\"${UUID}\"\t${MOUNTPOINT}\text4\tnoatime,nodiratime,nodev,noexec,nosuid\t1 2"
        echo -e "${LINE}" >> /etc/fstab
    fi
}

do_partition() {
# This function creates one (1) primary partition on the
# disk, using all available space
    _disk=${1}
    _type=${2}
    if [ -z "${_type}" ]; then
        # default to Linux partition type (ie, ext3/ext4/xfs)
        _type=83
    fi
    echo "n
p
1


t
${_type}
w"| fdisk "${_disk}"

#
# Use the bash-specific $PIPESTATUS to ensure we get the correct exit code
# from fdisk and not from echo
if [ ${PIPESTATUS[1]} -ne 0 ];
then
    echo "An error occurred partitioning ${_disk}" >&2
    echo "I cannot continue" >&2
    exit 2
fi
}
#end do_partition

scan_partition_format()
{
    log "Begin scanning and formatting data disks"

    DISKS=($(scan_for_new_disks))

    if [ "${#DISKS}" -eq 0 ];
    then
        log "No unpartitioned disks without filesystems detected"
        return
    fi
    echo "Disks are ${DISKS[@]}"
    for DISK in "${DISKS[@]}";
    do
        echo "Working on ${DISK}"
        is_partitioned ${DISK}
        if [ ${?} -ne 0 ];
        then
            echo "${DISK} is not partitioned, partitioning"
            do_partition ${DISK}
        fi
        PARTITION=$(fdisk -l ${DISK}|grep -A 1 Device|tail -n 1|awk '{print $1}')
        has_filesystem ${PARTITION}
        if [ ${?} -ne 0 ];
        then
            echo "Creating filesystem on ${PARTITION}."
    #        echo "Press Ctrl-C if you don't want to destroy all data on ${PARTITION}"
    #        sleep 10
            mkfs -j -t ext4 ${PARTITION}
        fi
        MOUNTPOINT=$(get_next_mountpoint)
        echo "Next mount point appears to be ${MOUNTPOINT}"
        [ -d "${MOUNTPOINT}" ] || mkdir -p "${MOUNTPOINT}"
        read UUID FS_TYPE < <(blkid -u filesystem ${PARTITION}|awk -F "[= ]" '{print $3" "$5}'|tr -d "\"")
        add_to_fstab "${UUID}" "${MOUNTPOINT}"
        echo "Mounting disk ${PARTITION} on ${MOUNTPOINT}"
        mount "${MOUNTPOINT}"
    done
}

create_striped_volume()
{
    DISKS=(${@})

    if [ "${#DISKS[@]}" -eq 0 ];
    then
        log "No unpartitioned disks without filesystems detected"
        return
    fi

    echo "Disks are ${DISKS[@]}"

    declare -a PARTITIONS

    for DISK in "${DISKS[@]}";
    do
        echo "Working on ${DISK}"
        is_partitioned ${DISK}
        if [ ${?} -ne 0 ];
        then
            echo "${DISK} is not partitioned, partitioning"
            do_partition ${DISK} fd
        fi

        PARTITION=$(fdisk -l ${DISK}|grep -A 2 Device|tail -n 1|awk '{print $1}')
        PARTITIONS+=("${PARTITION}")
    done

    MDDEVICE=$(get_next_md_device)    
    
    mdadm --create ${MDDEVICE} --level 0 --raid-devices ${#PARTITIONS[@]} ${PARTITIONS[*]}

    MOUNTPOINT=$(get_next_mountpoint)
    echo "Next mount point appears to be ${MOUNTPOINT}"
    [ -d "${MOUNTPOINT}" ] || mkdir -p "${MOUNTPOINT}"

    #Make a file system on the new device
    mkfs -t ext4 "${MDDEVICE}"

    read UUID FS_TYPE < <(blkid -u filesystem ${MDDEVICE}|awk -F "[= ]" '{print $3" "$5}'|tr -d "\"")

    add_to_fstab "${UUID}" "${MOUNTPOINT}"

    mount "${MOUNTPOINT}"
}

check_mdadm() {
    rpm -ql mdadm >/dev/null 2>&1
    if [ ${?} -ne 0 ]; then
        yum clean all
        yum update
        yum install mdadm -y
    fi
}

# Create Partitions
DISKS=$(scan_for_new_disks)

if [ "$RAID_CONFIGURATION" -eq 1 ]; then
    check_mdadm
    create_striped_volume "${DISKS[@]}"
else
    scan_partition_format
fi

Once the Azure Resource Manager Template completed its execution, I opened an SSH session using Putty and verified that our data disks have been stripped and mounted.

We are now ready to start working with this brand new CentOS Virtual Machine.

Resources

• Creating Azure VMs with ARM PowerShell cmdlets
• How to Use SSH with Linux on Azure
• How to Reset a Password or SSH for Linux Virtual Machines
• Azure Quickstart Templates

My previous post was about using the Azure Resource Manager to provision a Virtual Machine. It demonstrated how to use a custom PowerShell Desired State Configuration (DSC) to stripe, format and assign a drive letter to a storage space that contains thirty-two 1TB VHDs.

Debugging a custom PowerShell Desired State Configuration was a challenge, because I was completely new to PowerShell DSC and to the Azure Virtual Machine DSC Extension. This post is all about how I managed to debug, refine and test using the artifacts created by the Virtual Machine Extension.

Why is my DSC Failing?

The Azure Virtual Machine Extension for DSC gives us ways to test, debug and refine our Desired State Configurations. Finding that pesky bug can be challenging, don’t give up! Let’s take a look at what we should be looking for.

The Environment

To have something to work with, we need a Virtual Machine that uses the DSC Extension. So I used the Azure Resource Manager Template from my previous post to provision a brand-new VM.

Set-AzureSubscription -SubscriptionName (Get-AzureSubscription -Current).SubscriptionName -CurrentStorageAccountName 'briseboispackages'

$ModuleURL = "https://briseboispackages.blob.core.windows.net/windows-powershell-dsc/DiskConfiguration.ps1.zip"

$additionalParameters = New-Object -TypeName Hashtable
$additionalParameters["modulesUrl"] = $ModuleURL

Switch-AzureMode AzureResourceManager

New-AzureResourceGroup -Name "armmonstervm" `
                       -Location "westus" `
                       -TemplateFile 'WindowsVirtualMachine.json' `
                       -TemplateParameterFile 'WindowsVirtualMachine.param.dev.json' `
                       @additionalParameters `
                       -Force -Verbose

New-AzureResourceGroup Execution Trace

VERBOSE: 10:48:56 AM - Created resource group 'armmonstervm' in location 'westus'
VERBOSE: 10:48:57 AM - Template is valid.
VERBOSE: 10:48:58 AM - Create template deployment 'WindowsVirtualMachine'.
VERBOSE: 10:49:06 AM - Resource Microsoft.Storage/storageAccounts 'msarmbrisebois' provisioning status is running
VERBOSE: 10:49:11 AM - Resource Microsoft.Network/virtualNetworks 'msarmbriseboisvnet' provisioning status is running
VERBOSE: 10:49:13 AM - Resource Microsoft.Network/publicIPAddresses 'PublicIP' provisioning status is running
VERBOSE: 10:49:27 AM - Resource Microsoft.Network/publicIPAddresses 'PublicIP' provisioning status is succeeded
VERBOSE: 10:49:27 AM - Resource Microsoft.Network/virtualNetworks 'msarmbriseboisvnet' provisioning status is succeeded
VERBOSE: 10:49:32 AM - Resource Microsoft.Network/networkInterfaces 'VMNic' provisioning status is succeeded
VERBOSE: 10:49:32 AM - Resource Microsoft.Storage/storageAccounts 'msarmbrisebois' provisioning status is succeeded
VERBOSE: 10:49:40 AM - Resource Microsoft.Compute/virtualMachines 'msarmbrisebois' provisioning status is running
VERBOSE: 11:08:28 AM - Resource Microsoft.Compute/virtualMachines 'msarmbrisebois' provisioning status is succeeded
VERBOSE: 11:08:31 AM - Resource Microsoft.Compute/virtualMachines/extensions 'msarmbrisebois/dscExtension' provisioning status is running
VERBOSE: 11:16:27 AM - Resource Microsoft.Compute/virtualMachines/extensions 'msarmbrisebois/dscExtension' provisioning status is succeeded

Now that we have a Virtual Machine to work with, it’s time to use Remote Desktop.

Switch-AzureMode AzureResourceManager

Get-AzureRemoteDesktopFile -ResourceGroupName 'armmonstervm' `
                           -Name 'msarmbrisebois' `
                           -LocalPath 'C:\Users\brisebois'

Location of The DSC VM Extension Artifacts

When the DSC Virtual Machine Extension executes, it installs DSC in C:\Packages\Plugins\Microsoft.Powershell.DSC\1.7.0.0. It will also download and extract our DSC Module to C:\Program Files\WindowsPowerShell\Modules.

Location: C:\Packages\Plugins\Microsoft.Powershell.DSC\1.7.0.0

This folder contains 3 important sub-folders that help us understand what is push to the Virtual Machine from the Azure Resource Manager and the status of the DSC execution.

RuntimeSettings

This folder contains the Extension Settings that allow us to validate that we have the correct settings in our Azure Resource Manager Template.

{
    "runtimeSettings": [{
        "handlerSettings": {
            "publicSettings": {
                "ModulesUrl": "https://briseboispackages.blob.core.windows.net/windows-powershell-dsc/DiskConfiguration.ps1.zip",
                "SasToken": "",
                "ConfigurationFunction": "DiskConfiguration.ps1\\DiskConfiguration",
                "Properties": {
                    "StoragePoolFriendlyName": "storagespace0",
                    "VirtualDiskFriendlyName": "datadisk",
                    "DriveSize": 35184372088832,
                    "NumberOfColumns": 32
                }
            }
        }
    }]
}

DSCWork

This folder contains the configuration and the localhost.mof file. Use these assets to replay the DSC as if it were executed by the Extension.

Start-DscConfiguration -Path 'C:\Packages\Plugins\Microsoft.Powershell.DSC\1.7.0.0\DSCWork\DiskConfiguration.ps1.0\DiskConfiguration' `
                       -ComputerName localhost `
                       -Force `
                       -wait `
                       -Verbose `
                       -Debug

This execution produces an output that surfaces issues. Use this to identify where the DSC fails and what to address.

VERBOSE: Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = SendConfigurationApply,'className' = MSFT_DSC
LocalConfigurationManager,'namespaceName' = root/Microsoft/Windows/DesiredStateConfiguration'.
VERBOSE: An LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-21-3103417715-464229043-512622935-500.
VERBOSE: An LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-21-3103417715-464229043-512622935-500.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Set      ]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[StorageSpace]CreateStorageSpace]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[StorageSpace]CreateStorageSpace]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[StorageSpace]CreateStorageSpace]  in 1.5790 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Set      ]  [[StorageSpace]CreateStorageSpace]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Set      ]  [[StorageSpace]CreateStorageSpace]  in 62.5030 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[StorageSpace]CreateStorageSpace]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[VirtualDisk]CreateVirtualDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[VirtualDisk]CreateVirtualDisk]
DEBUG: [MSARMBRISEBOIS]:                            [[VirtualDisk]CreateVirtualDisk] datadisk
DEBUG: [MSARMBRISEBOIS]:                            [[VirtualDisk]CreateVirtualDisk] False
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[VirtualDisk]CreateVirtualDisk]  in 0.0710 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Set      ]  [[VirtualDisk]CreateVirtualDisk]
DEBUG: [MSARMBRISEBOIS]:                            [[VirtualDisk]CreateVirtualDisk] datadisk
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Set      ]  [[VirtualDisk]CreateVirtualDisk]  in 2.4500 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[VirtualDisk]CreateVirtualDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[Disk]InitializeDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[Disk]InitializeDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[Disk]InitializeDisk]  in 0.2260 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Set      ]  [[Disk]InitializeDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Set      ]  [[Disk]InitializeDisk]  in 0.3820 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[Disk]InitializeDisk]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[DiskPartition]CreateDiskPartition]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[DiskPartition]CreateDiskPartition]
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[DiskPartition]CreateDiskPartition]  in 0.1530 seconds.
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ Start  Set      ]  [[DiskPartition]CreateDiskPartition]
Invalid Parameter
    + CategoryInfo          : InvalidArgument: (:) [], CimException
    + FullyQualifiedErrorId : StorageWMI 5,New-Partition
    + PSComputerName        : localhost
 
VERBOSE: [MSARMBRISEBOIS]: LCM:  [ End    Set      ]  [[DiskPartition]CreateDiskPartition]  in 0.5880 seconds.
The PowerShell DSC resource cCreateDiskPartition threw one or more non-terminating errors while running the Set-TargetResource 
functionality. These errors are logged to the ETW channel called Microsoft-Windows-DSC/Operational. Refer to this channel for more 
details.
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : NonTerminatingErrorFromProvider
    + PSComputerName        : localhost
 
The SendConfigurationApply function did not succeed.
    + CategoryInfo          : NotSpecified: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : MI RESULT 1
    + PSComputerName        : localhost
 
VERBOSE: Operation 'Invoke CimMethod' complete.
VERBOSE: Time taken for configuration job to complete is 68.836 seconds

The error above is generated because of this following PowerShell.

    Get-VirtualDisk -FriendlyName $VirtualDiskFriendlyName `
    | Get-Disk `
    | New-Partition -UseMaximumSize `
                    -AssignDriveLetter `
                    -DriveLetter Y `
    | Format-Volume

It failed because I used both the AssignDriveLetter and the DriveLetter parameters. Removing the DriveLetter parameter allowed the script to auto assign the next available drive letter.

Status

This folder contains the DSC Status files. Are a great way to get insight into what happened when the VM Extension ran.

[{
    "status": {
        "substatus": [{
            "status": "success",
            "code": 1,
            "name": "DscConfigurationLog",
            "formattedMessage": {
                "message": "Perform operation \u0027Invoke CimMethod\u0027 with following parameters, \u0027\u0027methodName\u0027 = SendConfigurationApply,\u0027className\u0027 = MSFT_DSCLocalConfigurationManager,\u0027namespaceName\u0027 = root/Microsoft/Windows/DesiredStateConfiguration\u0027.\r\nAn LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-18.\r\nAn LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-18.\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Set      ]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[StorageSpace]CreateStorageSpace]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[StorageSpace]CreateStorageSpace]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[StorageSpace]CreateStorageSpace]  in 1.5470 seconds.\r\n[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[StorageSpace]CreateStorageSpace]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[StorageSpace]CreateStorageSpace]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[VirtualDisk]CreateVirtualDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[VirtualDisk]CreateVirtualDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[VirtualDisk]CreateVirtualDisk]  in 0.0780 seconds.\r\n[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[VirtualDisk]CreateVirtualDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[VirtualDisk]CreateVirtualDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[Disk]InitializeDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[Disk]InitializeDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[Disk]InitializeDisk]  in 0.1450 seconds.\r\n[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[Disk]InitializeDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[Disk]InitializeDisk]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[DiskPartition]CreateDiskPartition]\r\n[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[DiskPartition]CreateDiskPartition]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[DiskPartition]CreateDiskPartition]  in 0.1440 seconds.\r\n[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[DiskPartition]CreateDiskPartition]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[DiskPartition]CreateDiskPartition]\r\n[MSARMBRISEBOIS]: LCM:  [ End    Set      ]    in  2.4610 seconds.\r\nOperation \u0027Invoke CimMethod\u0027 complete.\r\nTime taken for configuration job to complete is 18.63 seconds",
                "lang": "en-US"
            }
        }],
        "status": "success",
        "code": 1,
        "formattedMessage": {
            "message": "DSC configuration was applied successfully.",
            "lang": "en-US"
        }
    },
    "timestampUTC": "2015-05-20 17:27:50Z",
    "version": "1.0"
}]

The substatus contains a capture of the log. This will contain details about the exceptions that occurred which DSC tried to apply the configuration.

Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = SendConfigurationApply,'className' = MSFT_DSCLocalConfigurationManager,'namespaceName' = root/Microsoft/Windows/DesiredStateConfiguration'.
An LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-18.
An LCM method call arrived from computer MSARMBRISEBOIS with user sid S-1-5-18.
[MSARMBRISEBOIS]: LCM:  [ Start  Set      ]
[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[StorageSpace]CreateStorageSpace]
[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[StorageSpace]CreateStorageSpace]
[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[StorageSpace]CreateStorageSpace]  in 1.5470 seconds.
[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[StorageSpace]CreateStorageSpace]
[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[StorageSpace]CreateStorageSpace]
[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[VirtualDisk]CreateVirtualDisk]
[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[VirtualDisk]CreateVirtualDisk]
[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[VirtualDisk]CreateVirtualDisk]  in 0.0780 seconds.
[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[VirtualDisk]CreateVirtualDisk]
[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[VirtualDisk]CreateVirtualDisk]
[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[Disk]InitializeDisk]
[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[Disk]InitializeDisk]
[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[Disk]InitializeDisk]  in 0.1450 seconds.
[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[Disk]InitializeDisk]
[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[Disk]InitializeDisk]
[MSARMBRISEBOIS]: LCM:  [ Start  Resource ]  [[DiskPartition]CreateDiskPartition]
[MSARMBRISEBOIS]: LCM:  [ Start  Test     ]  [[DiskPartition]CreateDiskPartition]
[MSARMBRISEBOIS]: LCM:  [ End    Test     ]  [[DiskPartition]CreateDiskPartition]  in 0.1440 seconds.
[MSARMBRISEBOIS]: LCM:  [ Skip   Set      ]  [[DiskPartition]CreateDiskPartition]
[MSARMBRISEBOIS]: LCM:  [ End    Resource ]  [[DiskPartition]CreateDiskPartition]
[MSARMBRISEBOIS]: LCM:  [ End    Set      ]    in  2.4610 seconds.
Operation 'Invoke CimMethod' complete.
Time taken for configuration job to complete is 18.63 seconds

Location: C:\Program Files\WindowsPowerShell\Modules

This folder contains the uncompressed DSC configuration and Modules that were packaged and uploaded to Azure Storage or any other accessible location.

Debugging, Testing and Refining

Now that we understand the environment that we have to work with, it’s time to dive in.

To start our debug session, we have to look at the status that was generated by DSC. This will give us something to work with. Then based on what we find, we can look at the configuration that was used to execute the DSC. Next based on our findings we can look at the DSC configuration and modules installed on the Virtual Machine.

Once we find out what has gone terribly wrong. We can fix the module in the target Virtual Machine and trigger the Start-DscConfiguration cmdlet. Using the produced output, we can iterate through dev & test cycles until we reach our goal.

When we have something that works, we copy the DSC configuration and module out of the vm. Then we insert it into our Azure Resource Manager Template Solution. From this point on, it’s business as usual. We package the DSC configuration and modules. And finally, we test the entire pipeline by deploying the template to a new Azure Resource Group.

A Monster VM Azure Resource Manager Template

In April I wrote a post about building a monster Virtual Machine using PowerShell on Microsoft Azure. Since then, Microsoft has released version 2 of the Azure Resource Manager (ARM). This version allows us to define a Virtual Machine, its data disks and its Desired State Configuration (DSC) VM Extensions as a template. Seeing this as a great opportunity, I decided to convert my first PowerShell script to an ARM template that would create a Virtual Machine and striped data disk.

The Target Virtual Machine Configuration

16 Cores
112 GB of RAM
800 GB of local SSD for temp disk
32 TB for the data disk
50,000 IOPS for the data disk
512 MB per second for the data disk

What is the Azure Resource Manager (ARM)?

Most applications that run in Microsoft Azure use a combination of resources (such as a database server, database, or website) to perform as designed. An Azure Resource Manager Template makes it possible for you to deploy and manage these resources together by using a JSON description of the resources and associated deployment parameters.

Azure Resource Manager Template

Using the Windows Server Virtual Machine template as a base, I delved into various templates from the Azure Quickstart Templates. These templates helped me figure out how to create an OS Disk in an Azure Premium Storage Account and how to attach thirty-two 1TB data disks.

The following is the template that resulted from this exercise. In order to, navigate and build this template, I used the Visual Studio JSON Explorer.

WindowsVirtualMachine.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.1",
    "parameters": {
        "newStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "Username for the Virtual Machine."
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "Password for the Virtual Machine."
            }
        },
        "location": {
            "type": "string",
            "metadata": {
                "description": "Data Center"
            },
            "defaultValue": "West US",
            "allowedValues": [
                "West US",
                "East US 2",
                "West Europe",
                "Southeast Asia",
                "Japan West"
            ]
        },
        "vmSize": {
            "type": "string",
            "metadata": {
                "description": "Virtual Machine Size"
            },
            "defaultValue": "Standard_DS14",
            "allowedValues": [
                "Standard_DS14"
            ]
        },
        "vmName": {
            "type": "string",
            "metadata": {
                "description": "Unique Name for the Virtual Machine"
            }
        },
        "dnsNameForPublicIP": {
            "type": "string",
            "metadata": {
                "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
            }
        },
        "windowsOSVersion": {
            "type": "string",
            "defaultValue": "2012-R2-Datacenter",
            "allowedValues": [
                "2008-R2-SP1",
                "2012-Datacenter",
                "2012-R2-Datacenter",
                "Windows-Server-Technical-Preview"
            ],
            "metadata": {
                "description": "The Windows version for the VM. This will pick a fully patched image of this given Windows version. Allowed values: 2008-R2-SP1, 2012-Datacenter, 2012-R2-Datacenter, Windows-Server-Technical-Preview."
            }
        },
        "modulesUrl": {
            "type": "string",
            "metadata": {
                "description": "URL for the DSC configuration module. NOTE: Can be a Github url(raw) to the zip file"
            }
        },
        "configurationFunction": {
            "type": "string",
            "metadata": {
                "description": "DSC configuration function to call"
            }
        }
    },
    "variables": {
        "imagePublisher": "MicrosoftWindowsServer",
        "imageOffer": "WindowsServer",
        "OSDiskName": "osdisk",
        "sizeOfDiskInGB": "1023",
        "nicName": "VMNic",
        "addressPrefix": "10.0.0.0/16",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "storageAccountType": "Premium_LRS",
        "publicIPAddressName": "PublicIP",
        "publicIPAddressType": "Dynamic",
        "vmStorageAccountContainerName": "vhds",
        "virtualNetworkName": "[concat(parameters('vmName'),'vnet')]",
        "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]",
        "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]"
    },
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('newStorageAccountName')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "StorageAccount"
            },
            "properties": {
                "accountType": "[variables('storageAccountType')]"
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[variables('publicIPAddressName')]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "PublicIPAddress"
            },
            "properties": {
                "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
                "dnsSettings": {
                    "domainNameLabel": "[parameters('dnsNameForPublicIP')]"
                }
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[variables('virtualNetworkName')]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "VirtualNetwork"
            },
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Network/networkInterfaces",
            "name": "[variables('nicName')]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "NetworkInterface"
            },
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
                "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
                            },
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            }
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "2015-05-01-preview",
            "type": "Microsoft.Compute/virtualMachines",
            "name": "[parameters('vmName')]",
            "location": "[parameters('location')]",
            "tags": {
                "displayName": "VirtualMachine"
            },
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
                "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
            ],
            "properties": {
                "hardwareProfile": {
                    "vmSize": "[parameters('vmSize')]"
                },
                "osProfile": {
                    "computername": "[parameters('vmName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "[variables('imagePublisher')]",
                        "offer": "[variables('imageOffer')]",
                        "sku": "[parameters('windowsOSVersion')]",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "osdisk",
                        "vhd": {
                            "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
                        },
                        "caching": "ReadWrite",
                        "createOption": "FromImage"
                    },
                    "dataDisks": [
                        {
                            "name": "datadisk1",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 0,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','1','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk2",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 1,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','2','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk3",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 2,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','3','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk4",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 3,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','4','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk5",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 4,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','5','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk6",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 5,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','6','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk7",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 6,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','7','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk8",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 7,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','8','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk9",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 8,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','9','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk10",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 9,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','10','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk11",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 10,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','11','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk12",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 11,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','12','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk13",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 12,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','13','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk14",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 13,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','14','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk15",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 14,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','15','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk16",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 15,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','16','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk17",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 16,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','17','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk18",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 17,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','18','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk19",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 18,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','19','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk20",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 19,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','20','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk21",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 20,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','21','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk22",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 21,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','22','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk23",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 22,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','23','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk24",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 23,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','24','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk25",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 24,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','25','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk26",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 25,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','26','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk27",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 26,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','27','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk218",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 27,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','28','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk29",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 28,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','29','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk30",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 29,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','30','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk31",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 30,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','31','.vhd')]"
                            },
                            "createOption": "Empty"
                        },
                        {
                            "name": "datadisk32",
                            "diskSizeGB": "[variables('sizeOfDiskInGB')]",
                            "lun": 31,
                            "caching": "ReadOnly",
                            "vhd": {
                                "Uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/','datadisk','32','.vhd')]"
                            },
                            "createOption": "Empty"
                        }
                    ]
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
                        }
                    ]
                }
            }
        },
        {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "name": "[concat(parameters('vmName'),'/', 'dscExtension')]",
            "apiVersion": "2015-05-01-preview",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"
            ],
            "properties": {
                "publisher": "Microsoft.Powershell",
                "type": "DSC",
                "typeHandlerVersion": "1.7",
                "settings": {
                    "ModulesUrl": "[parameters('modulesUrl')]",
                    "SasToken": "",
                    "ConfigurationFunction": "[parameters('configurationFunction')]",
                    "Properties": {
                        "StoragePoolFriendlyName" : "storagespace0",
                        "VirtualDiskFriendlyName" : "datadisk",
                        "DriveSize": 35184372088832,
                        "NumberOfColumns": 32
                    }
                },
                "protectedSettings": null
            }
        }
    ]
}

Azure Resource Manager Template Parameters

This file contains deployment specific parameter values for the Azure Resource Manager Template. As the name suggests, we can have multiple parameter files per project. If a parameter is missing from this file, it will be requested when the deployment is initiated.

WindowsVirtualMachine.param.dev.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.1",
    "parameters": {
        "newStorageAccountName": {
            "value": "msarmbrisebois"
        },
        "adminUsername": {
            "value": "brisebois"
        },
        "dnsNameForPublicIP": {
            "value": "msarmbrisebois"
        },
        "vmName": {
            "value": "msarmbrisebois"
        },
        "windowsOSVersion": {
            "value": "2012-R2-Datacenter"
        },
        "configurationFunction": {
            "value": "DiskConfiguration.ps1\\DiskConfiguration"
        }
    }
}

Creating a Custom DSC

This custom Desired State Configuration (DSC) stripes 32 Premium Data Disks using the DSC Virtual Machine Extension. To create this Module Iets borrowed from my earlier post that create a custom DSC Module to Stripe data disks on Azure Virtual Machines. First we need to make a few changes. We will start by breaking each step into its own module. Then we will add better test logic so that if one step fails, the DSC can be executed again.

The following scaffolding script builds out the files and folder structure for the DSC that we are building

Scaffolding the DSC Module

New-xDscResource –Name cCreateStorageSpace `
                 -Property (New-xDscResourceProperty –Name FriendlyName `
                                                     –Type String `
                                                     -Attribute Key) `
                 -Path 'C:\Program Files\WindowsPowerShell\Modules\' `
                 -ModuleName cDiskTools `
                 -FriendlyName StorageSpace `
                 -Verbose

New-xDscResource –Name cCreateVirtualDisk `
                 -Property (New-xDscResourceProperty –Name FriendlyName `
                                                     –Type String `
                                                     -Attribute Key), `
                           (New-xDscResourceProperty –Name StoragePoolFriendlyName `
                                                     –Type String `
                                                     -Attribute Required),
                           (New-xDscResourceProperty –Name DriveSize `
                                                     –Type UInt64 `
                                                     -Attribute Required), `
                           (New-xDscResourceProperty –Name NumberOfColumns `
                                                     -Type Uint32 `
                                                     -Attribute Required) `
                 -Path 'C:\Program Files\WindowsPowerShell\Modules\' `
                 -ModuleName cDiskTools `
                 -FriendlyName VirtualDisk `
                 -Verbose

New-xDscResource –Name cInitializeDisk `
                 -Property (New-xDscResourceProperty –Name VirtualDiskFriendlyName `
                                                     –Type String `
                                                     -Attribute Key) `
                 -Path 'C:\Program Files\WindowsPowerShell\Modules\' `
                 -ModuleName cDiskTools `
                 -FriendlyName Disk `
                 -Verbose

New-xDscResource –Name cCreateDiskPartition `
                 -Property (New-xDscResourceProperty –Name VirtualDiskFriendlyName `
                                                     –Type String `
                                                     -Attribute Key) `
                 -Path 'C:\Program Files\WindowsPowerShell\Modules\' `
                 -ModuleName cDiskTools `
                 -FriendlyName DiskPartition `
                 -Verbose

Now that we have something to work with, let’s port the code from DSC I created for my earlier post. The following files reside in the following location C:\Program Files\WindowsPowerShell\Modules\cDiskTools\

DSCResources\cCreateStorageSpace\cCreateStorageSpace.psm1

The first module tests whether the storage space already exists. If it’s missing, the module will create it.

function Get-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Collections.Hashtable])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName
	)
}

function Set-TargetResource
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName
	)

    New-StoragePool -FriendlyName $FriendlyName `
                    -StorageSubSystemUniqueId (Get-StorageSubSystem -FriendlyName '*Space*').uniqueID `
                    -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
}

function Test-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Boolean])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName
	)

    $result = [System.Boolean]

    try
    {
        $pool = Get-StoragePool -FriendlyName $FriendlyName -ErrorAction Ignore
        if(!$pool){
            $result = $false
        }else{
            $result = $true
        }
    }
    catch [System.Exception]
    {
        $result = $false
    }

    $result
}

Export-ModuleMember -Function *-TargetResource

DSCResources\cCreateVirtualDisk\cCreateVirtualDisk.psm1

The second module will check whether the virtual disk already exists. If it’s missing it will create it using the provided parameters.

function Get-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Collections.Hashtable])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName,

		[parameter(Mandatory = $true)]
		[System.String]
		$StoragePoolFriendlyName,

		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)

}

function Set-TargetResource
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName,

		[parameter(Mandatory = $true)]
		[System.String]
		$StoragePoolFriendlyName,

		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)

    Write-Debug $FriendlyName

	New-VirtualDisk -FriendlyName $FriendlyName `
                    -StoragePoolFriendlyName $StoragePoolFriendlyName `
                    -Size $DriveSize `
                    -NumberOfColumns $NumberOfColumns `
                    -ProvisioningType Thin `
                    -ResiliencySettingName Simple

}

function Test-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Boolean])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$FriendlyName,

		[parameter(Mandatory = $true)]
		[System.String]
		$StoragePoolFriendlyName,

		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)

    Write-Debug $FriendlyName

    $result = [System.Boolean]

    try
    {
        $disk = Get-VirtualDisk -FriendlyName $FriendlyName -ErrorAction Ignore
        if(!$disk){
            $result = $false
        }else{
            $result = $true
        }
    }
    catch [System.Exception]
    {
        $result = $false
    }

    Write-Debug $result

    $result
}

Export-ModuleMember -Function *-TargetResource

DSCResources\cInitializeDisk\cInitializeDisk.psm1

This module then makes sure that the disk is initialized properly.

function Get-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Collections.Hashtable])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)

}

function Set-TargetResource
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)

    Initialize-Disk -VirtualDisk (Get-VirtualDisk -FriendlyName $VirtualDiskFriendlyName) -PartitionStyle GPT -Confirm:$false
}

function Test-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Boolean])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)

	$result = [System.Boolean]

    try
    {
        $virtualDisk = Get-VirtualDisk -FriendlyName $VirtualDiskFriendlyName -ErrorAction Stop
        $disk = Get-Disk -VirtualDisk $virtualDisk

        if($disk.PartitionStyle -eq 'GPT')
        {
           $result = $true
        }
        else
        {
           $result = $false
        }
    }
    catch [System.Exception]
    {
        $result = $false
    }

    $result
}

Export-ModuleMember -Function *-TargetResource

DSCResources\cCreateDiskPartition\cCreateDiskPartition.psm1

Finally, this module creates a partition, assigns a drive letter and formats it.

function Get-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Collections.Hashtable])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)
}

function Set-TargetResource
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)

    Get-VirtualDisk -FriendlyName $VirtualDiskFriendlyName `
    | Get-Disk `
    | New-Partition -UseMaximumSize `
                    -AssignDriveLetter `
    | Format-Volume
}

function Test-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Boolean])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.String]
		$VirtualDiskFriendlyName
	)

    $result = [System.Boolean]

    try
    {
        $virtualDisk = Get-VirtualDisk -FriendlyName $VirtualDiskFriendlyName -ErrorAction Stop
        $disk = Get-Disk -VirtualDisk $virtualDisk

        $partition = Get-Partition -DiskNumber $disk.Number

        if(!$partition.DriveLetter)
        {
           $result = $false
        }
        else
        {
           $result = $true
        }
    }
    catch [System.Exception]
    {
        $result = $false
    }

    $result
}

Export-ModuleMember -Function *-TargetResource

DSC Configuration

This configuration executes the DSC Modules listed above and specifies the total size of the drive that will be created and the number of columns to use. On Azure, the rule of thumb is 1 column per data disk.

configuration DiskConfiguration {
 param (
   [String]$StoragePoolFriendlyName,
   [String]$VirtualDiskFriendlyName,
   [UInt64]$DriveSize = 32TB,
   [UInt32]$NumberOfColumns = 32
 )

 Import-DscResource -Module cDiskTools

 node localhost
 {
   Settings
   {
      DebugMode = 'All,ResourceScriptBreakAll'
      ConfigurationMode = 'ApplyAndAutoCorrect'
      RefreshMode = 'Disabled'
   }

   StorageSpace CreateStorageSpace
   {
     FriendlyName = $StoragePoolFriendlyName
   }

   VirtualDisk CreateVirtualDisk
   {
     FriendlyName = $VirtualDiskFriendlyName
     StoragePoolFriendlyName = $StoragePoolFriendlyName
     DriveSize = $DriveSize
     NumberOfColumns = $NumberOfColumns
   }

   Disk InitializeDisk
   {
     VirtualDiskFriendlyName = $VirtualDiskFriendlyName
   }

   DiskPartition CreateDiskPartition
   {
     VirtualDiskFriendlyName = $VirtualDiskFriendlyName
   }
 }
}

Packaging and Deployment

Getting the PowerShell DSC in Azure Storage can be done using the Publish-AzureVMDscConfiguration cmdlet. It can also be done by creating a zip file containing the DSC Module and the DSC Configuration. Then it’s up to you to upload the zip file to Azure Blob Storage. In the end the cmdlet is quite useful when you need to iterate quickly.

Switch-AzureMode AzureServiceManagement

$ErrorActionPreference = "stop"

# set the storage account that will contain the DSC packages

Set-AzureSubscription -SubscriptionName (Get-AzureSubscription -Current).SubscriptionName -CurrentStorageAccountName 'briseboispackages'

# package and publish the DSC to the storage account

Publish-AzureVMDscConfiguration -ConfigurationPath 'C:\Program Files\WindowsPowerShell\Modules\cDiskTools\DiskConfiguration.ps1' -Force

# 

$currentStorageAccount = (Get-AzureSubscription -Current).CurrentStorageAccountName

$ModuleURL = "https://$currentStorageAccount.blob.core.windows.net/windows-powershell-dsc/DiskConfiguration.ps1.zip"

$additionalParameters = New-Object -TypeName Hashtable

$additionalParameters["modulesUrl"] = $ModuleURL

Switch-AzureMode AzureResourceManager

# deploy the monster Virtual Machine

New-AzureResourceGroup -Name "armmonstervm" `
                       -Location "westus" `
                       -TemplateFile 'WindowsVirtualMachine.json' `
                       -TemplateParameterFile 'WindowsVirtualMachine.param.dev.json' `
                       @additionalParameters `
                       -Force -Verbose

The deployment produces the following output, that allows us to determine where things might have gone wrong.

VERBOSE: 5:04:36 PM - Created resource group 'armmonstervm' in location 'westus'
VERBOSE: 5:04:36 PM - Template is valid.
VERBOSE: 5:04:37 PM - Create template deployment 'WindowsVirtualMachine'.
VERBOSE: 5:04:42 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP' provisioning status is running
VERBOSE: 5:04:42 PM - Resource Microsoft.Network/virtualNetworks 'msarmbriseboisvnet' provisioning status is running
VERBOSE: 5:04:45 PM - Resource Microsoft.Storage/storageAccounts 'msarmbrisebois' provisioning status is running
VERBOSE: 5:04:54 PM - Resource Microsoft.Network/virtualNetworks 'msarmbriseboisvnet' provisioning status is succeeded
VERBOSE: 5:04:57 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP' provisioning status is succeeded
VERBOSE: 5:04:59 PM - Resource Microsoft.Network/networkInterfaces 'VMNic' provisioning status is succeeded
VERBOSE: 5:08:20 PM - Resource Microsoft.Compute/virtualMachines 'msarmbrisebois' provisioning status is running
VERBOSE: 5:08:20 PM - Resource Microsoft.Storage/storageAccounts 'msarmbrisebois' provisioning status is succeeded
VERBOSE: 5:22:03 PM - Resource Microsoft.Compute/virtualMachines/extensions 'msarmbrisebois/dscExtension' provisioning status is running
VERBOSE: 5:22:03 PM - Resource Microsoft.Compute/virtualMachines 'msarmbrisebois' provisioning status is succeeded
VERBOSE: 5:29:14 PM - Resource Microsoft.Compute/virtualMachines/extensions 'msarmbrisebois/dscExtension' provisioning status is succeeded

In a subsequent post, I will detail the steps that helped me to test and debug the Desired State Configuration used in this template.

This post marks my first adventure with PowerShell Desired State Configuration. Over the past few weeks, there have been a few announcements around Microsoft Azure. Once of these, is an extended version of the Azure Resource Manager which brings us the ability to run PowerShell DSC on Virtual Machines. This update changes everything and I will delve into it in an upcoming post.

DSC Module to Prepare a Stripe Volume

Using the xDSCResourceDesigner enabled by installing Windows Management Framework 5.0 Preview April 2015, I was able to use the following commands to scaffold a Customer PowerShell Desired State (DSC) Module that allows me to stripe data disks on an Azure Virtual Machine.

Install the xDSCResourceDesigner Module

Install-Module -Name xDSCResourceDesigner

scaffold a Customer PowerShell Desired State (DSC) Module

New-xDscResource –Name cStripeVHDsAndPrepareVolume `
                 -Property (New-xDscResourceProperty –Name DriveSize `
                                                     –Type UInt64 `
                                                     -Attribute Key), `
                           (New-xDscResourceProperty –Name NumberOfColumns `
                                                     -Type Uint32 `
                                                     -Attribute Required) `
                 -Path 'C:\Program Files\WindowsPowerShell\Modules\' `
                 -ModuleName cDiskTools `
                 -FriendlyName cDiskTools `
                 -Verbose

This will create many files in

C:\Program Files\WindowsPowerShell\Modules\cDiskTools

The file that interests us the most because this is where the logic is implemented.

cDiskTools\DSCResources\StripeVHDsAndPrepareVolume\StripeVHDsAndPrepareVolume.psm1

Implementing the Custom DSC Module

Get-TargetResource – This function will always return a hastable. It’ll gather information about the task we wanted to know, and return them in a hashtable.

function Get-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Collections.Hashtable])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)

    Try 
    {
        if (Test-Path $DriveLetter) 
        {
            Write-Debug 'F:/ exists on target.'
            @{
                DriveLetter = 'F'
                Mounted   = $true
            }
        }
        else {
             Write-Debug "F:/ can't be found."
             @{
                DriveLetter = 'F'
                Mounted   = $false
             }
        }
    }
    Catch 
    {
        throw "An error occured getting the F:/ drive informations. Error: $($_.Exception.Message)"
    }
}

Set-TargetResource – This function will apply what we want to do. In this case, it will create a Storage Space with all the disks that can be pooled. It will create a Virtual Disk and Initialize it. Then it will create a partition and assign a Drive Letter. Finally, it will Format the Volume.

function Set-TargetResource
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)

    Write-Verbose 'Creating Storage Pool'

	New-StoragePool -FriendlyName 'LUN-0' `
                    -StorageSubSystemUniqueId (Get-StorageSubSystem -FriendlyName '*Space*').uniqueID `
                    -PhysicalDisks (Get-PhysicalDisk -CanPool $true)

    Write-Verbose 'Creating Virtual Disk'

    New-VirtualDisk -FriendlyName 'Datastore01' `
                    -StoragePoolFriendlyName 'LUN-0' `
                    -Size $DriveSize `
                    -NumberOfColumns $NumberOfColumns `
                    -ProvisioningType Thin `
                    -ResiliencySettingName Simple

    Start-Sleep -Seconds 20

    Write-Verbose 'Initializing Disk'

    Initialize-Disk -VirtualDisk (Get-VirtualDisk -FriendlyName 'Datastore01')
 
    Start-Sleep -Seconds 20

    $diskNumber = ((Get-VirtualDisk -FriendlyName 'Datastore01' | Get-Disk).Number)
 
    Write-Verbose 'Creating Partition'

    New-Partition -DiskNumber $diskNumber `
              -UseMaximumSize `
              -AssignDriveLetter `
              -DriveLetter F
    
    Start-Sleep -Seconds 20

    Write-Verbose 'Formatting Volume and Assigning Drive Letter'
    
    Format-Volume -DriveLetter F `
              -FileSystem NTFS `
              -NewFileSystemLabel 'Data' `
              -Confirm:$false `
              -Force
}

Test-TargetResource – This function will test the system so see if the set-targetresource should be applied or not. It’ll always return a boolean.

function Test-TargetResource
{
	[CmdletBinding()]
	[OutputType([System.Boolean])]
	param
	(
		[parameter(Mandatory = $true)]
		[System.UInt64]
		$DriveSize,

		[parameter(Mandatory = $true)]
		[System.UInt32]
		$NumberOfColumns
	)
    
    $result = [System.Boolean]
    Try 
    {
        if (Test-Path F) 
        {
            Write-Verbose 'F:/ exists on target.'
            $result = $true
        }
        else 
        {
            Write-Verbose "F:/ can't be found."
            $result = $false
        }
    }
    Catch 
    {
        throw "An error occured getting the F:/ drive informations. Error: $($_.Exception.Message)"
    }
    $result
}

Providing Metadata About The Module

In order to be able to import a Module, it must provide metadata about itself, through a manifest file whose extension is psd1. In our case, it can be found at the root of our Module folder hierarchy.

C:\Program Files\WindowsPowerShell\Modules\DiskTools\DiskTools.psd1

If you have multiple resource in your module, you will need a new psd1 in each resource! And a root module should exists in the module manifest.

Now, we need to verify that the module is found by PowerShell

get-module -ListAvailable | ? Name -like "cD*"

ModuleType Version Name
---------- ------- -----------
Manifest   1.0     cDiskTools

Great, things seem to be in order. It’s time to write the DSC Configuration

configuration DataDisk {
    param (
        [UInt64]$DriveSize = 32TB,
        [UInt32]$NumberOfColumns = 32             
    )

    Import-DscResource -module cDiskTools
    node localhost
    {   
        cDiskTools StripeVHDsAndPrepareVolume 
        {
            DriveSize = $DriveSize
            NumberOfColumns = $NumberOfColumns
        }
    }
}
DataDisk

Testing The Module

This step can be a bit tricky especially in a scenario where we need disks to sacrifice. In other words, I need a temporary need a Monster Virtual Machine to test this out! Fortunately I have a script to create VM on Microsoft Azure with 32 data disks.

Using the script from my previous post, I create a brand new monster. To test this new DSC Module, I logged into the VM using Remote Desktop (RDP). Then I fired up Windows PowerShell and installed the DSC Service.

Add-WindowsFeature Dsc-Service

Copied my module to

C:\Program Files\WindowsPowerShell\Modules

And verified that it was found by PowerShell

get-module -ListAvailable | ? Name -like "cD*"

ModuleType Version Name
---------- ------- -----------
Manifest   1.0     cDiskTools

Then I copied the DataDisk.ps1 configuration file to

C:\Configuration\

Executing DataDisk.ps1 produces a file (contents shown below) named localhost.mof. In this scenario, the file is used to apply the desired configuration to the localhost.

/*
@TargetNode='localhost'
@GeneratedBy=brisebois
@GenerationDate=05/08/2015 05:37:20
@GenerationHost=MSBRISEBOIS
*/

instance of cStripeVHDsAndPrepareVolume as $cStripeVHDsAndPrepareVolume1ref
{
ResourceID = "[cDiskTools]StripeVHDsAndPrepareVolume";
 NumberOfColumns = 32;
 SourceInfo = "C:\\Configuration\\DataDisk.ps1::10::9::cDiskTools";
 DriveSize = 35184372088832;
 ModuleName = "cDiskTools";
 ModuleVersion = "1.0";

};

instance of OMI_ConfigurationDocument
{
 Version="1.0.0";
 Author="brisebois";
 GenerationDate="05/08/2015 05:37:20";
 GenerationHost="MSBRISEBOIS";
};

Now it’s time to test drive our Custom DSC

Start-DscConfiguration -Path C:\Configuration\DataDisk `
                       -ComputerName localhost `
                       -Force `
                       -wait `
                       -Verbose `
                       -Debug

If everything goes well we should get output that looks like the following

VERBOSE: Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = SendConfigurationApply,'className' = MSFT_DSCLocalConfiguration
Manager,'namespaceName' = root/Microsoft/Windows/DesiredStateConfiguration'.
VERBOSE: An LCM method call arrived from computer MSBRISEBOIS with user sid S-1-5-21-2374999437-3767087714-2116828215-500.
VERBOSE: [MSBRISEBOIS]: LCM:  [ Start  Set      ]
VERBOSE: [MSBRISEBOIS]: LCM:  [ Start  Resource ]  [[cDiskTools]StripeVHDsAndPrepareVolume]
VERBOSE: [MSBRISEBOIS]: LCM:  [ Start  Test     ]  [[cDiskTools]StripeVHDsAndPrepareVolume]
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] F:/ can't be found.
VERBOSE: [MSBRISEBOIS]: LCM:  [ End    Test     ]  [[cDiskTools]StripeVHDsAndPrepareVolume] in 0.2500 seconds.
VERBOSE: [MSBRISEBOIS]: LCM:  [ Start  Set      ]  [[cDiskTools]StripeVHDsAndPrepareVolume]
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] Creating Storage Pool
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] Creating Virtual Disk
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] Initializing Disk
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] Creating Partition
VERBOSE: [MSBRISEBOIS]:                            [[cDiskTools]StripeVHDsAndPrepareVolume] Formatting Volume and Assigning Drive Letter
VERBOSE: [MSBRISEBOIS]: LCM:  [ End    Set      ]  [[cDiskTools]StripeVHDsAndPrepareVolume] in 210.0800 seconds.
VERBOSE: [MSBRISEBOIS]: LCM:  [ End    Resource ]  [[cDiskTools]StripeVHDsAndPrepareVolume]
VERBOSE: [MSBRISEBOIS]: LCM:  [ End    Set      ]    in  226.3035 seconds.
VERBOSE: Operation 'Invoke CimMethod' complete.
VERBOSE: Time taken for configuration job to complete is 210.862 seconds

In Closing

Being a developer at heart, I never really took the time to play with PowerShell DSC. As Azure evolves, I’m pulled in a bunch of new directions and this is definitely one that I will continue to explore. Azure and Automation go hand in hand. It’s important to be capable to create environments in a repeatable and predictable manner. This is exactly what the Azure Resource Manager (ARM) and PowerShell DSC brings to the table.

This is a game changer for many ongoing projects who struggle with managing environment configurations across regions. Take some time and start digging into PowerShell DSC, your Infrastructure as a Service (IaaS) projects will definitely benefit from it.

Download Channel 9 Series

It’s learning season again, and I’m gearing up for lots of Azure this year. Between Build, Ignite, the Microsoft Virtual Academy and the endless stream of content on Channel 9, it’s easy to feel overwhelmed!

Since bandwidth is a precious commodity, and that I don’t always have the luxury of having an access point, I decided to re-purpose the script I used to download the Build and Ignite sessions. This new script has only one reason to be; it exists to download Series from Channel 9 so that I can watch them on the go.

Going through the list of Series, I was amazed by the amount of new content. I found MVA sessions and a bunch of other interesting topics related to Azure.

Feel free to use the comments section at the bottom of this post to suggest feeds that I should pay attention to.

The Script

To execute this script, I used Windows Powershell ISE and I manually download the Series that interest me. Browse through the series that I identified for myself. Who knows, you may find something of interest.

function Get-Media
{
    [CmdletBinding()]
    param
    (
        [Object]
        $url,
        [Object]
        $title,
        [Object]
        $path
    )

    $u = New-Object System.Uri($url)
    $name = $title
    $extension = [System.IO.Path]::GetExtension($u.Segments[-1])
    $fileName = $name + $extension

    $fileName = $fileName -replace "’", ''
    $fileName = $fileName -replace "\?", ''
    $fileName = $fileName -replace ":", ''
    $fileName = $fileName -replace '/', ''
    $fileName = $fileName -replace ",", ''
    $fileName = $fileName -replace '"', ''

    $fileName

    if (Test-Path($fileName)) {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
    }
    else
    {
        Invoke-WebRequest $url -OutFile (Join-Path -Path $path -ChildPath $fileName)
    }
}

function Get-VideosFromFeed
{
    [CmdletBinding()]
    param
    (
        [Object]
        $feedUrl,
        [Object]
        $folder,
        [Object]
        $path
    )

    $feed=[xml](New-Object System.Net.WebClient).DownloadString($feedUrl)

    $downloadPath = (Join-Path -Path $path -ChildPath $folder)

    if (Test-Path($downloadPath)) {
        Write-Host 'Skipping folder, already exists' -ForegroundColor Yellow
    }
    else
    {
        New-Item -Path $downloadPath -ItemType directory -WarningAction SilentlyContinue
    }

    foreach($i in $feed.rss.channel.item) {
        foreach($m in $i.group){
            foreach($u in $m.content `
                    | Where-Object { `
                            $_.url -like '*mid.mp4' `
                         } | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                     @{Name='title'; Expression = {$i.title}})
            {
                Get-Media -url $u.url -title $u.title -path $downloadPath
            }
        }
    }
}

$physicalPath = "V:\Videos\Series"

# Microsoft Datacenter vNext Preview: Bringing Azure to Your Datacenter
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Microsoft-Datacenter-vNext-Preview-Bringing-Azure-to-Your-Datacenter/RSS' -path $physicalPath -folder 'Data Center Preview'

# Microsoft Azure Fundamentals: Virtual Machines
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Microsoft-Azure-Fundamentals-Virtual-Machines/RSS' -path $physicalPath -folder 'Microsoft Azure Fundamentals Virtual Machines'

# Adding Microsoft Azure Search to Your Websites and Apps
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Adding-Microsoft-Azure-Search-to-Your-Websites-and-Apps/RSS' -path $physicalPath -folder 'Working with Azure Search'

# Docker for .NET Developers
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Docker-for-NET-Developers/RSS' -path $physicalPath -folder 'Docker for NET Devs'

# Storage Spaces Deep Dive
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Storage-Spaces-Deep-Dive/RSS' -path $physicalPath -folder 'Storage Spaces Deep Dive'

# Cloud DevCamp
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Cloud-DevCamp/RSS' -path $physicalPath -folder 'Cloud DevCamp'

# Microsoft Azure Back End for Gaming
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Microsoft-Azure-Back-End-for-Gaming/RSS' -path $physicalPath -folder 'Microsoft Azure Back End for Gaming'

# Azure App Service
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Windows-Azure-Web-Sites-Tutorials/RSS' -path $physicalPath -folder 'Azure App Service'

# Developing Solutions with Azure DocumentDB
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Developing-Solutions-with-Azure-DocumentDB/RSS' -path $physicalPath -folder 'Developing Solutions with Azure DocumentDB'

# Big Data Analytics with HDInsight
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Big-Data-Analytics-with-HDInsight/RSS' -path $physicalPath -folder 'Big Data Analytics with HDInsight'

# Microsoft Hybrid Cloud Best Practices for IT Pros
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Microsoft-Hybrid-Cloud-Best-Practices-for-IT-Pros/RSS' -path $physicalPath -folder 'Microsoft Hybrid Cloud Best Practices for IT Pros'

# Java on Microsoft Azure
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Java-on-Microsoft-Azure/RSS' -path $physicalPath -folder 'Java on Microsoft Azure'

# Azure Active Directory Core Skills
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Azure-Active-Directory-Core-Skills/RSS' -path $physicalPath -folder 'Azure Active Directory Core Skills'

# Managing Linux Workloads in Windows Server and System Center
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Managing-Linux-Workloads-in-Windows-Server-and-System-Center/RSS' -path $physicalPath -folder 'Managing Linux Workloads in Windows Server and System Center'

# All About Microsoft Azure Operational Insights
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/All-About-Microsoft-Azure-Operational-Insights/RSS' -path $physicalPath -folder 'All About Microsoft Azure Operational Insights'

# You ve Got Key Values! A Redis Jump Start
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Youve-Got-Key-Values-A-Redis-Jump-Start/RSS' -path $physicalPath -folder 'You ve Got Key Values! A Redis Jump Start'

# Azure Solution Training
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Azure-Solutions-Training/RSS' -path $physicalPath -folder 'Azure Solution Training'

# Advanced PowerShell Desired State Configuration (DSC) and Custom Resources
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Advanced-PowerShell-Desired-State-Configuration-DSC-and-Custom-Resources/RSS' -path $physicalPath -folder 'Advanced PowerShell Desired State Configuration (DSC) and Custom Resources'

# Getting Started with PowerShell Desired State Configuration (DSC)
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Getting-Started-with-PowerShell-Desired-State-Configuration-DSC/RSS' -path $physicalPath -folder 'Getting Started with PowerShell Desired State Configuration (DSC)'

# Polyglot Persistence: Choosing the Right Azure Storage Mix
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Polyglot-Persistence-Choosing-the-Right-Azure-Storage-Mix/RSS' -path $physicalPath -folder 'Polyglot Persistence Choosing the Right Azure Storage Mix'

# Getting Started with Microsoft Azure Active Directory
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Getting-Started-with-Microsoft-Azure-Active-Directory/RSS' -path $physicalPath -folder 'Getting Started with Microsoft Azure Active Directory'

# MEAN Stack Jump Start
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/MEAN-Stack-Jump-Start/RSS' -path $physicalPath -folder 'MEAN Stack Jump Start'

# Managing Your Systems on Microsoft Azure with Chef
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Managing-Your-Systems-on-Microsoft-Azure-with-Chef/RSS' -path $physicalPath -folder 'Managing Your Systems on Microsoft Azure with Chef'

# Connect(On Demand);
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/ConnectOn-Demand/RSS' -path $physicalPath -folder 'Connect(On Demand)'

# Partner Profitability: Cloud Transformation Series
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Partner-Profitability-Cloud-Transformation-Series/RSS' -path $physicalPath -folder 'Partner Profitability Cloud Transformation Series'

# Web API Design
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Web-API-Design/RSS' -path $physicalPath -folder 'Web API Design'

# Developing Microsoft SQL Server Databases
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Developing-Microsoft-SQL-Server-Databases/RSS' -path $physicalPath -folder 'Developing Microsoft SQL Server Databases'

# Monitor Workloads with System Center Operations Manager
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Monitor-Workloads-with-System-Center-Operations-Manager/RSS' -path $physicalPath -folder 'Monitor Workloads with System Center Operations Manager'

# Azure HDInsight Service
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Getting-started-with-Windows-Azure-HDInsight-Service/RSS' -path $physicalPath -folder 'Azure HDInsight Service'

# Automate Workloads with System Center Orchestrator
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Automate-Workloads-with-System-Center-Orchestrator/RSS' -path $physicalPath -folder 'Automate Workloads with System Center Orchestrator'

# Open Source for DevOps Practices
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Open-Source-for-DevOps-Practices/RSS' -path $physicalPath -folder 'Open Source for DevOps Practices'

# Developing Microsoft Azure Solutions
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Developing-Microsoft-Azure-Solutions/RSS' -path $physicalPath -folder 'Developing Microsoft Azure Solutions'

# Architecting Microsoft Azure Solutions
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Architecting-Microsoft-Azure-Solutions/RSS' -path $physicalPath -folder 'Architecting Microsoft Azure Solutions'

# Getting Great Performance out of Azure
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Getting-Great-Performance-out-of-Azure/RSS' -path $physicalPath -folder 'Getting Great Performance out of Azure'

# Azure Networking Fundamentals for IT Pros
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Azure-Networking-Fundamentals-for-IT-Pros/RSS' -path $physicalPath -folder 'Azure Networking Fundamentals for IT Pros'

# PHP for Azure
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/PHP-for-Azure/RSS' -path $physicalPath -folder 'PHP for Azure'

# You ve Got Documents! A MongoDB Jump Start
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Youve-Got-Documents-A-MongoDB-Jump-Start/RSS' -path $physicalPath -folder 'You ve Got Documents - A MongoDB Jump Start'

# Designing Solutions for SQL Server
Get-VideosFromFeed -feedUrl 'http://channel9.msdn.com/Series/Designing-Solutions-for-SQL-Server/RSS' -path $physicalPath -folder 'Designing Solutions for SQL Server'

Download All Sessions in SD Quality

$feedUrl = 'http://s.ch9.ms/Events/Ignite/2015/RSS'
 
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
function Get-Media
{
    [CmdletBinding()]
    param
    (
        [Object]
        $url,
        [Object]
        $title
    )
     
    $u = New-Object System.Uri($url)
    $name = $title
    $extension = [System.IO.Path]::GetExtension($u.Segments[-1])
    $fileName = $name + $extension

    $fileName = $fileName -replace "’", ''
    $fileName = $fileName -replace "\?", ''
    $fileName = $fileName -replace ":", ''
    $fileName = $fileName -replace '/', ''
    $fileName = $fileName -replace ",", ''
    $fileName = $fileName -replace '"', ''

    $fileName
            
    if (Test-Path($fileName)) {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
    }
    else
    {
        Invoke-WebRequest $url -OutFile $fileName
    }
}
  
$feed=[xml](New-Object System.Net.WebClient).DownloadString($feedUrl)
 
foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*mid.mp4' `
                     } | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

# Find and Download Keynotes

foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*KEY0*' `
                        -and $_.type -eq 'video/mp4' `
                       
                     } `
                     | Select-Object -Unique `
                     | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

Download Sessions Based on Filters in SD Quality

$feedUrl = 'http://s.ch9.ms/Events/Ignite/2015/RSS'
  
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
function Get-Media
{
    [CmdletBinding()]
    param
    (
        [Object]
        $url,
        [Object]
        $title
    )
      
    $u = New-Object System.Uri($url)
    $name = $title
    $extension = [System.IO.Path]::GetExtension($u.Segments[-1])
    $fileName = $name + $extension
 
    $fileName = $fileName -replace "’", ''
    $fileName = $fileName -replace "\?", ''
    $fileName = $fileName -replace ":", ''
    $fileName = $fileName -replace '/', ''
    $fileName = $fileName -replace ",", ''
    $fileName = $fileName -replace '"', ''
 
    $fileName
             
    if (Test-Path($fileName)) {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
    }
    else
    {
        Invoke-WebRequest $url -OutFile $fileName
    }
}
   
$feed=[xml](New-Object System.Net.WebClient).DownloadString($feedUrl)
  
foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*mid.mp4' `

                        # Filter by keywords
                        -and ($i.description.'#cdata-section' -like '*azure*' `
                              -or $i.title -like '*azure*' `

                         ) `
                     } | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            #Get-Media -url $u.url -title $u.title
            Write-Output $u.title
        }
    }
}
 
# Find and Download Keynotes
 
foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*KEY0*' `
                        -and $_.type -eq 'video/mp4' `                        
                     } `
                     | Select-Object -Unique `
                     | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Write-Output $u.title
            #Get-Media -url $u.url -title $u.title
        }             
    }
}

Download All Sessions in SD Quality

$feedUrl = 'http://s.ch9.ms/Events/Build/2015/RSS'
 
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
function Get-Media
{
    [CmdletBinding()]
    param
    (
        [Object]
        $url,
        [Object]
        $title
    )
     
    $u = New-Object System.Uri($url)
    $name = $title
    $extension = [System.IO.Path]::GetExtension($u.Segments[-1])
    $fileName = $name + $extension

    $fileName = $fileName -replace "’", ''
    $fileName = $fileName -replace "\?", ''
    $fileName = $fileName -replace ":", ''
    $fileName = $fileName -replace '/', ''
    $fileName = $fileName -replace ",", ''
    $fileName = $fileName -replace '"', ''

    $fileName
            
    if (Test-Path($fileName)) {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
    }
    else
    {
        Invoke-WebRequest $url -OutFile $fileName
    }
}
  
$feed=[xml](New-Object System.Net.WebClient).DownloadString($feedUrl)
 
foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*mid.mp4' `
                     } | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

# Find and Download Keynotes

foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*KEY0*' `
                        -and $_.type -eq 'video/mp4' `                       
                     } `
                     | Select-Object -Unique `
                     | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

Download Sessions Based on Filters in SD Quality

$feedUrl = 'http://s.ch9.ms/Events/Build/2015/RSS'
 
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
function Get-Media
{
    [CmdletBinding()]
    param
    (
        [Object]
        $url,
        [Object]
        $title
    )
     
    $u = New-Object System.Uri($url)
    $name = $title
    $extension = [System.IO.Path]::GetExtension($u.Segments[-1])
    $fileName = $name + $extension

    $fileName = $fileName -replace "’", ''
    $fileName = $fileName -replace "\?", ''
    $fileName = $fileName -replace ":", ''
    $fileName = $fileName -replace '/', ''
    $fileName = $fileName -replace ",", ''
    $fileName = $fileName -replace '"', ''

 
    $fileName
            
    if (Test-Path($fileName)) {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
    }
    else
    {
        Invoke-WebRequest $url -OutFile $fileName
    }
}
  
$feed=[xml](New-Object System.Net.WebClient).DownloadString($feedUrl)
 
foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*mid.mp4' `
                        -and ($i.description.'#cdata-section' -like '*azure*' `
                              -or $i.title -like '*azure*' `
                         ) `
                     } | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

# Find and Download Keynotes

foreach($i in $feed.rss.channel.item) {
    foreach($m in $i.group){
        foreach($u in $m.content `
                | Where-Object { `
                        $_.url -like '*KEY0*' `
                        -and $_.type -eq 'video/mp4' `
                     } `
                     | Select-Object -Unique `
                     | Select-Object -Property @{Name='url'; Expression = {$_.url}}, `
                                                 @{Name='title'; Expression = {$i.title}})
        {
            Get-Media -url $u.url -title $u.title
        }             
    }
}

In my last blog post I shared about how I use a Microsoft Azure Virtual Machine as my development machine. It’s been a month since I made this drastic change, and I must admit that it’s been pretty cool! The Azure Virtual Machine has provided me with more agility in my hectic schedule. I’ve been able to start my day from one machine, move to a different machine and continue from where I left off. I no longer deal with putting my machine to sleep or trying to remember where I was when I left off. When I log off from one machine, I log back in right where I left off.

It’s Getting Tight

As I move from project to project, the resource requirements vary, and sometimes I need more resources. Fortunately, we can scale our Virtual Machines up and down through PowerShell and the Azure Portal. I usually work with a Standard D2 Virtual Machine, that translates to 2 Cores 7 GB of RAM and a 100GB SSD Temp Disk. On some occasions, I scale-up to a Standard D4, that translates to 8 Cores 28GB of RAM and a 400GB SSD Temp Disk.

$serviceName = 'mydevmachine'
$name = 'mydevmachine'

# Stop the Azure VM
Stop-AzureVM -Name $name -ServiceName $serviceName -Force

# Get a reference to the Azure VM
$vm = Get-AzureVM -Name $name -ServiceName $serviceName

# Wait for the VM to reach the StoppedDeallocated State
Write-Output $('VM state is '+ $vm.InstanceStatus)

while( $vm.InstanceStatus -ne 'StoppedDeallocated')
{
    Start-Sleep -s 10
    $vm = Get-AzureVM -Name $name -ServiceName $serviceName
    Write-Output $('VM state is '+ $vm.InstanceStatus)
}

# Set the Instance Size & Update the WM
Set-AzureVMSize -InstanceSize Standard_D4 -VM $vm | Update-AzureVM

# Start the Azure VM
Start-AzureVM -Name $name -ServiceName $serviceName

# Wait for the VM to reach the ReadyRole State
$vm = Get-AzureVM -Name $name -ServiceName $serviceName
Write-Output $('VM state is '+ $vm.InstanceStatus)

while( $vm.InstanceStatus -ne 'ReadyRole')
{
    Start-Sleep -s 10
    $vm = Get-AzureVM -Name $name -ServiceName $serviceName
    Write-Output $('VM state is '+ $vm.InstanceStatus)
}

# VM Boot Grace Period
Start-Sleep -s 60

# Download and launch a Remote Desktop Session to the resized Azure VM
Get-AzureRemoteDesktopFile -Name $name -ServiceName $serviceName -Launch

UH OH!

The instance size we are looking for isn’t available? Don’t worry about it just yet. Not all Azure server racks are created equal. This means that if the Virtual Machine was initially created as a Standard A1, the configurations we are looking for might not be available. Consequentially, scaling to a newer Virtual Machine configuration requires us to take a different approach. First off, let’s find out if the Virtual Machine Instance Size is available in our Data Center.

Get-AzureLocation `
 | Where-Object {$_.VirtualMachineRoleSizes -contains 'A9'} `
 | Select-Object -Property Name

Name
----
South Central US
West US
East US
North Europe
West Europe
Japan East

If for some reason, the Data Center doesn’t have the desired Virtual Machine Instance Size available. We can review the list of Instance Sizes for a specific Data Center. Then we can decide whether we want to migrate to another Data Center or chose to use a different Virtual Machine Instance Size.

Get-AzureLocation `
| Where-Object -Property Name -EQ 'East US 2' `
| Select-Object -ExpandProperty VirtualMachineRoleSizes

A5
A6
A7
Basic_A0
Basic_A1
Basic_A2
Basic_A3
Basic_A4
ExtraLarge
ExtraSmall
Large
Medium
Small
Standard_D1
Standard_D11
Standard_D12
Standard_D13
Standard_D14
Standard_D2
Standard_D3
Standard_D4
Standard_DS1
Standard_DS11
Standard_DS12
Standard S_DS13
Standard_DS14
Standard_DS2
Standard_DS3
Standard_DS4
Standard_G1
Standard_G2
Standard_G3
Standard_G4
Standard_G5

As of April 30th 2015, the East US 2 Data Center has the D, DS and G Series but it does not have A8, A9, A10 or A11 Virtual Machine configurations. Scaling to A9 Virtual Machine, that translates to 16 Cores 112 GB of RAM, 382GB Temp Disk, 40Gbit/s InfiniBand network and remote direct memory access (RDMA) would require me to move my Virtual Machine to the East US Data Center.

It’s Time to Move

The first step towards moving, is to capture an Image of the Virtual Machine that we want to move. Before we start, be sure that the Virtual Machine is in a StoppedDeallocated state.

$servicename = "mydevmachine"
$vmname = "mydevmachine"

Get-AzureVM -ServiceName $servicename `
            -Name $vmname `
| Stop-AzureVM -Force

Then capture the Virtual Machine Image and give it a meaningful name

Save-AzureVMImage –ServiceName $servicename `
                  –Name $vmname `
                  –ImageName "BriseboisDevMachine" `
                  –OSState "Specialized" `
                  -Verbose

Create a Storage Account Context

# Source Storage Account Context
$sourceStorageAccountName = "briseboisms1"
$sourceKey = (Get-AzureStorageKey -StorageAccountName $sourceStorageAccountName).Primary
$sourceContext = New-AzureStorageContext –StorageAccountName $sourceStorageAccountName `
                                         -StorageAccountKey $sourceKey

Then find the VHD created by the Save-AzureVMImage cmdlet.

Get-AzureStorageBlob -Container 'vhds' -Context $sourceContext

Container Uri: https://briseboisms.blob.core.windows.net/vhds

Name                                   BlobType  Length        ContentType              LastModified                 SnapshotTime
----                                   --------  ------        -----------              ------------                 ------------
BriseboisDevMachine-os-2015-04-30.vhd  PageBlob  136367309312  application/octet-stream 4/30/2015 9:58:31 PM +00:00

Create a Storage Account in the Data Center that has the desired Virtual Machine Instance Size.

$destinationStorageAccountName = 'briseboisuseast1'

New-AzureStorageAccount -StorageAccountName $destinationStorageAccountName `
                        -Location 'East US' `
                        -Type 'Standard_GRS'

Copy the VHD created by the Save-AzureVMImage cmdlet to the newly Storage Account.

$blobName = 'BriseboisDevMachine-os-2015-04-30.vhd'
$container = 'vhds'

# Destination Storage Account Contex
$destinationKey = (Get-AzureStorageKey -StorageAccountName $destinationStorageAccountName).Primary
$destinationContext = New-AzureStorageContext –StorageAccountName $destinationStorageAccountName `
                                              -StorageAccountKey $destinationKey  

# Create the destination container
New-AzureStorageContainer -Name $container `
                          -Context $destinationContext 

# Copy the blob
$blobCopy = Start-AzureStorageBlobCopy -DestContainer $container `
                                       -DestContext $destinationContext `
                                       -SrcBlob $blobName `
                                       -Context $sourceContext `
                                       -SrcContainer $container

# Blob Copy Progress
while(($blobCopy | Get-AzureStorageBlobCopyState).Status -eq "Pending")
{
    Start-Sleep -s 30
    $blobCopy | Get-AzureStorageBlobCopyState
}

Now it’s time to re-hydrate the Virtual Machine with the desired Instance Size

# Add the Captured Virtual Machine as a VM Image
Add-AzureVMImage -ImageName BriseboisDevMachineEast `
                 -OS Windows `
                 -MediaLocation "https://$destinationStorageAccountName.blob.core.windows.net/vhds/$blobName"

# Verify that the VM Image has been added as expected
Get-AzureVMImage | Where-Object {$_.Location -eq 'East US'} `
                 | Where-Object {$_.ImageName -eq 'BriseboisDevMachineEast'} `
                 | Select-Object { $_.ImageName , $_.Location }

# Switch the CurrentStorageAccount to the destination Storage Account
Set-AzureSubscription -SubscriptionName 'Visual Studio Ultimate with MSDN' `
                      -CurrentStorageAccountName $destinationStorageAccountName

# Create the Virtual Machine using the newly created VM Image
New-AzureQuickVM -Windows `
-ServiceName BriseboisDev `
-Name BriseboisDev `
-ImageName BriseboisDevMachineEast `
-InstanceSize 'A9' `
-Password 'MY-PASSWORD' `
-Location 'East US' `
-AdminUsername 'MY-USERNAME' `
-Verbose

Once the Virtual Machine is running, we can RDP into it and resume our operations.

# Download and Launch RDP file for the new Azure VM
Get-AzureRemoteDesktopFile -ServiceName BriseboisDevMachineEast `
                           -Name BriseboisDevMachineEast `
                           -Launch

Note

Creating Larger Virtual Machines may take more time than what you would expect for smaller Virtual Machine Instance Sizes. This is because Azure has to find a location with enough resources to provision the instance. If you shut down a large Virtual Machine, you may encounter a situation where the Azure fabric is unable to start the Virtual Machine. One way to solve this, is to re-size the Virtual Machine by using a technique similar with the one found in the first section of this post. This will result in Azure to create an instance in a rack that has enough resources to meet the Virtual Machine Instance Size requirements.

Resources

• VM Image PowerShell ‘How To’ Blog Post
• How to Capture a Windows Virtual Machine to Use as a Template
• Migrate Azure Virtual Machines between Storage Accounts
• Recover Azure VM by attaching OS disk to another Azure VM
• Failed to Start Virtual Machine
• Allocation Failure and Remediation

Microsoft Azure is all about the opportunity to push back on known boundaries. In the last couple of months, I dealt with some scenarios where on-premises Data Centers ran out of capacity. Consequently, we were not able to push our tests as far as we would have liked. Taking the work loads to Microsoft Azure gave us the opportunity to stretch workloads to their limits.

In one of these scenarios, the workload was limited to 16 aging physical machines, and the time required to process the data was not acceptable. We needed to find ways to reduce the compute time, and had means to accomplish this on-premises. Pushing this workload to its limits, we deployed it on Microsoft Azure and provisioned it with well over 256 cores. We ended up processing workloads so fast that we now had the opportunity to run them multiple times a day. This newly discovered agility gave us the ability to refine the workload processes without disturbing ongoing business activities.

The first scenario was all about the lack of compute resources necessary to push a workload to its full potential. In a second scenario, we needed an impressive amount of resources within a single Virtual Machine. This blog post is all about how we created this Monster VM.

Building a Monster VM

The Requirement

16 Cores
112 GB of RAM
800 GB of local SSD for temp disk
32 TB for the data disk
50,000 IOPS for the data disk
512 MB per second for the data disk

Let’s think about that spec for a second. That’s monstrous! And we’re going to build it!

The BUILD

The first thing we need to do is set up our environment.

# Login to Azure
Add-AzureAccount

# Find a subscription to work with
Get-AzureSubscription

# set Azure Subscription
Select-AzureSubscription -SubscriptionName 'MSDN' `
                         -Default

Select-AzureSubscription -SubscriptionName 'MSDN' `
                         -Current

# Create a new Storage Premium Account
New-AzureStorageAccount -StorageAccountName 'monstervmstorage' `
                        -Location 'East US 2' `
                        -Type 'Premium_LRS'

# set Azure Subscription Storage Account
Set-AzureSubscription -SubscriptionName 'MSDN' `
                      -CurrentStorageAccountName 'monstervmstorage'

Ok, now it’s time to create our Virtual Machine. It’s composed of 32 1TB P3 Premium Storage Data Disks.

# Find a windows server VM Image Name
$imageName = (Get-AzureVMImage `
    | Where-Object -Property Label `
    -Match 'Windows Server 2012 R2 Datacenter')[0].ImageName

# Create a Standard DS14 Instance using as OS Disk
# stored in Azure Premium Storage
$storageAccount = 'monstervmstorage'
$adminName = 'MrAzure'
$adminPassword = 'SecurePassword!'
$vmName ='monstervmms'
$location = 'East US 2'
$vmSize ='Standard_DS14'

$vmConfig = New-AzureVMConfig -Name $vmName `
                              -ImageName $imageName `
                              -InstanceSize $vmSize

$vmConfig | Add-AzureProvisioningConfig -Windows `
                                        -AdminUsername $adminName `
                                        -Password $adminPassword

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 0' `
                              -LUN 0 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 1' `
                              -LUN 1 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 2' `
                              -LUN 2 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 3' `
                              -LUN 3 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 4' `
                              -LUN 4 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 5' `
                              -LUN 5 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 6' `
                              -LUN 6 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 7' `
                              -LUN 7 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 8' `
                              -LUN 8 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 9' `
                              -LUN 9 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 10' `
                              -LUN 10 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 11' `
                              -LUN 11 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 12' `
                              -LUN 12 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 13' `
                              -LUN 13 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 14' `
                              -LUN 14 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 15' `
                              -LUN 15 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 16' `
                              -LUN 16 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 17' `
                              -LUN 17 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 18' `
                              -LUN 18 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 19' `
                              -LUN 19 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 20' `
                              -LUN 20 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 21' `
                              -LUN 21 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 22' `
                              -LUN 22 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 23' `
                              -LUN 23 `
                              -HostCaching ReadOnly
$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 24' `
                              -LUN 24 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 25' `
                              -LUN 25 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 26' `
                              -LUN 26 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 27' `
                              -LUN 27 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 28' `
                              -LUN 28 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 29' `
                              -LUN 29 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 30' `
                              -LUN 30 `
                              -HostCaching ReadOnly

$vmConfig | Add-AzureDataDisk -CreateNew `
                              -DiskSizeInGB 1023 `
                              -DiskLabel 'Disk 31' `
                              -LUN 31 `
                              -HostCaching ReadOnly

New-AzureVM -ServiceName $vmName `
            -Location $location `
            -VMs $vmConfig -Verbose

Wait for it!

# Wait for the VM to reach the ReadyRole State
$vm = Get-AzureVM -Name $vmName -ServiceName $vmName
Write-Output $('VM state is '+ $vm.InstanceStatus)

while( $vm.InstanceStatus -ne 'ReadyRole')
{
    Start-Sleep -s 10
    $vm = Get-AzureVM -Name $vmName -ServiceName $vmName
    Write-Output $('VM state is '+ $vm.InstanceStatus)
}

The next steps need to occur on the Virtual Machine itself. Remote Desktop into the Virtual Machine.

Get-AzureRemoteDesktopFile -Name $vmName -ServiceName $vmName -Launch

Start Windows PowerShell ISE as an Administrator

Create a Storage Space to regroup the 32 1 TB data disks that we created along with the Virtual Machine.

New-StoragePool -FriendlyName 'LUN-32TB' `
-StorageSubSystemUniqueId (Get-StorageSubSystem -FriendlyName '*Space*').uniqueID `
-PhysicalDisks (Get-PhysicalDisk -CanPool $true)

Create a Virtual Disk using the Storage Space. Be sure to set the number of columns equal to the number of disks. This will make sure that the Virtual Machine is able to benefit from the combined stripe IO.

New-VirtualDisk -FriendlyName 'Datastore01' `
-StoragePoolFriendlyName 'LUN-32TB' -Size 32TB -NumberOfColumns 32 `
-ProvisioningType Thin -ResiliencySettingName Simple

Now it’s time to initialize the disk, to create a partition and to mount the disk.

Initialize-Disk -VirtualDisk (Get-VirtualDisk -FriendlyName 'Datastore01')

$diskNumber = ((Get-VirtualDisk -FriendlyName 'Datastore01' | Get-Disk).Number)

New-Partition -DiskNumber $diskNumber `
              -UseMaximumSize `
              -AssignDriveLetter

Finally, we need to format the disk

Format-Volume -DriveLetter F `
              -FileSystem NTFS `
              -NewFileSystemLabel 'Data' `
              -Confirm:$false `
              -Force

That’s it! Now we can go crazy and push this Monstrous Virtual Machine to its limits.

Resources

• Premium Storage: High-Performance Storage for Azure Virtual Machine Workloads
• Azure Storage Pricing

Back in 2013 I wrote about creating a Dev & Test environments in minutes. This post was all about the benefits and the flexibility of creating environments that support our development efforts.

Since then, I noticed that Microsoft regularly creates Virtual Machine templates for MSDN Subscribers that allow them to test software through their MSDN Benefits. Luckily, they have provided us with templates that are pre-configured with various flavors of Visual Studio.

As a developer, I am called to work on various projects, and I have many versions of Visual Studio installed. Over the past few months, I started working with multiple physical machines that I must maintain. By now, you can imagine that every tool that I decide to use, adds more maintenance. Updates, patches and betas take up a lot of my time away from actual work. To help me focus on producing value for the business, I use a good number of third party tools, with a limited number of licenses. This creates a challenge for me, because I have to pick and chose where I install these tools.

Recently, I decided to give myself a break. Using my MSDN Benefits, I provisioned a Virtual Machine with Visual Studio 2015 preview. Then I started to build out my brand-new developer machine. As a developer who’s been around azure for quite a while, I’m surprised that it took me this long to make this transition. I can now work from any physical computer. I can install all my tools, and I am no longer limited to 8GB of RAM.

I feel like a kid in a candy shop! I can scale my Virtual Machine to meet my needs and pay only for what I use. A few minutes before my work session, I use the following PowerShell script to start my Virtual Machine. Once, it’s ready, the Remote Desktop file is downloaded and launched.

$vm = Get-AzureVm -ServiceName 'my-dev-vm-name'

if($vm.InstanceStatus -ne 'ReadyRole')
{
    Start-AzureVM -ServiceName $vm.ServiceName -Name $vm.Name -Verbose
    
    # Wait for the VM to reach the ReadyRole State
    $vm = Get-AzureVM -Name $vm.Name -ServiceName $vm.ServiceName
    Write-Output $('VM state is '+ $vm.InstanceStatus)
 
    while( $vm.InstanceStatus -ne 'ReadyRole')
    {
        Start-Sleep -s 10
        $vm = Get-AzureVM -Name $vm.Name -ServiceName $vm.ServiceName

        Write-Output $('VM state is '+ $vm.InstanceStatus)
    }

    # Boot grace period of 60 seconds
    Start-Sleep -s 60
        
    Get-AzureRemoteDesktopFile -ServiceName $vm.ServiceName -Name $vm.Name -Launch

}else
{
    Write-Output 'Already ReadyRole'
    Get-AzureRemoteDesktopFile -ServiceName $vm.ServiceName -Name $vm.Name -Launch
}

Taking this a few steps further, you could create an app that would allow you to start the Virtual Machine from the bus using your Smart Watch.

Then when the time comes to call it quits, I use the following PowerShell script to shut down and deallocate my Virtual Machine. This is when I stop paying for compute.

$vm = Get-AzureVm -ServiceName 'my-dev-vm-name'

if($vm.InstanceStatus -eq 'StoppedDeallocated')
{
   Write-Output 'Already StoppedDeallocated'

}else
{
    Stop-AzureVM -ServiceName $vm.ServiceName -Name $vm.Name -Force -Verbose
}

This feature could also be added to the mobile app that I alluded to above, so that you could shutdown the Virtual Machine from your commute home.

Being in a hurry, I sometimes forget to shutdown the Virtual Machine, so I use Azure Automation to Stop Azure Virtual Machines on a Schedule. Usually, I scheduled the runbook to run at 19h00. This gives me a short grace period after 17h30 and helps me go offline for the night.

The Challenge

As developers, we deal with lots of complexity, and this is a good thing. It forces us to be creative, and sometimes to go beyond our known universe to overcome challenges.

Microsoft Azure is designed to help us make the right choices. It imposes performance targets through a multitude of mechanisms like throttling and quotas. One of which, I’m sure you have come to know, is that we cannot scale a Cloud Service to zero instances. Let’s stop for a moment and think about this limitation for a second. How would you creatively overcome this challenge?

Being Creative

Unlike Virtual Machines, Cloud Services incur costs when they are stopped. In order to stop paying, we must delete the deployment. These are the constraints that we must compose with.

Deleting a deployment comes with added complexity. It introduces significant latency, because it takes a few minutes for a Cloud Service to reach a ready state. Therefore, scenarios that are time sensitive are not satisfied by these workflows.

In a post about scaling Cloud Services, I shared a PowerShell runbook that can deploy packages from Azure Storage based on a schedule. It was originally built to deploy a scaled-up Cloud Service package to satisfy peak loads and to deploy a scaled-down Cloud Service package for the quite hours. This same script can be used as a starting point for the following scenarios.

Plan for a grace period between job executions, so that deployments have time to complete. As Cloud Services transition, they rarely provide any value, so try to come up with a good scheduling strategy. This will help you maximize your investment.

Scenario: Deploying a Package based on Queue Length

This scenario is depicted in the diagram above. Using Azure Automation, and a scheduled job, we can monitor the depth of a queue. When the queue reaches the targeted depth, the runbook deploys the Cloud Service.

The deployed Cloud Service is responsible for processing messages from the monitored queue.

Every time the job executes, it monitors the depth of a queue. If the queue is below the targeted depth, the runbook deletes the deployment.

Step By Step

1. Trigger Azure Automation deployment job.
2. Check whether queue depth is greater than target.
3. Check whether there is an active deployment.
4. Download Cloud Service Configurations and deploy the Cloud Service package from Azure Storage.
5. The deployed Cloud Service is functional and operating as expected.
6. Trigger Azure Automation deployment job.
7. Check whether queue depth is below the target.
8. Check whether there is an active deployment.
9. Delete the active deployment.

Scenario: Deploying a Package based on a Schedule

In this scenario, we use Azure Scheduler to deploy a Cloud Service to a Production slot. This can be quite useful for end of day processing. Scaling down to zero occurs based on a predefined schedule. This triggers an Azure Automation job that checks whether a Cloud Service is currently deployed. If it discovers an active deployment, it deletes it.

Step By Step

1. Trigger Azure Automation deployment job.
2. Check whether there is an active deployment.
3. Download Cloud Service Configurations and deploy the Cloud Service package from Azure Storage.
4. The deployed Cloud Service is functional and operating as expected.
5. Trigger Azure Automation delete deployment job.
6. Check whether there is an active deployment.
7. Delete the active deployment.

Deploying a Cloud Service

Taken from my previous post about scaling Cloud Services, this script can be taken as a starting point for your deployment efforts. In practice, this runbook deploys a Cloud Service package and its configuration from Azure Storage to a production slot.

Please refer to my previous post for detailed instructions and for a complete example.

workflow Invoke-DeployPackage
{
   param
   (
      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name given to the Windows PowerShell Credentials located in the Assets of this Azure Automation instance')]
      [String]
      $AzureCredentialsName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Microsoft Azure Subscription that contains the resources your wish to deploy')]
      [String]
      $AzureSubscriptionName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the service you wish to deploy')]
      [String]
      $ServiceName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The location of the service your with to deploy I.E. "East US"')]
      [String]
      $ServiceLocation,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Microsoft Azure storage account that contains the Cloud Service Packages')]
      [String]
      $StorageAccountName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Blob container that contains the Cloud Service Packages')]
      [String]
      $StorageContainerName,

      [parameter(Mandatory=$true, HelpMessage = 'The name of the Cloud Service Package file')]
      [String]
      $PackageBlobName,

      [parameter(Mandatory=$true, HelpMessage = 'The name of the Cloud Service Configurations file')]
      [String]
      $ConfigurationBlobName
    )

    $VerbosePreference = 'Continue'

    # Mark the start time of the script execution
    $StartTime = Get-Date

    $DeploymentLable = $ServiceName + ' (' + $StartTime +')'

    Write-Verbose ('Connecting to Microsoft Azure')

    $Credentials = Get-AutomationPSCredential `
                       -Name $AzureCredentialsName

    Add-AzureAccount `
       -Credential $Credentials

    InlineScript{

        Write-Verbose ('Selecting Azure Subscription')

        Select-AzureSubscription -SubscriptionName $Using:AzureSubscriptionName

        $StorageAccount = (Get-AzureStorageAccount -StorageAccountName $Using:StorageAccountName).Label

        Write-Verbose ('Setting the Azure Subscription and Storage Accounts')

        Set-AzureSubscription `
            -SubscriptionName $Using:AzureSubscriptionName `
            -CurrentStorageAccount $StorageAccount

        Write-Verbose ('[Start] Validating Azure cloud service environment {0}' -f $Using:ServiceName)

        try
        {
            $CloudService = Get-AzureService `
                                -ServiceName $Using:ServiceName

            Write-Verbose ('cloud service {0} in location {1} exist!' -f $Using:ServiceName, $Using:ServiceLocation)
        }
        catch
        {
            #Create
            Write-Verbose ('[Start] creating cloud service {0} in location {1}' -f $Using:ServiceName, $Using:ServiceLocation)

            New-AzureService `
                -ServiceName $Using:ServiceName `
                -Location $Using:ServiceLocation

            Write-Verbose ('[Finish] creating cloud service {0} in location {1}' -f $Using:ServiceName, $Using:ServiceLocation)
        }

        Write-Verbose ('[Finish] Validating Azure cloud service environment {0}' -f $Using:ServiceName)

        $TempFileLocation = "C:\$Using:ConfigurationBlobName"

        Write-Verbose ('Downloading Service Configurations from Azure Storage')

        Get-AzureStorageBlobContent `
            -Container $Using:StorageContainerName `
            -Blob  $Using:ConfigurationBlobName `
            -Destination $TempFileLocation `
            -Force

        Write-Verbose('Downloaded Configuration File: '+ $TempFileLocation)

        Write-Verbose('Getting Package Url from Azure Storage: '+ $Using:PackageBlobName)

        $blob = $(Get-AzureStorageBlob -Blob $Using:PackageBlobName -Container $Using:StorageContainerName)

        $PackageUri = $blob.ICloudBlob.Uri.AbsoluteUri

        Write-Verbose('Package Url: '+ $PackageUri)

        try
        {
            Write-Verbose('Attempting to Update an Existing Deployment')
            Set-AzureDeployment `
                -Package $PackageUri `
                -Configuration $TempFileLocation `
                -Slot Production `
                -Mode Simultaneous `
                -Label $Using:DeploymentLable `
                -ServiceName  $Using:ServiceName `
                -Upgrade `
                -Force `
                -Verbose

        }catch
        {
            Write-Output $error

            Write-Verbose('Attempting to Deploy the service')

            New-AzureDeployment `
                -Package $PackageUri `
                -Configuration $TempFileLocation `
                -Slot Production `
                -Label $Using:DeploymentLable `
                -ServiceName  $Using:ServiceName `
                -Verbose
        }
    }
}

Scaling an FTP Ingestion Service

Making an FTP Ingestion Service Highly Available (HA) can be a challenge. On Azure, we can take advantage of the Microsoft Azure Traffic Manager to direct users to the closest FTP Server. In this specific scenario, we assume that all FTP Servers are configured the same way and that users only have write access. When a document is uploaded over FTP, it is moved to a Microsoft Azure Storage Account that is used as persistent storage.

Note: Ingress Bandwidth is free on Azure. Sending a document from one Data Center to an other is considered Egress Bandwidth from the origin. So be sure that you take this into consideration when you estimate operational costs for this kind of scenario.

The Communication Flow

1. Using the CName FTP Client contacts Microsoft Azure Traffic Manager and is provided with the IP of a healthy Endpoint
2. The FTP Client connects directly with the healthy Endpoint
   1. The FTP Client uploads a document
3. The application moves the document to Microsoft Azure Storage

The Setup

To implement this scenario, we must start by creating 4 Virtual Machines. Then we configure each FTP Server to use Passive Mode. On Azure, we need to activate the Stateful FTP Filtering on the firewall of each Virtual Machine so that traffic is allowed to reach the FTP Server.

The following is a command line that can be used to enable Stateful FTP Filtering.

netsh advfirewall set global StatefulFtp enable

Configure FTP Server users on each Virtual Machine with write-only permissions. This ensures that communication is one-way and that transferred documents remain inaccessible from the outside. Furthermore, because the users are the same on all FTP Servers, clients can be load-balanced without any side effects.

Create the Microsoft Azure Storage Account that is used to collect the documents. Be sure to create the Storage Account in the Data Center that hosts the application that processes the transferred documents. Before we exit our Azure Management Environment (web portal or PowerShell), let’s make sure that we have created Input Endpoints for ports 80 and 21. These ports will be used by IIS and by the FTP Server.

Now that we have a Storage Account we can deploy, and configure auto-start for the application that moves documents to a Microsoft Azure Storage.

Configure Internet Information Services (IIS) on each Virtual Machine. This service hosts a static HTML page that is used by the Microsoft Azure Traffic Manager health probes. This enables the Traffic Manager to identify degraded nodes and to redirect traffic to healthy nodes.

Create a Microsoft Azure Traffic Manager and Endpoints from each machine. Provide FTP Clients with the Traffic Manager URL.

More on Azure Traffic Manager

Azure Traffic Manager gives us with a lot of flexibility. By providing us with 10 levels of Traffic Manager Profiles, we can compose complex routing strategies. We can add and remove Endpoints from Traffic Manager and we can use weighted distribution of network traffic for scenarios like Testing in Production (A/B Testing).

The in the following scenario, we the first Profile is used to find the closest deployment to the connected client. Then using a Round Robin Profile we distribute FTP Clients evenly over the Virtual Machines available to the selected region. Earlier I spoke of flexibility, well in this scenario, I have the possibility of adding more FTP Server instances in some regions in order to scale to demand. Adding and removing FTP Server instances can be achieved without downtime. Find out more by visiting the Traffic Manager Overview.

Moving Files to Azure Storage

The missing piece in this scenario is a program that moves documents from each Virtual Machine to Microsoft Azure Storage. The following sample uses the Dataflow (Task Parallel Library) to control the maximum degree of parallelism (the number of documents that can be uploaded in parallel).

The console application starts by creating a Blob Container. Then it queues move tasks for each document present in the monitored directory. The ActionBlock takes care of processing each task to completion. Once all documents are moved, the application waits for a few seconds and starts over.

internal class Program
{
  private static void Main(string[] args)
  {
    var mdop = ConfigurationManager.AppSettings["maxDegreeOfParallelism"];
    var maxDegreeOfParallelism = Convert.ToInt32(mdop);

    var cs = ConfigurationManager.AppSettings["connectionString"];
    var account = CloudStorageAccount.Parse(cs);
    var containerName = ConfigurationManager.AppSettings["container"];

    var c = account.CreateCloudBlobClient()
                    .GetContainerReference(containerName);
    c.CreateIfNotExists();

    while (true)
    {
      var block = new ActionBlock<FileInfo>(info =>
          TryMoveFile(account, containerName, info),
          new ExecutionDataflowBlockOptions
          {
              MaxDegreeOfParallelism = maxDegreeOfParallelism
          });

      var path = ConfigurationManager.AppSettings["directoryPath"];
      var directory = new DirectoryInfo(path);

      var fileInfos = directory.GetFiles();

      foreach (var info in fileInfos)
          block.Post(info);

      block.Complete();

      block.Completion.Wait();

      Task.Delay(TimeSpan.FromSeconds(5)).Wait();
    }
  }

  private static void TryMoveFile(CloudStorageAccount account,
                                  string containerName,
                                  FileInfo info)
  {
    try
    {
        MoveFile(account, containerName, info);
    }
    catch (Exception e)
    {
        Console.WriteLine(e.ToString());
    }
  }

  private static void MoveFile(CloudStorageAccount account,
                                  string containerName,
                                  FileInfo info)
  {
    var client = account.CreateCloudBlobClient();
    var container = client.GetContainerReference(containerName);

    var fileName = Path.GetFileName(info.FullName);
    var blobRerefence = container.GetBlockBlobReference(fileName);

    using (var fileStream = info.Open(FileMode.Open))
    {
        Console.WriteLine("Moving " + info.FullName);
        blobRerefence.UploadFromStream(fileStream);
    }
    info.Delete();
    Console.WriteLine("Moved " + info.FullName);
  }
}

The application’s configurations are listed below. Let’s take a moment to review them.

• The directory path is the folder where documents are placed once they are received by the FTP Server.
• The maximum degree of parallelism is the number of concurrent uploads. Be sure to keep this number reasonable. If the number is too high, it will have a negative impact on the overall solution.
• The connection string is the Microsoft Azure Storage connection string.
• The container is the place where document blobs are stored.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
   <startup>
      <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
   </startup>
   <appSettings>
      <add key="directoryPath" value="C:\ftp-files"/>
      <add key="maxDegreeOfParallelism" value="4"/>
      <add key="connectionString" value="[connection string]"/>
      <add key="container" value="ftp"/>
   </appSettings>
</configuration>

In order to start this application, we are presented with a few options. First we can configure it to execute when the Virtual Machine starts up. Secondly we can convert this to a Windows Service and install it on the Virtual Machine. And finally, we can start the application through Azure Automation and PowerShell. Which of these options should be implemented? I leave this choice to you.

Summary

The solution described in this post corresponds to the ingestion of documents. It does not cover what you do with the documents once they are persisted in Microsoft Azure Storage Account. I leave that to your creativity and for future adventures. The scenario described above, uses Microsoft Azure Traffic Manager to scale out and FTP Ingestion Service. The FTP Servers are configured the same way across all instances, and users only have write access. When a document is uploaded over FTP, it is moved to a Microsoft Azure Storage Account that is used as persistent storage.

This is a common scenario when we exchange information between companies. The FTP protocol is still quite relevant in 2015. It’s a trusted technology, that many understand fairly well. Most legacy systems integrate with it and it can be deployed rather quickly.

Scaling out FTP Severs works well when clients are not reading from the FTP Server. It’s especially useful when you need to receive a large quantity of files from your partners.

Remember, the Microsoft Azure Traffic Manager is a DNS based solution. That means that it works with TCP traffic and with HTTP traffic. Feel free to test out your scenarios and to share your experiences using the comments section below.

Resources

• How to Setup FTP on Azure VM
• Traffic Manager Overview
• About Traffic Manager Load Balancing Methods
• About Traffic Manager Monitoring
• How Do I Configure Traffic Manager?
• Azure Traffic Manager Cmdlets
• Dataflow (Task Parallel Library)
• How to: Specify the Degree of Parallelism in a Dataflow Block

Having written about lifting and shifting a Console Application, I decided to mirror the scenario and describe how a similar method can be used to lift and shift a Windows Service to Microsoft Azure.

Moving a Service to Microsoft Azure

Every time I dig a little deeper into Azure, I’m amazed at how much there is to know. Having done a few projects with Cloud Services in the past, I thought it would be interesting to see if it was possible to lift and shift a Windows Service into an Azure Worker Role.

Lift and Shift: The action of moving a workload to a new environment, without altering the application’s code.

Prerequisites

☁ Tools

Using the Microsoft Web Platform Installer install the latest Microsoft Azure PowerShell with Microsoft Azure SDK.

☁ Microsoft Azure Account

In order to deploy your workload to Microsoft Azure, be sure to have an account. You can create a Free Trial Account on the azure.com website.

Creating a Worker Role without Visual Studio

A Worker Role is a role that is useful for generalized development, and may perform background processing for a Web Role. When you have a need for a background process that performs long running or intermittent tasks, you should use this role.

Let’s fire-up Windows PowerShell ISE and navigate to the folder where you want to create your Cloud Service. For this blog post, I created a new folder, demo-lift-and-shift, in my Documents.

C:\users\<user id>\Documents\demo-lift-and-shift-service

In the PowerShell console, execute the following commands.

# Create the Cloud Service
New-AzureServiceProject -ServiceName 'service-lift-and-shift' -Verbose

# Create the Worker Role
Add-AzureWorkerRole -Name 'ServiceWorkerRole' -Instances 1 -Verbose

These two commands will produce the following files and documents. This represents the Cloud Service project. It contains the configurations and the Cloud Service definition.

The ServiceDefinition.csdef was created by PowerShell when we executed the New-AzureServiceProject command. Then it was updated when we executed the Add-AzureWorkerRole. The following is the unchanged version of the service definition. Since it is missing a few pieces, I will show you what to change in order to be able to deploy your brand-new service to Microsoft Azure.

The ServiceDefinition.csdef file contains the metadata that is required by the Azure environment for the requirements of your cloud service, including what roles it contains. This file also contains configuration settings that apply to all instances. These configuration settings can be read at runtime using the Azure Service Hosting Runtime API. This file cannot be updated while your service is running in Azure.

<?xml version="1.0" encoding="utf-16"?>
<ServiceDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="service-lift-and-shift" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="ServiceWorkerRole">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InputEndpoint name="HttpIn" protocol="tcp" port="80" />
    </Endpoints>
    <Runtime>
      <Environment>
        <Variable name="PORT">
          <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/Endpoints/Endpoint[@name='HttpIn']/@port" />
        </Variable>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  </WorkerRole>
</ServiceDefinition>

Specify a Role (Virtual Machine) size

The size of the role determines the number of CPU cores, the memory capacity, and the local file system size that is allocated to a running instance. The following

• ExtraSmall (1 CPU, 768 MB RAM, 19 GB DISK)
• Small (1 CPU, 1.75 GB RAM, 224 GB DISK)
• Medium (2 CPU. 2.5 GB RAM, 489 GB DISK)
• Large (4 CPU 7 GB RAM, 999 GB DISK)
• ExtraLarge (8 CPU, 14 GB RAM, 2039 GB DISK)

Considerations to help you decide on the correct size for your workload

1. Instances can now be configured to use a D-series VM. These are designed to run applications that demand higher compute power and temporary disk performance. D-series VMs provide faster processors, a higher memory-to-core ratio, and a solid-state drive (SSD) for the temporary disk. For details, see the announcement on the Azure blog, New D-Series Virtual Machine Sizes.
2. Web roles and worker roles require more temporary disk space than Azure Virtual Machines because of system requirements. The system files reserve 4 GB of space for the Windows page file, and 2 GB of space for the Windows dump file.
3. The OS disk contains the Windows guest OS and includes the Program Files folder (including installations done via startup tasks unless you specify another disk), registry changes, the System32 folder, and the .NET framework.
4. The local resource disk contains Azure logs and configuration files, Azure Diagnostics (which includes your IIS logs), and any local storage resources you define.
5. The apps (application) disk is where your .cspkg is extracted and includes your website, binaries, role host process, startup tasks, web.config, and so on.

More details about Virtual Machine and Cloud Service Sizes for Azure can be found in the official Azure documentation.

For this example we will use a Medium sized role (Virtual Machine) for our Cloud Service.

Replace

<WorkerRole name="ServiceWorkerRole">

By

<WorkerRole name="ServiceWorkerRole" vmsize="Medium">

Specify a Guest OS Version

Guest OS families and versions have a release date, a disabled date, and an expiration date. As of the release date, a Guest OS version can be manually selected in the management portal. A Guest OS is removed from the management portal on or after its “disabled” date. It is then “in transition” but is supported with limited ability to update a deployment. The expiration date is when a version or family is scheduled to be removed from the Azure system completely. Cloud services still running on a version when it expires will be stopped, deleted or force upgraded to a newer version, as detailed in the Azure Guest OS Supportability and Retirement Policy. Microsoft supports at least two recent versions of each supported Guest OS family.

Available Families as of December 2014

• FAMILY 4
  Windows Server 2012 R2
  Supports .NET 4.0, 4.5, 4.5.1, 4.5.2
• FAMILY 3
  Windows Server 2012
  Supports .NET 4.0, 4.5
• FAMILY 2
  Windows Server 2008 R2 SP1
  Supports .NET 3.5, 4.0

More details about Azure Guest OS Releases can be found in the official Azure documentation.

The ServiceConfiguration.{Environment}.cscfg was created by PowerShell when we executed the Add-AzureWorkerRole PowerShell command.

The configuration of the settings for your cloud service is determined by the values in the ServiceConfiguration.cscfg file. You specify the number of instances that you want to deploy for each role in this file. The values for the configuration settings that you defined in the service definition file are added to the service configuration file. The thumbprints for any management certificates that are associated with the cloud service are also added to the file. The Azure Service Configuration Schema (.cscfg File) provides the allowable format for a service configuration file.

The following is the unchanged version of the ServiceConfiguration.Cloud.cscfg file. Since it is missing a few pieces, I will show you what to change in order to be able to deploy your brand-new service to Microsoft Azure.

<?xml version="1.0" encoding="utf-16"?>
<ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                      serviceName="service-lift-and-shift"
                      osFamily="2"
                      osVersion="*"
                      xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="ServiceWorkerRole">
    <ConfigurationSettings />
    <Instances count="1" />
    <Certificates />
  </Role>
</ServiceConfiguration>

For this example we will keep Family 2 because our Windows Service is built using .Net 2.0.

We also kept osVersion=”*” to allow the Azure Fabric to continuously patch the Guest OS.

Packaging a Windows Service as a Cloud Service

The ServiceWorkerRole folder contains the files and documents that compose our new Azure Worker Role

Place the files that compose the Windows Service in this folder.

Be sure to include the application’s dependencies such that nothing stops the application from running once it is deployed to an Azure Worker Role. If, for some reason, the application does not seem to run as expected, refer to my earlier blog post that shared techniques to investigate and debug Azure Roles.

Before we package the Azure Cloud Service, we must update the startup.cmd file. This file contains the startup script that is executed before the worker.cmd. The following is the initial version of the file.

:: Placeholder for role startup

echo SUCCESS
exit /b 0

The required updates include, installing the Windows Service and making sure that it has started by executing the startup.ps1 PowerShell script.

D:\Windows\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe /LogToConsole=false DemoWindowsService.exe

REM   Attempt to set the execution policy by using PowerShell version 2.0 syntax.
PowerShell -Version 2.0 -ExecutionPolicy Unrestricted .\startup.ps1 >> "%TEMP%\StartupLog.txt" 2>&1

IF %ERRORLEVEL% EQU -393216 (
   REM   PowerShell version 2.0 isn't available. Set the execution policy by using the PowerShell version 1.0 calling method.

   PowerShell -Command "Set-ExecutionPolicy Unrestricted" >> "%TEMP%\StartupLog.txt" 2>&1
   PowerShell .\startup.ps1 >> "%TEMP%\StartupLog.txt" 2>&1
)

REM   If an error occurred, return the errorlevel.
EXIT /B %errorlevel%

echo SUCCESS
exit /b 0

The startup.ps1 PowerShell script Starts the DemoWindowsService Windows Service. This example is basic and would benefit from much error handling. For example, we could make sure that the service exists and that it is stopped before we call the Start-Service command. Then we could verify that it has indeed started.

$srv = 'DemoWindowsService'
Start-Service -Name $srv

The worker.cmd file contains the application loop that is executed by the Azure Worker role. The generated file contains a script that blocks the execution of the host thread and makes the Azure Fabric act as if the application was running.

:workerLoop

:: Do work
ping 123.45.67.89 -n 1 -w 1000000 > nul

goto workerLoop

In order to allow the Azure Cloud Service to self heal, I recommend that add logic that monitors the state of the Windows Service. If the Windows service stops, try to start it again. If that fails, your script should end the loop and allow the command line to complete its execution. This termination even signals the Azure Fabric to recycle the Role and to self heal.

When changes are made to CMD script files, we must ensure that they are saved using the ANSI Encoding. Failing to do so may prevent the CMD script from executing.

Now that we have a Windows Service configured to be installed and executed in a Worker Role, it’s time to package the Cloud Service. In Windows PowerShell ISE and navigate back to the folder that contains the Cloud Service.

The Cloud Service for this blog post is at the following location:

C:\users\<user id>\Documents\demo-lift-and-shift-service

In the PowerShell console, execute the following commands.

# Package the Cloud Service
Save-AzureServiceProjectPackage

This produces a package named cloud_package.cspkg, which is required for the service’s deployment.

Deploying the Cloud Service

This step has a prerequisite. We must start by importing a publish profile from our Microsoft Azure Subscription. PowerShell will use it to publish our Cloud Service to our Microsoft Azure.

In the PowerShell console, execute the following commands.

# Opens a website that allows us to download the Publish Profile
# associated with our Microsoft Azure Subscription
Get-AzurePublishSettingsFile

Save the publish settings files to your downloads folder then import it using the following command.

Import-AzurePublishSettingsFile -PublishSettingsFile 'C:\Users\<user id>\Downloads\<subscription name>-credentials.publishsettings'

We are now ready to publish our brand-new Cloud Service. The following command will create a new storage account to upload our package. This is the default behavior because we did not provide a storage account for the command. We specified a name for the service, a location (datacenter), a slot (production or staging) and a deployment name. For more information about the Publish-AzureServiceProject command please refer to the official documentation.

Publish-AzureServiceProject -ServiceName 'service-lift-and-shift'
                            -Location 'East US'
                            -Slot 'Production'
                            -DeploymentName '2015-02-14-15-46'
                            -Verbose

If the Publish-AzureServiceProject command is executed with the -Verbose flag, it will produce a detailed output that allows you to understand what is happening. It’s also very useful when you need to debug a failed deployment.

WARNING: Publishing service-lift-and-shift to Microsoft Azure. This may take several minutes...
WARNING: 3:47:10 PM - Preparing runtime deployment for service 'service-lift-and-shift'
WARNING: 3:47:11 PM - Verifying storage account 'serviceliftandshift'...
WARNING: 3:47:43 PM - Preparing deployment for service-lift-and-shift with Subscription ID: dfbedef7-xxxx-xxxx-xxxx-650ca03208ee...
WARNING: 3:47:47 PM - Connecting...
WARNING: 3:47:48 PM - Creating...
WARNING: 3:47:49 PM - Created hosted service 'service-lift-and-shift'.
WARNING: 3:47:49 PM - Uploading Package to storage service serviceliftandshift...
WARNING: 3:47:56 PM - Starting...
WARNING: 3:48:30 PM - Created Deployment ID: 03e36c62a5f74d5f9ebd0a2ec0b16880.
WARNING: 3:48:30 PM - Initializing...
WARNING: 3:51:49 PM - Instance ServiceWorkerRole_IN_0 of role ServiceWorkerRole is busy.
WARNING: 3:52:22 PM - Instance ServiceWorkerRole_IN_0 of role ServiceWorkerRole is ready.

PersistentVMDowntime : Microsoft.WindowsAzure.Commands.Utilities.CloudService.Model.PersistentVMDowntimeInfo
Name                 : 2015-02-14-15-46
DeploymentSlot       : Production
PrivateID            : 03e36c62a5f74d5f9ebd0a2ec0b16880
Status               : Running
Label                : service-lift-and-shift
Configuration        : ...
RoleInstanceList     : {ServiceWorkerRole}
UpgradeStatus        :
UpgradeDomainCount   : 1
RoleList             : {ServiceWorkerRole}
SdkVersion           : 2.5.6496.10
Locked               : False
RollbackAllowed      : False
VirtualNetworkName   :
CreatedTime          : 2015-02-14 3:47:58 PM
LastModifiedTime     : 2015-02-14 3:52:23 PM
ExtendedProperties   : {}
Dns                  :
VirtualIPs           : {Microsoft.WindowsAzure.Commands.Utilities.CloudService.Model.VirtualIP}

Once the Cloud Service has completed its deployment, you can log into the https://manage.windowsazure.com portal and you will be able to monitor and alter its configurations.

Because this solution does not rely on the .NET Azure SDK and since the CMD script has an infinite loop, we do not see a recycling status for a faulted Azure Worker Role. If we observe a high CPU, which could be unexpected, it could be because the Cloud Service was deployed with the wrong Guest OS Family. This can cause the Windows Service to continuously fail to start. The CMD worker script is blocked and prevents the Cloud Service from restarting. Consequently, updating the ServiceConfiguration.Cloud.cscfg file with the correct Guest OS Family and redeploying the Cloud Service can solve the issue.

Out of curiosity, you can configured the Remote Desktop Extension through the management portal and peaked into the provisioned instance. This will allow you to validate whether the Windows Service starts as expected.

Bringing it All Together

Azure Worker Roles support a surprising variety of scenarios! Here’s a quick recap of the steps taken to Lift & Shift a Windows Service to Microsoft Azure.

# Create the Cloud Service
New-AzureServiceProject -ServiceName 'service-lift-and-shift' -Verbose

# Create the Worker Role
Add-AzureWorkerRole -Name 'ServiceWorkerRole' -Instances 1 -Verbose

Update the configurations files with the proper Guest OS Family and with the desired VM Size for your Cloud Service. Then copy the Windows Service and its dependencies to the Role’s folder. Don’t forget to update the startup.cmd script file and to add the startup.ps1 file so that the service may install the Windows Service and ensure that it is started.

# Package the Cloud Service
Save-AzureServiceProjectPackage

# Publish the Package to Microsoft Azure
Publish-AzureServiceProject -ServiceName 'service-lift-and-shift'
                            -Location 'East US'
                            -Slot 'Production'
                            -DeploymentName '2015-02-14-15-46'
                            -Verbose

Opening Ports on Cloud Services

There are scenarios that warrants us to open ports of the Windows Firewall. Imagine an application that coordinates work across many compute nodes. The workload coordinator needs to know and manage each compute node.

The diagram above, depicts a Virtual Machine that has direct access to instances of a Cloud Service without going through a Load Balancer. Both the Virtual Machine and Cloud Service are deployed to a Virtual Network on Microsoft Azure. Using Internal Endpoints in this scenario, would not yield the desired configuration. Endpoints are defined at the Cloud Service boundary and are IP addresses belong to the Data Center’s Network Address Space. Therefore, In order to allow the Virtual Machine to communicate over specific ports to individual Cloud Service instances, we need to use PowerShell and a Startup Task to configure the Windows Firewall.

Using PowerShell to Open a Port

$port = New-Object -ComObject HNetCfg.FWOpenPort
$port.Port = 8080
$port.Name = 'Compute Node Inbound Port'
$port.Enabled = $true

$fwMgr = New-Object -ComObject HNetCfg.FwMgr
$profile = $fwMgr.LocalPolicy.CurrentProfile
$profile.GloballyOpenPorts.Add($port)

Using a Command Line to Open a Port

REM   Add a firewall rule in a startup task.

REM   Add an inbound rule requiring security and encryption for TCP port 8080 traffic.
netsh advfirewall firewall add rule name="Require Encryption for Inbound TCP/8080" protocol=TCP dir=in localport=8080 security=authdynenc action=allow >> "%TEMP%\StartupLog.txt" 2>&1

REM   If an error occurred, return the errorlevel.
EXIT /B %errorlevel%

Configuring a Cloud Service Startup Task

Startup tasks are actions that are taken before your roles begin. Startup tasks are defined in the ServiceDefinition.csdef file by using the Task element within the Startup element. Frequently startup tasks are batch files, but they can also be console applications, or batch files that start PowerShell scripts.

We have two options to open Ports on the Windows Firewall. For this post, I will use PowerShell because it has more steps. This example can be used as a starting point to run other PowerShell scripts from Startup Tasks.

Cloud Service – ServiceDefinition.csdef

The Startup Task is defined on a per role basis. In our case, we must elevate the execution context so that we may interact with the Windows Firewall.

<?xml version="1.0" encoding="utf-16"?>
<ServiceDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="ComputeService" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
	<WorkerRole name="Compute" vmsize="Standard_D1">
		<Startup>
			<Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
		</Startup>
		<Runtime executionContext="elevated">
			<EntryPoint>
				<ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
			</EntryPoint>
		</Runtime>
	</WorkerRole>
</ServiceDefinition>

Command Line – startup.cmd

This command file elevates the execution policy for PowerShell, then it executes the startup.ps1 file. It’s important to note that every time the command line is executed, it produces a StartupLog file, which helps to confirm that everything ran as expected. If an error occurs, this log file can save you countless hours of debugging by providing insights about what may have gone wrong.

REM   Attempt to set the execution policy by using PowerShell version 2.0 syntax.
PowerShell -Version 2.0 -ExecutionPolicy Unrestricted .\startup.ps1 >> "%TEMP%\StartupLog.txt" 2>&1

IF %ERRORLEVEL% EQU -393216 (
   REM   PowerShell version 2.0 isn't available. Set the execution policy by using the PowerShell version 1.0 calling method.

   PowerShell -Command "Set-ExecutionPolicy Unrestricted" >> "%TEMP%\StartupLog.txt" 2>&1
   PowerShell .\startup.ps1 >> "%TEMP%\StartupLog.txt" 2>&1
)

REM   If an error occurred, return the errorlevel.
EXIT /B %errorlevel%

echo SUCCESS
exit /b 0

PowerShell – startup.ps1

This file contains the PowerShell script listed above. This is also the place where we can configure other aspects of out Cloud Service via PowerShell.

$port = New-Object -ComObject HNetCfg.FWOpenPort
$port.Port = 8080
$port.Name = 'Compute Node Inbound Port'
$port.Enabled = $true

$fwMgr = New-Object -ComObject HNetCfg.FwMgr
$profile = $fwMgr.LocalPolicy.CurrentProfile
$profile.GloballyOpenPorts.Add($port)

Packaging

For this to work, we need to place the Command Line and PowerShell files at the root of the Cloud Service Role. If you are using Visual Studio to create your package, be sure to set both files to Copy Always otherwise they will be missing from the resulting package. If you are using PowerShell to create the package, the CmdLet will take everything from the role’s directory.

Deploying the Cloud Service

There is nothing different about deploying a Cloud Service that has Startup Tasks. Using your tool of preference, create a new Cloud Service package and deploy it to Microsoft Azure.

Using PowerShell to package the Cloud Service requires you to navigate to the Cloud Service directory and execute the Save-AzureServiceProjectPackage command. This will produce a cloud_package.cspkg package file.

To deploy Azure Cloud Services, I usually use PowerShell. This is an example of what you may write yourself.

New-AzureDeployment `
                -Package 'C:\\cloud_package.cspkg' `
                -Configuration 'C:\\ServiceConfiguration.Cloud.cscfg' `
                -Slot Production `
                -Label 'Compute-2015-02-17' `
                -ServiceName  'DemoCloudService' `
                -Name 'DemoCloudService' `
                -Verbose

Resources

• Run Startup Tasks in Azure
• Strategies for Writing Startup Tasks that can run Multiple Times
• Best Practices for Startup Tasks
• Define Startup Tasks for a Role
• Define Environment Variables Before a Role Starts
• Use Local Storage to Store Files During Startup
• Make a Startup Task Perform Different Actions on the Compute Emulator and the Cloud
• Use AppCmd.exe to Configure IIS at Startup
• Add Firewall Rules By Using a Startup Task
• WebRole Schema
• WorkerRole Schema

Configuring Internal Endpoints

There are many scenarios where Internal Endpoints make sense. Imagine a web application that needs to communicate with a middle tier. The communication between both Cloud Services does not need to leave Microsoft Azure networks. The diagram above depicts a scenario where a Web Role has access to a middle tier Cloud Service without going through the public Internet.

Role instances running in a Cloud Service on Microsoft Azure communicate through internal and external connections that vary depending on the type of communication that is needed. An internal endpoint can connect by using a protocol of HTTP, TCP or UDP.

The configuration of an Internal Endpoint is done through the Service Definition. Below is the template used to describe what can be done in terms of Endpoint configurations. 

Service Definition Endpoint Template

<Endpoints>
  <InputEndpoint certificate="<certificate-name>" ignoreRoleInstanceStatus="[true|false]" name="<input-endpoint-name>" protocol="[http|https|tcp|udp]" localPort="<port-number>" port="<port-number>" loadBalancerProbe=”<load-balancer-probe-name>” />
  <InternalEndpoint name="<internal-endpoint-name>" protocol="[http|tcp|udp|any]" port="<port-number>">
     <FixedPort port="<port-number>"/>
     <FixedPortRange min="<minium-port-number>" max="<maximum-port-number>"/>
  </InternalEndpoint>
  <InstanceInputEndpoint name="<instance-input-endpoint-name>" localPort="<port-number>" protocol="[udp|tcp]">
     <AllocatePublicPortFrom>
        <FixedPortRange min="<minium-port-number>" max="<maximum-port-number>"/>
     </AllocatePublicPortFrom>
  </InstanceInputEndpoint>
</Endpoints>

The Endpoints Element

Describes the collection of input (external), internal, and instance input endpoints for a role. This element is the parent of the InputEndpoint, InternalEndpoint, and InstanceInputEndpoint elements.

Input and Internal endpoints are allocated separately. A service can have a total of 25 input, internal, and instance input endpoints which can be allocated across the 25 roles allowed in a service. For example, if have 5 roles you can allocate 5 input endpoints per role or you can allocate 25 input endpoints to a single role or you can allocate 1 input endpoint each to 25 roles.

Sample ServiceDefinition

The following sample defines an Internal Endpoint names TcpListener which opens ports between 1000 and 2000.

<?xml version="1.0" encoding="utf-16"?>
<ServiceDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="briseboisDemo3" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="Compute">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InternalEndpoint name="TcpListener" protocol="tcp">
        <FixedPortRange min="1000" max="2000" />
      </InternalEndpoint>
    </Endpoints>
    <Runtime>
      <Environment>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  </WorkerRole>
</ServiceDefinition>

Deploying the Cloud Service

There is nothing different about deploying a Cloud Service that exposes Internal Endpoints. Using your tool of preference, create a new Cloud Service package and deploy it to Microsoft Azure.

Using PowerShell to package the Cloud Service requires you to navigate to the Cloud Service directory and execute the Save-AzureServiceProjectPackage command. This will produce a cloud_package.cspkg package file.

To deploy Azure Cloud Services, I usually use PowerShell. This is an example of what you may write yourself.

New-AzureDeployment `
                -Package 'C:\\cloud_package.cspkg' `
                -Configuration 'C:\\ServiceConfiguration.Cloud.cscfg' `
                -Slot Production `
                -Label 'Compute-2015-02-17' `
                -ServiceName  'DemoCloudService' `
                -Name 'DemoCloudService' `
                -Verbose

Resources

• Enable Communication for Role Instances in Azure
• WebRole Schema
• WorkerRole Schema

Configuring Elevated Privileges

There are various scenarios that require us to execute applications with elevated privileges. It’s common for this requirement to surface during lift and shift efforts.

This configuration is done in the Service Definition file. It requires us to specify the Runtime Execution Context to elevated.

Runtime Configuration Template

This template uses the ProgramEntryPoint as an EntryPoint. Roles also support NetFxEntryPoint.

<Runtime executionContext="[limited|elevated]">
   <Environment>
     <Variable name="<variable-name>" value="<variable-value>">
      <RoleInstanceValue xpath="<xpath-to-role-environment-settings>"/>
    </Variable>
  </Environment>
  <EntryPoint>
     <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
   </EntryPoint>
</Runtime>

Sample ServiceDefinition

This sample ServiceDefinition was generated using the Add-AzureWorkerRole PowerShell command.

<?xml version="1.0" encoding="utf-16"?>
<ServiceDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="briseboisDemo3" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="Compute">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InputEndpoint name="HttpIn" protocol="tcp" port="80" />
    </Endpoints>
    <Runtime executionContext="elevated">
      <Environment>
        <Variable name="PORT">
          <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/Endpoints/Endpoint[@name='HttpIn']/@port" />
        </Variable>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  </WorkerRole>
</ServiceDefinition>

Deploying the Cloud Service

There is nothing different about deploying a Cloud Service that executes with elevated privileges. Using your tool of preference, create a new Cloud Service package and deploy it to Microsoft Azure.

Using PowerShell to package the Cloud Service requires you to navigate to the Cloud Service directory and execute the Save-AzureServiceProjectPackage command. This will produce a cloud_package.cspkg package file.

To deploy Azure Cloud Services, I usually use PowerShell. This is an example of what you may write yourself.

New-AzureDeployment `
                -Package 'C:\\cloud_package.cspkg' `
                -Configuration 'C:\\ServiceConfiguration.Cloud.cscfg' `
                -Slot Production `
                -Label 'Compute-2015-02-17' `
                -ServiceName  'DemoCloudService' `
                -Name 'DemoCloudService' `
                -Verbose

Resources

• WebRole Schema
• WorkerRole Schema
• Microsoft Azure Cloud Services Part 2: Anatomy of a Cloud Service

Deploying a Cloud Service to a VNet

In a recent post about Microsoft Azure Virtual Networks, I made the recommendation that Cloud Services should be deployed to Microsoft Azure Virtual Networks. The driving factor behind this recommendation comes from my personal real-world experience, where I learned the hard way, that moving a deployed Cloud Service to a Virtual Network meant downtime…

Getting it Done

In order to deploy a Cloud Service to a Virtual Network, we must add Network Configuration section to the Service Configuration file. The following is the updated ServiceConfiguration.Cloud.cscfg that specified that my Cloud Service is to be deployed to my Virtual Network called AzureVNet and more specifically to its Subnet called MiddleTier.

<?xml version="1.0" encoding="utf-16"?>
<ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serviceName="briseboisDemo3" osFamily="4" osVersion="*" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">

  <Role name="Compute">
    <Instances count="5" />
  </Role>

  <NetworkConfiguration>
    <!-- Name of the target Virtual Network -->
    <VirtualNetworkSite name="AzureVNet" />

    <!-- Associating a Role to a Specific Subnet by name -->
    <AddressAssignments>
      <InstanceAddress roleName="Compute">
        <Subnets>
          <Subnet name="MiddleTier" />
        </Subnets>
      </InstanceAddress>
    </AddressAssignments>
  </NetworkConfiguration>
</ServiceConfiguration>

Looking at these configuration updates, you may be asking whether we have to do anything else to successfully deploy to a Virtual Network. The answer is no, there is nothing more to it. Fire up PowerShell or your favorite tool to deploy your updated Cloud Service package to Azure.

I used the following Network Configuration to create my test Virtual Network.

<?xml version="1.0" encoding="utf-8"?>
<NetworkConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <Dns />
    <VirtualNetworkSites>
      <VirtualNetworkSite name="AzureVNet" Location="East US 2">
        <AddressSpace>
          <AddressPrefix>10.0.0.0/24</AddressPrefix>
        </AddressSpace>
        <Subnets>
          <Subnet name="MiddleTier">
            <AddressPrefix>10.0.0.0/27</AddressPrefix>
          </Subnet>
        </Subnets>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

Why are Virtual Networks Important?

As a developer, I used to forget about Virtual Networks. And to be fair, I shied away from pretty much everything that can be considered as infrastructure.

Microsoft Azure is a game changer! It requires Developers and IT Pros to collaborate on projects. Let’s take a moment to set things right. Developers and IT Pros are not competing against each other in this new world. They collaborate in order to produce value for the business and customers.

As we move to a Cloud First and Mobile First world, security is more important than it ever was. On Microsoft Azure, the first step towards securing your application resources is to create a Virtual Network.

Microsoft Azure is constantly evolving and augmenting its security. Everything starts at the edge of the public Internet. This is where mechanisms like DDoS protection kick in. Then traffic is routed through multiple network layers and finally reaches public Endpoints that are configured on Cloud Services.

Cloud Services are containers. Like a firewall, they protect resources such as Roles and Virtual Machines from the outside world. They provide our resources with an internal network. At this point, it’s important to note that Cloud Services can contain up to 25 Roles. However, it is recommended to apply the Single Responsibility Rule (SRP) and to keep the Role count low. This helps reduce accidental complexity when it comes to deployments and maintenance.

Since Cloud Services are internal networks shielded from the outside world, they communicate over public Endpoints. The diagram below illustrates a 3-tier Cloud Service that is accessible from the outside world.

Each tier is discoverable and accessible from the public Internet. In some scenarios, this can be favorable. However, it can be considered as an attack vector.

In order to secure our applications, best practices recommend the use of Virtual Networks and Subnets. As shown below, we can take advantage of Network Security Groups (NSG) and Access Control Lists (ACL) to control network traffic.

This diagram illustrates that a Front End tier of an application is accessible from the public Internet. It also shows that the Middle tier is only accessible from the Front End and that access to the Back End is limited to the Middle tier.

Getting to Know Azure Virtual Networks (VNet)

What is a Virtual Network (VNet)

A Virtual Network (VNet) is a network overlay that you can configure in Azure. Virtual Machines and Services that are part of the same Virtual Network can access each other. However, services outside the virtual network have no way to identify or connect to services hosted within virtual networks unless you decide to configure that specific type of connection, as in the case of VNet to VNet configurations. This provides an added layer of isolation to your services. Azure Virtual Network also lets you extend your network into Azure and treat deployments as a natural extension to your on-premises network.

Take Notice

Virtual Machines and Cloud Services acquire their network settings during deployment. This means you can’t move something into a Virtual Network if it is already deployed. In order to move a deployed resource into a Virtual Network, you need redeploy it.

When Do You Need A Virtual Networks (VNet)?

Whether you need a Virtual Network depends entirely on what you are trying to do. Azure currently supports two categories of Virtual Networks. The first is a Cloud-Only Virtual Network. And the second is a Cross-Premises Virtual Network (Hybrid Cloud Solutions). If you start with a Cloud-Only Virtual Network, you can convert it to a Cross-Premises Virtual Network. Going through a migration from no Virtual Network to a Cross-Premises Virtual Network can be painful and incur unwanted downtime.

The major advantage of starting with the Virtual Network is flexibility. You can create multi-site configurations, VNet to VNet configurations, ExpressRoute connections, and combinations of multiple configuration types.

Considerations for Using a Virtual Network (VNet)

• Name Resolution (DNS) – If you want to connect to your Virtual Machines and Cloud Services by hostname or SRV records, rather than using the IP address, you need a Name Resolution (DNS).
• Enhanced Security and Isolation – Only virtual machines and services that are part of the same network can access each other.
• Extended Trust and Security Boundary – Extends the trust boundary from a single service to the virtual network boundary.
• Extend Your On-Premises Network to the Cloud – Leverage all on-premises investments around monitoring and identity for your services hosted in Azure.
• User persistent Private IP Addresses – Virtual machines within a Virtual Network will have a stable private IP address. Virtual Machines can be configured with a Static Internal IP. However, if you decide to use this feature it is recommended that you separate Virtual Machines that have static DIPs from PaaS instances by creating separate subnets.
• Internal Load Balancing – Enables scenarios like Highly Available (HA) line of business (LOB) apps and SQL Always On.

Using PowerShell to Create a Virtual Network

To create a Microsoft Azure Virtual Network, we must start by creating a Network Configuration File. Referring to the Virtual Network Configuration Schema I created the following XML configuration File.

<?xml version="1.0" encoding="utf-8"?>
<NetworkConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <Dns />
    <VirtualNetworkSites>
      <VirtualNetworkSite name="AzureVNet" Location="East US 2">
        <AddressSpace>
          <AddressPrefix>10.0.0.0/24</AddressPrefix>
        </AddressSpace>
        <Subnets>
          <Subnet name="FrontEnd">
            <AddressPrefix>10.0.0.0/27</AddressPrefix>
          </Subnet>
          <Subnet name="MiddleTier">
            <AddressPrefix>10.0.0.32/27</AddressPrefix>
          </Subnet>
          <Subnet name="BackEnd">
            <AddressPrefix>10.0.0.64/27</AddressPrefix>
          </Subnet>
        </Subnets>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

Subnets are defined within the Virtual Network’s Address Space. Make sure to give yourself enough room to grow. Address Prefixes are used to define available IPs for each subnet. To create a new Subnet on a deployed Virtual Network, export the current Virtual Network Configuration from your Azure Subscription and add a new node to the Subnets XML node. Then import the configuration back to your Azure Subscription through PowerShell or the Azure Management Portal.

Azure Virtual Networks use the CIDR notation to define Usable Address Ranges.

CIDR notation is a compact representation of an IP address and its associated routing prefix. The notation is constructed from the IP address and the prefix size, the latter being equivalent to the number of leading 1 bits in the routing prefix mask. The IP address is expressed according to the standards of IPv4 or IPv6. It is followed by a separator character, the slash (‘/’) character, and the prefix size expressed as a decimal number.

The address may denote a single, distinct interface address or the beginning address of an entire network. The maximum size of the network is given by the number of addresses that are possible with the remaining, least-significant bits below the prefix. This is often called the host identifier.

For example, ‘192.168.100.0/24′ represents the given IPv4 address and its associated routing prefix 192.168.100.0, or equivalently, its subnet mask 255.255.255.0, which has 24 leading 1-bits.

For those who are not comfortable with the CIDR notation, The Azure team has provided us with detailed dropdown menus to guide us through this configuration.

I saved the above XML configuration as C:\demo\AzureVNet.netcfg. Then Using PowerShell, I setup the proper PowerShell console context and applied the Virtual Network configurations to my subscription.

# Switch to Azure Service Management

Switch-AzureMode -Name AzureServiceManagement

# Refresh Credentials

Add-AzureAccount

# Select Subscription

$subscriptions = Get-AzureSubscription

# Select the first Subscription
# (I did not specify my target Subscription for this demo)

$firstSubscription = $subscriptions `
                        | Select-Object -First 1 `
                                        -Verbose

# Select the Azure Subscrition and set it as Default

Select-AzureSubscription -SubscriptionId $firstSubscription.SubscriptionId `
                         -Default `
                         -Verbose

# Create Virtual Network 

Set-AzureVNetConfig -ConfigurationPath 'C:\demo\AzureVNet.netcfg' `
                    -Verbose

Once the command completes, I navigated to the Azure Management Portal and was able to verify that everything had been created.

Take Notice

Using the Set-AzureVNetConfig from PowerShell will replace all the Virtual Networks associated with your subscription and apply the new configurations present in the Network Configuration File.

Best Practices

• Use Subnets to create multi-tier typologies
• Control flow over Network Segments using Network Security Groups
• Use Access Control Lists (ACL) to filter conditions with allow / deny
• Configure internal Load Balancing for Highly Available tiers
• RDP to internal endpoints for added security
• Design Virtual Networks with the following in mind
  • Maximum of 2048 machines (Virtual Machines and Web/Worker role instances).
  • 500 000 concurrent TCP connections for a virtual machine or role instance.
  • 50 Access Control Lists (ACLs) per endpoint.
  • 10 local network sites per Virtual Network.
  • Each Virtual Network supports a single virtual network gateway.

More Resources

Be sure to read through the Virtual Network FAQ, you’ll find answers to most of your queries. Then go though these additional resources for more in depth information.

• Azure Subscription and Service Limits, Quotas, and Constraints
• Virtual Network Overview
• Azure Networking Guidance
• Designing Networking and Hybrid Connectivity Infrastructure
• About Secure Cross-Premises Connectivity
• Virtual Network Configuration Tasks
• Azure Virtual Network Configuration Schema
• How does a Network Security Group Work
• About Network Access Control Lists (ACLs)
• Configure a Point-to-Site VPN
• Configure a Static Internal IP Address for a Virtual Machine
• Configure a Cloud-Only Virtual Network in the Management Portal
• General Availability of Network Security Groups
• Great Blog Post about Network Security Groups

Find The Closest #Azure Data Center

I regularly get asked about how to choose a Microsoft Azure Data Center for optimal deployments. The answer is to pick the Data Center that is nearest to your users. For applications that have a broad user base, it’s favorable to deploy multiple instances of the application and to use Microsoft Azure Traffic Manager to direct users to the closest Data Center. This typically provides the best user experience.

Finding that closest Data Center requires a couple of tests, because physical networks differ from location to location.

To get a ping from my location to all the available Microsoft Azure Data Centers, I use a community tool called Azure Site Ping (GitHub).
Then, to compare the Azure Blob Storage response time from multiple Microsoft Azure Data Centers, I use another community tool called Microsoft Azure Speed Test. This tool produces a chart (see featured image of this post) that allows us to visualize and understand the variance in latency.

Armed with this information, I can identify a Data Center. In some scenarios, the Data Center that appears to be the nearest, may not be the closest to the application’s users. Just to be sure, run these tools from locations where your users will consume your application. This will help you identify the best Data Center for your scenario.

Remember, it’s acceptable to deploy to one Data Center for Dev & Test scenarios and to deploy to another Data Center for Production. Latency obeys the laws of physics, and we need to take them under consideration when we make choices that can affect the overall user experience.

When you can’t get close enough to your users with Azure Blog Storage, consider using the Azure Content Delivery Network (CDN). This service is composed of many Edge Nodes that place your content closer to your users.

Ultimately, the bottom line is that when latency is low, users are happy.

Uploading Content to Azure Blobs

I use Azure Blobs on a regular basis. They’re generally really useful and help me through some tough situations. Working with Blobs is simple. You can interact with them using Visual Studio, third-party tools, REST and even through PowerShell.

The following PowerShell command demonstrates how I upload content to Azure Storage.

Set-BlobContent -StorageAccountName 'scaleupdowndemopkgs' `
                -StorageContainer 'packages' `
                -FilePath 'C:\Service\cloud_package.cspkg' `
                -BlobName 'extra_small_vm_cloud_package.cspkg'

The following is the the PowerShell function invoked in the previous example. It finds your Azure Storage keys and creates a Blob Storage Container is it does not already exist. Then it uploads the contents of the provided file path to an Azure Blob.

function Set-BlobContent
{
    Param
    (
        [Parameter(Mandatory=$true)]
        [string]
        $StorageAccountName,
 
        [Parameter(Mandatory=$true)]
        [string]
        $StorageContainer,
 
        [Parameter(Mandatory=$true)]
        [string]
        $FilePath,
 
        [Parameter(Mandatory=$true)]
        [string]
        $BlobName
    )
 
    # Setup the Storage Context used for Storage Operations
 
    $AccountKeys = Get-AzureStorageKey -StorageAccountName $StorageAccountName `
                                       -Verbose `
                                       -WarningAction Stop `
                                       -ErrorVariable $e `
 
    $StorageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName `
                                              -StorageAccountKey $AccountKeys.Primary `
                                              -Verbose `
                                              -WarningAction Stop `
                                              -ErrorVariable $e `
 
    try
    {
        Write-Verbose('Try create Container')
        $newcontainer = New-AzureStorageContainer -Context $StorageContext `
                                              -Container $StorageContainer `
                                              -Verbose `
                                              -WarningAction Ignore `
                                              -ErrorVariable $e `
    }
    catch
    {
        Write-Verbose('Container already exists')
    }
 
    # Upload and rename the Cloud Service package to Blob Storage
 
    Set-AzureStorageBlobContent -File $FilePath `
                                -Container $StorageContainer `
                                -Context $StorageContext `
                                -Blob $BlobName `
                                -Force `
                                -Verbose `
                                -WarningAction Stop `
                                -ErrorVariable $e `
}

Saving Microsoft ARM Templates as JSON files on disk

If you are creating a brand new environment on Azure, you should be using Windows PowerShell and Resource Manager. Instead of creating and managing individual resources, use a template (resource model) to create a resource group that has the resources need to support your service. You can create your own templates or pick a template from the gallery.

If you’re starting off with the Azure Resource Manager (ARM), I recommend taking a look at what others have published. Then modify them to create a resource model that represents your service.

The following code will download the Azure Resource Manager (ARM) templates that are published and maintained by Microsoft.

Switch-AzureMode -Name AzureResourceManager
 
New-Item -path 'C:\Azure\Templates\' -type directory -Force

$microsoftTemplates = Get-AzureResourceGroupGalleryTemplate -Publisher 'Microsoft'
 
foreach ($item in $microsoftTemplates)
{
    $identity = $item.'Identity'
 
    Save-AzureResourceGroupGalleryTemplate -Identity $identity `
             -Path C:\Azure\Templates\ `
             -Force `
             -Verbose
}

NB: be sure that ‘C:\Azure\Templates\’ exists before running this script.

This list is growing fast. Microsoft is adding new features on a regular basis. And there’s no better way to learn, than to look at what’s already out there.

Next Steps

To learn more about using Windows PowerShell with Resource Manager:

• Azure Resource Manager Cmdlets: Learn to use the cmdlets in the AzureResourceManager module.
• Using Resource groups to manage your Azure resources: Learn how to create and manage resource groups in the Azure Management Portal.
• Using the Azure Cross-Platform Command-Line Interface with the Resource Manager: Learn how to create and manage resource groups with command-line tools that work on many operating system platforms.

Finding an Invisible NAS on a Network

Over the weekend I decided to migrate data from an older NAS (Network Attached Storage) device to a newer device. So I plugged in the new NAS and set it up with two brand new HDDs. Provisioned a RAID 1 (Mirror) and finally I secured the device with a better password than “blank“.

Plugging in the older device, I expected that it would show up in Windows Explorer, but it did not. I spent the next couple hours trying to figure out why I could not find the device’s IP. I connected to my router, looked at the DHCP map and it wasn’t showing up.

Obviously, I hadn’t used the device for some time and had forgotten the devices name. So here I was, with no IP or Name for the device. The RJ45 plugged in and the flashing lights telling me that something was going on…

In trying to find the device, I started to set up meaningful DHCP Reservations for all known devices my network. Surprisingly, I found myself adding more devices than I was expecting. This was an eye opener, but left me in the cold. The NAS was still invisible.

Having spent about an hour creating DHCP Reservations by finding the MAC Address of each device, I went ahead and created a new reservation for the NAS device. Fortunately, most network devices like routers, have a hard copy of their MAC Address printed on a label. The new reservation assigned 192.168.1.153 as the IP Address for my NAS device.

I proceeded by rebooting both my router and the NAS device. Then I executed the following in a Command Prompt.

NET VIEW \\192.168.1.153

Seeing the following output put a huge smile on my face, because I finally had a known IP for my NAS device.

Shared resources at 192.168.1.153

DNS-325

Share name  Type   Used as  Comment

--------------------------------------------
lp          Print           USB Printer
Volume_1    Disk
The command completed successfully.

I was then able to map the network drive by using this new address \\192.168.1.153\Volume_1. Then using the same IP Address, I browsed to the device’s admin website. Out of curiosity I tried to find the cause and this is when I realized that I had previously configured the device with a Static IP. I had forgotten the device’s name and IP.

The lesson I’m pulling from this adventure, is that using DHCP Reservations easier to diagnose than forgotten Static IPs.

The Challenge

As developers, we are up against odds that push us to make trade-offs in order to go into production on time. More often than not, it’s a race where security becomes an afterthought.

Securing Azure SQL Databases

Security mechanisms come in many flavors. It is a requirement that needs to be defined and implemented on day 1. These rituals (policies and practices), must become natural in your application life cycle management. Consider these as a starting point from which you can develop your own security practices.

• Do not use the default user for development, testing or for deployments
  • Create a user specifically for deployments (can perform schema alterations)
  • Create a user on a per application basis (cannot alter schema and has limited write access)
  • Create a user for support investigations (this should be read-only)
  • Create individual accounts for members of DevOps who will need to act upon the database. (these accounts should have limited write access)
• Reference data should be read-only (immutable versions) and should only be updated through deployments. This type of data can be stored in NoSQL data services to augment the overall scalability of your application.
• Enable Auditing for Azure SQL Database, this feature will give you deep insight in how the database is manipulated and about how it is used.
• Use SQL Database Projects to design, build, version and deploy
• Use schemas to segregate tenants, reference data, activity data and resouce (shared) data.
• Use schemas to keep track of ownership chaining
• Encrypt connection string passwords at rest
• Use strong passwords
• Set Trusted_Connection=False in the connection string. This forces server certificate validation
• Set Encrypt=True in the connection string to force the client to use SSL
• Ensure that you are covered against SQL Injection
• SQL Database Firewall rules should block everything except the consuming applications
  • SQL Database Firewall rules should be set on both the Server and Database

Automating Azure SQL Server Firewall Rule Maintenance

All too often our Azure SQL Server firewall rules are riddled with unused and forgotten entries.

Although we try to be vigilant, we are human and can forget to remove unnecessary firewall rules. These rules, become attack vectors that can be leverage against our application and we should proactively do everything we can to reduce potential risks. This vulnerability can affect business continuity in two ways. The first is a malicious attack meant to steal or destroy data. The second way business continuity can be affected is by human error. Imagine for a moment that a developer updates a database and targets the production database by mistake, the results can be disastrous. Rolling back isn’t always possible and downtime can be costly. On the other hand, if the Azure SQL Server firewall rules are maintained, pruned and specifically targeted at the consuming applications, developers would be blocked from targeting the wrong database during their development cycle. In order to update the database, developers would need to manually add a firewall rule to the Azure SQL Server and target the database.

Protecting SQL Database instance against human error got me thinking and I decided to write a runbook that could be schedule using Azure Automation.

<#
.Synopsis
   The purpose of this script is to maintain a clean list of firewall rules
   for Azure SQL Database Server. It will add a rule for each Role it finds
   within the targeted Cloud Service. It will update rules that are out of
   sync with the deployed cloud service. Then it will delete all unrelated rules.
.EXAMPLE
   Set-CloudServiceSqlDatabaseFirewallRules -AzureCredentialsName 'runbooks' `
                                            -SqlServerName $SqlServerName `
                                            -CloudServiceName $CloudServiceName `
                                            -DeploymentSlot Production `
                                            -Verbose
#>
workflow Set-CloudServiceSqlDatabaseFirewallRules
{
    Param
    (
        [parameter(Mandatory=$true, `
                 HelpMessage = 'The name given to the Windows PowerShell Credentials located# in the Assets of this Azure Automation instance')]
        [String]
        $AzureCredentialsName,

        # Azure SQL Server name
        [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Azure SQL Server')]
        [string]
        $SqlServerName,

        # Azure Cloud Service Name
        [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Azure Cloud Service')]
        [string]
        $CloudServiceName,

        # Azure Cloud Service Deployment Slot
        [parameter(Mandatory=$true, `
                 HelpMessage = 'The Azure Cloud Service deployment Slot')]
        [ValidateSet('Production', 'Staging')]
        [string]
        $DeploymentSlot
    )

    $Credentials = Get-AutomationPSCredential `
                       -Name $AzureCredentialsName

    Add-AzureAccount `
       -Credential $Credentials

    $rules = Get-AzureSqlDatabaseServerFirewallRule -ServerName $SqlServerName
    'Rules Obtained from Azure SQL Server'
    $rules

    InlineScript
    {
      $instances = Get-AzureRole -ServiceName $Using:CloudServiceName -Slot $Using:DeploymentSlot -InstanceDetails `
                 | Select-Object ServiceName,  @{Name="Vip";Expression={$_.InstanceEndpoints[0].Vip}}

      $instances | `
      ForEach-Object {

        $name = $_.ServiceName

        $vip =$_.Vip

        $rule = $null

        $rules = Get-AzureSqlDatabaseServerFirewallRule -ServerName $Using:SqlServerName

        if($rules)
        {
           $rule = $rules | Where-Object {$_.RuleName -eq $name}
        }

        if($vip)
        {
          if($rule)
          {
            if($rule.StartIpAddress -ne $vip -and $rule.EndIpAddress -ne $vip)
            {
              Set-AzureSqlDatabaseServerFirewallRule -ServerName $Using:SqlServerName `
                                                     -RuleName $name `
                                                     -StartIpAddress $vip `
                                                     -EndIpAddress $vip `
                                                     -Verbose
            }
          }
          else
          {
            New-AzureSqlDatabaseServerFirewallRule -ServerName $Using:SqlServerName `
                                                   -RuleName $name `
                                                   -StartIpAddress $vip `
                                                   -EndIpAddress $vip `
                                                   -Verbose
          }
        }
      }

      $rules = Get-AzureSqlDatabaseServerFirewallRule -ServerName $Using:SqlServerName

      $rules | `
      ForEach-Object {
        $name = $_.RuleName

        $instance = $instances | Where-Object {$_.ServiceName -eq $name}

        if(!$instance)
        {
          Remove-AzureSqlDatabaseServerFirewallRule -ServerName $Using:SqlServerName `
                                                    -RuleName $name `
                                                    -Verbose
        }
      }
    }

    'Updated on Azure SQL Server'
    $rules = Get-AzureSqlDatabaseServerFirewallRule -ServerName $SqlServerName
    $rules
}

Executing this runbook on a regular schedule requires us to set up credentials and a schedule in Azure Automation Assets. My earlier post about scaling Azure Cloud Services up and down like clockwork can help you get up and running.

Everytime Set-CloudServiceSqlDatabaseFirewallRules is executed, it will give you an output similar to the following. Let’s walk through the output and understand what we can extract from it. The first part is a list of the Azure SQL Server firewall rules prior to maintenance. Then it will list all removed and added firewalls rules. Finally it will list the updated set of Azure SQL Server firewall rules.

Rules Obtained from Azure SQL Server

RuleName       : ClientIPAddress_Alex_home
StartIpAddress : 69.130.90.185
EndIpAddress   : 69.130.90.185
ServerName     : bnv4xb0akd
PSComputerName : localhost

RuleName       : ClientIPAddress_Alex_work
StartIpAddress : 69.130.90.19
EndIpAddress   : 69.130.90.19
ServerName     : bnv4xb0akd
PSComputerName : localhost

RuleName       : console
StartIpAddress : 23.99.200.245
EndIpAddress   : 23.99.200.245
ServerName     : bnv4xb0akd
PSComputerName : localhost

1/8/2015 9:25:05 AM, Verbose: 606a4214-a079-47a8-9105-f65dc92f9e8f:[localhost]:Removing firewall rule "ClientIPAddress_Alex_home" for Microsoft Azure
Sql Database server "bnv4xb0akd".

1/8/2015 9:25:06 AM, Verbose: 606a4214-a079-47a8-9105-f65dc92f9e8f:[localhost]:Removing firewall rule "ClientIPAddress_Alex_work" for Microsoft Azure
Sql Database server "bnv4xb0akd".

Updated on Azure SQL Server

RuleName       : console
StartIpAddress : 23.99.200.245
EndIpAddress   : 23.99.200.245
ServerName     : bnv4xb0akd
PSComputerName : localhost

Once the runbook execution is complete, the Microsoft Azure management portal should look much cleaner. The newly created firewall rules will have the same name as the Cloud Service that they represent.

Next Steps

This runbook is an example that you can use to build out your maintenance and security strategy around Azure SQL Dababase firewall rules.

The above runbook has short comings that need to be addressed. It does not support a scenarios where a database is shared by more than one Cloud Service. It also does not configure the database level firewall. As illustrated below, Azure SQL Databases has two levels of Firewalls and you should take advantage of this.

Azure SQL Server firewall rules give access to all of its databases. If you need to restrict access to a single database within an Azure SQL Server, you will need to execute SQL commands against the Azure SQL Database. Refer to this documentation about database-level firewall rules.

Azure SQL Database Resources

• How to: Configure Firewall Settings (Azure SQL Database)
• Best Practices for Securing Azure SQL Database
• Azure SQL Database Firewall
• Guidelines for Connecting to Azure SQL Database
• Managing Databases and Logins in Azure SQL Database
• Azure SQL Database Guidelines and Limitations

TSQL Resources

• Create Firewall Rule
• Delete Firewall Rule
• Get Firewall Rule
• List Firewall Rules

PowerShell Resources

• New-AzureSqlDatabaseServerFirewallRule
• Remove-AzureSqlDatabaseServerFirewallRule
• Set-AzureSqlDatabaseServerFirewallRule
• Get-AzureSqlDatabaseServerFirewallRule

Scaling Azure Cloud Services

When we build applications on the cloud, we usually favor scaling-out over scaling-up. Consequently, scaling in and out is supported out of the box on Microsoft Azure. On rare occasions, there is a business need to scale-up during peak hours and to scale-down for the quiet hours. This blog post will show you how to achieve this with the help of Microsoft Azure Automation and with PowerShell.

Keep in mind, that scaling up and down requires us to be creative.

Getting Ready

Azure Cloud Services define Virtual Machine size in the ServiceDefinition.csdef file. Unfortunately this file is part of the Cloud Service package. Therefore, one needs to redeploy to scale to a different Virtual Machine size.

This limitation requires us to create an individual package for each Virtual Machine size that we will use to scale our Cloud Service up and down.

For this blog post, we will use the Cloud Service I created in my previous post about deploying a Console Application to Microsoft Azure to create my packages. The first package will be sized to an ExtraSmall (1 CPU, 768 MB RAM, 19 GB DISK) Virtual Machine. This will be the default deployment package. We will schedule this package to be deployed to the production slot between the hours of 8 PM and 6 AM. The second package will be sized to an ExtraLarge (8 CPU, 14 GB RAM, 2039 GB DISK) Virtual Machine. It will be deployed to the production slot of my Cloud Service between the hours of 6 AM and 8 PM.

To create the first package, we must edit the ServiceDefinition.csdef file and set the vmsize of the ConsoleWorkerRole to ExtraSmall. The following is the updated ServiceDefinition.csdef that we will use to created the scaled-down package.

<!--?<span class="hiddenSpellError" pre="" data-mce-bogus="1"-->xml version="1.0" encoding="utf-16"?>
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   name="console-lift-and-shift"
                   xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="ConsoleWorkerRole" vmsize="ExtraSmall">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InputEndpoint name="HttpIn" protocol="tcp" port="80" />
    </Endpoints>
    <Runtime>
      <Environment>
        <Variable name="PORT">
          <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/Endpoints/Endpoint[@name='HttpIn']/@port" />
        </Variable>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  <!--<span class="hiddenSpellError" pre="" data-mce-bogus="1"-->WorkerRole>
</ServiceDefinition>

In order to create the scaled-down package for the Cloud Service, we must open Windows PowerShell ISE and navigat to the folder that contains the Cloud Service.

The Cloud Service for this blog post is at the following location:

C:\users\<user id>\Documents\demo-lift-and-shift\console-lift-and-shift

In the PowerShell console, we execute the following command to create a deployment package for the Cloud Service.

# Package the Cloud Service
Save-AzureServiceProjectPackage

This produces a package named cloud_package.cspkg. It is located in the folder that contains the Cloud Service.

Next, we must upload the cloud_package.cspkg file and the ServiceConfiguration.Cloud.cscfg file to Azure Blob Storage. This can be done from tools like Visual Studio, Azure Management Studio or PowerShell.

For this example, we will use PowerShell to rename and upload the Cloud Service artifacts (Package & Configurations) to Azure Blob Storage.

In order for this to work, we need to import the publish profile from our Microsoft Azure Subscription. PowerShell will then use it to connect to Microsoft Azure.

In the PowerShell console, execute the following commands.

# Opens a website that allows us to download the Publish Profile
# associated with our Microsoft Azure Subscription
Get-AzurePublishSettingsFile

# Save the publish settings files to your downloads folder
# Then import it using the following command.
Import-AzurePublishSettingsFile -PublishSettingsFile 'C:\Users\\Downloads\-credentials.publishsettings'

Then make sure to set our Azure subscription as the default subscription for the local PowerShell environment.

Select-AzureSubscription -Default -SubscriptionName 'my subscription name'

Before we upload anything to Azure Storage, we must first make sure that we have a provisioned Storage Account.
Using the following commands I will create a new Storage Account that will be used to store the Cloud Service artifacts.

New-AzureStorageAccount -StorageAccountName 'scaleupdowndemopkgs' `
                        -Label 'scale up down demo deployment packages' `
                        -Location 'East US' `
                        -Type 'Standard_GRS' `
                        -Verbose -WarningAction SilentlyContinue `
                        -ErrorVariable $e `

Now using the following PowerShell function, we will upload the artifacts to our new Azure Storage Account.

function Set-BlobContent
{
    Param
    (
        [Parameter(Mandatory=$true)]
        [string]
        $StorageAccountName,

        [Parameter(Mandatory=$true)]
        [string]
        $StorageContainer,

        [Parameter(Mandatory=$true)]
        [string]
        $FilePath,

        [Parameter(Mandatory=$true)]
        [string]
        $BlobName
    )

    # Setup the Storage Context that is used for Storage Operations

    $AccountKeys = Get-AzureStorageKey -StorageAccountName $StorageAccountName `
                                       -Verbose `
                                       -WarningAction Stop `
                                       -ErrorVariable $e `

    $StorageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName `
                                              -StorageAccountKey $AccountKeys.Primary `
                                              -Verbose `
                                              -WarningAction Stop `
                                              -ErrorVariable $e `

    try
    {
        Write-Verbose('Try create Container')
        $newcontainer = New-AzureStorageContainer -Context $StorageContext `
                                              -Container $StorageContainer `
                                              -Verbose `
                                              -WarningAction Ignore `
                                              -ErrorVariable $e `
    }
    catch
    {
        Write-Verbose('Container already exists')
    }

    # Upload and rename the Cloud Service package to Blob Storage

    Set-AzureStorageBlobContent -File $FilePath `
                                -Container $StorageContainer `
                                -Context $StorageContext `
                                -Blob $BlobName `
                                -Force `
                                -Verbose `
                                -WarningAction Stop `
                                -ErrorVariable $e `
}

Uploading the Cloud Service package provides us with the chance to give the package file a meaningful name. For the purpose of this example, I renamed the package file to extra_small_vm_cloud_package.cspkg.

Set-BlobContent -StorageAccountName 'scaleupdowndemopkgs' `
                -StorageContainer 'packages' `
                -FilePath 'C:\Users\\Documents\demo-lift-and-shift\console-lift-and-shift\cloud_package.cspkg' `
                -BlobName 'extra_small_vm_cloud_package.cspkg'

Then we must update the ServiceConfiguration.Cloud.cscfg file and set the instance count to 1. This will allow the Cloud Service to scale-in the as it scales-down.

<!--?<span class="hiddenSpellError" pre="" data-mce-bogus="1"-->xml version="1.0" encoding="utf-16"?>
xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serviceName="lift-and-shift-demo" osFamily="4" osVersion="*" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="ConsoleWorkerRole">
    <ConfigurationSettings />
    <Instances count="1" />
    <Certificates />
  </Role>
</ServiceConfiguration>

Using the same PowerShell function we will upload the updated ServiceConfiguration.Cloud.cscfg and give it a meaningful name. For the purpose of this example, I renamed the package file to ServiceConfiguration.Scaled-In-Cloud.cscfg.

Set-BlobContent -StorageAccountName 'scaleupdowndemopkgs' `
                -StorageContainer 'packages' `
                -FilePath 'C:\Users\\Documents\demo-lift-and-shift\console-lift-and-shift\ServiceConfiguration.Cloud.cscfg' `
                -BlobName 'ServiceConfiguration.Scaled-In-Cloud.cscfg'

To create the second package, we must edit the ServiceDefinition.csdef file and set the vmsize of the ConsoleWorkerRole to ExtraLarge. The following is the updated ServiceDefinition.csdef that will be used to created the scaled-up package.

<?xml version="1.0" encoding="utf-16"?>
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   name="console-lift-and-shift"
                   xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="ConsoleWorkerRole" vmsize="ExtraLarge">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InputEndpoint name="HttpIn" protocol="tcp" port="80" />
    </Endpoints>
    <Runtime>
      <Environment>
        <Variable name="PORT">
          <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/Endpoints/Endpoint[@name='HttpIn']/@port" />
        </Variable>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  </WorkerRole>
</ServiceDefinition>

In the PowerShell console, we must execute the following command to create the deployment package for the Cloud Service.

# Package the Cloud Service
Save-AzureServiceProjectPackage

This will produce a package named cloud_package.cspkg. It will be created in the folder that contains the Cloud Service.

Before we upload the new package to Azure Storage, update the ServiceConfiguration.Cloud.cscfg file and set the instance count to 5. This will allow the Cloud Service to scale-out as it scales-up.

<?xml version="1.0" encoding="utf-16"?>
xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serviceName="lift-and-shift-demo" osFamily="4" osVersion="*" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="ConsoleWorkerRole">
    <ConfigurationSettings />
    <Instances count="5" />
    <Certificates />
  </Role>
</ServiceConfiguration>

Uploading the Cloud Service package provides us with the chance to give the package file a meaningful name. For the purpose of this example, I renamed the package file to extra_large_vm_cloud_package.cspkg. I also renamed the updated ServiceConfiguration.Cloud.cscfg to ServiceConfiguration.Scaled-Out-Cloud.cscfg.

Set-BlobContent -StorageAccountName 'scaleupdowndemopkgs' `
                -StorageContainer 'packages' `
                -FilePath 'C:\Users\\Documents\demo-lift-and-shift\console-lift-and-shift\cloud_package.cspkg' `
                -BlobName 'extra_large_vm_cloud_package.cspkg'

Set-BlobContent -StorageAccountName 'scaleupdowndemopkgs' `
                -StorageContainer 'packages' `
                -FilePath 'C:\Users\\Documents\demo-lift-and-shift\console-lift-and-shift\ServiceConfiguration.Cloud.cscfg' `
                -BlobName 'ServiceConfiguration.Scaled-Out-Cloud.cscfg'

Automate and Schedule!

Microsoft Azure Automation is all about automating frequent, time-consuming, and error-prone cloud management tasks. Azure Automation helps you spend more of your time focused on work that adds business value. By reducing errors and boosting efficiency, it can also help lower your operational costs.

Setting this up for the first time can be tricky, but don’t worry, there are some great resources that will help us overcome this challenge.

By default, Microsoft Azure Automation does not have access to your Azure resources. It must be configured appropriately and to do so, we should followed the steps described in Azure Automation: Authenticating to Azure using Azure Active Directory.

Note: Those who have a microsoft.onmicrosoft.com subscription may hit a wall using this method. If you know of a workaround please share it through the comment section on this post.

OK. Now we’re ready to build a Windows PowerShell workflow (runbook). Having no prior experience with runbooks, I peaked at pre-built samples from the Runbook Gallery. This Gallery allows you to browse and import runbooks to your Automation account from the Azure Management Portal. This gave me a great deal of insight as to what I was up against and got me asking a bunch of questions. Here are a few of the resources that answered most of my questions.

• Get started with Azure Automation
• Microsoft Azure Automation on MSDN
• Authoring Automation Runbooks
• Runbook Concepts
• Runbook Execution in Azure Automation
• Azure Automation: Controlling Runbook Streams for Testing and Troubleshooting
• Azure Automation: Runbook Input, Output, and Nested Runbooks
• Azure Automation in Depth: Runbook Authoring
• Microsoft Azure Automation Forums

Armed with these resources I wrote the following runbook. It deploys a Cloud Service package and desired configurations from Azure Storage to the production slot of a Cloud Service. If the Cloud Service does not exist, it creates it. Then it tries to update the Cloud Service. If that fails, it created a brand new deployment.

workflow Invoke-DeployPackage
{
   param
   (
      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name given to the Windows PowerShell Credentials located in the Assets of this Azure Automation instance')]
      [String]
      $AzureCredentialsName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Microsoft Azure Subscription that contains the resources your wish to deploy')]
      [String]
      $AzureSubscriptionName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the service you wish to deploy')]
      [String]
      $ServiceName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The location of the service your with to deploy I.E. "East US"')]
      [String]
      $ServiceLocation,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Microsoft Azure storage account that contains the Cloud Service Packages')]
      [String]
      $StorageAccountName,

      [parameter(Mandatory=$true, `
                 HelpMessage = 'The name of the Blob container that contains the Cloud Service Packages')]
      [String]
      $StorageContainerName,

      [parameter(Mandatory=$true, HelpMessage = 'The name of the Cloud Service Package file')]
      [String]
      $PackageBlobName,

      [parameter(Mandatory=$true, HelpMessage = 'The name of the Cloud Service Configurations file')]
      [String]
      $ConfigurationBlobName
    )

    $VerbosePreference = 'Continue'

    # Mark the start time of the script execution
    $StartTime = Get-Date

    $DeploymentLable = $ServiceName + ' (' + $StartTime +')'

    Write-Verbose ('Connecting to Microsoft Azure')

    $Credentials = Get-AutomationPSCredential `
                       -Name $AzureCredentialsName

    Add-AzureAccount `
       -Credential $Credentials

    InlineScript{

        Write-Verbose ('Selecting Azure Subscription')

        Select-AzureSubscription -SubscriptionName $Using:AzureSubscriptionName

        $StorageAccount = (Get-AzureStorageAccount -StorageAccountName $Using:StorageAccountName).Label

        Write-Verbose ('Setting the Azure Subscription and Storage Accounts')

        Set-AzureSubscription `
            -SubscriptionName $Using:AzureSubscriptionName `
            -CurrentStorageAccount $StorageAccount

        Write-Verbose ('[Start] Validating Azure cloud service environment {0}' -f $Using:ServiceName)

        try
        {
            $CloudService = Get-AzureService `
                                -ServiceName $Using:ServiceName

            Write-Verbose ('cloud service {0} in location {1} exist!' -f $Using:ServiceName, $Using:ServiceLocation)
        }
        catch
        {
            #Create
            Write-Verbose ('[Start] creating cloud service {0} in location {1}' -f $Using:ServiceName, $Using:ServiceLocation)

            New-AzureService `
                -ServiceName $Using:ServiceName `
                -Location $Using:ServiceLocation

            Write-Verbose ('[Finish] creating cloud service {0} in location {1}' -f $Using:ServiceName, $Using:ServiceLocation)
        }

        Write-Verbose ('[Finish] Validating Azure cloud service environment {0}' -f $Using:ServiceName)

        $TempFileLocation = "C:\$Using:ConfigurationBlobName"

        Write-Verbose ('Downloading Service Configurations from Azure Storage')

        Get-AzureStorageBlobContent `
            -Container $Using:StorageContainerName `
            -Blob  $Using:ConfigurationBlobName `
            -Destination $TempFileLocation `
            -Force

        Write-Verbose('Downloaded Configuration File: '+ $TempFileLocation)

        Write-Verbose('Getting Package Url from Azure Storage: '+ $Using:PackageBlobName)

        $blob = $(Get-AzureStorageBlob -Blob $Using:PackageBlobName -Container $Using:StorageContainerName)

        $PackageUri = $blob.ICloudBlob.Uri.AbsoluteUri

        Write-Verbose('Package Url: '+ $PackageUri)

        try
        {
            Write-Verbose('Attempting to Update an Existing Deployment')
            Set-AzureDeployment `
                -Package $PackageUri `
                -Configuration $TempFileLocation `
                -Slot Production `
                -Mode Simultaneous `
                -Label $Using:DeploymentLable `
                -ServiceName  $Using:ServiceName `
                -Upgrade `
                -Force `
                -Verbose

        }catch
        {
            Write-Output $error

            Write-Verbose('Attempting to Deploy the service')

            New-AzureDeployment `
                -Package $PackageUri `
                -Configuration $TempFileLocation `
                -Slot Production `
                -Label $Using:DeploymentLable `
                -ServiceName  $Using:ServiceName `
                -Verbose
        }
    }
}

Using your browser let’s navigated to http://manage.windowsazure.com and log into our Azure Account. From this management portal I will show you how to schedule your runbook to execute once at 8 PM to scale-down and once at 6 AM to scale-up.

From the bottom of the management portal click on the + button and make the selections seen below. Then create a new runbook whose name is the same as the workflow listed above (for this to work, both names must match). For the purpose of this blog post, I also opted to create a new Automation Account. This account will be used to host the runbook, schedules and assets.

Once the Automation Account and Runbook have been created, navigate to the new Automation Account and click on the Assets tab.

Click on ADD SETTING and then on ADD CREDENTIAL

Select Windows PowerShell Credential from the CREDENTIAL TYPE dropdown. Then give this asset a name. It will be used by the runbook, to retrieve the required credentials that provide access the Azure Storage Account that contains the packages and configurations of the Cloud Service.

Enter the user name and password for the Azure Active Directory user that was created earlier in this post. This user must be co-administrator to your Azure Subscription.

Once the asset is created, it will appear in the assets list. This list is composed of Connections, Credentials, Variables and Schedules that can be used or linked to runbooks.

Navigate to the RUNBOOKS tab and select the Invoke-DeployPackage runbook.

Navigate to the AUTHOR tab. The DRAFT tab should be active by default. By now the runbook is composed of an empty workflow. Replace it with the full runbook script listed above. Click SAVE and then TEST. Provide all the necessary information and see whether the runbook is able to deploy the Cloud Service. Once everything look OK, click PUBLISH.

Navigate to the SCHEDULE tab and click on LINK TO A NEW SCHEDULE.

There are two schedules that need to be created. The first one is to scale-down the Cloud Service. Give the schedule a meaningful name and description.

Then configure the schedule. As originally stated, we want to scale-down everyday at 8 PM.

Provide the configuration values for the scheduled execution of the Invoke-DeployPackage runbook.

Once the schedule is created, it will be listed under the SCHEDULE tab.

It is possible to go back and view the schedule’s details. Click on VIEW DETAILS located at the bottom of the list.

Now it’s time to create the scale-up schedule. Click on LINK located at the bottom of the schedule list and select Link to a new Schedule.

To create the scale-up schedule, the wizard will walk through the same screens as it did for the scale-down schedule. Be sure to provide a meaningful name and description. These values can be quite useful when the list of schedules gets to be lengthy.

As previously stated, we want to scale-up the Cloud Service everyday at 6:00 AM.

Provide the configuration values for the scheduled execution of the Invoke-DeployPackage runbook. This time, specify the extra large package and configuration file names.

The schedule will appear in the schedule list. Note that this is a great place to get information about the schedule’s next execution time.

Once you navigate back to the DASHBOARD tab, you will notice that 2 schedules are active.

Within 24 hours, the DASHBOARD will light up with useful information about the execution of the runbooks. In this screen capture, we can observe that I had two successful executions and that I consumed 2 minutes of the available 500 minutes. This runbook is currently scaled at the free tier and can easily be scaled up if I need more execution time.

Wrapping it up

This was a long one, but I hope it can help you through this process. Getting this setup for the first time required that I read up on runbooks and PowerShell. It also required me to do a lot of testing, and I found that using the portal slowed me down. So I decided to develop and test my runbook from Windows PowerShell ISE. This allowed me to go through development cycles at a much faster rate.

Configuring the Credentials necessary for Microsoft Azure Automation also took me some time to figure out because I was using the wrong Azure Active Directory.

In summary, where are the steps required to setup this scenario.

1. Create the Cloud Service packages
   1. The first with the scaled-up VM size
   2. The second with the scaled-down VM size
2. Prepare 2 Cloud Service configuration files
   1. The first with the scaled-in instance count
   2. The second with the scaled-out instance count
3. Upload the Cloud Service packages and configuration files to Azure Storage
4. Create an Automation account
5. Setup Azure Automation: Authenticating to Azure using Azure Active Directory
6. Author the runbook (draft, test and publish)
7. Schedule therunbook
   1. Once to scale-up everyday at 6 AM
   2. Once to scale-down everyday at 8 PM

How does this scenario translate to lower operational costs?

The following comparison uses the pay-as-you-go pricing from December 2014

Let’s take a look at the operational cost difference between both scenarios

I think the numbers speak for themselves. At this time, it’s important to note that scaling dynamically using features like Auto Scaling and scaling based on a schedule can dramatically affect your overall operational costs. Furthermore, Azure gives us the ability to scale up and down as well as in and out. Finding the right combination can enable us to provide the appropriate amount of resources to satisfy the application’s needs.

Be careful, over provisioning means overspending. On the other hand, under provisioning can result in decreased revenue and dissatisfied customers. It can even push customers to your competitors. Use telemetry data to identify the amount of resources required to strike a balance point between operational costs, availability and responsiveness of your application.

Moving to Microsoft Azure

Every time I dig a little deeper into Azure, I’m amazed at how much there is to know. Having done a few projects with Cloud Services in the past, I thought it would be interesting to see if it was possible to lift and shift a Console Application into an Azure Worker Role.

Lift and Shift: The action of moving a workload to a new environment, without altering the application’s code.

Prerequisites

☁ Tools

Using the Microsoft Web Platform Installer install the latest Microsoft Azure PowerShell with Microsoft Azure SDK.

☁ Microsoft Azure Account

In order to deploy your workload to Microsoft Azure, be sure to have an account. You can create a Free Trial Account on the azure.com website.

Creating a Worker Role without Visual Studio

A Worker Role is a role that is useful for generalized development, and may perform background processing for a Web Role. When you have a need for a background process that performs long running or intermittent tasks, you should use this role.

Let’s fire-up Windows PowerShell ISE and navigate to the folder where you want to create your Cloud Service. For this blog post, I created a new folder, demo-lift-and-shift, in my Documents.

C:\users\<user id>\Documents\demo-lift-and-shift

In the PowerShell console, execute the following commands.

# Create the Cloud Service
New-AzureServiceProject -ServiceName 'console-lift-and-shift' -Verbose

# Create the Worker Role
Add-AzureWorkerRole -Name 'ConsoleWorkerRole' -Instances 1 -Verbose

These two commands will produce the following files and documents. This represents the Cloud Service project. It contains the configurations and the Cloud Service definition.

The ServiceDefinition.csdef was created by PowerShell when we executed the New-AzureServiceProject command. Then it was updated when we executed the Add-AzureWorkerRole. The following is the unchanged version of the service definition. Since it is missing a few pieces, I will show you what to change in order to be able to deploy your brand-new service to Microsoft Azure.

The ServiceDefinition.csdef file contains the metadata that is required by the Azure environment for the requirements of your cloud service, including what roles it contains. This file also contains configuration settings that apply to all instances. These configuration settings can be read at runtime using the Azure Service Hosting Runtime API. This file cannot be updated while your service is running in Azure.

<?xml version="1.0" encoding="utf-16"?>
<ServiceDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   name="console-lift-and-shift"
                   xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="ConsoleWorkerRole">
    <Startup>
      <Task commandLine="setup_worker.cmd &gt; log.txt" executionContext="elevated">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="RUNTIMEID" value="" />
          <Variable name="RUNTIMEURL" value="" />
        </Environment>
      </Task>
      <Task commandLine=".\startup.cmd &gt; startup_log.txt" executionContext="elevated" />
    </Startup>
    <Endpoints>
      <InputEndpoint name="HttpIn" protocol="tcp" port="80" />
    </Endpoints>
    <Runtime>
      <Environment>
        <Variable name="PORT">
          <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/Endpoints/Endpoint[@name='HttpIn']/@port" />
        </Variable>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="worker.cmd" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
  </WorkerRole>
</ServiceDefinition>

Specify a Role (Virtual Machine) size

The size of the role determines the number of CPU cores, the memory capacity, and the local file system size that is allocated to a running instance. The following

• ExtraSmall (1 CPU, 768 MB RAM, 19 GB DISK)
• Small (1 CPU, 1.75 GB RAM, 224 GB DISK)
• Medium (2 CPU. 2.5 GB RAM, 489 GB DISK)
• Large (4 CPU 7 GB RAM, 999 GB DISK)
• ExtraLarge (8 CPU, 14 GB RAM, 2039 GB DISK)

Considerations to help you decide on the correct size for your workload

1. Instances can now be configured to use a D-series VM. These are designed to run applications that demand higher compute power and temporary disk performance. D-series VMs provide faster processors, a higher memory-to-core ratio, and a solid-state drive (SSD) for the temporary disk. For details, see the announcement on the Azure blog, New D-Series Virtual Machine Sizes.
2. Web roles and worker roles require more temporary disk space than Azure Virtual Machines because of system requirements. The system files reserve 4 GB of space for the Windows page file, and 2 GB of space for the Windows dump file.
3. The OS disk contains the Windows guest OS and includes the Program Files folder (including installations done via startup tasks unless you specify another disk), registry changes, the System32 folder, and the .NET framework.
4. The local resource disk contains Azure logs and configuration files, Azure Diagnostics (which includes your IIS logs), and any local storage resources you define.
5. The apps (application) disk is where your .cspkg is extracted and includes your website, binaries, role host process, startup tasks, web.config, and so on.

More details about Virtual Machine and Cloud Service Sizes for Azure can be found in the official Azure documentation.

For this example we will use a Medium sized role (Virtual Machine) for our Cloud Service.

Replace

<WorkerRole name="ConsoleWorkerRole">

By

<WorkerRole name="ConsoleWorkerRole" vmsize="Medium">

Specify a Guest OS Version

Guest OS families and versions have a release date, a disabled date, and an expiration date. As of the release date, a Guest OS version can be manually selected in the management portal. A Guest OS is removed from the management portal on or after its “disabled” date. It is then “in transition” but is supported with limited ability to update a deployment. The expiration date is when a version or family is scheduled to be removed from the Azure system completely. Cloud services still running on a version when it expires will be stopped, deleted or force upgraded to a newer version, as detailed in the Azure Guest OS Supportability and Retirement Policy. Microsoft supports at least two recent versions of each supported Guest OS family.

Available Families as of December 2014

• FAMILY 4
  Windows Server 2012 R2
  Supports .NET 4.0, 4.5, 4.5.1, 4.5.2
• FAMILY 3
  Windows Server 2012
  Supports .NET 4.0, 4.5
• FAMILY 2
  Windows Server 2008 R2 SP1
  Supports .NET 3.5, 4.0

More details about Azure Guest OS Releases can be found in the official Azure documentation.

The ServiceConfiguration.{Environment}.cscfg was created by PowerShell when we executed the Add-AzureWorkerRole PowerShell command.

The configuration of the settings for your cloud service is determined by the values in the ServiceConfiguration.cscfg file. You specify the number of instances that you want to deploy for each role in this file. The values for the configuration settings that you defined in the service definition file are added to the service configuration file. The thumbprints for any management certificates that are associated with the cloud service are also added to the file. The Azure Service Configuration Schema (.cscfg File) provides the allowable format for a service configuration file.

The following is the unchanged version of the ServiceConfiguration.Cloud.cscfg file. Since it is missing a few pieces, I will show you what to change in order to be able to deploy your brand-new service to Microsoft Azure.

<?xml version="1.0" encoding="utf-16"?>
<ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                      serviceName="console-lift-and-shift"
                      osFamily="2"
                      osVersion="*"
                      xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="ConsoleWorkerRole">
    <ConfigurationSettings />
    <Instances count="1" />
    <Certificates />
  </Role>
</ServiceConfiguration>

For this example we will use Family 4 because our Console Application is built using .Net 4.5.

Replace

<ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                      serviceName="console-lift-and-shift"
                      osFamily="2"
                      osVersion="*"
                      xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">

By

<ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                      serviceName="console-lift-and-shift"
                      osFamily="4"
                      osVersion="*"
                      xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">

We left osVersion=”*” to allow the Azure Fabric to continuously patch the Guest OS.

Packaging The Console Application as a Cloud Service

The ConsoleWorkerRole folder contains the files and documents that compose our new Azure Worker Role

Place the files that compose the Console Application in this folder. The example used for this Blog is an application that writes messages to an Azure Storage Queue.

Be sure to include the application’s dependencies such that nothing stops the application from running once it is deployed to an Azure Worker Role. If, for some reason, the application does not seem to run as expected, refer to my earlier blog post that shared techniques to investigate and debug Azure Roles.

Before we package the Azure Cloud Service, we must update the worker.cmd file. This file contains the application loop that is executed by the Azure Worker role. The following is the initial version of the file.

:workerLoop

:: Do work
ping 123.45.67.89 -n 1 -w 1000000 > nul

goto workerLoop

In order to start the application, replace the ping command by a direct call to the application’s executable. Every time the application terminates, it will be restarted. This is the first of many mechanisms in place to ensure that the application is resilient and available.

:workerLoop

:: This EXE will be re-invoked every time it completes it's execution
DemoAppWritingToAQueue.exe

goto workerLoop

Be sure to select the ANSI Encoding then you save your changes. Failing to do so may prevent the CMD script from executing.

Now that we have the Console Application wired up to the Worker Role, it’s time to package the Cloud Service. In Windows PowerShell ISE and navigate back to the folder that contains the Cloud Service.

The Cloud Service for this blog post is at the following location:

C:\users\<user id>\Documents\demo-lift-and-shift\console-lift-and-shift

In the PowerShell console, execute the following commands.

# Package the Cloud Service
Save-AzureServiceProjectPackage

This produces a package named cloud_package.cspkg, which is required for the service’s deployment.

Deploying the Cloud Service

This step has a prerequisite. We must start by importing a publish profile from our Microsoft Azure Subscription. PowerShell will use it to publish our Cloud Service to our Microsoft Azure.

In the PowerShell console, execute the following commands.

# Opens a website that allows us to download the Publish Profile
# associated with our Microsoft Azure Subscription
Get-AzurePublishSettingsFile

Save the publish settings files to your downloads folder then import it using the following command.

Import-AzurePublishSettingsFile -PublishSettingsFile 'C:\Users\<user id>\Downloads\<subscription name>-credentials.publishsettings'

We are now ready to publish our brand-new Cloud Service. The following command will create a new storage account to upload our package. This is the default behavior because we did not provide a storage account for the command. We specified a name for the service, a location (datacenter), a slot (production or staging) and a deployment name. For more information about the Publish-AzureServiceProject command please refer to the official documentation.

Publish-AzureServiceProject -ServiceName 'lift-and-shift-demo' 
                            -Location 'East US' 
                            -Slot 'Production' 
                            -DeploymentName '2014-12-14-15-46' 
                            -Verbose

If the Publish-AzureServiceProject command is executed with the -Verbose flag, it will produce a detailed output that allows you to understand what is happening. It’s also very useful when you need to debug a failed deployment.

WARNING: Publishing lift-and-shift-demo to Microsoft Azure. This may take several minutes...
WARNING: 3:47:10 PM - Preparing runtime deployment for service 'lift-and-shift-demo'
WARNING: 3:47:11 PM - Verifying storage account 'liftx2dandx2dshiftx2ddem'...
WARNING: 3:47:43 PM - Preparing deployment for lift-and-shift-demo with Subscription ID: dfbedef7-xxxx-xxxx-xxxx-650ca03208ee...
WARNING: 3:47:47 PM - Connecting...
WARNING: 3:47:48 PM - Creating...
WARNING: 3:47:49 PM - Created hosted service 'lift-and-shift-demo'.
WARNING: 3:47:49 PM - Uploading Package to storage service liftx2dandx2dshiftx2ddem...
WARNING: 3:47:56 PM - Starting...
WARNING: 3:48:30 PM - Created Deployment ID: 03e36c62a5f74d5f9ebd0a2ec0b16880.
WARNING: 3:48:30 PM - Initializing...
WARNING: 3:51:49 PM - Instance ConsoleWorkerRole_IN_0 of role ConsoleWorkerRole is busy.
WARNING: 3:52:22 PM - Instance ConsoleWorkerRole_IN_0 of role ConsoleWorkerRole is ready.
WARNING: 3:52:23 PM - Created Website URL: http://lift-and-shift-demo.cloudapp.net/.

PersistentVMDowntime : Microsoft.WindowsAzure.Commands.Utilities.CloudService.Model.PersistentVMDowntimeInfo
Name                 : 2014-12-14-15-46
DeploymentSlot       : Production
PrivateID            : 03e36c62a5f74d5f9ebd0a2ec0b16880
Status               : Running
Label                : lift-and-shift-demo
Url                  : http://lift-and-shift-demo.cloudapp.net/
Configuration        : <?xml version="1.0" encoding="utf-16"?><ServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serviceName="lift-and-shift-demo" osFamily="4" osVersion="*"
                       xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">  <Role name="ConsoleWorkerRole">
                       <ConfigurationSettings />    <Instances count="1" />    <Certificates />  </Role></ServiceConfiguration>
RoleInstanceList     : {ConsoleWorkerRole}
UpgradeStatus        :
UpgradeDomainCount   : 1
RoleList             : {ConsoleWorkerRole}
SdkVersion           : 2.5.6496.10
Locked               : False
RollbackAllowed      : False
VirtualNetworkName   :
CreatedTime          : 2014-12-14 3:47:58 PM
LastModifiedTime     : 2014-12-14 3:52:23 PM
ExtendedProperties   : {}
Dns                  :
VirtualIPs           : {Microsoft.WindowsAzure.Commands.Utilities.CloudService.Model.VirtualIP}

Once the Cloud Service has completed its deployment, you can log into the https://manage.windowsazure.com portal and you will be able to monitor and alter its configurations.

Because this solution does not rely on the .NET Azure SDK and since the CMD script has an infinite loop, I did not see a recycling status for my Azure Worker Role when it wasn’t configured properly. In the previous screen capture, we can observe a high CPU, which is unexpected for my Console Application. This was because I had deployed my Cloud Service on Guest OS Family 2 instead of 4 and that the .NET Framework 4.5 was missing. This cause my Console Application to continuously fail to start. The CMD worker script kept restarting my Console Application and ate up the entire CPU. Consequently, I updated the ServiceConfiguration.Cloud.cscfg file with the correct Guest OS Family and redeployed the Cloud Service.

Out of curiosity, I configured the Remote Desktop Extension through the management portal and peaked into the provisioned instance. I was happy to find my Console Application running normally.

Bringing it All Together

Azure Worker Roles support a surprising variety of scenarios! Here’s a quick recap of the steps taken to Lift & Shift a Console Application to Microsoft Azure.

# Create the Cloud Service
New-AzureServiceProject -ServiceName 'console-lift-and-shift' -Verbose

# Create the Worker Role
Add-AzureWorkerRole -Name 'ConsoleWorkerRole' -Instances 1 -Verbose

Update the configurations files with the appropriate Guest OS Family and with the desired VM Size for your Cloud Service. Then copy the Console Application and its dependencies to the Role’s folder. Don’t forget to update the worker.cmd script file so that it executes the desired Console Application.

# Package the Cloud Service
Save-AzureServiceProjectPackage

# Publish the Package to Microsoft Azure 
Publish-AzureServiceProject -ServiceName 'lift-and-shift-demo' 
                            -Location 'East US' 
                            -Slot 'Production' 
                            -DeploymentName '2014-12-14-15-46' 
                            -Verbose

Doing Something
Is Better Than Doing Nothing at All

There are times where uncertainty is high, the process is not suitable and the risk of missed deadlines looms on the horizon.

This is when we find ourselves in a sort of no man’s land, where everyone is terrified to make the call. At this point it’s important to note that inaction is a collective decision that we must all live with for the remainder of the project. More often than not, doing nothing puts your team on edge and produces an extraordinary amount of avoidable stress.

A manager once told me that even if you make the best decision today, chances are that it will be revised at a later time. From this I understood the importance of balancing facts, urgency and foresight. Don’t try to make the ultimate decision that will cover all your bases. Failing at making the best choice today, will provide you with insight that will influence subsequent decisions. This process is iterative and must be driven by a vision.

The important thing to remember here is that attaining your goal is more important than how you reach you goal. Keep your eye on the ball.

On the path to achieving a goal, your team may adjust and course correct multiple times. Keep an eye out for irritants, blockers, delays and process issues. Address them quickly and keep moving forward. A decision made today may provide further details about the root cause. In turn, providing you with the tools necessary to keep your team motivated and on track.

I find Kanban to be very effective at identifying improvement opportunities. It highlights slowdowns and process issues. But most important of all, it allows you to continuously observe and adapt your process.

Teams that adopted this “can do” attitude have reaped the benefits of lower stress levels, better quality and better flow.

Encouraging teams to learn from their mistakes empowers individual team members to speak up and take ownership. This ultimately results in rich opportunities that would have gone unnoticed.

Ideas, solutions, challenges and opportunities have to originate from everyone on a team. I personally love discussions based on facts. These discussions typically start by exposing a challenge and ends with a plan of action. It’s important to keep these brief and on topic. The best way I have found to achieve this, is to be prepared. Schedule a talk a head of time. Detail the topics and be ready to make a call. Don’t worry, make the best call based on the facts at hand. Test your hypothesis and iterate.

Acting sooner rather than later will allow you to be agile and adapt to unforeseen events.

Remember, keep your eye on the ball and seize the opportunity to define your fate. Doing something is better than doing nothing at all.

Using StartsWith to Filter on RowKeys

There are many scenarios where filtering on partial RowKeys makes sense. One of these scenarios is Azure Diagnostics Log analysis where events are partitioned by time based PartitionKeys and by compound RowKeys. This allows us to filter and find information effectively.

Event RowKeys are composed of deployment IDs, role names, instance names, categories and other information:

8637d014bcf94452a1e48f393a11674b___Brisebois.WorkerRole___Brisebois.WorkerRole_IN_0___0000000001652031520___WADLogsLocalQuery

Querying WADLogsTable Effectively

The following example, shows how to target a specific table partition and filter events based on a StartsWith pattern.

var storageAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(connectionString);
var client = storageAccount.CreateCloudTableClient();

var table = client.GetTableReference("WADLogsTable");

// Querying Windows Azure Diagnostics by Partition for a partial RowKey
var query = new FindWithinPartitionStartsWithByRowKey("0635204061600000000", "8637d014bcf94452a");
var result = query.Execute(table);

The FindWithinPartitionStartsWithByRowKey query returned many events from my pet project ‘This day on Azure‘. As we can observe, the RowKey structure allows us to partially filter from left to right. This example shows data from a single Cloud Service. But there are times when many Cloud Service uses the same Azure Storage Account to persist Azure Diagnostics.

{Microsoft.WindowsAzure.Storage.Table.DynamicTableEntity}
    ETag: "W/\"datetime'2013-11-18T21%3A17%3A38.0626707Z'\""
    PartitionKey: "0635204061600000000"
    Properties: Count = 9
    RowKey: "8637d014bcf94452a1e48f393a11674b___Brisebois.WorkerRole___Brisebois.WorkerRole_IN_0___0000000001652031489___WADLogsLocalQuery"
    Timestamp: {2013-11-18 9:17:38 PM +00:00}

{Microsoft.WindowsAzure.Storage.Table.DynamicTableEntity}
    ETag: "W/\"datetime'2013-11-18T21%3A17%3A38.0626707Z'\""
    PartitionKey: "0635204061600000000"
    Properties: Count = 9
    RowKey: "8637d014bcf94452a1e48f393a11674b___Brisebois.WorkerRole___Brisebois.WorkerRole_IN_0___0000000001652031490___WADLogsLocalQuery"
    Timestamp: {2013-11-18 9:17:38 PM +00:00}

Filtering Within a Partition

In order to produce a StartsWith search pattern with Azure Table Storage queries, one must query based on RowKey (this also works for PartitionKeys) and set the lower bound greater or equal to the StartsWith pattern. Then we must increment the last character from the lower bound string in order to produce the upper bound of the query. This will produce the desired StartsWith capability that we all have come to cherish.

public class FindWithinPartitionStartsWithByRowKey
    : IQuery<CloudTable, List<DynamicTableEntity>>
{
    private readonly string partitionKey;
    private readonly string startsWithPattern;

    public FindWithinPartitionStartsWithByRowKey(string partitionKey,
        string startsWithPattern)
    {
        if (partitionKey == null)
            throw new ArgumentNullException("partitionKey");
        if (startsWithPattern == null)
            throw new ArgumentNullException("startsWithPattern");

        this.partitionKey = partitionKey;
        this.startsWithPattern = startsWithPattern;
    }

    public List<DynamicTableEntity> Execute(CloudTable model)
    {
        var query = new TableQuery();

        var length = startsWithPattern.Length - 1;
        var lastChar = startsWithPattern[length];

        var nextLastChar = (char) (lastChar + 1);

        var startsWithEndPattern = startsWithPattern.Substring(0, length) + nextLastChar;

        var prefixCondition = TableQuery.CombineFilters(
            TableQuery.GenerateFilterCondition("RowKey",
                QueryComparisons.GreaterThanOrEqual,
                startsWithPattern),
            TableOperators.And,
            TableQuery.GenerateFilterCondition("RowKey",
                QueryComparisons.LessThan,
                startsWithEndPattern)
            );

        var filterString = TableQuery.CombineFilters(
            TableQuery.GenerateFilterCondition("PartitionKey",
                QueryComparisons.Equal,
                partitionKey),
            TableOperators.And,
            prefixCondition
            );

        var entities = model.ExecuteQuery(query.Where(filterString));

        return entities.ToList();
    }
}

Filtering Within a Partition Range

This query is essentially the same as the previous query, but it allows us to extend our search to multiple partitions. Remember that smaller data sets produce faster results.

public class FindWithinPartitionRangeStartsWithByRowKey
    : IQuery<CloudTable, List<DynamicTableEntity>>
{
    private readonly string partitionLowerBound;
    private readonly string partitionUpperBound;
    private readonly string startsWithPattern;

    public FindWithinPartitionRangeStartsWithByRowKey(string partitionLowerBound,
        string partitionUpperBound,
        string startsWithPattern)
    {
        if (partitionLowerBound == null)
            throw new ArgumentNullException("partitionLowerBound");
        if (partitionUpperBound == null)
            throw new ArgumentNullException("partitionUpperBound");
        if (startsWithPattern == null)
            throw new ArgumentNullException("startsWithPattern");

        this.partitionLowerBound = partitionLowerBound;
        this.partitionUpperBound = partitionUpperBound;
        this.startsWithPattern = startsWithPattern;
    }

    public List<DynamicTableEntity> Execute(CloudTable model)
    {
        var query = new TableQuery();

        var length = startsWithPattern.Length - 1;
        var lastChar = startsWithPattern[length];

        var nextLastChar = (char) (lastChar + 1);

        var startsWithEndPattern = startsWithPattern.Substring(0, length) + nextLastChar;

        var prefixCondition = TableQuery.CombineFilters(
            TableQuery.GenerateFilterCondition("RowKey",
                QueryComparisons.GreaterThanOrEqual,
                startsWithPattern),
            TableOperators.And,
            TableQuery.GenerateFilterCondition("RowKey",
                QueryComparisons.LessThan,
                startsWithEndPattern)
            );

        var partitionFilterString = TableQuery.CombineFilters(
            TableQuery.GenerateFilterCondition("PartitionKey",
                QueryComparisons.GreaterThanOrEqual,
                partitionLowerBound),
            TableOperators.And,
            TableQuery.GenerateFilterCondition("PartitionKey",
                QueryComparisons.LessThanOrEqual,
                partitionUpperBound)
            );

        var filterString = TableQuery.CombineFilters(
            partitionFilterString,
            TableOperators.And,
            prefixCondition
            );

        var entities = model.ExecuteQuery(query.Where(filterString));

        return entities.ToList();
    }
}

Reusable Queries

In 2012, I wrote about building reusable and testable queries. This query object interface has been tested on several of my projects and has proven to clean up solutions. It provides a way to name questions so that we can discover them by using tools like R# (ReSharper). It also has the added benefit of promoting code reuse and strongly encourages self documenting code.

New developers can browse the solution and discover the application’s capabilities by reading file names. Each name describes a question that is used throughout the application.

I have used this technique on Cloud Services, Web APIs and even on Clients. And every time I put this into practice, the code got cleaner and easier to work with.

A query isn’t just SQL or LINQ, it’s a question that is asked about or to something. In this specific case, we are asking questions to Azure Table Storage, but this same pattern can be applied to business domains, databases and plain old objects.

public interface IQuery<in TModel, out TResult>
{
    TResult Execute(TModel model);
}

Don’t Frown on CSVs

In Microsoft Azure (Azure) CSV and Avro can help you deal with unpredictable amounts of data.

CSV files are surprisingly compact. They compresses really well and allows us to work with datasets that do not fit in RAM. This low-tech solution is often overlooked and frowned upon by developers who don’t get the opportunity to work with very large datasets.

Root cause analysis scenarios have led me to comb through several days’ worth of logs. More often than not, this represents gigabytes worth of data. Exporting application logs to a CSV files, I was able to parse and analyze them with minimal resources.

With these two options available to us, why should we consider using the CSVs? Well, Avro is still fairly new and unsupported by most systems. CSVs can be imported into Databases, Azure Table Storage, Hadoop (HDInsight), ERPs… And a slew of other systems with minimal effort. Heck, you can even open CSV files in Microsoft Excel!

CSVs are used for electronic data interchange (EDI) between companies. Of course, it requires the consumers to have knowledge about columns and intended data types. XML was a great idea. It brought data, structure and semantics together. Although the idea was good, it resulted in huge files.

Then came Avro. Avro is interesting, because like XML, it packs the structure and semantics with the data. It’s a very compact serialization protocol and even if it’s fairly new, it is natively supported by HDInsight.

So how do you choose between CSV and Avro? It all depends on intent. How will the files be consumed? Are these destined to be used for Big Data or are these files used to share datasets with partner companies? Who will consume the dataset?

Notice that I haven’t mentioned Json. That’s because Json isn’t as compact as CSV or Avro. I use Json and XML for service responses, for object graphs or for documents. I use CSV and Avro for archives, batching and cold storage.

Playing Around

Using CsvHelper I wrote 1 million times the same object in a CSV file. I’m using a Solid State Disk (SSD) and my results show that a similar operation using Avro isn’t much slower.

For both tests I used the following entity

[DataContract]
public class Data
{
    [DataMember]
    public int Id { get; set; }
    [DataMember]
    public string Name { get; set; }
    [DataMember]
    public string Email { get; set; }
    [DataMember]
    public double Lat { get; set; }
    [DataMember]
    public double Long { get; set; }
    [DataMember]
    public string Description { get; set; }
}

And the following instance

static readonly Data DataInstance = new Data
{
    Id = 1000,
    Description = "Established in 1985, Microsoft Canada Inc. " +
                  "is the Canadian subsidiary of Microsoft Corp." +
                  " (Nasdaq \"MSFT\"), the worldwide leader in software," +
                  " services and solutions that help people and" +
                  " businesses realize their full potential.",
    Email = "example@microsoft.com",
    Lat = 45.502399d,
    Long = -73.573653d,
    Name = "Microsoft, Montreal Canada"
};

To Write the entities using CsvHelper I used the following mapping

public class DataCsvMap : CsvClassMap<Data>
{
    public DataCsvMap()
    {
        Map(data => data.Id).Index(0)
                            .TypeConverterOption(NumberStyles.Integer);
        Map(data => data.Name).Index(1)
                              .TypeConverterOption(CultureInfo.InstalledUICulture);
        Map(data => data.Description).Index(2)
                                     .TypeConverterOption(CultureInfo.InstalledUICulture);
        Map(data => data.Lat).Index(3)
                             .TypeConverterOption(NumberStyles.AllowDecimalPoint);
        Map(data => data.Long).Index(4)
                              .TypeConverterOption(NumberStyles.AllowDecimalPoint);
        Map(data => data.Email).Index(5)
                               .TypeConverterOption(CultureInfo.InstalledUICulture);
    }
}

Created an empty file and executed the following to fill the file.

private static void WriteCsv(string filePath)
{
    var streamWriter = new StreamWriter(File.Create(filePath));
    var csvSerializer = new CsvSerializer(streamWriter);
    var writer = new CsvWriter(csvSerializer);

    writer.Configuration.HasHeaderRecord = true;
    writer.Configuration.RegisterClassMap(new DataCsvMap());

    writer.WriteHeader<Data>();
    for (var i = 0; i < 1000000; i++)
        writer.WriteRecord(DataInstance);

    writer.Dispose();
}

Then next part of my little experiment was reading all the entities sequentially from the file

private static void ReadCsv(string filePath)
{
    using (var file = File.Open(filePath, FileMode.Open, FileAccess.Read))
    {
        var reader = new CsvReader(new StreamReader(file));

        while (reader.Read())
        {
            var data = reader.GetRecord<Data>();
        }
    }
}

My results were surprising!

Writing  1 000 000 records to : C:\archive.csv

Archive Size 292 MB

CSV Write Completed in 192 milliseconds

CSV Read Completed in < 1 milliseconds

Then I used the Microsoft .NET Library for Avro to conduct the same experiment.

Using the following code and the entity instance from the CSV experiment I created and filled a new archive file.

private static void WriteAvro(string filePath)
{
    using (var file = File.Open(filePath, FileMode.Create, FileAccess.Write))
    {
        var container = AvroContainer.CreateWriter<Data>(file, Codec.Null);
        for (var i = 0; i < 1000000; i++)
        {
            var avroWriterBlock = container.CreateBlock();
            avroWriterBlock.Write(DataInstance);
            container.WriteBlock(avroWriterBlock);
        }
        container.Close();
        file.Flush();
    }
}

Then I read all the entities sequentially from the file in order to observe possible latency.

private static void ReadAvro(string filePath)
{
    using (var file = File.Open(filePath, FileMode.Open, FileAccess.Read))
    {
        var container = AvroContainer.CreateReader<Data>(file);
        while (container.MoveNext())
        {
            var data = container.Current.Objects;
        }
    }
}

Once again my results caught me by surprise. I was expecting that Avro would take more time to complete the serialization.

Writing  1 000 000 records to : C:\archive.avro

Archive Size 297 MB

Avro Write Completed in 211 milliseconds

Avro Read Completed in < 1 milliseconds

Although Avro performed admirably well, I believe that both formats answer to different scenarios. CSVs are a wonderful low-tech solution to a common challenge. Avro is emerging and is still limited in terms of adoption. Choosing the right format comes down to how you plan to consume the resulting dataset.

CSVs on Azure

When developing Cloud Services on Azure, CSVs can help keep operational costs under control by reducing the amount of resources required to process large datasets. Imagine a service whose purpose is to parse logs in order to extract telemetry. There are two approaches to this challenge.

The first is to load the dataset in memory and extract metrics. This can require a substantial amount of memory. Luckily, Microsoft announced the G-series (Godzilla) Virtual Machines which will provide more memory and more local Solid State Drive (SSD) storage. The largest Godzilla VM will offer 448 GB RAM and 6.5 TB of local SSD storage. But think of the price tag that comes with these reserved resources…

As developers we forget about the Virtual Machine’s local storage (temporary storage). This is where the second approach come into to play. Using CSVs we can stream large amounts of loosely structured data from Azure Table Storage to local storage. They allow us to consume datasets line by line with a very small memory footprint. When you take a step back and think about it, this approach resembles MapReduce. And because of this small memory footprint, we can get away with small A1 (1 CPU, 1.75 GB RAM, 224 GB HD) VM for about ~$63.27/mo. If the analysis requires more IOPS we can use a D1 (1 CPU, 3.5 GB RAM, 50GB SSD) VM for about ~$133.93/mo.

As developers we must strive to use resources responsibly. Finding ways to reduce the amount of resources necessary to support our applications, means that we can take on more load without adding more resources. Now, this is when things get interesting for everyone. By handling more load with the same amount of resources, you are effectively raising profit margins and creating new business opportunities.

The Greenfield to Brownfield Transition

There’s a special moment in every project where it transitions from a Greenfield (new) project to a Brownfield (legacy) project. This moment usually occurs close to the second release.

As developers, we love Greenfield projects. It’s a new adventure where creativity runs wild and free.

The transition of a project from Greenfield to a Brownfield is an event that we can all appreciate. It’s delicate and requires attention from everyone on the team.

During this transition the architecture starts to settle, patterns and practices start to immerge. New features now require developers to examine what can be reused. Quality Assurance personnel has expectations and can put more effort on exploratory testing. Team members are often called to contribute to new projects and knowledge transfer becomes crucial.

On Microsoft Azure (a.k.a Azure), this transition comes rather quickly. Azure is a platform that evolves at a very fast pace. Each release is accompanied with new features and services that create new opportunities. These opportunities drive teams to reassess their strategy in order to gain an edge on competitors.

Regular reassessment of a team’s strategy is crucial in modern development practices. The Application Lifecycle is often composed of an analysis phase, a build phase, a release phase and an observation phase.

On regular occasions, we forget about the observation phase. This is where Brownfield projects go bad, because this forgotten phase is an invaluable source of feedback. Feedback that must be used to drive subsequent efforts.

Types of Feedback

• Application Metrics – metrics are great at providing insight and context about a live application. They can be used to troubleshoot, monitor and identify areas that require more effort.
• Consumer Feedback – applications often use services like UserVoice to empower consumers to provide feedback about their experiences and suggestions about missing features. Remember, consumers use your application because it satisfies their needs. Missing features, quality issues and lack of support will drive them away.
• Development Metrics – development metrics including but not limited to test coverage, number of lines of code, number of defects, average number story points covered per sprint, cyclomatic complexity and number of defects versus number of new features are crucial key performance indicators (KPI) that help teams refine their practices.

The success of a project depends on a successful transition from a Greenfield project to a mature Brownfield project. This transition must be prepared adequately. Failure to do so usually results in developers demanding rewrites. Those who find themselves in this delicate situation are rarely happy. They lose interest, passion and fear the code base. Our industry has named this Fear Driven Development (FDD).

Preparing for the transition starts on day one. A solid organic architecture that follows best practices is the foundation for everything else. Architectures are hard to change. Therefore, it requires constant attention from everyone on the team. Developers, Architects, Team Leads, Testers and Management must be critical and raise issues as they come up.

YAGTNI (you aren’t going to need it) is a principle that helps to manage technical debt. Its premise lies in the practice of building only what you need for the current release. Building for future releases often results in rigidity that we must deal with for the project’s lifetime.

Rigidity robs us of future opportunities. I see this regularly, when project efforts are geared towards a database centric design. The database becomes the business domain and the application itself. A change to the database results in lots of effort because the entire application is affected. Good architectures strive to protect against this sort of rigidity by creating opportunities to use different storage strategies for different workloads.

Azure is all about opportunities. Using design principles like Dependency Inversion, teams build flexible designs that can evolve and adapt to an ever changing ecosystem.

Preserving a project’s agility is an ongoing process that starts on day one. Take time to address technical debt early. Iterate, assess and course correct on a regular basis. If your development process hinders you from producing quality software, try something different. Adapt your process as your team and project mature.

So the next time you’re invited to participate on a Brownfield, don’t panic. Iterate, assess and course correct. If you get invited to participate on a Greenfield project, tell yourself that it’s your chance to experience the transition. Your efforts and your open mind will make the difference.

Working with Microsoft Azure Resources

On September 22 2014, I had the pleasure of speaking to the MSDEVMTL community about working with Microsoft Azure Resources.

Microsoft Azure Resources include Blob Storage, Table Storage, Queue Storage, Service Bus, Virtual Machines, Cloud Services and SQL Database. During my talk, I introduced a couple of tools that allow us to work with these resources. Some tools are built by Microsoft others are built by companies like Cerebrata, Cloud Berry and Zudio.

I compared and contrasted Azure Explorer by Cerebrata, Cerebrata Azure Management Studio and the Azure tools found in Visual Studio. I spoke about my personal experiences and about how I chose which tool I use based on what I need to do.

For example, Azure Management Studio has a great feature that allows me to export a storage table to CSV. Another great feature mentioned during my talk was the ability to copy a deployed package (cscfg and cspkg) to blob storage. This is an awesome feature! Imagine starting on a new project and being asked to redeploy a Cloud Service to a new Azure subscription. The catch, is that the person you’re replacing, used to deploy directly from Visual Studio. You have no historical Packages and absolutely no guarantee that the deployed package is the latest and greatest. Chances are that the package is a few weeks behind the source… The only option that you have, is pulling the deployed package from the Azure deployment and using it to deploy to the new Azure Subscription.

Tip: Importing Azure Subscriptions to Azure Management Studio is easily done by importing Publish Settings. To download your Publish Settings, log into manage.windowsazure.com and select your desired subscription. Then browse to http://bit.ly/getmyazurecert.

Talk Slides

Releasing Memory Pressure

Traditionally, in .NET applications we adjust the Garbage Collector to operate in server mode by modifying the App.config or Web.config files. However, Worker Roles work differently because your code is actually hosted by the WaWorkerHost.exe process. So you must modify the WaWorkerHost.exe.config file instead of app.config and web.config.

This article covers the Worker Role. However, the same concepts and techniques may be used to change the Web Roles mode by changing WaWebHost.exe.config instead of WaWorkerHost.exe.config.

WaWorkerHost.exe.config is in %approot%\base\x64

What is ServerGC & Why is it Important?

The Server Garbage Collection process allows us to reclaims objects that are no longer being used, clears memory, and makes it available for future allocations. In contrast to the Work Station Garbage Collector mode, the Server Garbage Collection mode aggressively releases memory and can prevent your roles from running out of memory due to memory bloat.

Reducing memory bloat allows us to increased density, which typically reduces overall operational costs in a pay for what you consume environment. Furthermore, Background Server Garbage Collection can increase performance.

Conditions for a Garbage Collection Include

• The system has low physical memory.
• The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.
• The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing.

Enabling Background Server Garbage Collection

Leveraging Azure Role Starup Tasks, we call a CMD that executes a PowerShell script if the Azure Role is not executing in the Azure Emulator.

<!--
//*********************************************************
//
//    Copyright (c) Microsoft. All rights reserved.
//    This code is licensed under the Microsoft Public License.
//    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
//    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
//    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
//    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//*********************************************************
-->
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="ServerModeGC.Solution" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2013-10.2.2">
  <WorkerRole name="GCServer.Worker" vmsize="Large">
    <Startup>
      <Task commandLine="ServerGC.cmd" executionContext="elevated" taskType="simple">
        <Environment>
          <Variable name="UseServerGC" value="True" />
          <Variable name="UseBackgroundGC" value="True" />
        </Environment>
      </Task>
    </Startup>
  </WorkerRole>
</ServiceDefinition>

The following Command Line pulls the desired configuration and applies them by executing the PowerShell script. Note that Azure Roles executing inside the Azure Emulator will not execute the PowerShell script.

REM *********************************************************
REM
REM     Copyright (c) Microsoft. All rights reserved.
REM     This code is licensed under the Microsoft Public License.
REM     THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
REM     ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
REM     IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
REM     PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
REM
REM *********************************************************

REM Check if the script is running in the Azure emulator and if so do not run
IF "%IsEmulated%"=="true" goto :EOF 

If "%UseServerGC%"=="False" GOTO :ValidateBackground
If "%UseServerGC%"=="0" GOTO :ValidateBackground
SET UseServerGC="True"

:ValidateBackground
If "%UseBackgroundGC%"=="False" GOTO :CommandExecution
If "%UseBackgroundGC%"=="0" GOTO :CommandExecution
SET UseBackgroundGC="True"

:CommandExecution

PowerShell.exe -executionpolicy unrestricted -command ".\GCSettingsManagement.ps1" -serverGC %UseServerGC% -backgroundGC %UseBackgroundGC%

Exit /b

This PowerShell script will create the required configuration file if it is missing or update the existing WaWorkerHost.exe.config file.

<#
//*********************************************************
//
//    Copyright (c) Microsoft. All rights reserved.
//    This code is licensed under the Microsoft Public License.
//    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
//    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
//    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
//    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//*********************************************************
#>
Param
(
	$serverGC = $True,
	$backgroundGC = $True
)

[string]$configFilePath = "$(${env:RoleRoot})\base\x64\WaWorkerHost.exe.config"

function Create-ConfigFileIfNotExists
{
	# Only create the Xml document if it does not already exist
	if(-not (Test-Path -Path $configFilePath -PathType Leaf))
	{
		[System.Xml.XmlDocument]$document = New-Object System.Xml.XmlDocument

		# config file doesn't exist create a now one
 		[System.Xml.XmlDeclaration]$prolog = $document.CreateXmlDeclaration("1.0", "utf-8", $null)
 		[System.Xml.XmlNode]$child = $document.AppendChild($prolog)
		[System.Xml.XmlElement]$configurationElement = Append-ElementIfNotExists $document $document.DocumentElement "configuration"

		# Save a copy of the document
		$document.Save($configFilePath)
	}
}

function Load-ConfigFile
{
	[System.Xml.XmlDocument]$document = New-Object System.Xml.XmlDocument

	#Check if the document already exists and load it if it does not
	if(Test-Path -Path $configFilePath -PathType Leaf)
	{
		$document.Load($configFilePath)
	}

	return $document
}

function Append-ElementIfNotExists
{
	param
	(
		[System.Xml.XmlDocument]$document,
		[System.Xml.XmlElement]$parent,
		[string]$elementName
	)
	[System.Xml.XmlElement]$element = $null
	[System.Xml.XmlNode]$parentNode = $parent

	if($document -ne $null)
	{
		if($parentNode -eq $null)
		{
			$parentNode = $document
		}

		$element = $parentNode.SelectSingleNode("./$($elementName)")

		if($element -eq $null)
		{
			$element = $document.CreateElement($elementName)
			[System.Xml.XmlElement]$child = $parentNode.AppendChild($element)
		}
	}

	return $element
}

function Create-ElementStructureIfNotExists
{
	param
	(
		[System.Xml.XmlDocument]$document
	)
	[bool]$isSuccess = $false

	if($document -ne $null)
	{
		[System.Xml.XmlElement]$configurationElement = Append-ElementIfNotExists $document $null "configuration"

		if($configurationElement -ne $null)
		{
			[System.Xml.XmlElement]$element = Append-ElementIfNotExists $document $configurationElement "runtime"

			$isSuccess = $element -ne $null
		}
	}

	return $isSuccess
}

# Create the document if required
Create-ConfigFileIfNotExists

# Load the configuration file into the XML document
[System.Xml.XmlDocument]$configurationDocument = Load-ConfigFile

if($configurationDocument -ne $null)
{
	if(Create-ElementStructureIfNotExists $configurationDocument)
	{
		# All of the entries are on the runtime element
		[System.Xml.XmlElement]$runtimeElement = $configurationDocument.DocumentElement.SelectSingleNode('./runtime')

		if($runtimeElement -ne $null)
		{
			# Set the Server GC to enabled if requested
			[System.Xml.XmlElement]$serverGCElement = Append-ElementIfNotExists $configurationDocument $runtimeElement "gcServer"
			$serverGCElement.SetAttribute("enabled", $serverGC.ToString([System.Globalization.CultureInfo]::InvariantCulture).ToLower()) 

			# Set the concurrent GC to enabled if requested
			[System.Xml.XmlElement]$concurrentGCElement = Append-ElementIfNotExists $configurationDocument $runtimeElement "gcConcurrent"
			$concurrentGCElement.SetAttribute("enabled", $backgroundGC.ToString([System.Globalization.CultureInfo]::InvariantCulture).ToLower())
		}
	}

	# Save the document
	$configurationDocument.Save($configFilePath)
}

Adding the Files to the Solution

Add the command and PowerShell files to the Worker Role that will be elevated. Place both files in the root directory of the Worker Role. Set the “Build Action” to “Content”, and set “Copy to Output Directory” to “Copy Always”.

Find Out More

• Server Garbage Collection Mode in Microsoft Azure
• Enable Server GC mode for your worker role
• Fundamentals of Garbage Collection
• Garbage Collection and Performance

Personally, I’m all about Cloud Native Hyper-Scale applications. WOW! that’s a lot of buzz words in a row so let’s distill that to something more meaningful. I love the flexibility I get from Platform as a Service (PaaS). This model provides me with granular control over my application’s structure without thinking too much about the actual Virtual Machines (VMs).

The Azure ecosystem is rich with Software as a Service (SaaS) features that developers can use to build their own applications. SaaS provides tremendous value to developers and stakeholders because we as developers don’t have to take ownership. These are some of the SaaS features available developers on Azure today.

• Azure SQL Database
• Azure Table Storage
• Azure Blob Storage
• Azure Service Bus
• Azure API Management
• Azure DocumentDB
• Azure Search
• Azure Traffic Manager
• Azure Cache
• Azure Content Delivery Network (CDN)
• Azure Active Directory
• Azure Push Notification
• Azure Automation
• Azure Scheduler
• …

Every year, the list of services keep growing. Each service is built and maintained by Microsoft. It comes with Service Level Agreements (SLAs) and assurances around data and privacy, SLAs also means that Microsoft is responsible for support. In other words, it means that we as developers get to focus on adding business value to our application and let Microsoft worry about the details. As long as we consider performance targets while we design, we get to spend time on the code that matters most to our businesses.

What’s your Cloud flavor? Mine is like a buffet, I compose with PaaS, Saas, IaaS and Hybrid Cloud solutions to provide the best end user experiences.

The underlying connection was closed

For the past weeks I’ve used PowerShell to test REST APIs exposed over HTTPS. Everything was wonderful until I had to execute my script against a local instance of the service. My machine is not setup with proper certificates and PowerShell doesn’t see to like that. When ever I execute an Invoke-RestMethod command I get the following error message:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel

Looking around I was fortunate to find this one-liner that fixes everything!

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

By turning off certificate validation you are disabling security. Beware that this one-liner is a patch, and should not be used for anything else than development purposes.

These days I’m all about automation. As most of us are focused on Python, C# JavaScipt and Node I’m taking a different approach to Azure DocumentDB. This experiment’s goal is to facilitate the creation and seeding of DocumentDBs with very little effort from JSON documents stored an Azure Blob Storage container.

Meet DocumentDB

Azure DocumentDB is a NoSQL document database service designed from the ground up to natively support JSON and JavaScript directly inside the database engine. It’s the right solution for web and mobile applications when predictable throughput, low latency, and flexible query are key. Microsoft consumer applications like OneNote already use DocumentDB in production to support millions of users.

Getting Started

We need a DocumentDB and the best way to create one is to navigate to portal.azure.com. Then click on + NEW then Everything then Data, storage, cache, + backup and finally DocumentDB. Fill in all the required information and find something to do for the next 5 minutes (while you wait for Azure to provision your DocumentDB instance).

Creating the Database & Collection

This adventure initially started with me trying to use the REST API directly from PowerShell. I used available documentation and fiddler to produce the following test script. Unfortunately I was unable to generate a valid authorization token from the Get-Key function.

$DocumentDbUrl = 'https://{name}.documents.azure.com'

$DocumentDbKey = '{your key}'

$DocumentDbApiVersion = '2014-08-21'

function Get-Key {
  param
  (
    [System.String]
    $Verb,

    [System.String]
    $ResourceId = '',

    [System.String]
    $ResourceType,

    [HashTable]
    $Headers
  )

  $message = $($Verb + '\n' + $ResourceType + '\n' + $ResourceId +'\n' + $Headers.'x-ms-date' + '\n\n')

  $key = [System.Convert]::FromBase64String($DocumentDbKey)

  $hmacsha = new-object -TypeName System.Security.Cryptography.HMACSHA256 -ArgumentList (,$key) 

  $messageBytes =[Text.Encoding]::UTF8.GetBytes($message.ToLowerInvariant())
  $hash = $hmacsha.ComputeHash($messageBytes)
  $signature = [System.Convert]::ToBase64String($hash)

  return [System.Web.HttpUtility]::UrlEncode($('type=master&ver=1.0&sig=' + $signature))
}

function Add-Database{
  param
  (
    [System.String]
    $DbName
  )  

  $ResourceUrl = '/dbs'

  $date = Get-Date
  $utcDate = $date.ToUniversalTime()
  $dateString = $utcDate.ToString('r',[System.Globalization.CultureInfo]::InvariantCulture)

  $headers = @{
             'x-ms-date'= $dateString
              }

  $verb = 'POST'
  $resourceType ='dbs'
  $resourceId = [String]::Empty

  $authorization = Get-Key -Verb $verb -ResourceType $resourceType -ResourceId $resourceId -Headers $headers

  $headers.Add('authorization', $authorization)

  $body = '{"id":"'+$DbName+'" }'

  $url = $($DocumentDbUrl+$ResourceUrl) 

  $response = Invoke-RestMethod -Method $verb -Uri $url -Headers $headers -Body $body

  $response | Format-List
}

Add-Database -DbName 'DocumentDatabaseName'

Executing this consistently returns

Invoke-RestMethod : The remote server returned an error: (401) Unauthorized.

Not accepting this failure and wanting to setup my environments through PowerShell I decided to create my own Cmdlets.

Using Cmdlets to Wrap the .NET DocumentDB Client

After a couple hours of (401) Unauthorized, I decided to write a series of Cmdlets that wrap the .NET DocumentDB client library. This turned out to be faster than trying to figure out the APIs authorization token.

The Cmdlets

• New-Context – this builds a PowerShell object that contains the DocumentDB URL and Key
• Add-Database – this creates the Database in DocumentDB and returns the Database object from which we extract the Self Link.
• Add-DocumentCollection – this creates a new Document Collection and returns the Document Collection object from which we extract the Self Link
• Import-FromBlobStorageContainer – this reads all blobs from the target container and adds each blob as a new Document to the target Document Collection

Using the Cmdlets

Import-Module 'DocumentDB.Cmdlet' -Force

$ctx = New-Context -Url 'https://{name}.documents.azure.com' -Key '{key}'

$database = Add-Database -Context $ctx -Name '{name}'

$collection = Add-DocumentCollection -Context $ctx -DatabaseLink $database.SelfLink -Name '{name}'

Import-FromBlobStorageContainer -Context $ctx -DocumentCollectionLink $collection.SelfLink -ContainerName '{name}' -StorageConnectionString 'DefaultEndpointsProtocol=https;AccountName={name};AccountKey={key}'

Getting Started With Development

Cmdlets are created by inheriting from one of these classes:

• Cmdlet: A simple cmdlet using a .NET class derived from the Cmdlet base
  class. This type of cmdlet does not depend on the Windows PowerShell runtime
  and can be called directly from a .NET language.
• PSCmdlet: A more complex cmdlet based on a .NET class that derives from the
  PSCmdlet base class. This type of cmdlet depends on the Windows PowerShell
  runtime, and therefore executes within a runspace.

I decided to inherit from PSCmdlet, because I will not use these Cmdlets without Windows PowerShell. Once you’ve chosen a base class, the next challenge is finding the DLL that contains the System.Management.Automation namespace.

There are two ways to get this DLL. The first is to install the Windows SDK which will install the DLL in Program Files (x86)\Reference Assemblies\Microsoft\WindowsPowerShell\3.0. The second option is to run the following PowerShell command.

Copy ([PSObject].Assembly.Location) C:\

DocumentDB Connection Context

This Cmdlet creates a Context PSObject used to pass DocumentDB connection information for the other Cmdlets.

[Cmdlet(VerbsCommon.New, "Context")]
public class NewContext : PSCmdlet
{
    [Parameter(Position = 0,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "DocumentDB Url")]
    public string Url { get; set; }

    [Parameter(Position = 0,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "DocumentDB Key")]
    public string Key { get; set; }

    protected override void ProcessRecord()
    {
        var obj = new PSObject();
        obj.Properties.Add(new PSVariableProperty(new PSVariable("Url",Url)));
        obj.Properties.Add(new PSVariableProperty(new PSVariable("Key",Key)));
        WriteObject(obj);
    }
}

DocumentDB Cmdlet Base Class

This is the base class for the subsequent Cmdlets. It adds the Context object parameter at position 0.

public abstract class DocumentDbCmdlet : PSCmdlet
{
    [Parameter(Position = 0,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "DocumentDB Connection Information")]
    public PSObject Context { get; set; }
}

Add-Database

Using a Context and a Name, this Cmdlet uses the .NET DocumentDB client to create a new Database and returns the Database resource from which we can get a SelftLink.

[Cmdlet(VerbsCommon.Add, "Database")]
public class AddDatabase : DocumentDbCmdlet
{
    [Parameter(Position = 1,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Database Name")]
    public string Name { get; set; }

    protected override void ProcessRecord()
    {
        var url = base.Context.Properties["Url"].Value.ToString();
        var key = base.Context.Properties["Key"].Value.ToString();

        var client = new DocumentClient(new Uri(url), key);
        var task = client.CreateDatabaseAsync(new Database { Id = Name });

        task.Wait();

        WriteObject(task.Result.Resource);
    }
}

Add-DocumentCollection

Using a Context, a Name and a Database Link, this Cmdlet uses the .NET DocumentDB client to create a new Document Collection and returns the Document Collection resource from which we can get a SelftLink.

[Cmdlet(VerbsCommon.Add, "DocumentCollection")]
public class AddDocumentCollection : DocumentDbCmdlet
{
    [Parameter(Position = 1,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Database Link")]
    public string DatabaseLink { get; set; }

    [Parameter(Position = 2,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Collection Name")]
    public string Name { get; set; }

    protected override void ProcessRecord()
    {
        var url = base.Context.Properties["Url"].Value.ToString();
        var key = base.Context.Properties["Key"].Value.ToString();

        var client = new DocumentClient(new Uri(url), key);

        var task = client.CreateDocumentCollectionAsync(DatabaseLink,new DocumentCollection()
        {
            Id = Name
        });

        task.Wait();

        WriteObject(task.Result.Resource);
    }
}

Add-FromBlobStorageContainer

Using a Context, a Document Collection Link, a Storage Connection String and a Container Name, this Cmdlet uses the .NET DocumentDB client to create a new Document for each JSON file it finds in the Azure Blob Storage Container.

[Cmdlet(VerbsData.Import, "FromBlobStorageContainer")]
public class ImportFromBlobStorageContainer : DocumentDbCmdlet
{
    [Parameter(Position = 1,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Document Collection Link")]
    public string DocumentCollectionLink { get; set; }

    [Parameter(Position = 2,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Azure Storage Connection String")]
    public string StorageConnectionString { get; set; }

    [Parameter(Position = 3,
        Mandatory = true,
        ValueFromPipeline = true,
        ValueFromPipelineByPropertyName = true,
        HelpMessage = "Blob Container Name")]
    public string ContainerName { get; set; }

    protected override void ProcessRecord()
    {
        WriteProgress(new ProgressRecord(0, "Downloading List of Blobs", "Reading from " + ContainerName));

        var account = CloudStorageAccount.Parse(StorageConnectionString);
        var blobClient = account.CreateCloudBlobClient();
        var container = blobClient.GetContainerReference(ContainerName);

        var blobs = container.ListBlobs(useFlatBlobListing: true).ToList();

        var count = blobs.Count;

        string url = base.Context.Properties["Url"].Value.ToString();
        string key = base.Context.Properties["Key"].Value.ToString();

        var client = new DocumentClient(new Uri(url), key);

        var progress = new ProgressRecord(0, "Importing Blobs from " + ContainerName, "Importing Blobs from " + ContainerName);
        WriteProgress(progress);

        for (var i = 0; i < count; i++)
        {
            var blockBlob = blobs[i] as CloudBlockBlob;

            if (blockBlob == null) continue;

            progress.StatusDescription = blockBlob.Name;
            WriteProgress(progress);

            blockBlob.DownloadTextAsync()
                .ContinueWith(t => client.CreateDocumentAsync(DocumentCollectionLink, t.Result))
                .Wait();
        }

        var collection = client.ReadDocumentCollectionAsync(DocumentCollectionLink);
        collection.Wait();

        WriteObject(collection.Result.Resource);
    }
}

SnapIn

In order to test the Cmdlets in Windows PowerShell ISE, I implemented a Custom PowerShell SnapIn and placed the the Cmdlet DLLs in Documents\WindowsPowerShell\Modules. This folder is read by PowerShell to find Modules. In my case, it found a folder named DocumentDB.Cmdlet which I used to import the Module.

[RunInstaller(true)]
public class SnapIn : CustomPSSnapIn
{
    private Collection cmdlets = new Collection();
    private Collection providers = new Collection();
    private Collection types = new Collection();
    private Collection formats = new Collection();

    public SnapIn()
        : base()
    {
        cmdlets.Add(new CmdletConfigurationEntry("New-Context", typeof(NewContext), null));
        cmdlets.Add(new CmdletConfigurationEntry("Add-Database", typeof(AddDatabase), null));
        cmdlets.Add(new CmdletConfigurationEntry("Add-DocumentCollection", typeof(AddDocumentCollection), null));
        cmdlets.Add(new CmdletConfigurationEntry("Import-FromBlobStorageContainer", typeof(ImportFromBlobStorageContainer), null));
    }
    public override string Name
    {
        get { return "DocumentDB Cmdlets"; }
    }
    public override string Vendor
    {
        get { return "Alexandre Brisebois"; }
    }
    public override string VendorResource
    {
        get { return "Alexandre Brisebois"; }
    }
    public override string Description
    {
        get { return "DocumentDB Cmdlet example"; }
    }
    public override string DescriptionResource
    {
        get { return "DocumentDB Cmdlet example"; }
    }
    public override Collection Cmdlets
    {
        get { return cmdlets; }
    }

    public override Collection Providers
    {
        get { return providers; }
    }

    public override Collection Types
    {
        get { return types; }
    }

    public override Collection Formats
    {
        get { return formats; }
    }
}

When you start the Out-of-Process service be sure to specify the LocalSystem account.

SemanticLogging-svc.exe -s -a=LocalSystem

Failure to execute under the right account will prevent the Out-of-Process service from logging Events to Azure Table Storage.

• Semantic Logging Application Block – Out-of-process Service
• Semantic Logging Application Block – Source Code & Samples

What is SLAB?

The Semantic Logging Application Block (SLAB) provides a set of destinations (sinks) to persist application events published using a subclass of the EventSource class from the System.Diagnostics.Tracing namespace. Sinks include Azure table storage, SQL Server databases, file, console and rolling files with several formats and you can extend the block by creating your own custom formatters and sinks. The console sink is part of this nuget package. Other Sinks mentioned above are available as separate nuget packages. For the sinks that can store structured data, the block preserves the full structure of the event payload in order to facilitate analysing or processing the logged data.

What is Azure API Management

Microsoft Azure API Management is a service that helps protect your mission critical systems with authentication, rate limiting, quotas and caching to ease load under pressure. Rest easy knowing that only the partners, developers and applications you’ve authorized have access to your APIs and that those groups are acting in accordance with your policies. Find out more on Azure.com

REST APIs

Like many services on Azure, API Management provides us with a comprehensive REST API. This API allows us to manage Users, Groups, Products and Subscriptions.

Working on a multi-region solution, I was really happy to see these APIs. One of the recurring challenges I face everyday, is to replicate my efforts across multiple deployments sprawled over many Azure regions. Now the only way to do this effectively is to automating everything!

As of August 2014 API Management is still in public preview and is going through constant evolution. New features make their way to production and pieces fall together. The newly released REST APIs are just that, a piece that was missing. Wanting to reduce my workload I decided to create a PowerShell Module to help automate some of my repetitive tasks.

Note: The API does not allow you to define APIs, Representations. The Developer Portal CMS is not accessible through these APIs. Although these are things I would love to interact with through the REST API, I’m hopeful that something will come along.

Before we start, there are a couple things we need to do. First we need to activate the Management APIs on our API Management service. Then we need to generate an access token. I opted for the manual process which you can follow on the Azure API Management REST API Authentication page.

Then in our PowerShell ISE set both the REST API URL and the Access Token to global variable and we’re good to go.

Set-Variable -Name authorizatoin -Value 'SharedAccessSignature uid=5{id}&ex={date-time}&sn={signature}'
Set-Variable -Name serviceUrl -Value 'https://{service}.management.azure-api.net'

The first thing I tried was to Add a User. Then I needed a way to validate this first action, so I decided to List Users. And finally, I needed a way to clean up after my test so I Removed the User I created.

#This is the ID I will use to Add and Remove the User
$userID = [guid]::NewGuid()

Add-User -ID $userID -FirstName 'alexandre' -LastName 'brisebois' -Email 'alexandre@test.com' -Password 'test' -State active -Note 'test account will be delete'

#Listing Users so that I can validate that it was added
$users = Get-Users
$users.value | Format-Table -AutoSize

Remove-User -id $('/users/'+$userID)

#Listing Users so that I can validate that it was removed
$users = Get-Users
$users.value | Format-Table -AutoSize

The next step in my adventure was to traverse the graph from each resource group. I started by traversing the User graph.

# Getting a list of all Users
$users = Get-Users

foreach($u in $users.value)
{
    # Getting a User by ID
    $user = Get-User -id $u.id
    $user | Format-List

    # Getting the Groups to which a User belogs
    $groups = Get-UserGroupList -id $user.id
    $groups.value | Format-Table -AutoSize

    # Getting the Users Subscriptions
    $subscriptions = Get-UserSubscriptionList -id $user.id
    $subscriptions | Format-Table -AutoSize
}

Traversing the Groups was very similar and allows us to drill down to list Users per group.

# Getting a list of all the Groups
$groups = Get-Groups

foreach($g in $groups.value)
{
  # Getting a Group by ID
  $group = Get-Group -id $g.id

  # Getting a Group's User list
  $users = Get-Groupusers -id $group.id

  foreach($u in $users.value)
  {
    $u | Format-List
  }
}

Traversing Products came just as naturally

# Getting a list of Products
$products = Get-Products

foreach($p in $products.Value)
{
  # Getting a Product by ID
  Get-Product -id $p.id | Format-List

  # Getting a list of APIs that belong to a Product
  $apiList = Get-ProductApiList -id $p.id

  foreach($a in $apiList.Value)
  {
    # Getting an API by ID
    $api = Get-API -id $a.id 

    $api | Format-List
  }
}

REST API Wrapper PowerShell Module

function Add-Version
{
  param
  (
    [System.String]
    $url
  )

  $version ='2014-02-14'

  return $($url+'?api-version='+$version)
}

function Invoke-ApiManagementResource
{
  param
  (
    [System.String]
    $MethodUrl,

    [System.String]
    $Authorization,

    [System.String]
    $Body,

    [System.String]
    $ContentType

  )

  $dict = @{}
  $dict.Add('Authorization',$authorization)

  $response = Invoke-RestMethod -Method 'POST' -Uri $(Add-Version -url $MethodUrl) -Headers $dict -ContentType $ContentType -Body $body

  return $response
}

function Remove-ApiManagementResource
{
  param
  (
    [System.String]
    $MethodUrl,

    [System.String]
    $Authorization,

    [System.String]
    $Body,

    [System.String]
    $ETag,

    [System.String]
    $ContentType

  )

  $dict = @{}
  $dict.Add('Authorization',$authorization)
  $dict.Add('If-Match',$ETag)

  $response = Invoke-RestMethod -Method 'DELETE' -Uri $(Add-Version -url $MethodUrl) -Headers $dict -ContentType $ContentType
  return $response
}

function Update-ApiManagementResource
{
  param
  (
    [System.String]
    $MethodUrl,

    [System.String]
    $Authorization,

    [System.String]
    $Body,

    [System.String]
    $ContentType

  )

  $dict = @{}
  $dict.Add('Authorization',$authorization)

  $response = Invoke-RestMethod -Method 'PUT' -Uri $(Add-Version -url $MethodUrl) -Headers $dict -ContentType $ContentType -Body $body

  return $response
}

function Get-ApiManagementResourceMetadata
{
  param
  (
    [System.String]
    $MethodUrl,

    [System.String]
    $Authorization,

    [System.String]
    $ContentType

  )

  $dict = @{}
  $dict.Add('Authorization',$authorization)

  $response = Invoke-WebRequest -Method 'HEAD' -Uri $(Add-Version -url $MethodUrl) -Headers $dict -ContentType $ContentType

  return $response
}

function Get-ApiManagementResource
{
  param
  (
    [System.String]
    $MethodUrl,

    [System.String]
    $Authorization,

    [System.String]
    $ContentType

  )

  $dict = @{}
  $dict.Add('Authorization',$authorization)

  $response = Invoke-RestMethod -Method 'GET' -Uri $(Add-Version -url $MethodUrl) -Headers $dict -ContentType $ContentType

  return $response
}

function Get-Products
{
    return Get-ApiManagementResource -MethodUrl $($serviceUrl+'/products') -Authorization $authorizatoin -ContentType $contentType
}

function Get-Product
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType
}

function Get-API
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType
}

function Get-ProductApiList
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id+'/apis') -Authorization $authorizatoin -ContentType $contentType
}

function Get-ProductMetadata
{
   param
   (
     [System.String]
     $id
   )

    $response = Get-ApiManagementResourceMetadata -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType

    return $response.Headers.'ETag'
}

function Get-Users
{
    return Get-ApiManagementResource -MethodUrl $($serviceUrl+'/users') -Authorization $authorizatoin -ContentType $contentType
}

function Get-User
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType
}

function Get-UserMetadata
{
   param
   (
     [System.String]
     $id
   )

    $response = Get-ApiManagementResourceMetadata -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType

    return $response.Headers.'ETag'
}

function Remove-User
{
   param
   (
     [System.String]
     $id
   )

    $etag = Get-UserMetadata  -id $id
    return Remove-ApiManagementResource -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType -ETag $etag
}

function Add-User
{
   param
   (
     [System.String]
     $ID,

     [System.String]
     $FirstName,

     [System.String]
     $LastName,

     [System.String]
     $Email,

     [System.String]
     $Password,

     [System.String]
     [ValidateSet('active','blocked')]
     $State,

     [System.String]
     $Note = ''
   )

    $body = @{}
    $body.Add('firstName',$FirstName)
    $body.Add('lastName',$LastName)
    $body.Add('email',$Email)
    $body.Add('note',$Note)
    $body.Add('password',$Password)
    $body.Add('state',$State)

    return Update-ApiManagementResource -MethodUrl $($serviceUrl+'/users/'+$ID) -Authorization $authorizatoin -Body $(ConvertTo-Json $body) -ContentType $contentType
}

function Get-UserGroupList
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id+'/groups') -Authorization $authorizatoin -ContentType $contentType
}

function Get-UserSubscriptionList
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id+'/subscriptions') -Authorization $authorizatoin -ContentType $contentType
}

function Get-Groups
{
    return Get-ApiManagementResource -MethodUrl $($serviceUrl+'/groups') -Authorization $authorizatoin -ContentType $contentType
}

function Get-Group
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id) -Authorization $authorizatoin -ContentType $contentType
}

function Get-GroupUsers
{
   param
   (
     [System.String]
     $id
   )

    return Get-ApiManagementResource -MethodUrl $($serviceUrl+''+$id+'/users') -Authorization $authorizatoin -ContentType $contentType
}

function Get-Subscriptions
{
    return Get-ApiManagementResource -MethodUrl $($serviceUrl+'/subscriptions') -Authorization $authorizatoin -ContentType $contentType
}

Set-Variable -Name contentType -Value 'application/json'

Export-ModuleMember 'Add-Version'
Export-ModuleMember 'Get-ApiManagementResource'
Export-ModuleMember 'Get-ApiManagementResourceMetadata'
Export-ModuleMember 'Remove-ApiManagementResource'
Export-ModuleMember 'Update-ApiManagementResource'
Export-ModuleMember 'Invoke-ApiManagementResource'
Export-ModuleMember 'Get-Subscriptions'
Export-ModuleMember 'Add-User'
Export-ModuleMember 'Remove-User'
Export-ModuleMember 'Get-Users'
Export-ModuleMember 'Get-User'
Export-ModuleMember 'Get-UserGroupList'
Export-ModuleMember 'Get-UserSubscriptionList'
Export-ModuleMember 'Get-Products'
Export-ModuleMember 'Get-Product'
Export-ModuleMember 'Get-ProductMetadata'
Export-ModuleMember 'Get-ProductApiList'
Export-ModuleMember 'Get-Groups'
Export-ModuleMember 'Get-Group'
Export-ModuleMember 'Get-GroupUsers'
Export-ModuleMember 'Get-API'

Release Note

This is my preliminary attempt at using the API Management REST APIs for Automation. This script is a great starting point for those who wish to follow the same path. The code is available on GitHub https://github.com/brisebois/Brisebois.Azure.PowerShell

My first impression of these APIs is that it’s discoverable, well documented and easy to use from PowerShell.

Rebuilding SQL Database Indexes

A few months ago I wrote a blog post titled “Don’t Forget About Index Maintenance on Azure SQL Database“. Since then, Microsoft Azure SQL Database has changed a lot. We aren’t as concerned about the size of the database anymore, because databases can reach 500 GB in size. Take a moment to think about that number. 500GB is a lot of data! Before you get excited and move on to more important things, ask yourself this question, does all that data really belong in my SQL Database? Put some thought into it, you may be surprised by the answers you come up with.

So if we’re not as concerned about size, why should we care about index fragmentation. Well, if indexes aren’t maintained, your application will slow down. Queries may timeout and users may look for alternatives. Rebuilding indexes is something we overlook on a regular basis, along with missing indexes, it can really harm the overall performance of an application.

Since I started automating everything, I decided it was time to revamp the console application from my earlier post. I reused bits from a couple of blog posts and produced something that will make my life easier. The following PowerShell script is a mix of C# and PowerShell. It takes in the Server name, the Database name, the Username and the Password to build its execution context.

Start-SqlDatabaseIndexRebuild  -Server '' -Database '' -User '' -Password ''

Optionally you can override the default Fragmentation Percentage Threshold of 10%. Ideally, you don’t want to rebuild costly indexes too often, but you don;t want them to get out of hand either. Since you cannot REORGANIZE indexes on SQL Database, I tent to rebuild earlier than we’re used to on SQL Server.

Use this script to rebuild indexes on your Microsoft SQL Databases, it will skip SQL Data Sync tables and will inspect User Tables for fragmented indexes. Feel free to adapt it to your needs. This code is available on my Azure PowerShell GitHub Repository.

<#
.Synopsis
  This will scan a Microsoft Azure SQL Database for Indexes whose Index fragmentation is greater than the defined threshold percentage and will rebuild them sequentially.
.DESCRIPTION
   Be sure to rebuild fragmented indexes on a regular basis on Azure SQL Database. Fragmentation increases Index size and results in slower queries.
.EXAMPLE
   Start-SqlDatabaseIndexRebuild -Server '' -Database '' -User '' -Password ''
.EXAMPLE
   Start-SqlDatabaseIndexRebuild  -Server '' -Database '' -User '' -Password '' -FragmentationPercentageThreshold 10
#>
Function Start-SqlDatabaseIndexRebuild
{
  [CmdletBinding()]
  Param
  (
    # SQL Database Server
    [Parameter(Mandatory=$true, ValueFromPipelineByPropertyName=$true, Position=0)]
    [string]
    $Server,

    # SQL Database Name
    [Parameter(Mandatory=$true, ValueFromPipelineByPropertyName=$true, Position=1)]
    [string]
    $Database,

    # SQL Database Username
    [Parameter(Mandatory=$true, ValueFromPipelineByPropertyName=$true, Position=2)]
    [string]
    $User,

    # SQL Database Password
    [Parameter(Mandatory=$true, ValueFromPipelineByPropertyName=$true, Position=3)]
    [string]
    $Password,

    # Fragmentation Percentage Threshold used to select indexes to defragment
    [Parameter(Mandatory=$false, ValueFromPipelineByPropertyName=$true, Position=4)]
    [int]
    $FragmentationPercentageThreshold = 10,

    # A flag used to skip tables created by Azure SQL DataSync
    [Parameter(Mandatory=$false, ValueFromPipelineByPropertyName=$true, Position=5)]
    [bool]
    $SkipSqlDataSyncTables = $true
  )

  $assemblies = (
    'System.Data',
    'System.Collections',
    'System.Data.DataSetExtensions',
    'System.Xml',
    'System.Linq'
  ) 

  $source = @'
    using System;
    using System.Linq;
    using System.Collections;
    using System.Collections.Generic;
    using System.Data;
    using System.Data.SqlClient;
    using System.Globalization;

    public class DbContext
    {
        public string Server { get; set; }
        public string Database { get; set; }
        public string User { get; set; }
        public string Password { get; set; }

        private string ConnectionString
        {
            get
            {
                return String.Format(CultureInfo.InvariantCulture,
                                     "Server=tcp:{0}.database.windows.net,1433;Database={1};User ID={2}@{0};Password={3};Trusted_Connection=False;Encrypt=True;Connection Timeout=3600;",
                                     Server,
                                     Database,
                                     User,
                                     Password);
            }
        }

        const string listTablesQuery = @"SELECT o.name AS [table_name]
                                            FROM  sys.objects AS o
                                            WHERE o.is_ms_shipped = 0 AND o.[type] = 'U'";
        public List GetTableNames()
        {
            return Query(listTablesQuery)
                        .AsEnumerable()
                        .Select(r => r[0].ToString())
                        .ToList();
        }

        const string findFragmentedIndexesQuery = @"SELECT name, avg_fragmentation_in_percent
                                                    FROM sys.dm_db_index_physical_stats (
                                                            DB_ID(N'DatabaseName')
                                                            , OBJECT_ID('{0}')
                                                            , NULL
                                                            , NULL
                                                            , NULL) AS a
                                                    JOIN sys.indexes AS b
                                                    ON a.object_id = b.object_id AND a.index_id = b.index_id
                                                    WHERE avg_fragmentation_in_percent > {1}";

        public Hashtable GetFragmentedTableIndexes(string table, int fragmentationThreshold)
        {
            return ToIndexTableMap(table, Query(string.Format(CultureInfo.InvariantCulture,
                findFragmentedIndexesQuery,
                table,
                fragmentationThreshold)));
        }

        private static Hashtable ToIndexTableMap(string table, DataTable indexes)
        {
            var t = new Hashtable();
            foreach (var i in indexes.AsEnumerable())
                t.Add(i[0],i[1]);    

            return t;
        }

        private DataTable Query(string query)
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                connection.Open();

                var command = new SqlCommand(query, connection);
                var reader = new SqlDataAdapter(command);
                var table = new DataTable();
                reader.Fill(table);
                return table;
            }
        }

        const string rebuildIndex = "ALTER INDEX [{0}] ON [{1}] REBUILD WITH (ONLINE=ON);";

        public int RebuildIndex(string indexName, string table)
        {
            return Command(string.Format(CultureInfo.InvariantCulture, rebuildIndex, indexName, table));
        }

        private int Command(string cmd)
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                connection.Open();

                var command = new SqlCommand(cmd, connection);
                return command.ExecuteNonQuery();
            }
        }
    }
'@

  Add-Type -Language CSharp -TypeDefinition $source -ReferencedAssemblies $assemblies

  $db = New-Object DBContext
  $db.Server = $Server
  $db.Database = $Database
  $db.User = $User
  $db.Password = $Password    

  Write-Verbose 'Connecting'

  $tables = $db.GetTableNames()

  Write-Verbose $('Found ' + $tables.Count + ' Tables')

  $percent = 0
  $tableIndex = 0

  foreach($t in $tables){

    if($SkipSqlDataSyncTables -and $t.Contains('dss'))
    {
      Write-Warning $('Skipping  Table ' + $t)
      continue
    }

    $indexes = $db.GetFragmentedTableIndexes($t, $FragmentationPercentageThreshold)
    foreach($i in $indexes.Keys)
    {
      $message = 'Successful'
      try
      {
        $result = $db.RebuildIndex($i,$t)
      }
      catch
      {
        $message = 'Failed'
        Write-Warning $("Exception Message: $($_.Exception.Message)")
      }
      finally
      {
        Write-Verbose $($message + ' Rebuild > Table '+ $t+ ' Index ' + $i + ' fragmented @ '+ $indexes[$i] + '%')
      }
    }
    $tableIndex ++
    $percent = ($tableIndex/$tables.Count) * 100
    Write-Progress $t -PercentComplete $percent
  }
}

Using PowerShell to Authenticate Against OAuth

From development to deployment, PowerShell is becoming the ‘go to’ automation technology on Microsoft Azure. So, I decided to use PowerShell to perform automated tests against a Web API (a.k.a REST service). These tests are built to run during the execution of a Continuous Release cycle and confirm that the API is responding as expected.

The Web API I’m testing is exposed to the outside world through Microsoft Azure API Management. Therefore, consumers of my API must subscribe and must use a Subscription Key for every request. This key is used to authenticate against Microsoft Azure API Management, which acts as a proxy between the outside world and the Web API. The Subscription Key does not authenticate the client against the Web API. Therefore, clients must authenticate by posting requests to the OAuth endpoint.

Getting an Access Token from OAuth on Azure API Management

#OAuth Basic Token
$basicToken = 'Basic lots-of-alphanumerics=='

#API Management URL & Key
$url = 'https://test.azure-api.net/api/token'
$subscriptionKey = 'lots-of-alphanumerics'

#Service User Info
$username ='brisebois'
$password = 'zxcvbnm'

#Getting an Access Token
$context = Get-ApiToken -methodUrl $url -username $username -password $password -authorization $basicToken -subscriptionkey $subscriptionKey

#Testing the AccessToken by making a GET request against the API
$peoplelistUrl = 'https://test.azure-api.net/api/people'

$peopleList = Get-ApiResource -methodurl $peoplelistUrl -oauthContext $context -subscriptionkey $subscriptionKey

Wait! How Does That Work?

First I need a Data Transfer Object (DTO). This DTO will be used to make subsequent secured calls to the Web API.

Add-Type @'
    using System;
    public class OAuthContext{
        public string AccessToken{get;set;}
        public string TokenType{get;set;}
        public string ExpiresIn{get;set;}
        public string RefreshToken{get;set;}
    }
'@

Using Invoke-RestMethod to authenticate against an OAtuh endpoint, I encapsulate the response in an OAuthContext instance. I decided to use Invoke-RestMethod instead of Invoke-WebRequest because it automatically applies ConvertFrom-Json to the response. This is quite useful, because I can work directly with the object.

Headers for the request need an IDictionary collection. In PowerShell, we use a Hash collection to store key/value pairs. I really want to highlight this, because it took me a couple of minutes to find out how to satisfy the IDictionary dependency. The following will create a collection to which you can add your headers.

$dict = @{}
$dict.Add('Authorization',$authorization)

I’m sure I could have done this differently, but considering that I’m still new to PowerShell, I’m pretty happy with the outcome. The following builds a request that is used to authenticate against an OAuth endpoint that is exposed through Microsoft Azure API Management. Notice that I systematically add the Subscription Key as a query string, this is done so that Microsoft Azure API Management allows us to reach the API.

function Get-ApiToken([string]$methodUrl, [string]$username,[string] $password,[string] $authorization, [string] $subscriptionkey)
{
  $methodtype = 'POST'
  $contentType = 'application/x-www-form-urlencoded'

  $dict = @{}
  $dict.Add('Authorization',$authorization)

  $methodURL = $methodURL+'?subscription-key='+$subscriptionkey

  $body = [System.Text.Encoding]::UTF8.GetBytes('grant_type=password&username='+$username+'&password='+$password) 

  $response = Invoke-RestMethod -Method $methodtype -Uri $methodUrl -Headers $dict -ContentType $contentType -Body $body

  $context = New-Object OAuthContext

  $context.AccessToken = $response.access_token
  $context.ExpiresIn = $response.expires_in
  $context.RefreshToken = $response.refresh_token
  $context.TokenType = $response.token_type

  return $context
}

To test the acquired Access Token I perform a GET request against a resource exposed by my Web API. The following creates a GET request and returns the Json payload as an object to the caller.

function Get-ApiResource([string]$methodurl,[OAuthContext]$oauthContext,[string]$subscriptionkey)
{
	$headers = @{}
	$headers.Add('Authorization',$oauthContext.TokenType + ' ' + $oauthContext.AccessToken)

	$methodtype = 'GET'
	$methodurl = $methodurl+'?subscription-key='+$subscriptionkey

	return Invoke-RestMethod -Method $methodtype -Uri $methodurl -Headers $headers
}

That was Cool, What’s Next?

The goal of this post was provide a happy path REST call against a Web API exposed through Microsoft Azure API Management. These examples lack error handling, retry policies and everything else needed to create robust solutions. Use this is a starting point, but remember that it still needs a lot of love.

Do you use PowerShell to make requests to Web APIs (a.k.a REST services)? Have you done this using different a technique? I very curious to find out what I missed.

How do You Version Packages?

This topic seems to come up regularly. And is usually a friction point for teams because many of us do this differently. In an attempt to standardize versioning and to remove this undesired friction I decided to promote Semantic Versioning (http://semver.org/).

Familiarizing Ourselves with Semantic Versioning

This is not a new or revolutionary idea. In fact, you probably do something close to this already. The problem is that “close” isn’t good enough. Without compliance to some sort of formal specification, version numbers are essentially useless for dependency management. By giving a name and clear definition to the above ideas, it becomes easy to communicate your intentions to the users of your package. Once these intentions are clear, flexible (but not too flexible) dependency specifications can finally be made.

Given a version number is composed of the following 4 parts: MAJOR.MINOR.PATCH and INCREMENT. The recommended approach to incrementing each part is as follows:

1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backwards-compatible manner
3. PATCH version when you make backwards-compatible bug fixes
4. INCREMENT version when you make a build (should be automatically set by the build server)

Additional labels for pre-release and build metadata are available as extensions to this format.

NuGet has wonderful documentation about versioning. The following package versions illustrate the flexibility that we get from using metadata. In this list, we can clearly observe the distinctions created by the use of metadata.

1.0.0.0-preview
1.0.0.0-rc
1.0.0.0-alpha
1.0.0.0-beta
1.0.0.0

To summarize, the template for a package version is MAJOR.MINOR.PATCH.INCREMENT
optionally we can add -METADATA to denote various stages of preview packages.

Why is this Standard Important?

Having a standard in place simplifies the communication of intent to the consumers of your packages. Seeing that a package has changed MAJOR versions is a clear indicator for consumers to review a packages’s release notes. In turn, helping the consumers save countless hours of troubleshooting because of breaking changes.

Noticing a MINOR, PATCH or INCREMENT version change should not be alarming to the consumers of the package. This type of version change should be welcomed with greater confidence. Release notes remain important, but should not convey any breaking changes.

Should we Produce Release Notes?

The short answer is YES. Release notes should documents fundamental changes to the component. These are especially important for MAJOR versions. MINOR and PATCH versions also deserve to have release notes, but I consider these to be relaxed because these versions should be backward compatible and should be tested accordingly.

Safely Updating Packages Through NuGet

In a blog post by Maxime Rouiller titled NuGet–Upgrading your packages like a boss we get introduced to NuGet’s -Safe flag. If everyone applies Semantic Versioning to their packages, this flag can definitely become your best friend, because it can save you countless hours of refactoring and debugging. By using this flag you are assured that only PATCH versions are applied to your solution. Find out how this work by reading his blog post.

Busy… Starting Role… Repeat

So you deployed a Cloud Service and it’s status is stuck on “Busy (Starting role… Sites were deployed.)” By now you’ve probably checked the SDK assembly versions, checked that your Cloud Service runs in the Azure Emulator and probably everything else that you could think of. So what’s next?

In August 2013 Kevin Williamson published an awesome blog series titled Windows Azure PaaS Compute Diagnostics Data. It contains a wealth of priceless information that ended my 24 hour debug session.

Since my Cloud Service already had a Remote Desktop Connection setup at publish time, I went a head with the proposed path to diagnose the issue. This adventure ended abruptly as I was presented with this prompt.

I decided remove the Remote Desktop configurations from my Cloud Service and to redeploy it to Azure. This allowed me to try to dynamically enable Microsoft Azure Remote Desktop. I used this DIY Microsoft Azure Troubleshooting Tutorial which takes you through the steps.

After a few minutes, I was finally able to open a Remote Desktop session to one of my Cloud Service Role instances. Finally I got a glimpse of light from the other end of the tunnel!

Troubleshooting effectively and in a timely manner requires you to install a couple tools. I was happy to find out that Microsoft has made this process easy for us. On the Azure VM hosting the Cloud Service Role open a PowerShell console and Copy/Paste and Run the following script

md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe;c:\tools\AzureTools.exe

*This only works on Guest OS Family 2 or later

PowerShell will start AzureTools and present you with the following screen. Right click on the tool of you choice to select download from the context menu.

Generally when I troubleshoot, I usually need a way to persist files to blob storage. The second tab in AzureTools allows you to do just that.

The third tab is full of great shortcuts and tools that help you focus on finding the root cause you’re after.

From this tab you can start by clicking the Open Log Files. This will open logs that could shed some light into what’s going on. I started by looking at the WaAppAgent.log from which I got a view at a log showing the Role starting and stopping on a regular interval. Then I looked at the AppAgentRuntime.log, this provided me with a lot more information and I got lost for a little while. If something is going wrong, this is a great place to start. it’s got lots and lots of detailed information about the Roles.

Open Log Files – This will open the current log file for all of the log files most commonly used when troubleshooting on an Azure VM (see Windows Azure PaaS Compute Diagnostics Data).  This is useful when you are troubleshooting an issue and you don’t quite know where to begin looking so you want to see all of the data without having to go searching for all of it

Then I navigated to the Event Viewer to get a feel for the exceptions that could be contributing to the Role’s behavior.

From the Windows Azure Event Log I was able to find clues as to what was happening.

WaIISHost
Role entrypoint could not be created: System.TypeLoadException: Unable to load the role entry point due to the following exceptions:
-- System.IO.FileLoadException: Could not load file or assembly 'System.Web.Http, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'System.Web.Http, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

This started to be helpful, but which library is actually depending on this dependency? To find this information, go to the third tab in the AzureTools and click on Fusion Logging. This turns on or off .NET Fusion Logging verbose output. Remember that this slows down the Role, but it provides you with insights about Bindings and dependencies between assemblies.

=== Pre-bind state information ===
LOG: DisplayName = System.Web.Http, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///E:/approot/bin
LOG: Initial PrivatePath = E:\approot\bin
Calling assembly : WebApi.OutputCache.V2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: E:\base\x64\WaIISHost.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Web.Http, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Attempting download of new URL file:///E:/approot/bin/System.Web.Http.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Minor Version
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

Having identified the dependency that cause this issue I decided to contribute to the GitHub project by updating all of it’s dependencies and it’s target framework to 4.5.1. The code can be downloaded from my GitHub repository. Subsequently, I created a pull request so that everyone working on the latest version of Web API may benefit from this adventure.

If you’re curious about Azure Roles and about troubleshooting them I recommend that you browse through the following resources. I found them to be very good at pointing me in the right direct when I was out of ideas.

• Windows Azure – Troubleshooting & Debugging
• The Diagnostic Utility used by the Windows Azure Developer Support Team
• Troubleshooting Scenario 1 – Role Recycling
• Troubleshooting Scenario 2 – Role Recycling After Running Fine For 2 Weeks
• Troubleshooting Scenario 3 – Role Stuck in Busy
• Troubleshooting Scenario 4 – Windows Azure Traffic Manager Degraded Status
• Troubleshooting Scenario 5 – Internal Server Error 500 in WebRole
• Troubleshooting Scenario 6 – Role Recycling After Running For Some Time
• Troubleshooting Scenario 7 – Role Recycling
• DIY Microsoft Azure Troubleshooting
• Enabling Microsoft Azure Remote Desktop When Publishing

Diagnostic Data Locations

This list includes the most commonly used data sources used when troubleshooting issues in a PaaS VM, roughly ordered by importance (ie. the frequency of using the log to diagnose issues).

• Windows Azure Event Logs – Event Viewer –> Applications and Services Logs –> Windows Azure
  • Contains key diagnostic output from the Windows Azure Runtime, including information such as Role starts/stops, startup tasks, OnStart start and stop, OnRun start, crashes, recycles, etc.
  • This log is often overlooked because it is under the “Applications and Services Logs” folder in Event Viewer and thus not as visible as the standard Application or System event logs.
  • This one diagnostic source will help you identify the cause of several of the most common issues with Azure roles failing to start correctly – startup task failures, and crashing in OnStart or OnRun.
  • Captures crashes, with callstacks, in the Azure runtime host processes that run your role entrypoint code (ie. WebRole.cs or WorkerRole.cs).
• Application Event Logs – Event Viewer –> Windows Logs –> Application
  • This is standard troubleshooting for both Azure and on-premise servers.  You will often find w3wp.exe related errors in these logs.
• App Agent Runtime Logs – C:\Logs\AppAgentRuntime.log
  • These logs are written by WindowsAzureGuestAgent.exe and contain information about events happening within the guest agent and the VM.  This includes information such as firewall configuration, role state changes, recycles, reboots, health status changes, role stops/starts, certificate configuration, etc.
  • This log is useful to get a quick overview of the events happening over time to a role since it logs major changes to the role without logging heartbeats.
  • If the guest agent is not able to start the role correctly (ie. a locked file preventing directory cleanup) then you will see it in this log.
• App Agent Heartbeat Logs – C:\Logs\WaAppAgent.log
  • These logs are written by WindowsAzureGuestAgent.exe and contain information about the status of the health probes to the host bootstrapper.
  • The guest agent process is responsible for reporting health status (ie. Ready, Busy, etc) back to the fabric, so the health status as reported by these logs is the same status that you will see in the Management Portal.
  • These logs are typically useful for determining what is the current state of the role within the VM, as well as determining what the state was at some time in the past.  With a problem description like “My website was down from 10:00am to 11:30am yesterday”, these heartbeat logs are very useful to determine what the health status of the role was during that time.
• Host Bootstrapper Logs – C:\Resources\WaHostBootstrapper.log
  • This log contains entries for startup tasks (including plugins such as Caching or RDP) and health probes to the host process running your role entrypoint code (ie. WebRole.cs code running in WaIISHost.exe).
  • A new log file is generated each time the host bootstrapper is restarted (ie. each time your role is recycled due to a crash, recycle, VM restart, upgrade, etc) which makes these logs easy to use to determine how often or when your role recycled.
• IIS Logs – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\LogFiles\Web
  • This is standard troubleshooting for both Azure and on-premise servers.
  • One key problem scenario where these logs are often overlooked is the scenario of “My website was down from 10:00am to 11:30am yesterday”.  The natural tendency is to blame Azure for the outage (“My site has been working fine for 2 weeks, so it must be a problem with Azure!”), but the IIS logs will often indicate otherwise.  You may find increased response times immediately prior to the outage, or non-success status codes being returned from IIS, which would indicate a problem within the website itself (ie. in the ASP.NET code running in w3wp.exe) rather than an Azure issue.
• Performance Counters – perfmon, or Windows Azure Diagnostics
  • This is standard troubleshooting for both Azure and on-premise servers.
  • The interesting aspect of these logs in Azure is that, assuming you have setup WAD ahead of time, you will often have valuable performance counters to troubleshoot problems which occurred in the past (ie. “My website was down from 10:00am to 11:30am yesterday”).
  • Other than specific problems where you are gathering specific performance counters, the most common uses for the performance counters gathered by WAD is to look for regular performance counter entries, then a period of no entries, then resuming the regular entries (indicating a scenario where the VM was potentially not running), or 100% CPU (usually indicating an infinite loop or some other logic problem in the website code itself).
• HTTP.SYS Logs – D:\WIndows\System32\LogFiles\HTTPERR
  • This is standard troubleshooting for both Azure and on-premise servers.
  • Similar to the IIS Logs, these are often overlooked but very important when trying to troubleshoot an issue with a hosted service website not responding.  Often times it can be the result of IIS not being able to process the volume of requests coming in, the evidence of which will usually show up in the HTTP.SYS logs.
• IIS Failed Request Log Files – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\FailedReqLogFiles
  • This is standard troubleshooting for both Azure and on-premise servers.
  • This is not turned on by default in Windows Azure and is not frequently used.  But if you are troubleshooting IIS/ASP.NET specific issues you should consider turning FREB tracing on in order to get additional details.
• Windows Azure Diagnostics Tables and Configuration – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\Monitor
  • This is the local on-VM cache of the Windows Azure Diagnostics (WAD) data.  WAD captures the data as you have configured it, stores in in custom .TSF files on the VM, then transfers it to storage based on the scheduled transfer period time you have specified.
  • Unfortunately because they are in a custom .TSF format the contents of the WAD data are of limited use, however you can see the diagnostics configuration files which are useful to troubleshoot issues when Windows Azure Diagnostics itself is not working correctly.  Look in the Configuration folder for a file called config.xml which will include the configuration data for WAD.  If WAD is not working correctly you should check this file to make sure it is reflecting the way that you are expecting WAD to be configured.
• Windows Azure Caching Log Files – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\AzureCaching
  • These logs contain detailed information about Windows Azure role-based caching and can help troubleshoot issues where caching is not working as expected.
• WaIISHost Logs – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\WaIISHost.log
  • This contains logs from the WaIISHost.exe process which is where your role entrypoint code (ie. WebRole.cs) runs for WebRoles.  The majority of this information is also included in other logs covered above (ie. the Windows Azure Event Logs), but you may occasionally find additional useful information here.
• IISConfigurator Logs – C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\IISConfigurator.log
  • This contains information about the IISConfigurator process which is used to do the actual IIS configuration of your website per the model you have defined in the service definition files.
  • This process rarely fails or encounters errors, but if IIS/w3wp.exe does not seem to be setup correctly for your service then this log is the place to check.
• Role Configuration Files – C:\Config\{DeploymentID}.{DeploymentID.{Rolename}.{Version}.xml
  • This contains information about the configuration for your role such as settings defined in the ServiceConfiguration.cscfg file, LocalResource directories, DIP and VIP IP addresses and ports, certificate thumbprints, Load Balancer Probes, other instances, etc.
  • Similar to the Role Model Definition File, this is not a log file which contains runtime generated information, but can be useful to ensure that your service is being configured in the way that you are expecting.
• Role Model Definition File – E:\RoleModel.xml (or F:\RoleModel.xml)
  • This contains information about how your service is defined according to the Azure Runtime, in particular it contains entries for every startup task and how the startup task will be run (ie. background, environment variables, location, etc).  You will also be able to see how your <sites> element is defined for a web role.
  • This is not a log file which contains runtime generated information, but it will help you validate that Azure is running your service as you are expecting it to.  This is often helpful when a developer has a particular version of a service definition on his development machine, but the build/package server is using a different version of the service definition files.

* A note about ETL files

If you look in the C:\Logs folder you will find RuntimeEvents_{iteration}.etl and WaAppAgent_{iteration}.etl files.  These are ETW traces which contain a compilation of the information found in the Windows Azure Event Logs, Guest Agent Logs, and other logs.  This is a very convenient compilation of all of the most important log data in an Azure VM, but because they are in ETL format it requires a few extra steps to consume the information.  If you have a favorite ETW viewing tool then you can ignore several of the above mentioned log files and just look at the information in these two ETL files.

Find more details from : Windows Azure PaaS Compute Diagnostics Data

How do you Update Your Wetware?

I’m passionate about technology, but sometimes I need a wetware update. A reminder that technology needs people to change the world.

Wetware is a term drawn from the computer-related idea of hardware or software, but applied to biological life forms. Here the prefix “wet” is a reference to the water found in living creatures. Wetware is used to describe the elements equivalent to hardware and software found in a person, namely the central nervous system (CNS) and the human mind. The term wetware finds use both in works of fiction and in scholarly publications.

Communicating effectively doesn’t come easily. Surprisingly, the skills required to be heard include paying attention to others. Listening with intent and genuine interest has a greater effect than one can imagine.

As a technology enthusiast, and as a dreamer, it’s important that I remember that I need to communicate to succeed. Ideas grow and mature to fruition through collaboration and care.

Curiosity drives me to update my wetware through various activities. For one, I am an active participant in my local and worldwide community where I learn about technology. By interacting with others, I learn from their experiences and expand my understanding. I listen to podcasts and comment on blog posts. I ask and answer questions on social networks like Stackoverflow and I participate in forums.

Human interaction is at the center of my personal growth and this fact has driven me to two wonderful books. The first book that I had the pleasure of discovering is “Speaking as a Leader“. This book is rich with advice about how to move others with your vision. Using the right approach and the right tools makes a world of difference. The second book is time-tested and has captured the interest of millions. “How to Win Friends and Influence People” is a book that conveys principles through stories about people who have left their mark on history.

Both books complement each other really well. And both are rich with a wealth of wisdom that begs for a second read.

How do you update your wetware?

Automate Everything!

These two words immediately caught my attention! This is one of the hardest things for me as a developer. I spend most of my time designing and coding away, but I hardly spend anytime with PowerShell… so I’ll be blunt, as a developer, I must learn PowerShell and I must change my Definition of Done (DoD). I am not done until my feature can be deployed repeatedly and reliably. Automation makes me sleep better at night because my deployments yield predictable results.

This new e-book  is all about taking advantage of what the cloud has to offer. It’s packed with best practices for DevOps, data storage and high availability. Since the authors took a pattern-based approach, each chapter can be read independently.

Download all formats (PDF, Mobi and ePub) as well as link to the companion content hosted by the Microsoft Virtual Academy.

Getting Acquainted With #Azure Service Bus Event Hubs

The Microsoft Azure ecosystem just keeps growing. This week Microsoft unveiled a very welcomed addition to the Microsoft Azure Service Bus. Event Hubs join ranks with Queues, Topics and Relays to offer options adapted to your needs.

Contrasting available Service Bus Flavors?

• Relays – are used to bridge communications over the cloud in a secure and transparent manner.
• Queues – are pipes that allow for many publishes and many consumers to communicate over a single channel. This is great for Competing Consumers and for Queue-based Load Leveling.
• Topics – are pipes that allow fan out scenarios, where each consumer gets his own copy of the inbound queue. It also has some handy features like filters. Use this flavor to implement Pipes and Filters.
• Event Hubs – are a bit more complex. Event Hubs enable the collection of event streams at high throughput, from a diverse set of devices and services. In other words, they help us deal with the 3 Vs.
  • Volume (amount of data)
  • Velocity (speed of data in and out)
  • Variety (range of data types and sources).

Microsoft Azure Service Bus Event Hubs

Event Hub join ranks with Queues, Topics and Relays to offer options adapted to your needs. They province the mechanisms necessary to collection of event streams at high throughput, from a diverse set of devices and services. They are composed of a Published Policy, of Consumer Groups and of Partitions.

Event Hubs support the following scenarios:

• Collecting event streams at high throughput from devices/services for use in real-time and batch processing.
• Connecting millions of devices from diverse platforms for sending data (with individual authentication and flow control).
• Process event streams per device “in order” using several backend services (publish/subscribe).

Considerations Prior to Creating an Event Hub

You must put some effort in capacity planning before you create an Event Hub. In order to make the right decisions let’s go over a couple details about Event Hubs.

Event Hubs are partitioned. The minimum number of partitions is 8 and the maximum (public) number of partitions is 32. If you need more partitions you must go through a support ticket. At that time, you can request up to 1024 (and higher) partitions for your Event Hub.

Each partition has a performance target of 1 MB ingress or 1000 operations and 2 MB egress per second. By default, each Event Hub is created with 16 partitions. This corresponds to 16 MB ingress or 16,000 operations and 32 MB egress per second.

At first glance this seems like way too much so let’s put some perspective on this.

Imagine that our messages are 5 KB in size. We could stream 3276.8 events per second. If messages were 1 KB in size we could technically hit the performance target of 1000 operations per second. This could technically represent 16,000 messages per second. In order to reach the performance targets on all partitions of the Event Hub, we would need as many uniquely identified publisher as we have partitions. Each publisher is pinned to one partition by the Event Hub Service.

Calculating the number of partitions you need starts by calculating potential throughput in megabytes. Then you need to calculate the number of messages per second. This will give you the first part of the equation. The second part of the equation is found by calculating the number of processing nodes required to meet your performance targets.

Since a single partition cannot be processed concurrently by multiple nodes, you must have at least as many partitions as you have backend node instances. For example, a system that must process 30 messages simultaneously, where each process requires a dedicated processing node, requires the Event Hub to have at least 30 partitions.

Number of partitions = MAX (cumulative throughput required for the stream – given 1 MB per partition, number of nodes needed by the backend processing application)

Calculating the number of partitions based on performance targets is important because it cannot be changed after the creation of the Event Hub.

Changing the number of partitions once in production can cause quite a bit of headaches because it means that we need to create a new Event Hub and reconfigure publishers to use the new Event Hub. While events are piling up let the backend nodes empty the Event Hub. Once it’s empty, you can reconfigure your backend nodes to consumer events from the new Event Hub. Switching to the backend processing nodes to the new Event Hub too early would break the order of events. From a billing perspective, the number of partitions is irrelevant because there is not charge for partitions.

Once the application is deployed you can provision throughput units to scale the Event Hub’s throughput capacity. A single throughput unit (as of July 2014) has the capacity of 1 MB per second of ingress events (events sent into an Event Hub), but no more than 1000 ingress events, management operations or control API calls per second. It has 2 MB per second of egress events (events consumed from an Event Hub) and 84 GB of event storage (sufficient for the default 24-hour retention period). In order to max out the total possible capacity of an Event Hub, you will need to provision one throughput unit for each partition. Event Hub throughput units are billed hourly, based on the maximum number of units selected during the given hour.

Creating an Event Hub

Click on the +NEW area at the bottom of the portal and navigate to the Event Hub service.

A wizard will show on screen to assist you with the creation. Enter an Event Hub name, select a Region that hosts the services that will process the events and click next. Be sure that you are creating the resource on the correct subscription and desired Azure Service Bus Namespace.

The next panel prompts for important information. This is where you specify the number of partitions for your Event Hub. Then because this is a Standard Event Hub, you can specie the number of days that an event stays in the Event Hub. This is especially handy when you need to replay events from a week ago. The maximum number of days is set to 30, the minimum is set to 1 day.

Publishing Events

In order to publish events to an Event Hub, you must create Shared Access Policies. I strongly recommend creating a policy that allows single actions. In order to publish an event we need a Shared Access Policy that allows us to send events. Navigate to the Service Bus namespace and click on the Event Hubs tab. From the list choose the newly created Event Hub.

Then click on Configure. In this screen, you will be able to create a Send enabled Shared Access Policy.

Now we need to retrieve a SAS key connection strings to authenticate and interact with the Event Hub. Select the Dashboard tab and click on the View Connection String link.

From the wizard copy the Connection String that is enabled for Send operations. Click the copy button to store the connection string in you clipboard. This will allow you to paste the connection string in your project’s configurations.

Getting down to the code. Use the connection string we copied from the portal to send events to our newly create Microsoft Azure Service Bus Event Hub. The EventData object is the equivalent of the Service Bus BrokeredMessage with a twist. The EventData object requires a Partition Key. This key is used internally to decide which Event Hub partition is used to storage and convey the event. The EventHubClient used to send the EventData is part of the Microsoft Azure Service Bus 2.4.1.1 NuGet package.

var cs = @"[Service Bus Connection String] ";

var builder = new ServiceBusConnectionStringBuilder(cs)
                    {
                        TransportType = TransportType.Amqp
                    };

var client = EventHubClient.CreateFromConnectionString(builder.ToString(), "brisebois");

try
{
    var e = new Event
    {
        Message = "Test Message"
    };

    var serializedString = JsonConvert.SerializeObject(e);
    var data = new EventData(Encoding.Unicode.GetBytes(serializedString))
    {
        PartitionKey = "test-app"
    };

    // Set user properties if needed
    data.Properties.Add("Type", "Event");

    await client.SendAsync(data);
}
catch (Exception exp)
{
    Console.WriteLine("Error on send: " + exp.Message);
}

The data encapsulated inside an EventData object is stored in bytes. This allows us to use DataContracs and efficient serialization libraries like Json.NET, Protobuf and Avro. Keep in mind that like Service Bus Queues and Topics, messages are charged by slices of 64 KB.

[DataContract]
public class Event
{
    [DataMember]
    public string Message { get; set; }
}

Consuming Events

The first step in consuming events from the Event Hub is to implement IEventProcessor. The Event Processor is used to process events. It’s a great way to build an event processing pipeline. By using the Properties collection of the EventData object we can store the type of the event, which we can use to choose whether to deserialize and process the body of the EventData object.

class EventProcessor : IEventProcessor
{
    public Task OpenAsync(PartitionContext context)
    {
        return Task.FromResult<object>(null);
    }

    public Task ProcessEventsAsync(PartitionContext context, IEnumerable events)
    {
        foreach (var eventData in events)
        {
            if (eventData.Properties["Type"].ToString() != "Event")
                continue;

            var bytes = Encoding.Unicode.GetString(eventData.GetBytes());
            var data = JsonConvert.DeserializeObject<Event>(bytes);

            Console.WriteLine("Processing EVENT [(Message: {0}) PartitionKey: {1}] at PartitionId: {2}",
                                data.Message,
                                eventData.PartitionKey,
                                context.Lease.PartitionId);

            foreach (KeyValuePair<string, object=""> p in eventData.Properties)
            {
                if (!p.Key.Equals("ContentType"))
                {
                    Console.WriteLine("  [Property: {0} = {1}]", p.Key, p.Value);
                }
            }
        }

        return Task.FromResult<object>(null);
    }

    public Task CloseAsync(PartitionContext context, CloseReason reason)
    {
        throw new NotImplementedException();
    }
}

There are a couple ways to consume messages from the Event Hub. The easiest is to use the EventProcessorHost. It takes care of scaling and distributing compute resources to process the Event Hub’s partitions. Create an instance by providing a Connection String that is capable of reading from the Event Hub. Provide an Azure Storage Connection String, it will be used by the EventProcessorHost to scale and checkpoint the event stream offset to Blob Storage.

This example reads from the default Consumer Group. In practice, we can use multiple Consumer Groups as we use multiple Subscription on Service Bus Topics. Consumer groups are Partitioned Subscriptions. Each Consumer Group gets a copy of the event. Each Consumer Group can be processed in parallel.

var cs = @"[Service Bus Connection String] ";

var blobConnectionString = @"[Storage Connection String]";

var builder = new ServiceBusConnectionStringBuilder(cs)
                    {
                        TransportType = TransportType.Amqp
                    };

var client = EventHubClient.CreateFromConnectionString(builder.ToString(), "brisebois");

try
{
    var eventProcessorHost = new EventProcessorHost("singleworker",
                                                    client.Path,
                                                    client.GetDefaultConsumerGroup().GroupName,
                                                    builder.ToString(),
                                                    blobConnectionString);

    eventProcessorHost.RegisterEventProcessorAsync<EventProcessor>();
}
catch (Exception exp)
{
    Console.WriteLine("Error on send: " + exp.Message);
}

We can also manage the way we process events from the Event Hub partitions. The first step is to implement an ICheckpointManager. This is used to persist the offset location of the last processed event from an Event Hub partition.

public class EventProcessorCheckpointManager : ICheckpointManager
{
    public Task CheckpointAsync(Lease lease, string offset)
    {
        Console.WriteLine("PartitionId: " + lease.PartitionId);
        Console.WriteLine("Offset: " + offset);
        return Task.FromResult<object>(null);
    }
}

Then by using the EventHubClient and the NamespaceManager we can pick and choose Consumer Groups and partitions to subscribe to. In order to get this solution working, the connection string must contain a SAS key allowing the consuming code to manage the EventHub.

var cs = @"[Service Bus Connection String] ";

var blobConnectionString = @"[Storage Connection String]";

var builder = new ServiceBusConnectionStringBuilder(cs)
                    {
                        TransportType = TransportType.Amqp
                    };

var client = EventHubClient.CreateFromConnectionString(builder.ToString(), "brisebois");

try
{
    var subscriberGroup = client.GetDefaultConsumerGroup();

    var eventHub = NamespaceManager.CreateFromConnectionString(builder.ToString()).GetEventHub("brisebois");

    // Register event processor with each shard to start consuming messages
    foreach (var partitionId in eventHub.PartitionIds)
    {
        subscriberGroup.RegisterProcessor<EventProcessor>(new Lease
        {
            PartitionId = partitionId
        }, new EventProcessorCheckpointManager());

        Console.WriteLine("Processing : " + partitionId);
    }

}
catch (Exception exp)
{
    Console.WriteLine("Error on send: " + exp.Message);
}

Summary

The Azure Service Bus Event Hubs enables us to stream millions of events. It allows us to partition and organize the consumption of events. Furthermore, we can use offsets to replay event streams. This is a powerful service. It has its particularities and takes a bit of effort to tame, but once you nailed down the details… its flat out awesome!

To grok Event Hubs, one has to get to know each of its parts. The service that has many great use cases like the Internet of things (IoT).

In this post I tried to focus on working with Event Hubs from C# using the NuGet packages, these packages are made available by the Azure Service Bus team. Its important to note that you can interact with Even Hubs though a REST API over HTTP. An example of this usage pattern can be found in the sample projects listed below.

Take a look at the following resources. As of July 2014, the documentation isn’t abundant and it requires us to explore. The next few months will probably bring forth much more documentation and use cases. Until then, please share your findings and your thoughts about this new service.

NuGet Packages

• Microsoft Azure Service Bus
• Microsoft Azure Service Bus Event Hub – EventProcessorHost

Documentation

• Event Hubs Developer Guide
• Event Hubs API Overview
• Event Hubs Authentication and Security Model Overview
• Event Hubs Preview Availability and Support

Sample Projects

• Service Bus Event Hub Getting Started
• Service Bus Event Hub Large Scale Secure Publishing
• Service Bus Event Hub Direct Receivers

Microsoft Azure MVP (2014)

Being recognized for something I’m so passionate about is amazing! The past year has truly been rewarding. It was a year of change, opportunity and challenges.

I’m honored to receive the 2014 Microsoft Azure MVP award.

I joined the Microsoft Azure MVPs last July and was welcomed by a group of remarkable individuals. Inspired by our energetic community, I strive to learn and share about what it means to build applications for the cloud.

2013 was a year of change for the Microsoft Azure community. Services, Partners and Technologies have made their way to the cloud. Consumers have a rich ecosystem of services with which to compose their solutions. These are very exciting times for developers who tinker with the cloud. Competition, creativity and business opportunities create the perfect setting for a fast pace evolution. I really think that we’re only seeing the tip of the iceberg, 2014 is going inspiring!

I’d like to thank everyone who contributed over the past year. I’m very grateful for your questions, insights and opinions. The adventure has been very interesting and I truly hope that you will continue to be part of it.

Thank you for this exceptional award.

The Microsoft Most Valuable Professional (MVP) Award is our way of saying thank you to exceptional, independent community leaders who share their passion, technical expertise, and real-world knowledge of Microsoft products with others. It is part of Microsoft’s commitment to supporting and enriching technical communities. Even before the rises of the Internet and social media, people have come together to willingly offer their ideas and best practices in technical communities.

Microsoft MVP web site

Microsoft Azure is a vast ecosystem of services that keeps on growing. As a new comer to this ecosystem, you may feel overwhelmed. But don’t worry, I wrote this post to answer questions like “where should I start?” and “What resources are available to me?”

Where Should I Start?

This is often the hardest thing to identity when we start tinkering with Azure. I started exploring Azure by reading free e-books that you can download from the Patterns & Practices website.

The first books I read were

• Moving Applications to the Cloud, Third Edition on Microsoft Azure
• Developing Multi-tenant Applications for the Cloud, Third Edition on Microsoft Azure
• Building Hybrid Applications in the Cloud on Microsoft Azure

Then I read Cloud Design Patterns. The Patterns & Practices team has also written a book about CQRS Journey, which can greatly influence your overall design.

If you’re looking to buy a book, I highly recommend Cloud Architecture Patterns: Using Microsoft Azure, a book written by Bill Wilder. This should probably be the first book you read. It’s packed with tons of valuable information that applies to most cloud vendors.

Other e-books

• Developing big data solutions on Microsoft Azure HDInsight
• Building an On-Demand Video Service with Microsoft Azure Media Services
• Enterprise Library 5.0 Integration Pack for Microsoft Azure

I found these e-books to be full of valuable information. They helped me define and architect various cloud native solutions. Microsoft has also created a section about Architecture on the Azure portal. This section covers patterns, best practices and provides you with a Visio stencil that I find very helpful when I design Azure solutions.

If you’re like me, you’ll also like to take some time to browse through guidance around best practices on design, deployment, troubleshooting, security, and performance provided by Microsoft.

More Guidance on Azure

Microsoft has also published scenario-based diagrams that help you build on Azure. These blueprints can provide you with a starting point for your own solution architecture.

Don’t forget about case studies, these can help you build arguments to help you promote the cloud within your organization. One of the biggest challenges for cloud adoption, is resistance to change and the lack of context around one’s desire to move to the cloud.

More Resources on Azure

Microsoft has an amazing collection of free online courses about Azure.

• Designing Applications for Windows Azure Jump Start
• Moving to Hybrid Cloud with Microsoft Azure
• Windows Azure SQL Database
• Windows Azure Storage – Design and Implementation Jump Start
• Windows Azure Security Overview
• Windows Azure for IT Pros Jump Start
• DevOps: An IT Pro Guide

Visit Microsoft Virtual Academy (MVA) for the complete list.

More Posts on Azure

From time to time, I like to write about how to approach Azure. These are a few of my posts about the platform and my adventures on Azure.

• What is Azure?
• Is your application built for the cloud?
• Create a Dev & Test Environment in Minutes!
• How do you build features on #Azure?
• Brownfield Azure solutions & hidden treasures
• Why is Software Architecture important?
• Reaching performance targets on Azure by preparing and pre-calculating data
• Times have changed, Keep Up!
• Designing Cloud Services – consider bulkiness & chattiness
• Keep your privates private!
• Turbo charging your Azure Cloud Services
• Azure Storage Best Practices
• Does your data really belong in your SQL Database?
• Continuous Services require continuous attention

In Retrospect

Azure is alive and kicking, its release cycle is quite short and we continuously see new features make their way to general availability. Keep a close eye on the official Azure blog and Azure podcasts. Don’t forget about events like Build whose recorded sessions can be found on Channel 9.

HTTP/1.1 503 Service Unavailable

On Azure we must design with cost of operation and service constraints. I recently had an interesting event where my REST Service, deployed on a small (A1) cloud service instance, started to respond with HTTP Status Code 503 Service Unavailable.

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response. [Source: HTTP Status Code Definitions]

Faced with this interesting challenged I started looking in the usual places, which include SQL Database metrics, Azure Storage Metrics, application logs and performance counters.

Throughout my investigation, I noticed that the service was hit on average 1.5 million times a day. Then, I noticed that the open socket count was quite high. Trying to make some sense out of the situation, I started identifying resource contentions.

Looking at the application logs didn’t yield much information about why the network requests were piling up, but it did hint at internal process slowdowns following peak loads.

Working on extracting logs from Azure Table Storage I finally got a break. I noticed that the service was generating 5 to 10 megabytes of application logs per minute. To put this into perspective, the service requires enough IO to respond to consumer requests, to push performance counter data to Azure storage, to persist application logs to Azure storage and enough IO capacity to satisfy the application’s need to interact with external resources.

Base on my observations I came to the conclusion that my resource contention was around IO. Now, I rarely recommend scaling up, but in this case it made sense because the service move lots of data around. One solution would have been to turn off telemetry. But doing so would have made me as comfortable as if I were flying a jumbo jet with a blindfold on. In other words, this isn’t something I want to consider because I believe that telemetry is crucial when it comes to understanding how an application behaves on the cloud.

Just to be clear, scaling up worked in this situation, but it may not resolve all your issues. There are times when we need to tweak IIS configurations to make better use of resources and to handle large amounts of requests.Scaling up should remain the last option of your list.

Related Posts

• Lessons Learned – On #Azure, Log Everything!
• Choosing The Right Compute Virtual Machine Size

#Azure Storage Tables – DateTime.MinValue is not Within the Supported DateTime Range

Azure Table Storage is a NoSQL storage key/value based part of Microsoft Azure Storage services. It’s really good at absorbing massive amounts of data. It’s really good at massive parallel operations work on small amounts of data. But it’s horrible when it comes time to extract large amounts of data in a serial manner. We’ll get to that topic in a future post.

DateTime is a type that is used to represent time in .NET. We use it very liberally without much afterthought. But when we start playing with Azure Table Storage, we have to think about the supported DateTime range. At this time, it’s also important to note that the local Azure Storage Emulators run on various database flavors like SQL Server and LocalDB. These databases do not impose the same limitations for DateTime values. Therefore, this bug will only show up on Microsoft Azure.

In order to limit headaches, I’m providing the following table to help identify what is supported and what isn’t.

Find out More about Azure Storage

• Understanding the Table Service Data Model
• Migrating Data to Azure Table Storage
• Azure Table Storage and Azure SQL Database – Compared and Contrasted