GauravMantri.com

Windows Azure Blob Storage – Dealing With “The specified blob or block content is invalid” Error

Gaurav — Sat, 18 May 2013 07:42:33 +0000

If you’re uploading blobs by splitting blobs into blocks and you get the error – The specified blob or block content is invalid, then this post is for you.

Short Version

If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error.

Long Version

Now for the longer version of this post . A few days back I was working with storage client library especially around uploading blobs in chunks and with one particular blob I was constantly getting the error – The specified blob or block content is invalid. I tried numerous combinations even resorting to REST API directly but to no avail. It only happened with just one blob. Furthermore if I uploaded the same blob without splitting it into blocks, all was well. I was at my wits’ end. Tried searching the Internet for this error but could not find a conclusive answer to my problem.

After much trial and error, I was able to simulate the same problem on other blobs as well. Here’s how you can recreate it:

Start uploading the blob by splitting it into blocks. For block id, let’s do a 7 character long string e.g. intValue.ToString(“d7”). This will ensure that my block ids would be “0000001”, “0000002”, …, ”0000010” …..
After one or two blocks are uploaded, cancel the operation.
Now re-upload the blob by splitting it into blocks. However this time for block id, let’s do a 6 character long string e.g. intValue.ToString(“d6”).
You’ll get the error as soon as you try to upload the 1^st block.

Possible Solutions

Now that we know the root cause of this problem, let’s look at some of the possible solutions to solve this problem.

Wait out

One possible solution is to wait out. I know its lame but still a possible solution. We know that Windows Azure Blob Storage Service keeps all uncommitted blocks for a duration of 7 days and if within 7 days those uncommitted blocks are not committed, the storage service purges them.

I wish storage service provided some mechanism to purge uncommitted blocks programmatically.

Commit uncommitted blocks

You could possibly commit the blocks which are in uncommitted state so that at least you get a blob (which would not be the blob we wanted to upload in the first place). You can then delete that blob and re-upload the blob by specifying block ids which are of same length. To fetch the list of uncommitted blocks, if you’re using REST API directly you can perform “Get Block List” operation and pass “blocklisttype=uncommitted” as one of the query string parameters. If you’re using storage client library (assuming you’re using the version 2.x of .Net storage client library), you can do something like the code below:

        private static List GetUncommittedBlockIds(CloudBlockBlob blob)
        {
            var sasUri = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
            {
                SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5),
                Permissions = SharedAccessBlobPermissions.Read,
            });
            var blobUri = new Uri(string.Format("{0}{1}", blob.Uri, sasUri));
            List uncommittedBlockIds = new List();
            var request = BlobHttpWebRequestFactory.GetBlockList(blobUri, null, null, BlockListingFilter.Uncommitted, null, null);
            //request.Headers.Add("Authorization", 
            using (var resp = (HttpWebResponse)request.GetResponse())
            {
                using (var stream = resp.GetResponseStream())
                {
                    var getBlockListResponse = new GetBlockListResponse(stream);
                    var blocks = getBlockListResponse.Blocks;
                    foreach (var block in blocks.Where(b => !b.Committed))
                    {
                        uncommittedBlockIds.Add(Encoding.UTF8.GetString(Convert.FromBase64String(block.Name)));
                    }
                }
            }
            return uncommittedBlockIds;
        }

A few things to keep in mind here:

Microsoft.WindowsAzure.Storage.Blob namespace does not have the capability to get the list of uncommitted blocks. You would need to make use of Microsoft.WindowsAzure.Storage.Blob.Protocol namespace.
Because we’re kind of invoking the REST API by executing an HttpWebRequest, I created a shared access signature on the blob so that I don’t have to create “Authorization” header.

Fetch uncommitted blocks to see block id length

You could fetch the list of uncommitted blocks just to find out the length of the block id used. You could then use that block id length for your new upload session and do the upload. Please see the code snippet above to find this information.

Upload another blob with same name without splitting it into blocks

You could also upload another blob with the same name without splitting it into blocks. It could very well be a zero byte blob. That way your uncommitted block list will be wiped clean. Then you could delete that dummy blob and re-upload the actual blob.

A Few Words About Blocks

Since we’re talking about blocks, I thought it might be useful to mention a few points about them:

Blocks and block related operations are only applicable for “Block Blobs”. Duh!! You’ll get an error if you’re trying to do these operations on a “Page Blob”.
For uploading large blobs, it is recommended that you split your blob into blocks. In fact if your blob size is more than 64 MB, then you have to split it into blocks.
Minimum size of a block is 1 Byte and the maximum size of a block is 4 MB. It is recommended that you choose a block size based on your internet connectivity and number of parallel threads you want use to upload these blocks.
A blob can be split into a maximum of 50000 blocks. It’s important to remember this limitation because you are reminded of this limit when you’re trying to upload 50001^st block.
The length of all the block ids must be same. So if you’re using an integer value to denote block id, you make sure that you pad that integer value with “0” so that you get same length. So you could do something like int.ToString(“d6”).
When passing the block id as a parameter, it must be Base64 encoded.
While the order in which the blocks are uploaded is not important, the order is important when you commit the block list because that’s when the blob is constructed by the service. For example, let’s say you’re uploading a blob by splitting it into 5 blocks (with ids “000001”, “000002”, “000003”, “000004”, and “000005”). You could upload these blocks in any order – 000004, 000001, 000003, 000005, 000002 however when you commit the block list, ensure that the block ids are passed in proper order i.e. 000001, 000002, 000003, 000004, 000005.

Summary

That’s it for this post. I hope you’ve found this information useful. I spent considerable amount of time trying to fix this problem so I hope it will help some folks out. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Happy Coding!

Windows Azure SDK 2.0 For .Net – Taking A Second Look At Windows Azure Diagnostics

Gaurav — Sun, 12 May 2013 15:43:33 +0000

A few days ago I wrote a post about newly released Windows Azure SDK 2.0 for .Net. You can read that post here: http://gauravmantri.com/2013/04/30/introducing-windows-azure-sdk-2-0-for-net/. In that post I briefly talked about the improvements in the latest SDK with regards to Windows Azure Diagnostics (WAD). In this post, we’ll talk about WAD in more details. We’ll talk about improvements in SDK as well as changes in the Service Management API to facilitate these changes.

So let’s start.

Changes to Windows Azure Diagnostics

First let’s talk about changes done to Windows Azure Diagnostics.

Windows Azure Diagnostics is now an Extension

Windows Azure Diagnostics (WAD) module is now an extension to your cloud service. I wrote a blog post on extensions which you can read here: http://gauravmantri.com/2013/05/06/windows-azure-cloud-services-extensions-and-service-management-api-fun-with-remote-desktop/. What that means is that you can enable / disable diagnostics functionality on the fly. If you’ve been working with cloud services, I think you’ll greatly appreciate this enhancement. In the cloud services which used previous version (say 1.8) of the SDK, enabling diagnostics was a pain. One would need to remember a lot of things to get the diagnostics working and if one forgets to do those things, the only option to make diagnostics working was to go back to your code and make the changes and redeploy your application. NOT ANYMORE! Even if you forget to enable diagnostics when you first deploy your cloud service, you can enable the diagnostics on the fly using the extensions mechanism. Furthermore, you don’t really need to write any code in your role’s OnStart() method to enable the diagnostics.

We’ll see an example of how you can do this later in this post.

Configuring Windows Azure Diagnostics has never been easier

With the latest SDK, it’s extremely easy to configure Windows Azure Diagnostics (WAD). Visual Studio provides you with an easy to use user interface to configure the diagnostics. To configure WAD, open up your cloud service configuration by double clicking on the role’s name and go to “Configuration” tab. Under the “Diagnostics” section, you’ll see various configuration options as shown in the screenshot below:

If you’ve been using previous versions of SDK, you’re used to seeing this kind of screen:

As you can see from the screenshots, not only you get to choose whether the diagnostics should be enabled or disabled you also get a simple to use user interface to configure the diagnostics. Let’s talk about these options.

Errors only

When you choose “Errors Only” option, WAD extension transfers only the log entries with log level type “error” and “critical” to storage. A few things to keep in mind:

This option will only enable following log types – trace and event logs.
This option will not do anything with the performance counters. So no performance counters data will be transferred to storage.
As said earlier, only “error” and “critical” log entries will be transferred to storage.
By default the logs will be transferred to storage every minute or in other words the scheduled transfer period is one minute.

Following code segment shows what kind of trace log entries will be transferred if this option is chosen:

            Trace.WriteLine("This is verbose trace entry", "Verbose");//Will NOT be transferred.
            Trace.WriteLine("This is information trace entry", "Information");//Will NOT be transferred.
            Trace.WriteLine("This is warning trace entry", "Warning");//Will NOT be transferred.
            Trace.WriteLine("This is error trace entry", "Error");//Will BE transferred.
            Trace.WriteLine("This is critical trace entry", "Critical");//Will BE transferred.

All information

When you choose “All information” option, WAD extension transfers all log entries to storage. A few things to keep in mind:

This option will only enable following log types – trace and event logs.
This option will not do anything with the performance counters. So no performance counters data will be transferred to storage.
As said earlier, all log entries will be transferred to storage. This would include the following log level types: Verbose, Information, Warning, Error and Critical.
By default the logs will be transferred to storage every minute or in other words the scheduled transfer period is one minute.

Following code segment shows what kind of trace log entries will be transferred if this option is chosen:

            Trace.WriteLine("This is verbose trace entry", "Verbose");// Will BE transferred.
            Trace.WriteLine("This is information trace entry", "Information");// Will BE transferred.
            Trace.WriteLine("This is warning trace entry", "Warning");// Will BE transferred.
            Trace.WriteLine("This is error trace entry", "Error");//Will BE transferred.
            Trace.WriteLine("This is critical trace entry", "Critical");//Will BE transferred.

Custom plan

Custom plan is where things get interesting. Here you get full flexibility for configuring WAD. This is where you’re in complete control of what is captured and what gets transferred to storage. Here’re a few screenshots of the window where you can customize the diagnostics:

Trace Logs

Event Logs

Performance Counters

My guess is that most of the time developers would use these screens to configure diagnostics.

Diagnostics.wadcfg file

If you look closely in your solution explorer, you’ll notice a diagnostics.wadcfg file beneath your role name. You can read more about this file here: http://msdn.microsoft.com/en-us/library/windowsazure/hh411551.aspx.

Essentially when you configure diagnostics using one of the options above, Visual Studio automatically updates this file. This is not a new concept and has been around for quite some time. What’s new is that now Visual Studio creates this file for you. So if you want to tweak some settings after Visual Studio has updated the file, just modify the XML file manually and you should be good to go.

Good Bye Old Storage Client Library

SDK 1.8 came with a new version of storage client library (2.0.0.0) however Windows Azure Diagnostics (WAD) was not updated to make use of this library. It was still using older version of storage client library. Needless to say, it was a big mess. Not only you would need to remember to add reference to both versions of the library but also take into consideration the different namespaces of older and newer versions of storage client library. With the release of SDK 2.0, WAD library is updated to make use of the latest version of storage client library which is 2.0.5.1 at the time SDK 2.0 was made live.

Diagnostics Extension

In this section we’ll talk about diagnostics extension and how it can be used to enable/disable or change diagnostics on the fly. I would recommend going through my blog post on extensions first because we’ll make use of code from that post. You can read that post here: http://gauravmantri.com/2013/05/06/windows-azure-cloud-services-extensions-and-service-management-api-fun-with-remote-desktop/.

For fun sake, we’ll create a cloud service (a simple worker role will do) and choose not to enable diagnostics. When we do that, out diagnostics.wadcfg file looks something like this:

This is how my worker role’s code looks like:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.Storage;

namespace WorkerRole1
{
    public class WorkerRole : RoleEntryPoint
    {
        public override void Run()
        {
            // This is a sample worker implementation. Replace with your logic.
            Trace.TraceInformation("WorkerRole1 entry point called", "Information");

            while (true)
            {
                Thread.Sleep(10000);
                Trace.WriteLine("This is verbose trace entry", "Verbose");//Will NOT be transferred.
                Trace.WriteLine("This is information trace entry", "Information");//Will NOT be transferred.
                Trace.WriteLine("This is warning trace entry", "Warning");//Will BE transferred.
                Trace.WriteLine("This is error trace entry", "Error");//Will BE transferred.
                Trace.WriteLine("This is critical trace entry", "Critical");//Will BE transferred.
            }
        }

        public override bool OnStart()
        {
            // Set the maximum number of concurrent connections 
            ServicePointManager.DefaultConnectionLimit = 12;

            // For information on handling configuration changes
            // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.

            return base.OnStart();
        }
    }
}

Nothing special really. One thing you’ll notice in the OnStart() method is that I’ve not included initialization code for diagnostics. In prior versions of the SDK, I would need to add the following 2 lines of code for diagnostics to work:

            DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);

And this is how my configuration file looks like. If you notice, there’s no diagnostics connection string there.

Let’s publish this cloud service. To ensure Visual Studio is not doing anything “funky”, we’ll just package the file and upload it using Azure Management Studio. If you want, you can upload it through the portal as well.

Now it’s time to get a bit dramatic . For the sake of adding drama to the whole equation, let’s assume that you’re working with a remote team with significant time difference so that you don’t really talk with your remote team in real time but using emails and stuff. The remote team is responsible for the code and deployment and in all their wisdom, they just deployed the service with code as above i.e. with no diagnostics enabled but you don’t know that. Suddenly you start getting calls from your users that the application is running slow and they are not able to do anything there. On top of that, your boss is breathing down your neck to find the problem. Your first reaction is to fire your favorite tools to see diagnostics data to figure out what’s going on but then you realize that your “smart ass remote team ” forgot to enable diagnostics. Following sections will help you get out of the murky water and make you an office superhero!!! Let’s begin.

Step 1: Get the details of diagnostics extension

Using the code from my previous post, you get the details of diagnostics extension so that you can fetch public and private configuration schema for diagnostics:

            var allExtensions = ListAvailableExtensions(subscriptionId, cert);
            var diagnostics = allExtensions.FirstOrDefault(e => e.Type == "Diagnostics");

Public Configuration Schema:

The maximum amount of file system storage allocated for the specified data.

The interval between scheduled transfers for this data, rounded up to the nearest minute.

The minimum log severity to transfer.

The name of the container where the content of the directory is to be transferred.

The maximum size of the directory in megabytes.

The absolute path to the directory to monitor.

If true, then environment variables in the path will be expanded.

The path relative to the local resource to monitor.

The local resource that contains the directory to monitor.

The absolute path to the directory to monitor.

The path relative to a local resource to monitor.

The directory of log files to monitor.

The IIS log directory.

The failed request log directory.

The crash dump directory.

Additional log directories.

The path to the performance counter to collect.

The rate at which the performance counter should be sampled.

The performance counter to collect.

An XPath expression specifying the logs to collect.

The event log to monitor.

Configures the logs generated by the underlying diagnostics infrastructure. The diagnostic infrastructure logs are useful for troubleshooting the diagnostics system itself.

Describes the configuration of a directory to which file-based logs are written.

Configures basic Windows Azure logs.

Configures performance counter collection.

Configures Windows event log collection.

The interval at which the diagnostic monitor polls for configuration changes.

The total amount of file system storage allocated for all logging buffers.

Private Configuration Schema:

Based on these, you can generate XML template for public and private configurations.

Public Configuration XML Template:



  
    ConnectionQualifiers1
    https
    Name1

Private Configuration XML Template:



  StorageKey1

Step 2: Create XML files

Using the templates from the previous steps, we create public and private configuration XML files. Let’s assume that we’re only interested in capturing all trace log entries, “Application” event logs and 2 performance counters – “\Processor(_Total)\% Processor Time” and “\Memory\Available Mbytes”. Based on this, this is how our XML files look like:

Public Configuration XML:



  
    
    https
    [your storage account name]

Private Configuration XML:



  [your storage account key]

Step 3: Add WAD Extension

To add WAD extension, we’ll use public and private key configuration for the extension and add that extension to the cloud service.

            string publicConfiguration = @"
                
                  
                    
                    https
                    [your storage account name]
                  
                  
                    
                      
                      
                        
                        
                      
                      
                      
                        
                        
                      
                      
                        
                      
                    
                  
                ";

            string privateConfiguration = @"
                
                  [your storage account key]
                ";

            AzureExtension wadExtension = new AzureExtension()
            {
                ProviderNamespace = "Microsoft.Windows.Azure.Extensions",
                Type = "Diagnostics",
                Id = "DiagnosticsExtensionWhichWillSaveMyButt",
                PublicConfiguration = publicConfiguration,
                PrivateConfiguration = privateConfiguration,
            };

            AddExtension(subscriptionId, cert, cloudServiceName, wadExtension);

Step 4: Update Configuration

Once the extension is added, next step would be to update the cloud service configuration by performing “Change Deployment Configuration” operation. For this, first we may need to fetch the current configuration. To do so, we’ll perform “Get Hosted Service Properties” operation and try to get both production and staging configuration settings. Here’s the sample code to get the configuration settings:

        private static string[] GetCloudServiceDeploymentConfigurations(string subscriptionId, X509Certificate2 cert, string cloudServiceName)
        {
            string[] deploymentConfigurations = new string[2];//We'll try to get both production and staging deployment configurations. The 1st element will always be production and the 2nd will be staging.
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}?embed-detail=true", subscriptionId, cloudServiceName);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "GET";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (var streamReader = new StreamReader(resp.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    XElement cloudServiceProperties = XElement.Parse(result);
                    var deployments = cloudServiceProperties.Elements(XName.Get("Deployments", "http://schemas.microsoft.com/windowsazure"));
                    foreach (var deployment in deployments.Elements())
                    {
                        var slotElement = deployment.Element(XName.Get("DeploymentSlot", "http://schemas.microsoft.com/windowsazure"));
                        var configElement = deployment.Element(XName.Get("Configuration", "http://schemas.microsoft.com/windowsazure"));
                        var deploymentSlot = slotElement.Value;
                        var configurationSettings = Encoding.UTF8.GetString(Convert.FromBase64String(configElement.Value));
                        switch (deploymentSlot.ToUpper())
                        {
                            case "PRODUCTION":
                                deploymentConfigurations[0] = configurationSettings;
                                break;
                            case "STAGING":
                                deploymentConfigurations[1] = configurationSettings;
                                break;
                        }
                    }
                }
            }
            return deploymentConfigurations;
        }

Now that we’ve got the configurations for both production and staging slots, let’s apply the changes. Here’s the sample code for change deployment configuration operation:

        private static void ChangeDeploymentConfiguration(string subscriptionId, X509Certificate2 cert, string cloudServiceName, string slot, string configuration)
        {
            try
            {
                string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/deploymentslots/{2}/?comp=config", subscriptionId, cloudServiceName, slot);
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
                request.Method = "POST";
                request.ContentType = "application/xml";
                request.Headers.Add("x-ms-version", "2013-03-01");
                request.ClientCertificates.Add(cert);
                byte[] content = Encoding.UTF8.GetBytes(configuration);
                request.ContentLength = content.Length;
                using (var requestStream = request.GetRequestStream())
                {
                    requestStream.Write(content, 0, content.Length);
                }

                using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
                {
                }
            }
            catch (WebException webEx)
            {
                using (var streamReader = new StreamReader(webEx.Response.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    Console.WriteLine(result);
                }
            }

        }

And this is how you call it:

            string updateConfigurationFormat = @"
                
                  {0}
                
                    
                      
                        {1}
                      
                    
                 
                ";

            var productionConfig = configurations[0];
            if (!string.IsNullOrWhiteSpace(productionConfig))
            {
                var productionConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(productionConfig)), wadExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Production", productionConfigurationSetting);
            }

Putting it all together nicely in just one function:

        private static void AddDiagnosticsExtension(string subscriptionId, X509Certificate2 cert, string cloudServiceName)
        {
            string publicConfiguration = @"
                
                  
                    
                    https
                    [your storage account name]
                  
                  
                    
                      
                      
                        
                        
                      
                      
                      
                        
                        
                      
                      
                        
                      
                    
                  
                ";

            string privateConfiguration = @"
                
                  [your storage account key]
                ";

            string updateConfigurationFormat = @"
                
                  {0}
                
                    
                      
                        {1}
                      
                    
                 
                ";

            AzureExtension wadExtension = new AzureExtension()
            {
                ProviderNamespace = "Microsoft.Windows.Azure.Extensions",
                Type = "Diagnostics",
                Id = "DiagnosticsExtensionWhichWillSaveMyButt",
                PublicConfiguration = publicConfiguration,
                PrivateConfiguration = privateConfiguration,
            };

            AddExtension(subscriptionId, cert, cloudServiceName, wadExtension);

            var configurations = GetCloudServiceDeploymentConfigurations(subscriptionId, cert, cloudServiceName);
            var productionConfig = configurations[0];
            if (!string.IsNullOrWhiteSpace(productionConfig))
            {
                var productionConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(productionConfig)), wadExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Production", productionConfigurationSetting);
            }
            var stagingConfig = configurations[1];
            if (!string.IsNullOrWhiteSpace(stagingConfig))
            {
                var stagingConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(stagingConfig)), wadExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Staging", stagingConfigurationSetting);
            }
        }

That’s pretty much it!!! Once the operation is complete, you should be able to see the diagnostics data for your cloud service in some time.

Summary

I strongly believe that changes done to diagnostics in SDK 2.0 are pretty significant. From my personal experience working with various users of Cerebrata Azure Diagnostics Manager, I do believe that it’s a boon to developers. Now it is very easy and straight forward to configure the diagnostics and change it on the fly. That’s it for this post. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Happy Coding!

Windows Azure Cloud Services, Extensions and Service Management API – Fun with Remote Desktop

Gaurav — Mon, 06 May 2013 11:47:43 +0000

I want you to try something for me (pretty please with cherry on top ). Fire up Visual Studio, create a simple Windows Azure Cloud Service and then without making any changes just publish that service. When you publish the service, DO NOT ENABLE REMOTE DESKTOP!

Once the service is deployed, head on to Windows Azure Portal (https://manage.windowsazure.com), navigate to the cloud service you just deployed and click on the “CONFIGURE” tab. Once there, just click on the “REMOTE” button below. You’ll be greeted with the following screen:

Now follow the instructions there and enable remote desktop. Once the process is complete, go to “INSTANCES” tab and click on “CONNECT” button below. What you’ll notice is that you’re able to RDP into your instances. If you’ve been using Windows Azure for some time, you know that in order to remote desktop into your instances you would have to enable this functionality when you’re publishing the service. If you forgot doing this at the time of deployment, you would need to go through an upgrade or new deployment process just to enable this functionality.

However what you saw just now is that you did not do anything of that sort. You published the service and then enabled this functionality on the fly! So, how did this happen? I’ve got one sentence for you:

Your Cloud Service just got Extensions!

In this blog post, we’ll talk about “Cloud Services Extensions”. We’ll talk about what they are, how you can work with them and finally we’ll see some code which will allow you to enable/disable this functionality on the fly.

So, let’s get started!

Extensions

Extensions are the newest functionality available through Windows Azure Service Management API. The way I understand it is that extensions allow you to dynamically add/remove some functionality in your cloud services without redeploying your code. At the time of writing this blog, there’re two things which you could do – Enable/Disable Remote Desktop (RDP) and Enable/Disable Windows Azure Diagnostics (WAD). Yes, you read it right!! Now you can enable/disable diagnostics for your cloud services on the fly without redeploying your service. No more messy code for diagnostics configuration and trying to remember things. I’ll cover diagnostics in a separate post. For now let’s focus on core extensions functionality.

The way it works today is that Windows Azure team has published some pre-defined extensions and you could enable/disable these extensions in your cloud service using Service Management API. Service Management API has added some functions to work with these extensions. The way the extensions functionality work is that you define settings associated with that extension; some of these settings will be public (i.e. you can fetch them later on e.g. RDP username) while some of the settings are private (i.e. you can’t fetch them after they’re set e.g. RDP password). You would define public settings in a public configuration file and private settings in a private configuration file. The configuration file are XML files. In order to determine the structure of these XML files, Service Management API exposes relevant schemas (public configuration schema and private configuration schema) using which you can construct XML files. Once the XML configuration files are defined, you can add an extension to your cloud service. However adding an extension does not do any magic. You would need to apply that extension to your cloud service which you can do through “Change Deployment Configuration” operation.

In this section we’ll focus on these functions and we’ll put them to practical use in next section. What I did was create a simple console application and defined some classes and methods to work with the extensions.

Based on the data returned by various functions for extensions management, I created a simple class called “AzureExtension” and this is what it looks like:

    public class AzureExtension
    {
        /// 
        /// The provider namespace of the extension. The provider namespace for 
        /// Windows Azure extensions is Microsoft.WindowsAzure.Extensions.
        /// 
        public string ProviderNamespace
        {
            get;
            set;
        }

        /// 
        /// The type of the extension e.g. RDP for Remote Desktop.
        /// 
        public string Type
        {
            get;
            set;
        }

        /// 
        /// The version of the extension.
        /// 
        public string Version
        {
            get;
            set;
        }

        /// 
        /// The label that is used to identify the extension.
        /// 
        public string Label
        {
            get;
            set;
        }

        /// 
        /// The description of the extension.
        /// 
        public string Description
        {
            get;
            set;
        }

        /// 
        /// The type of resource that supports the extension. This value can be 
        /// WebRole, WorkerRole, or WebRole|WorkerRole.
        /// 
        public string HostingResource
        {
            get;
            set;
        }

        /// 
        /// The thumbprint algorithm of the certificate that is used for encryption.
        /// 
        public string ThumbprintAlgorithm
        {
            get;
            set;
        }

        /// 
        /// The thumbprint of the certificate that is used to encrypt the configuration specified in PrivateConfiguration. 
        /// If this element is not specified, a certificate may be automatically generated and added to the cloud service.
        /// 
        public string Thumbprint
        {
            get;
            set;
        }

        /// 
        /// The base64-encoded schema of the public configuration.
        /// 
        public string PublicConfigurationSchema
        {
            get;
            set;
        }

        /// 
        /// XML configuration based on PublicConfigurationSchema
        /// 
        public string PublicConfiguration
        {
            get;
            set;
        }

        /// 
        /// The base64-encoded schema of the private configuration.
        /// 
        public string PrivateConfigurationSchema
        {
            get;
            set;
        }

        /// 
        /// XML configuration based on PrivateConfigurationSchema
        /// 
        public string PrivateConfiguration
        {
            get;
            set;
        }

        /// 
        /// Extension id.
        /// 
        public string Id
        {
            get;
            set;
        }

        public static AzureExtension Parse(XElement extensionXml)
        {
            var extension = new AzureExtension();
            foreach (var xe in extensionXml.Elements())
            {
                var elementName = xe.Name.LocalName;
                var elementValue = xe.Value;
                switch (elementName.ToUpperInvariant())
                {
                    case "PROVIDERNAMESPACE":
                        extension.ProviderNamespace = elementValue;
                        break;
                    case "TYPE":
                        extension.Type = elementValue;
                        break;
                    case "VERSION":
                        extension.Version = elementValue;
                        break;
                    case "THUMBPRINT":
                        extension.Thumbprint = elementValue;
                        break;
                    case "THUMBPRINTALGORITHM":
                        extension.ThumbprintAlgorithm = elementValue;
                        break;
                    case "PUBLICCONFIGURATION":
                        extension.PublicConfiguration = Encoding.UTF8.GetString(Convert.FromBase64String(elementValue));
                        break;
                    case "PRIVATECONFIGURATION":
                        extension.PrivateConfiguration = Encoding.UTF8.GetString(Convert.FromBase64String(elementValue));
                        break;
                    case "PUBLICCONFIGURATIONSCHEMA":
                        extension.PublicConfigurationSchema = Encoding.UTF8.GetString(Convert.FromBase64String(elementValue));
                        break;
                    case "PRIVATECONFIGURATIONSCHEMA":
                        extension.PrivateConfigurationSchema = Encoding.UTF8.GetString(Convert.FromBase64String(elementValue));
                        break;
                    case "LABEL":
                        extension.Label = elementValue;
                        break;
                    case "DESCRIPTION":
                        extension.Description = elementValue;
                        break;
                    case "HOSTINGRESOURCE":
                        extension.HostingResource = elementValue;
                        break;
                    case "ID":
                        extension.Id = elementValue;
                        break;
                }
            }
            return extension;
        }

        public XElement ConvertToXml()
        {
            XElement extensionXElement = new XElement(XName.Get("Extension", "http://schemas.microsoft.com/windowsazure"));
            extensionXElement.Add(new XElement(XName.Get("ProviderNameSpace", "http://schemas.microsoft.com/windowsazure"), this.ProviderNamespace));
            extensionXElement.Add(new XElement(XName.Get("Type", "http://schemas.microsoft.com/windowsazure"), this.Type));
            extensionXElement.Add(new XElement(XName.Get("Id", "http://schemas.microsoft.com/windowsazure"), this.Id));
            if (!string.IsNullOrWhiteSpace(this.Thumbprint))
            {
                extensionXElement.Add(new XElement(XName.Get("Thumbprint", "http://schemas.microsoft.com/windowsazure"), this.Thumbprint));
                extensionXElement.Add(new XElement(XName.Get("ThumbprintAlgorithm", "http://schemas.microsoft.com/windowsazure"), this.ThumbprintAlgorithm));
            }
            extensionXElement.Add(new XElement(XName.Get("PublicConfiguration", "http://schemas.microsoft.com/windowsazure"), Convert.ToBase64String(Encoding.UTF8.GetBytes(this.PublicConfiguration))));
            extensionXElement.Add(new XElement(XName.Get("PrivateConfiguration", "http://schemas.microsoft.com/windowsazure"), Convert.ToBase64String(Encoding.UTF8.GetBytes(this.PrivateConfiguration))));
            return extensionXElement;
        }
    }

Now that the entity to manipulate extensions is defined, let’s look at the functions.

List Available Extensions

This function returns the list of all extensions available to your subscription. Here’s the sample code for listing all extensions available to you:

        /// 
        /// Gets a list of all available extensions.
        /// 
        /// 
        /// Subscription id of the subscription.
        /// 
        /// 
        /// Management certificate for authenticating Service Management API requests.
        /// 
        /// 
        /// 
        private static IEnumerable ListAvailableExtensions(string subscriptionId, X509Certificate2 cert)
        {
            List extensions = new List();
            string uri = string.Format("https://management.core.windows.net/{0}/services/extensions", subscriptionId);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "GET";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (var streamReader = new StreamReader(resp.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    Console.WriteLine(result);
                    XElement extensionImagesXelement = XElement.Parse(result);
                    foreach (var extensionImageXelement in extensionImagesXelement.Elements(XName.Get("ExtensionImage", "http://schemas.microsoft.com/windowsazure")))
                    {
                        extensions.Add(AzureExtension.Parse(extensionImageXelement));
                    }
                }
            }
            return extensions;
        }

For more details, please click here: http://msdn.microsoft.com/en-us/library/windowsazure/dn169559.aspx.

Add Extension

This function adds an extension to the list of available extensions in a cloud service. As mentioned above, after adding an extension you would need to perform “Change Deployment Configuration” operation for that extension to be enabled on a cloud service. Here’s the sample code for adding an extension to a cloud service:

        /// 
        /// Adds an extension to a cloud service. Just by adding an extension won't do any good!
        /// You must call the "Change Deployment Configuration" to apply this extension to the 
        /// running instance of your cloud service.
        /// 
        /// 
        /// Subscription id of the subscription.
        /// 
        /// 
        /// Management certificate for authenticating Service Management API requests.
        /// 
        /// 
        /// Name of cloud service.
        /// 
        /// 
        /// Extension which needs to be added.
        /// 
        private static void AddExtension(string subscriptionId, X509Certificate2 cert, string cloudServiceName, AzureExtension extension)
        {
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/extensions", subscriptionId, cloudServiceName);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "POST";
            request.ContentType = "application/xml";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            string requestPayload = string.Format("{0}", extension.ConvertToXml());
            byte[] content = Encoding.UTF8.GetBytes(requestPayload);
            request.ContentLength = content.Length;
            using (var requestStream = request.GetRequestStream())
            {
                requestStream.Write(content, 0, content.Length);
            }

            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
            }
        }

Please note that when adding extension, if certificate thumbprint is not provided Service Management API automatically creates a certificate for you and associate that with the cloud service.

For more details, please click here: http://msdn.microsoft.com/en-us/library/windowsazure/dn169558.aspx.

List Extensions

This function returns all the extensions defined for a particular cloud service. Here’s the sample code to get the list of all extensions for a cloud service:

        /// 
        /// Gets a list of all extensions enabled in a cloud service.
        /// 
        /// 
        /// Subscription id of the subscription.
        /// 
        /// 
        /// Management certificate for authenticating Service Management API requests.
        /// 
        /// 
        /// Name of cloud service.
        /// 
        /// 
        private static IEnumerable ListExtensions(string subscriptionId, X509Certificate2 cert, string cloudServiceName)
        {
            List extensions = new List();
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/extensions", subscriptionId, cloudServiceName);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "GET";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (var streamReader = new StreamReader(resp.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    Console.WriteLine(result);
                    XElement extensionImagesXelement = XElement.Parse(result);
                    foreach (var extensionImageXelement in extensionImagesXelement.Elements(XName.Get("Extension", "http://schemas.microsoft.com/windowsazure")))
                    {
                        extensions.Add(AzureExtension.Parse(extensionImageXelement));
                    }
                }
            }
            return extensions;
        }

For more details, please click here: http://msdn.microsoft.com/en-us/library/windowsazure/dn169561.aspx.

Get Extension

This function returns the details of a particular extension in a cloud service. Here’s the sample code for getting extension details:

        /// 
        /// Gets the details of a particular extension in a cloud service.
        /// 
        /// 
        /// Subscription id of the subscription.
        /// 
        /// 
        /// Management certificate for authenticating Service Management API requests.
        /// 
        /// 
        /// Name of cloud service.
        /// 
        /// 
        /// Extension id.
        /// 
        /// 
        private static AzureExtension GetExtension(string subscriptionId, X509Certificate2 cert, string cloudServiceName, string extensionId)
        {
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/extensions/{2}", subscriptionId, cloudServiceName, extensionId);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "GET";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (var streamReader = new StreamReader(resp.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    Console.WriteLine(result);
                    XElement extensionDetailsXElement = XElement.Parse(result);
                    return AzureExtension.Parse(extensionDetailsXElement);
                }
            }
        }

For more details, please click here: http://msdn.microsoft.com/en-us/library/windowsazure/dn169557.aspx.

Delete Extension

This function removes an extension. Here’s the sample code for deleting an extension from a cloud service:

        /// 
        /// Removes a  particular extension from a cloud service.
        /// 
        /// 
        /// Subscription id of the subscription.
        /// 
        /// 
        /// Management certificate for authenticating Service Management API requests.
        /// 
        /// 
        /// Name of cloud service.
        /// 
        /// 
        /// Extension id.
        /// 
        private static void DeleteExtension(string subscriptionId, X509Certificate2 cert, string cloudServiceName, string extensionId)
        {
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/extensions/{2}", subscriptionId, cloudServiceName, extensionId);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "DELETE";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
            }
        }

For more details, please click here: http://msdn.microsoft.com/en-us/library/windowsazure/dn169560.aspx.

That’s pretty much to it!

Remote Desktop (RDP)

Now that we’ve seen how extensions work, let’s put this together for some practical use. What we’ll do is enable remote desktop on the fly using the extensions. For the sake of simplicity, we’ll enable RDP on all the roles in our cloud service.

Step 1: Get the list of all extensions

To do so, we’ll make use of “List Available Extensions” and find out the public and private configuration schema for RDP. Here’s the sample code to do so:

            var allExtensions = ListAvailableExtensions(subscriptionId, cert);
            var rdpExtension = allExtensions.FirstOrDefault(e => e.Type == "RDP");

Once we get the details about the RDP extension, we can just take the configuration schemas. Here’s what the service currently returns:

Public Configuration Schema:

Private Configuration Schema:

Based on these, we’ll create public and private configuration:

Public Configuration:



  {0}
  {1}

Private Configuration:

{0}

Step 2: Add Extension

Next step is adding extension to the cloud service. To do so, we’ll make use of “Add Extension” functionality. Here’s the sample code to do so.

            AzureExtension rdpExtension = new AzureExtension()
            {
                ProviderNamespace = "Microsoft.Windows.Azure.Extensions",
                Type = "RDP",
                Id = "RDP-" + Guid.NewGuid().ToString(),
                PublicConfiguration = string.Format(publicConfigurationFormat, userName, expiryDate.ToString("yyyy-MM-dd")),
                PrivateConfiguration = string.Format(privateConfigurationFormat, password),
            };

            AddExtension(subscriptionId, cert, cloudServiceName, rdpExtension);

Here I’m passing the username, password and the RDP expiry date. Please note that since I’ve not specified a certificate thumbprint Windows Azure will automatically create a new certificate and associate with my cloud service. Also I noticed that the password needs to be a strong password. I actually spent quite some time to realize this. If you provide a simple password (like “password”), the operation would complete however you will not be able to connect to your role instances using RDP. You’ll get a “login failed” message. Your password should have 3 of the following – a lowercase character, an uppercase character, a number, and a special character.

Step 3: Update Configuration

        private static string[] GetCloudServiceDeploymentConfigurations(string subscriptionId, X509Certificate2 cert, string cloudServiceName)
        {
            string[] deploymentConfigurations = new string[2];//We'll try to get both production and staging deployment configurations. The 1st element will always be production and the 2nd will be staging.
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}?embed-detail=true", subscriptionId, cloudServiceName);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "GET";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (var streamReader = new StreamReader(resp.GetResponseStream()))
                {
                    string result = streamReader.ReadToEnd();
                    XElement cloudServiceProperties = XElement.Parse(result);
                    var deployments = cloudServiceProperties.Elements(XName.Get("Deployments", "http://schemas.microsoft.com/windowsazure"));
                    foreach (var deployment in deployments.Elements())
                    {
                        var slotElement = deployment.Element(XName.Get("DeploymentSlot", "http://schemas.microsoft.com/windowsazure"));
                        var configElement = deployment.Element(XName.Get("Configuration", "http://schemas.microsoft.com/windowsazure"));
                        var deploymentSlot = slotElement.Value;
                        var configurationSettings = Encoding.UTF8.GetString(Convert.FromBase64String(configElement.Value));
                        switch (deploymentSlot.ToUpper())
                        {
                            case "PRODUCTION":
                                deploymentConfigurations[0] = configurationSettings;
                                break;
                            case "STAGING":
                                deploymentConfigurations[1] = configurationSettings;
                                break;
                        }
                    }
                }
            }
            return deploymentConfigurations;
        }

Now that we’ve got the configurations for both production and staging slots, let’s apply the changes. Here’s the sample code for change deployment configuration operation:

        private static void ChangeDeploymentConfiguration(string subscriptionId, X509Certificate2 cert, string cloudServiceName, string slot, string configuration)
        {
            string uri = string.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/deploymentslots/{2}/?comp=config", subscriptionId, cloudServiceName, slot);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.Method = "POST";
            request.ContentType = "application/xml";
            request.Headers.Add("x-ms-version", "2013-03-01");
            request.ClientCertificates.Add(cert);
            byte[] content = Encoding.UTF8.GetBytes(configuration);
            request.ContentLength = content.Length;
            using (var requestStream = request.GetRequestStream())
            {
                requestStream.Write(content, 0, content.Length);
            }

            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
            }
        }

And this is how you can call it.

            string updateConfigurationFormat = @"
                
                    {0}
                    
                        
                          
                            {1}
                          
                        
                     
                ";

                var productionConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(productionConfig)), rdpExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Production", productionConfigurationSetting);

Putting it all together nicely in just one function:

        private static void EnableRemoteDesktop(string subscriptionId, X509Certificate2 cert, string cloudServiceName, string userName, string password, DateTime expiryDate)
        {
            string publicConfigurationFormat = @"
                
                    {0}
                    {1}
                ";

            string privateConfigurationFormat = @"
                
                    {0}
                ";

            string updateConfigurationFormat = @"
                
                    {0}
                    
                        
                          
                            {1}
                          
                        
                     
                ";

            //Define RDP Extension
            AzureExtension rdpExtension = new AzureExtension()
            {
                ProviderNamespace = "Microsoft.Windows.Azure.Extensions",
                Type = "RDP",
                Id = "RDP-" + Guid.NewGuid().ToString(),
                PublicConfiguration = string.Format(publicConfigurationFormat, userName, expiryDate.ToString("yyyy-MM-dd")),
                PrivateConfiguration = string.Format(privateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(password))),
            };

            //Add extension
            AddExtension(subscriptionId, cert, cloudServiceName, rdpExtension);

            //Get deployment configurations
            var configurations = GetCloudServiceDeploymentConfigurations(subscriptionId, cert, cloudServiceName);

            var productionConfig = configurations[0];
            if (!string.IsNullOrWhiteSpace(productionConfig))
            {
                //Apply RDP extension to production slot.
                var productionConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(productionConfig)), rdpExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Production", productionConfigurationSetting);
            }
            var stagingConfig = configurations[1];
            if (!string.IsNullOrWhiteSpace(stagingConfig))
            var stagingConfig = configurations[1];
            if (!string.IsNullOrWhiteSpace(stagingConfig))
            {
                //Apply RDP extension to staging slot.
                var stagingConfigurationSetting = string.Format(updateConfigurationFormat, Convert.ToBase64String(Encoding.UTF8.GetBytes(stagingConfig)), rdpExtension.Id);
                ChangeDeploymentConfiguration(subscriptionId, cert, cloudServiceName, "Staging", stagingConfigurationSetting);
            }
        }

That’s pretty much it!!! Once the operation is complete, you should be able to RDP into your instances.

Wish List

My only wish here is that the API team has not clubbed this extensions functionality with change deployment configuration and provided direct operations for enabling/disabling the extensions. Clubbing this functionality with change deployment configuration may lead to some inadvertent errors. Other than that, I wish they had an “Update Extension” operation. Currently, I don’t know how I would update an extension. That functionality can be real handy.

Summary

I think the extensions functionality is pretty awesome. In due course of time when Windows Azure team starts accepting components from ISVs, it would open up a lot of opportunities. Even now, the flexibility offered by built-in extensions is quite helpful to the developers. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Happy Coding!

Introducing Windows Azure SDK 2.0 for .Net

Gaurav — Tue, 30 Apr 2013 18:21:42 +0000

Today Microsoft announced the availability of Windows Azure SDK 2.0 for .Net. You can read more about the announcement in Scott Guthrie’s blog post here: http://weblogs.asp.net/scottgu/archive/2013/04/30/announcing-the-release-of-windows-azure-sdk-2-0-for-net.aspx. In this blog post, we’ll cover some of the features introduced in this newest version.

Downloading/Installing SDK 2.0

You can download and install SDK 2.0 using Web Platform Installer from Windows Azure Developer center: http://www.windowsazure.com/en-us/develop/net/.

Improved Tooling in Visual Studio

A number of improvements done in Visual Studio as part of this release. Some of these changes are:

Windows Azure Diagnostics

IMO, this is the most important improvement in the tooling. Windows Azure Diagnostics (WAD) is a crucial part of any cloud service deployment and unfortunately it was really tough to get diagnostics working. In the latest version, the tooling team has made it extremely easy to manage WAD.

You get an easy to use way of configuring diagnostics. See the screenshot below for example.

You get 3 options – “Errors only”, “All information” and “Custom plan”.

When you choose “Errors only”, only log entries (trace, event, and infrastructure) with “Error” type will be transferred to storage. When you choose “All information”, all logs entries will be transferred to storage. Please note that in both the cases the transfer interval is one minute and performance counters data will not be transferred. If you want to transfer performance counters data or wish to have more granular control, you would need to choose “Custom plan” where you would be able to configure not only performance counters but also scheduled transfer period and buffer size as well among other things.

Another important change with WAD is that it is no longer dependent on Storage Client Library 1.7. It makes use of version 2.0.5.1 of the storage client library.

I’ll probably write another blog post focusing exclusively on Windows Azure Diagnostics and SDK 2.0 to cover that in more details.

Support for Windows Azure Websites

Now you can view and manage Windows Azure Websites through Visual Studio. To manage your websites, you would go to VIEW à Server Explorer à Windows Azure Websites. There you will see all your websites. You will need to provide a publish profile file before you can use this functionality.

Things you could do with Windows Azure Websites in Visual Studio are:

· Manage configuration – as shown in the screenshot above.

· View / download website logs.

· Improved publishing support. Unfortunately you can’t create a new website through Visual Studio just yet.

Support for Larger VM Size (A6, A7)

When Windows Azure Virtual Machine (IaaS) went GA a few days ago, they made 2 new VM sizes available there (A6 and A7). They’re now available in Cloud Services as well i.e. you can deploy cloud services in A6 and A7 sized VMs as well. You get that option in Visual Studio as well.

Library Improvements

Latest libraries

SDK 2.0 by default includes latest version of various libraries which at the time of release was 2.0.5.1 for Windows Azure Storage, 2.0 for Windows Azure Service Bus, 2.0 for Windows Azure Caching and 2.0 for Windows Azure Configuration Manager. You can also get these libraries from NuGet as well because these libraries will be constantly updated with latest features in between SDK releases.

Windows Azure Service Bus client library has been updated with support for browsing messages, the event-driven message programming model, and auto-deleting idle messaging entities.

Breaking Changes

Following are some of the breaking changes in SDK 2.0 which you would need to keep in mind when upgrading your solution to the latest SDK version:

.NET Framework Minimum Version

The minimum version of the .NET Framework required for building web and worker roles is now .NET Framework 4.0. You can still reference .NET Framework 3.5 assemblies in your project, but the project itself must be built targeting .NET Framework 4.0 or 4.5. Furthermore ServiceRuntime, Configuration and Caching assemblies are now built against the .NET Framework 4.0 runtime. Therefore, any of your applications that target the .NET Framework 3.5 should now target the .NET Framework 4.0 before upgrading to the Windows Azure SDK for .NET 2.0.

Side-by-Side Cloud Service Debugging

Side-by-Side cloud service debugging is only supported with Windows Azure SDK for .NET versions 1.7, 1.8, and 2.0. For a full debugging experience, projects built using versions prior to version 1.7 must be upgraded. This is done in line with the SDK expiration policy announced some days back.

Summary

That’s it for this post. All in all I am quite pleased with the changes introduced in the new SDK and I think they will help users working with Windows Azure immensely. In the next post, I will dig deep into Windows Azure Diagnostics and SDK 2.0.

Stay tuned!

Windows Azure Primer–Basics

Gaurav — Fri, 12 Apr 2013 12:27:46 +0000

There’re many articles which cover Windows Azure and it’s functionality in great details however I could not find much information when it comes to novice users (or may be I need to brush up on my searching skills ). Anyways, through these series of blog posts I will try and explain Windows Azure as I understand it to users who are just starting out with Windows Azure. We’ll cover some basic functionalities available in Windows Azure and try to understand that. If you’ve worked with Windows Azure, you may find these posts trivial but I still want you to read this so that you can correct me if I have provided some incorrect information.

With that intent, let’s start!

Before we get into the details about what Windows Azure and what it is not, let’s take a moment to define what Windows Azure is. I would define it like the following:

Windows Azure is Public Cloud Computing Platform offering by Microsoft

However before we talk about Windows Azure, let’s take a moment and talk about Cloud Computing first as I feel folks who are starting out have a different notion of what cloud computing is (and what it’s not). Then we get into Windows Azure.

Cloud Computing

In this section we’ll focus on cloud computing. We’ll start with definition, then talk about key characteristics of a cloud environment and then talk about some application scenarios where cloud can be used.

Definition

Wikipedia defines Cloud Computing as follows:

Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet)

However as I understand it (and at a very basic level):

It’s Just a Hosting Platform (with some differences)!

Let me explain. Let’s say I have built a website and now I want to host it. Depending on my application requirement, I may decide to go with a hosting provider and they would offer me different environments to choose from (Shared, VPS, Dedicated) to run my application; different storage options (disk quota, choice of data stores); bandwidth quota and some value added services etc. This is exactly the same with any cloud provider. At the very least, each cloud provider provides an environment to run the application (typically referred to as “compute”), choice of data stores (Relational, NoSql, Object) and network infrastructure (bandwidth, firewall etc.). So the application I would host with any hosting provider could easily be hosted with any cloud provider. There’re some differences and I’ll talk about those in the next section.

Key Principles/Characteristics

Any cloud computing infrastructure is built on the certain key principles and we talk about them below. Also I see a number of folks want to build their own cloud infrastructure (my advice to them: please, don’t! It’s bloody expensive ). So if you’re one the guys who want to build your own cloud infrastructure, make sure it has the characteristics described below.

Shared Resources

When you run your applications in your data centers, you own each and every part of it. You own the servers, you own the infrastructure, everything. However with cloud, it’s different. You don’t own anything there other than the application (and application data) you’re hosting.

Hosting an application in the cloud is quite similar to booking a room in a hotel.

You get a room which you rent for a few days. You can’t claim ownership of that room but at the same time you’re the only tenant in that room. In a cloud environment, you get one or more virtual machines in which you would run your applications but you don’t own those virtual machines. Like your hotel room, cloud provider would guarantee you that you’re the only tenant of those virtual machines. They won’t put more tenants in that virtual machine.

Hotel may offer other amenities (like Internet access, Gym, Restaurant etc.) but you share those amenities with other guests in the hotel. Similarly in a cloud environment, you share resources like internet bandwidth, physical infrastructure (like routers and stuff) with other tenants. Like you can’t have another guest sharing the same treadmill with you at the Gym , cloud provider provides you with some isolation for these shared resources and ensures that you’re not impacted by “rude/noisy neighbor behavior” but that’s pretty much it.

Self-Service Model

All cloud provider provides a self-service model. As I understand it, a self-service model allows you to manage the resources you consume in a cloud environment by yourself without going to the cloud provider.

Again taking the analogy of the hotel, think of it that way: Let’s say you want to extend the reservation or may be change a room type. One way you could do it is to go to front desk and ask them to make those changes however that’s not the self-service model. Now let’s assume that the hotel provides a web site or a kiosk using which you can make the changes you want. Now that’s a self-service model.

Similarly in a cloud environment, each cloud provider provides a self-service model (either in form of raw API or through easy-to-use applications) using which you can manipulate the resources you want to consume e.g. you may want to run your application in a larger virtual machine or you want to increase (or decrease) the number of virtual machines in which your application runs. You don’t have to contact your cloud provider (by phone/email etc.) to do the same.

Another aspect of self-service model is “self healing”. Wouldn’t it be nice if the hotel automatically detects that your room is dirty and needs cleaning or you’ve run out of coffee in your room and automatically replenishes it! It doesn’t happen in hotel (at least in my experience) but it sure happens in the cloud. Through their advanced monitoring, a cloud provider knows that there’s something wrong with one or more virtual machines in which your application is running and it tries to heal them automatically. Self healing could happen in the form of taking out a troubled virtual machine out of the picture and spinning off a new virtual machine automatically.

Elasticity

In cloud computing, elasticity means the ability to expand or shrink resource consumption based on the need. It is sometimes related to scalability also. To give an example, let’s say you’ve an application which gets used more on weekdays than on weekends. In cloud environment you have the flexibility to put more resources (typically virtual machines) to serve your application during weekdays and then reduce the resources over the weekend. Combined with “self-service model” characteristic of the cloud infrastructure makes it quite attractive hosting choice for some kinds of applications (we’ll discuss that in a bit).

Another aspect of elasticity is the notion of infinite limit. To give you an example, when we buy a computer we get it with a hark disk with a pre-defined size e.g. 512 GB, 1 TB etc. Now we can’t go beyond that. If we need to go beyond that, we would either need to purchase an additional hard disk with higher capacity and replace the current one with a new one or get additional hard disk and attach it along with existing hard drive. But again there’s a limit to which you could go. In cloud you have the notion of infinite limit. You can increase as much as you want to.

I used the word “notion” and there’s a reason for that. It may seem like there’s no limit but in reality there’s one however generally speaking the limit is too high so we may never reach that limit.

Pay Per Use

Pay per use or consumption based billing is the most important characteristic of a cloud environment. In a typical cloud environment, you only pay for the resources you consume – nothing more, nothing less. So if you’re running a virtual machine for 12 hours, you only pay for 12 hours. This becomes really attractive for some companies (especially startups) because they don’t have to make upfront investment in procuring infrastructure. Today you need one server to serve your application, you use one server and you pay for just one. Tomorrow you need say ten servers to server your application, you use ten servers and you pay for those ten servers. Day after tomorrow you need again just one server, you use and pay for that one server.

When I talk to students about cloud computing, a common example I give is the computer buying experience. Let’s say you’re in the market for buying a computer and you don’t really need 4 GB of RAM or 512 GB of hard disk to begin with but then when you’re working on a special project you might need that kind of capability. You can’t tell your computer vendor that since you’ll be using a part of the resources available on the computer, you’ll only pay for that. They’ll laugh at you. You would either have to buy a low-end computer (and then worry about upgrading it later) or buy a high-end computer and underutilize it. Either way you look at it, you lose.

Not in cloud though!!! You need a low-end virtual machine to begin with, you go with that and pay a lower price for that (and only for the time you use) but then later on you need a high-end virtual machine, you just rent that high-end machine and use that and pay only for the time you use that machine. Makes complete sense, doesn’t it!

Key Application Scenarios/Patterns

Since a cloud computing infrastructure is a hosting environment and you could possibly host any kind of application there. However to take full advantage of the benefits offered by cloud, there’re a few key application scenarios/patterns. If your application fits those scenarios/patterns, then you may want to consider hosting your application in the cloud.

Here’re some of those scenarios/patterns:

On and Off Pattern

Let’s say you’re making a digital movie. Once the movie is complete, you would need massive compute capacity to create a render farm to render the movie. However once the rendering is done, you really don’t need that compute capacity. That’s “On and Off” pattern and cloud could be a perfect choice for that. Another example could be monthly data processing where you would need to generate lots and lots of reports for your clients however the whole exercise of generating reports is a few days task. This kind of application can also be considered to be deployed in the cloud.

Growing Fast Pattern

This is where typical startups come into picture. Let’s assume that you’re running a startup. Like every startup, you expect your startup to be extremely successful and wish to have millions of users using your application. As much as you want to do it, it’s impractical (both physically and financially) to start with having an infrastructure to support extreme load. So what you do is host your application in the cloud. If the startup is successful and you start getting more and more visitors, you just tap into “elastic” characteristic of your cloud provider and increase the resources really needed to support your user base.

Unpredictable Bursting Pattern

Everybody likes pleasant surprises. Let’s say you’re running a startup and unknown to you, suddenly you got “Techcrunched”!!! As a result, you see a sudden spike in the visitation to your application. Again the “elastic” characteristic of your cloud provider would allow you to increase the resources in no time to handle is unpredictable increase in load. Another example is that you’re providing an event management application as a service and one of the subscriber announces this really special event and that kind of catches you off guard. Having hosted this application in the cloud would give the ability and flexibility to meet with this unexpected demand.

Predictable Bursting Pattern

In one of my previous jobs, we developed a system where people would make reservations for camping sites all across the US. Summer time, especially long weekends (memorial day, labor day) were dreaded period for us because we knew that our application will be hammered badly on those days. Also there were some National Parks (like Yosemite National Park in California) which opened their booking on a particular date so we knew that we’ll get extensive load on that day. My only wish today is that we had cloud infrastructure during that time so that we didn’t have those nightmarish days and weekends .

When you know well in advance that you would get more visitors on a particular day/time, you may want to consider cloud as a hosting option. You don’t have to build that excess capacity in-house just to take care of peak demand.

Another example I give here in India is Indian Railways train reservation website!!! They open their booking every morning at 8:00 AM and it is next to impossible to book a train ticket between 8:00 – 10:00 AM if you’re trying to book a train for summer vacation as everybody’s on that site. They could certainly use cloud, provision excess capacity for those two hours and have a lot of happy customers.

Windows Azure

Now that we’ve some basic understanding of cloud computing, let’s talk about Windows Azure. As I said above:

Windows Azure is Public Cloud Computing Platform offering by Microsoft

At a very high level it provides a compute environment to run your application, different storage options to store your application data and some value added services apart from core infrastructure. In this section we’ll briefly talk about these components (we’ll cover them in details in subsequent posts), answer the question if Windows Azure is right for your application and then answer some common questions (or debunk some myths). Plus we’ll also talk about how to get started with Windows Azure.

Components

First, let’s talk about components. In this post we’ll keep the discussion real short as we’ll cover them in great details in the subsequent posts.

Compute

This is the execution environment for your applications. Windows Azure provides both shared and dedicated execution environments with varied compute resource size (in terms of processing power, RAM etc.) and a wide range of operating systems (including but not limited to Windows Server 2012, 2008, some flavors of Linux etc.).

Storage

Every application either need to store some data or work on a stored data. Windows Azure provides many storage options including but not limited to Relational Databases (SQL Server, MySQL as a service), NoSql Databases (Table Storage) and Object Storage (Blob Storage). Through it’s Infrastructure as a Service (IaaS) offering, it is possible for you to support other relational and NoSql databases as well.

Value Added Services

This post is already quite long so for the sake of not making it longer I’ve clubbed all other services under just one bucket – value added services. These include messaging (Windows Azure Queues/Service Bus), CDN, Caching, Identity Management (Windows Azure Active Directory/Windows Azure Access Control Service), Windows Azure Mobile Service, Windows Azure Media Service etc. It also offers a store which allows you to avail other value added services like monitoring etc. from 3rd party providers.

Is Windows Azure Right For Me?

I guess the right question to ask: “Is Cloud Right For Me?”. Simple answer to the question: it depends .

Well, one criteria is the key scenarios/patterns described above. If your application fits one or more of the scenarios described above then yes you could consider Windows Azure.

Yet another criteria would be the application business domain. You would need to remember is that Windows Azure is Public Cloud and you don’t get physical access to the underlying infrastructure or data centers where your application is hosted. If your application business domain is constrained by certain rules and regulations (like HIPPA for healthcare, SOX for Finance etc.) you may want to check that first.

Also you would need to consider data sovereignty laws as Windows Azure does not have data center in every country/region. If your application has a requirement that the application data can’t leave the physical boundary of your country/region and Windows Azure does not have a data center in your country/region, you’re kind of out of luck! Windows Azure won’t stop you from hosting the application but then you would be in breach of your local laws.

Some Questions Answered

I participate in a number of forums and when people start with Windows Azure, they have some common set of questions. I’ve picked a few of them:

If I choose Windows Azure, do I let go of my existing skills and need to learn new ones?

No. Most folks think that cloud computing is yet another programming language they would need to master. Remember, Windows Azure is just another hosting platform for your application. You would still develop the application the way you’ve been developing them and when you’re ready to deploy your applications, you just deploy them in Windows Azure.

Obviously, to take full advantage of the platform you would need to learn some platform specific tricks but you don’t discard your existing skills. You learn new skills to complement your existing skills. One common example here is that when we build traditional web applications we either store images on the file system or in the application database. You could still do that but that’s not recommended as the file system in the cloud is volatile and there’re better alternatives than application database e.g. Blob Storage. So you would need to learn how to save images in blob storage.

A key benefit of Windows Azure is scalability however to achieve this you may have to build applications which have loosely coupled components (or modules) so that each component can be scaled independently. So in order to do that you may need to learn how to build loosely coupled components.

Isn’t Windows Azure all about .Net?

Absolutely not! Just because Windows Azure is from Microsoft we tend to assume that it’s all about .Net but that’s just not true. Windows Azure supports many languages and platforms including but not limited to .Net, Java, PHP, Python, Node.Js etc. If your programming environment is not supported natively you could still use Windows Azure by making use of IaaS feature.

Getting Started

Let’s wrap up this post by how to get started with Windows Azure. Well, Windows Azure offers a 90 day free trial which you can sign up for. You would be asked to provide a credit card number but it won’t be charged (with the exception of a $1.00 charge to verify your card) so it’s completely risk free. However you must realize that free trial comes with a preset quota for a month and if you exceed that quota, Windows Azure will suspend your account for the rest of that month. So please ensure that you don’t exceed that quota. You can avoid account suspension by removing the spending limit but during trial phase I would strongly recommend NOT to do that. This is one common mistake we do is not realize that in Windows Azure we pay based on consumption. We think that the consumption is the “consumption of our application” i.e. we only pay if somebody is using our application. That’s not true. Consumption is how much of Windows Azure’s resources are we consuming. When we deploy our application, we’re consuming virtual machines provided by Windows Azure. So please keep that in mind when using Windows Azure. To get started with Windows Azure free 90 day trial, please click here: http://www.windowsazure.com/en-us/pricing/free-trial/.

Windows Azure also provides a compute and storage emulator using which you can try some features without even signing up for the trial. So you may want to look into that as well. Compute and storage emulator are part of Windows Azure SDK which you can download from here: http://www.windowsazure.com/en-us/downloads/.

Other than that, I would strongly recommend going through development centers for different languages (http://www.windowsazure.com/en-us/documentation/) and trying out Windows Azure Training Kit (http://www.microsoft.com/en-US/download/details.aspx?id=8396). Please ensure that you’re downloading the latest training kit.

Summary

That’s it for this post. Pretty long post, eh!!! In the subsequent posts, we’ll explore functionality in greater details. As always, your feedback is most welcome. If I have made a mistake in the post, please let me know and I’ll fix it ASAP.

Moving On – Goodbye Cerebrata, Hello Cynapta!

Gaurav — Thu, 28 Mar 2013 19:27:42 +0000

Today’s my last day at Cerebrata. Wow!!! It has been an incredible journey for the last 5 years. I vividly remember when around the same time 5 years ago, I told my team in the US about my decision to move back to India to “do my own thing”. I was not 100% sure what I’ll be doing; there was no concrete plan – only thing I was sure of was to build a product development company in India. Never in my dream, I thought of achieving the success I achieved – First, built a pure product company that too in a small town in India. Second, successfully running and selling the company. Third, awarded Windows Azure MVP three years in a row. Honestly, I couldn’t have asked for more.

Looking Back – Some Random Thoughts

Looking back, some thoughts on few important things:

On My Team

You may get sick of hearing this thing but I’m really fond of my team! They trusted me with their careers and for that I’m grateful. Going down the memory lane, I see young graduates coming in becoming part of my dream and helping me achieve those. When they came in, they were raw out of college having no idea how industry worked. Now I look at them, I see them all grown up and thorough professionals. Together we built a culture of mutual respect and openness in the company and I thank them for that. I’ll always cherish their love, affection and adulation. They see me more as their mentor than their boss and I take pride in that. My only hope is that when their time comes, they would do the same. Only then, I would be able to pat my back. I’ll certainly miss working with them.

Picture of current & ex-team members taken during our grand farewell party

On Our Users

Our products are successful only because of our users. Like I said in the past, neither I nor my team were experts in programming or Windows Azure for that matter but our users were, and they helped us shape our products. Our software had issues (in fact many issues) but our users were patient and were willing to help us out and I thank them from the bottom of my heart for that. Some of them worked with us diligently during the beta phases of our products while some of them found issues and provided resolutions in the released products. There were countless occasions where our users went out of the way to help us out from doing code reviews, providing solutions to just making our software better. My wish is to have same users helping me out in the next phase of my professional life.

On Red Gate

Prior to Red Gate, I was just an entrepreneur. Because of Red Gate, I am now a successful entrepreneur!!! I thank Red Gate for that. I’ve been working with them for past 18 months now (since the acquisition) and I can tell you they’re the best guys to work with. Not only these guys are real smart, their practice of “delightful user experience before anything else” is something I’ll take away from this relationship.

Every relationship has ups and downs and we had our share of the same as well. But they treated me like one of their own (in fact, sometimes better than their own). They simply pampered me. I’ve gone through many acquisitions in my professional career and generally speaking acquisitions suck (big time) however I can say that this was the best experience I had.

On Microsoft

We built products exclusively for Microsoft platform (that too Windows Azure only). If they wanted, they could have cannibalized us in no time but they didn’t, and I thank Microsoft for doing that. They really know how to build and nurture a true partner eco-system.

On top of that, they recognized the work I was doing in the community and awarded me with prestigious Most Valuable Professional (MVP) award. This is really huge for me!!!

One last thing I would like to mention about Microsoft is their BizSpark program. For a guy who was starting out on his own, I can’t tell you how valuable was it for me. Their MSDN subscription was a big boon for me. I was once doing an interview with somebody and he asked me about the biggest benefit of this program to me and my response was – BizSpark let me stay legit with software . Otherwise it would have been tough!!! Anybody who’s starting out must try it out whether you love or hate Microsoft (or in other words whether or not building products for Microsoft platform).

Kind of cheesy confession: Back in the day when I was starting out, anybody with a microsoft.com email address was a celebrity in my eye .

Whats’ Next

Next up for me is Cynapta (http://www.cynapta.com – really, there’s nothing to see there just yet). It’s my new company. I’ve some product ideas which I want to explore. Building products is more of a gamble and more than likely I will fail but I still want to scratch that itch (so to speak) . So wish me luck!!!

I’ll be building my team again from ground up so that should be an interesting experience to relive and I’m really looking forward to that. Hopefully I won’t make the same mistakes I made with my previous team.

Other than that, I would be spending my time on two of my favorite places on the Internet – MSDN Forums & StackOverflow, and obviously on this blog sharing things and on Twitter.

Going forward, my Cerebrata email account won’t work (though Red Gate folks have been kind enough to keep it active and if you write to me there, you’ll probably get an OOF) so if you wish to reach out to me you can do so by writing me an email at gaurav @ cynapta.com.

Closing Thoughts

I would like to end this post thanking my team and our users. If I didn’t have the support and encouragement from you guys, I would not have been writing this post. So from the bottom of my heart (and one last time) thank you, thank you, thank you!!!

Cheers!!!

Uploading Large Files in Windows Azure Blob Storage Using Shared Access Signature, HTML, and JavaScript

Gaurav — Sat, 16 Feb 2013 07:54:09 +0000

In my last post I talked about Shared Access Signature feature of Windows Azure Storage. You can read that post here: http://gauravmantri.com/2013/02/13/revisiting-windows-azure-shared-access-signature/. In this post we’ll put that to some practical use .

One thing I wanted to accomplish recently is the ability to upload very large files into Windows Azure Blob Storage from a web application. General approach is to read the file through your web application using “File” HTML control and upload that entire file to some server side code which would then upload the file in blob storage. This approach would work best for smaller files but would fail terribly when it comes to moderately to very large files as the file upload control would upload the entire file to the server (for bigger files, this would cause timeouts depending on your Internet connection) and then that file resides in the server memory before any action can be taken on that file (again for bigger files, this would cause performance issues assuming you have thousands of users uploading thousands of large files).

In this blog post, we’ll talk about how we can accomplish that using File API feature in HTML5.

Please be warned that I’m no JavaScript/HTML/CSS expert . About 4 – 5 years ago I used to do a lot of JavaScript development and built some really insane applications using JavaScript (like mail merge kind of features, office automation, native device interfacing all using JS) but that was then. For last 4 – 5 years, I have been extensively doing desktop application development. So most of the JS code you’ll see below is copied from various sites and StackOverflow .

Objectives

Following were some of the objectives I had in mind:

The interface should be web based (possibly pure HTML).
I don’t have to share my storage account credentials with the end users.
If possible, there should not be any server side code (ASP.Net/PHP etc.) i.e. it should be pure HTML and the communication should be between user’s web browser and storage account.
If possible, the solution should upload the file in chunks so that bigger files can be uploaded without reading them in completely in a single go.

Now let’s see how we can meet our objectives.

Solution

Here’re some of the things I did to achieve the objectives listed above.

Chunking File

The first challenge I ran into is how to upload the file in chunks. All the examples I saw were uploading an entire file however this is not what I wanted. Finally I ran into this example on HTML5 Rocks where they talked about the File API: http://www.html5rocks.com/en/tutorials/file/dndfiles/. Basically what caught my interest there is the “slicing” feature available in HTML 5’s File interface. In short, what this slice feature does is that it reads a portion of a file asynchronously and returns that data. This is exactly I was looking at. Once I found that, I knew 75% of my work was done! Everything else was a breeze

Securing Storage Account Credentials

Next thing on my list was securing storage account credentials and this was rather painless as I knew exactly what I had to do – Shared Access Signature. SAS provided me with a secured URI using which users will be able to upload files in my storage account without me giving them access to storage account credentials. This was covered extensively in my post about the same: Revisiting Windows Azure Shared Access Signature.

Direct Communication Between Client Application and Windows Azure Storage

Next thing was to facilitate direct communication between the client application and storage. As we know Windows Azure Storage is built on REST so that means I can simply use AJAX functionality to communicate with REST API. One important thing to understand is that Windows Azure Storage still does not support Cross-Origin Resource Sharing (CORS) at the time of writing this blog. What that means is that your web application and blob storage must be in the same domain. The solution to this problem is to host your HTML application in a public blob container in the same storage account where you want your users to upload the files. I’ve been told that CORS support is coming soon in Windows Azure Storage and once that happens then you need not host this application in that storage account but till then you would need to live with this limitation.

The Code

Now let’s look at the code.

HTML Interface

Since I was trying to hack my way through the code, I kept the interface rather simple. Here’s how my HTML code looks like:


    
        
            File Uploader
            
                SAS URI: 
                

                
                    
                
            
            
                File To Upload: 
                

                
                    
                
            
            
                
                File Properties:
                

                
                    Name: 
                
                
                    File Size:  bytes.
                
                
                    File Type: 
                
                
                    
                
                
                    Progress: 0.00 %

All it has a textbox for a user to enter SAS URI and HTML File control. Once the user selects a file, I display the file properties and the “Upload” button to start uploading the file.

Reading File Properties

To display file properties, I made use of “onchange” event of File element. The event gave me a list of files. Since I was uploading just one file, I picked up the first file from that list and got its name (blob would have that name), size (blob’s size and determining chunk size) and type (for setting blob’s content type property).

//Bind the change event.
$("#file").bind('change', handleFileSelect);

        
        //Read the file and find out how many blocks we would need to split it.
        function handleFileSelect(e) {
            var files = e.target.files;
            selectedFile = files[0];
            $("#fileName").text(selectedFile.name);
            $("#fileSize").text(selectedFile.size);
            $("#fileType").text(selectedFile.type);
        }

Chunking

In my application I made an assumption that I will split the file in chunks of 256 KB size. Once I found the file’s size, I just found out the total number of chunks.

        
        //Read the file and find out how many blocks we would need to split it.
        function handleFileSelect(e) {
            maxBlockSize = 256 * 1024;
            currentFilePointer = 0;
            totalBytesRemaining = 0;
            var files = e.target.files;
            selectedFile = files[0];
            $("#output").show();
            $("#fileName").text(selectedFile.name);
            $("#fileSize").text(selectedFile.size);
            $("#fileType").text(selectedFile.type);
            var fileSize = selectedFile.size;
            if (fileSize < maxBlockSize) {
                maxBlockSize = fileSize;
                console.log("max block size = " + maxBlockSize);
            }
            totalBytesRemaining = fileSize;
            if (fileSize % maxBlockSize == 0) {
                numberOfBlocks = fileSize / maxBlockSize;
            } else {
                numberOfBlocks = parseInt(fileSize / maxBlockSize, 10) + 1;
            }
            console.log("total blocks = " + numberOfBlocks);
        }

Endpoint for File Uploading

The SAS URI actually represented a URI for blob container. Since I had to create an endpoint for uploading file, I split the URI (path and query) and appended the file name to the path and then re-appended the query to the end.

            var baseUrl = $("#sasUrl").val();
            var indexOfQueryStart = baseUrl.indexOf("?");
            submitUri = baseUrl.substring(0, indexOfQueryStart) + '/' + selectedFile.name + baseUrl.substring(indexOfQueryStart);
            console.log(submitUri);

Reading Chunk

This is where File API’s chunk function would come in picture. What happens in the code is that when the user clicks the upload button, I read a chunk of that file asynchronously and get a byte array. That byte array will be uploaded.


        var reader = new FileReader();
        var fileContent = selectedFile.slice(currentFilePointer, currentFilePointer + maxBlockSize);
        reader.readAsArrayBuffer(fileContent);

Uploading Chunk

Since I wanted to implement uploading in chunk, as soon as a chunk is read from the file, I create a Put Block request based on Put Block REST API specification using jQuery’s AJAX function and pass that chunk as data. Once this request is successfully completed, I read the next chunk and repeat the process till the time all chunks are processed.

        reader.onloadend = function (evt) {
            if (evt.target.readyState == FileReader.DONE) { // DONE == 2
                var uri = submitUri + '&comp=block&blockid=' + blockIds[blockIds.length - 1];
                var requestData = new Uint8Array(evt.target.result);
                $.ajax({
                    url: uri,
                    type: "PUT",
                    data: requestData,
                    processData: false,
                    beforeSend: function(xhr) {
                        xhr.setRequestHeader('x-ms-blob-type', 'BlockBlob');
                        xhr.setRequestHeader('Content-Length', requestData.length);
                    },
                    success: function (data, status) {
                        console.log(data);
                        console.log(status);
                        bytesUploaded += requestData.length;
                        var percentComplete = ((parseFloat(bytesUploaded) / parseFloat(selectedFile.size)) * 100).toFixed(2);
                        $("#fileUploadProgress").text(percentComplete + " %");
                        uploadFileInBlocks();
                    },
                    error: function(xhr, desc, err) {
                        console.log(desc);
                        console.log(err);
                    }
                });
            }
        };

Committing Blob

Last step in this process is to commit the blob in blob storage. For this I create a Put Block List request based on Put Block List REST API specification and process that request again using jQuery’s AJAX function and pass the block list as data. This completed the process.

        function commitBlockList() {
            var uri = submitUri + '&comp=blocklist';
            console.log(uri);
            var requestBody = '';
            for (var i = 0; i < blockIds.length; i++) {
                requestBody += '' + blockIds[i] + '';
            }
            requestBody += '';
            console.log(requestBody);
            $.ajax({
                url: uri,
                type: "PUT",
                data: requestBody,
                beforeSend: function (xhr) {
                    xhr.setRequestHeader('x-ms-blob-content-type', selectedFile.type);
                    xhr.setRequestHeader('Content-Length', requestBody.length);
                },
                success: function (data, status) {
                    console.log(data);
                    console.log(status);
                },
                error: function (xhr, desc, err) {
                    console.log(desc);
                    console.log(err);
                }
            });

Complete Code

Here’s the complete code. For CSS, I actually used Metro UI CSS – http://metroui.org.ua/. If you’re planning on building web applications and want to give them Windows 8 applications style look and feel, do give it a try. It’s pretty awesome + it’s open source. Really no reason for you to not give it a try!




    File Uploader
    
    
    


    
        
            File Uploader
            
                SAS URI: 
                

                
                    
                
            
            
                File To Upload: 
                

                
                    
                
            
            
                
                File Properties:
                

                
                    Name: 
                
                
                    File Size:  bytes.
                
                
                    File Type: 
                
                
                    
                
                
                    Progress: 0.00 %

Some Caveats

This makes use of HTML5 File API and while all new browsers support that, same can’t be said about older browsers. If your users would be accessing an application like this using older browsers, you would need to think about alternative approaches. You could either make use of SWF File Uploader or could write an application using Silverlight. Steve Marx wrote a blog post about uploading files using Shared Access Signature and Silverlight which you can read here: http://blog.smarx.com/posts/uploading-windows-azure-blobs-from-silverlight-part-1-shared-access-signatures.

I found the code working in IE 10, Google Chrome (version 24.0.1312.57 m) on my Windows 8 machine. I got error when I tried to run the code in FireFox (version 18.0.2) and Safari (version 5.1.7) browsers so obviously one would need to keep the browser incompatibility in mind.

Enhancements

I hacked this code in about 4 hours or so and obviously my knowledge is somewhat limited when it comes to JavaScript and CSS so a lot can be improved on that front . However some other features I could think of are:

Generate SAS on demand: You could possibly have a server side component which would generate SAS URI on demand instead of having a user enter that manually.

Multiple file uploads: This application can certainly be extended to upload multiple files. A user would select multiple files (or may be even a folder) and have the application upload multiple files.

Drag/drop support: This application can certainly be extended to support drag/drop scenario where users could drag files from their desktop and drop them to upload.

Do upload in Web Worker: This is another improvement that can be done where uploads are done through web worker capability in HTML5.

Parallel uploads: Currently the code uploads one chunk at a time. A modification could be to upload multiple chunks simultaneously.

Transient error handling: Since Windows Azure Storage is a remote shared resource, you may encounter transient errors. You could modify the application to handle these transient errors. For more details on transient errors, please see this blog post of mine: http://gauravmantri.com/2013/01/11/some-best-practices-for-building-windows-azure-cloud-applications/.

Summary

So that’s it for this post! As you saw, it is quite easy to implement a very simple HTML/JS based application for getting data into Windows Azure Blob Storage. Obviously there’re some limitations and there’s cross-browser compatibility issues one would need to consider but once those are sorted out, it opens up a lot of exciting opportunities. I hope you’ve found this post useful. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Happy Coding!

Revisiting Windows Azure Shared Access Signature

Gaurav — Wed, 13 Feb 2013 08:03:14 +0000

In this blog post, we’ll talk about Shared Access Signature (SAS) functionality in Windows Azure. Steve Marx (@smarx) wrote excellent blog posts on the same subject a few years ago which you can read here: http://blog.smarx.com/posts/shared-access-signatures-are-easy-these-days, http://blog.smarx.com/posts/uploading-windows-azure-blobs-from-silverlight-part-1-shared-access-signatures. However a lot has changed since Steve wrote the posts and I thought it might be useful to write about it taking those new things into consideration (hence the word “Revisiting” in the title ). This blog will start with a basic explanation of shared access signature and the purpose it serves and then I will show you some code to perform some basic operations against Windows Azure Storage especially Blob Storage.

Now that the agenda is set, let’s start

Shared Access Signature (SAS) – Why and What?

This section describes why we need SAS and what exactly it is. If you’re familiar with these concepts, please feel free to skip this section otherwise read on!

Why?

First let’s understand why we need SAS. As you may already know that access to Windows Azure Storage is not anonymous. Rather you would need an “account key” to access resources in a Storage Account. This account key (actually there’re 2 account keys – primary and secondary) is generated automatically for you by Windows Azure Storage and you also get the ability to regenerate the key on demand. The problem with account key based access is that the account key grants “Admin” privileges on the storage account in question. Anybody who has access to this account key can perform any allowed operation on the storage account. This is obviously a security concern and one must attempt to keep the account key secret at all times and guard it with full zeal.

However there’re scenarios where you would want to grant a user (who may be a partner or a customer etc.) restricted access to your storage account. You would that user to perform only a few operations and that too for a limited amount of time. Obviously handing that user your account key won’t do you any good because that would make the user an admin. The answer to this problem is SAS. A SAS allows you to grant temporary access to private resources in your storage account.

I was recently reading a book by Bill Wilder (@codingoutloud) on Cloud Architecture Patterns and he has explained this concept very nicely by using a “Valet Key” analogy. If you’re planning on building a cloud application in Windows Azure, I would strongly recommend reading this book first before writing code.

What?

Simply put, SAS is a URL (or rather a part of it). You, as a creator of SAS define the validity duration of SAS and what a user in possession of this SAS can do with the resources protected by SAS. For example, you could define a SAS which would allow a user to upload a file into a blob container in your storage account and would expire after say 1 hour. Within that hour for which the SAS is valid, any user who has access to this SAS will be able to upload the files in the blob container. After that hour, the link will become useless.

You could create a SAS for a blob resources (blob containers and blobs), table resources (table and PartitionKey/RowKey ranges) and queue resources (queues and messages). You could give one or more of CRUD operation permissions in a SAS. Lastly, you could define a start and an end date for which the SAS is valid.

With blob storage, you could create a SAS for a blob container or a blob. Following table summarizes the permissions and the purpose they serve on various blob resources:

Permission Type	Blob Container	Blob
Read	Not applicable on blob container.	Gives the ability to read the contents of the blob. Also gives the ability to read the metadata and properties of the blob as well.
Write	Gives the ability to upload one or more blobs in blob container. Also gives the ability to update properties, metadata and create snapshots of a blob.	Give the ability to upload a new blob (by the same name) or overwrite an existing blob (by the same name). Also gives the ability to update properties, metadata and create snapshots of a blob.
List	Lists the blobs in the blob container.	Not applicable on blobs.
Delete	Not applicable on blob container.	Gives the ability to delete the blob.

Update – 22nd Feb 2013:
By “Not applicable on blob container” what I mean is that using a SAS that particular operation can’t be performed on the blob container in question. For example, “Delete” permission in SAS on a blob container will not allow the users to delete that blob container. One would need to use storage account credentials to delete the blob container.

With table storage, you could create a SAS for a table or entity ranges in a table. For entity ranges, you could specify PartitionKey/RowKey ranges. Following table summarizes the permissions and the purpose they serve on various table resources:

Permission Type	Table	Entity Ranges (PartitionKey/RowKey Ranges)
Query	Gives the ability to query the table.	Gives the ability to query a table but the query is limited to entity range specified in the SAS.
Add	Gives the ability to add an entity to the table.	Gives the ability to add an entity to the table provided the entity’s PartitionKey/RowKey values are within range.
Update	Gives the ability to update an entity in the table.	Gives the ability to update an entity in the table provided the entity’s PartitionKey/RowKey values are within range.
Delete	Gives the ability to delete an entity from the table.	Gives the ability to delete an entity from the table provided the entity’s PartitionKey/RowKey values are within range.

With queue storage, you could create a SAS for a queue. Following table summarizes the permissions and the purpose they serve on queue resources:

Permission Type	Queue
Read or Peek	Gives the ability to peek at the messages in the queue. Please note that peek messages functionality allows the user to view messages in “Read-Only” mode and does not alter the visibility of the messages.
Add	Gives the ability to add messages in the queue.
Process	Gives the ability to get messages from the queue. Please note that this would give users the ability to delete the messages as well and it alters the visibility of the messages.
Update	Gives the ability to update a message in the queue. For this permission, Process permission is required.

A few more comments about SAS before we jump into the code .

Revocable SAS

You could make a SAS revocable. What that means is that you could make a SAS invalid before its expiry time. This is achieved through what is known as “Blob Container Access Policy”. When you create a SAS using blob container access policy, to revoke the SAS, you could just change the access policy identifier or delete that access policy all together. At the time of writing, a blob container can have a maximum of 5 access policies.

Anonymous SAS

When a SAS is created without a blob container access policy, it’s called anonymous SAS. Since these are not revocable, proper care must be taken in handling this SAS. Generally speaking, you should try and create very short duration anonymous SAS.

Never Expiring SAS

Though not a recommended practice, it is possible to create a SAS which never expires. To achieve this, you don’t specify the expiry date when creating SAS. In the earlier version of storage service, only revocable SAS can be never expiring and anonymous SAS are only valid for 1 hour duration. However with the latest version of storage service, it is possible to create never expiring anonymous SAS.

SAS Start Date/Time

Please note that SAS start and expiry date/time are in UTC. Furthermore these are validated in Windows Azure. There may be a possibility that there may be a mismatch between time on your local computer (where SAS is generated) and time in Windows Azure. To counter this issue, it is generally recommended that you make your SAS start time by subtracting about 5 – 10 minutes from your local computer’s time (in UTC). For example:

            var sas = blobContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy()
                {
                    Permissions = permission,
                    SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-5),//SAS Start time is back by 5 minutes to take clock skewness into consideration
                    SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(15),
                });

Update – 22nd Feb 2013:
Please note that SAS start time is optional and if you want the SAS to be valid immediately, you can omit this parameter.

MSDN has nice documentation about SAS functionality and much more which I recommend you to read: http://msdn.microsoft.com/en-us/library/windowsazure/ee393343.aspx

The Code!

OK, enough talking! Let’s see some code. In the code examples below, I have used latest version of storage client library (2.0.4 at the time of writing this post). I’ve used the storage client library to create SAS URIs however you could use tools like Cloud Storage Studio which supports this functionality. Then using that SAS URI, I’m going to demonstrate how you’re going to perform some common operations against blob storage. For each operation, I’ve included how you’re going to perform the operation using storage client library as well as if you’re going to do this by consuming REST API directly using HttpWebRequest/HttpWebResponse. The common theme with storage client library is that it allows you to create instances of CloudBlobContainer and CloudBlockBlob (I’ve focused only on block blobs in this post) using the SAS URI only. Again for simplicity, I just went with anonymous SAS and did not use blob container access policy for creating SAS URIs.

Helper Functions

I created two helper functions to create SAS on a blob container and blob. For the sake of simplicity, I hard coded the start and expiry time for SAS:

        /// 
        /// Creates a SAS URI for the blob container.
        /// 
        /// 
        /// 
        /// 
        static string GetSaSForBlobContainer(CloudBlobContainer blobContainer, SharedAccessBlobPermissions permission)
        {
            var sas = blobContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy()
                {
                    Permissions = permission,
                    SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-5),//SAS Start time is back by 5 minutes to take clock skewness into consideration
                    SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(15),
                });
            return string.Format(CultureInfo.InvariantCulture, "{0}{1}", blobContainer.Uri, sas);
        }

        /// 
        /// Creates a SAS URI for the blob.
        /// 
        /// 
        /// 
        /// 
        static string GetSaSForBlob(CloudBlockBlob blob, SharedAccessBlobPermissions permission)
        {
            var sas = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
            {
                Permissions = permission,
                SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-5),
                SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(15),
            });
            return string.Format(CultureInfo.InvariantCulture, "{0}{1}", blob.Uri, sas);
        }

Listing Blobs

Here’s the code for listing blobs in a blob container. For listing blobs, “List” permission must be defined in SAS.

Using Storage Client Library

        /// 
        /// List blobs in a blob container using storage client library.
        /// 
        /// 
        static void ListBlobsWithStorageClientLibrary(string blobContainerSasUri)
        {
            CloudBlobContainer blobContainer = new CloudBlobContainer(new Uri(blobContainerSasUri));
            var blobs = blobContainer.ListBlobs(null, true);
            foreach (var blob in blobs)
            {
                Console.WriteLine(blob.Uri);
            }
        }

Using REST API

        /// 
        /// List blobs in a blob container using REST API.
        /// 
        /// 
        static void ListBlobsWithRestAPI(string blobContainerSasUri)
        {
            string requestUri = string.Format(CultureInfo.InvariantCulture, "{0}&comp=list", blobContainerSasUri);
            HttpWebRequest request = (HttpWebRequest) WebRequest.Create(requestUri);
            request.Method = "GET";
            using (HttpWebResponse resp = (HttpWebResponse) request.GetResponse())
            {
                using (Stream s = resp.GetResponseStream())
                {
                    using (StreamReader reader = new StreamReader(s, true))
                    {
                        string blobsData = reader.ReadToEnd();
                        Console.WriteLine(blobsData);
                    }
                }
            }
        }

You must append “&comp=list” to the URI for listing blob containers. Also the output is XML.

I could just take the URI (in requestUri variable above) and paste it in browser’s address window and see the list of blobs in the blob container. See screenshot below for example.

Upload Blobs

Here’s the code for uploading a blob in a blob container. For uploading a blob, you could create a SAS with “Write” permission on either a blob container or a blob. The blob need not be present in the blob container. If the blob is not present, it will be created otherwise it will be overwritten.

Using Storage Client Library

        /// 
        /// Uploads a blob in a blob container where SAS permission is defined on a blob container using storage client library.
        /// 
        /// 
        static void UploadBlobWithStorageClientLibrarySasPermissionOnBlobContainer(string blobContainerSasUri)
        {
            CloudBlobContainer blobContainer = new CloudBlobContainer(new Uri(blobContainerSasUri));
            CloudBlockBlob blob = blobContainer.GetBlockBlobReference("sample.txt");
            string sampleContent = "This is sample text.";
            using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(sampleContent)))
            {
                blob.UploadFromStream(ms);
            }
        }

        /// 
        /// Uploads a blob in a blob container where SAS permission is defined on a blob using storage client library.
        /// 
        /// 
        static void UploadBlobWithStorageClientLibrarySasPermissionOnBlob(string blobSasUri)
        {
            CloudBlockBlob blob = new CloudBlockBlob(new Uri(blobSasUri));
            string sampleContent = "This is sample text.";
            using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(sampleContent)))
            {
                blob.UploadFromStream(ms);
            }
        }

Using REST API

        /// 
        /// Uploads a blob in a blob container where SAS permission is defined on a blob container using REST API.
        /// 
        /// 
        static void UploadBlobWithRestAPISasPermissionOnBlobContainer(string blobContainerSasUri)
        {
            string blobName = "sample.txt";
            string sampleContent = "This is sample text.";
            int contentLength = Encoding.UTF8.GetByteCount(sampleContent);
            string queryString = (new Uri(blobContainerSasUri)).Query;
            string blobContainerUri = blobContainerSasUri.Substring(0, blobContainerSasUri.Length - queryString.Length);
            string requestUri = string.Format(CultureInfo.InvariantCulture, "{0}/{1}{2}", blobContainerUri, blobName, queryString);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
            request.Method = "PUT";
            request.Headers.Add("x-ms-blob-type", "BlockBlob");
            request.ContentLength = contentLength;
            using (Stream requestStream = request.GetRequestStream())
            {
                requestStream.Write(Encoding.UTF8.GetBytes(sampleContent), 0, contentLength);
            }
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                
            }
        }

        /// 
        /// Uploads a blob in a blob container where SAS permission is defined on a blob using REST API.
        /// 
        /// 
        static void UploadBlobWithRestAPISasPermissionOnBlob(string blobSasUri)
        {
            string sampleContent = "This is sample text.";
            int contentLength = Encoding.UTF8.GetByteCount(sampleContent);
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(blobSasUri);
            request.Method = "PUT";
            request.Headers.Add("x-ms-blob-type", "BlockBlob");
            request.ContentLength = contentLength;
            using (Stream requestStream = request.GetRequestStream())
            {
                requestStream.Write(Encoding.UTF8.GetBytes(sampleContent), 0, contentLength);
            }
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {

            }
        }

When working with REST API, you must specify “x-ms-blob-type” request header. For block blobs, its value must be “BlockBlob” and for page blobs, the value must be “PageBlob”.

Upload Blobs in Blocks

Here’s the code if you want to upload a blob by splitting it into blocks. For uploading a blob, you could create a SAS with “Write” permission on either a blob container or a blob. The blob need not be present in the blob container. If the blob is not present, it will be created otherwise it will be overwritten.

Using Storage Client Library

        /// 
        /// Uploads a blob by splitting into blocks in a blob container where SAS permission is defined on a blob container using storage client library.
        /// 
        /// 
        static void UploadBlobInBlocksWithStorageClientLibrarySasPermissionOnBlobContainer(string blobContainerSasUri)
        {
            CloudBlobContainer blobContainer = new CloudBlobContainer(new Uri(blobContainerSasUri));
            CloudBlockBlob blob = blobContainer.GetBlockBlobReference(Guid.NewGuid().ToString() + ".txt");
            List blockIds = new List();
            for (int i = 0; i < 10; i++)
            {
                string sampleContent = string.Format(CultureInfo.InvariantCulture, "Line {0}: This is sample text.\r\n", i);
                string blockId = i.ToString("d4");
                blockIds.Add(blockId);
                using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(sampleContent)))
                {
                    blob.PutBlock(blockId, ms, null);
                }
            }
            blob.PutBlockList(blockIds);
        }

        /// 
        /// Uploads a blob by splitting into blocks in a blob container where SAS permission is defined on a blob using storage client library.
        /// 
        /// 
        static void UploadBlobInBlocksWithStorageClientLibrarySasPermissionOnBlob(string blobSasUri)
        {
            CloudBlockBlob blob = new CloudBlockBlob(blobSasUri);
            List blockIds = new List();
            for (int i = 0; i < 10; i++)
            {
                string sampleContent = string.Format(CultureInfo.InvariantCulture, "Line {0}: This is sample text.\r\n", i);
                string blockId = i.ToString("d4");
                blockIds.Add(blockId);
                using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(sampleContent)))
                {
                    blob.PutBlock(blockId, ms, null);
                }
            }
            blob.PutBlockList(blockIds);
        }

Using REST API

        /// 
        /// Uploads a blob by splitting into blocks in a blob container where SAS permission is defined on a blob container using REST API.
        /// 
        /// 
        static void UploadBlobInBlocksWithRestAPISasPermissionOnBlobContainer(string blobContainerSasUri)
        {
            string blobName = Guid.NewGuid().ToString() + ".txt";
            string queryString = (new Uri(blobContainerSasUri)).Query;
            string blobContainerUri = blobContainerSasUri.Substring(0, blobContainerSasUri.Length - queryString.Length);
            List blockIds = new List();
            for (int i = 0; i < 10; i++)
            {
                string sampleContent = string.Format(CultureInfo.InvariantCulture, "Line {0}: This is sample text.\r\n", i);
                string blockId = i.ToString("d4");
                blockIds.Add(blockId);
                int contentLength = Encoding.UTF8.GetByteCount(sampleContent);
                string requestUri = string.Format(CultureInfo.InvariantCulture, "{0}/{1}{2}&comp=block&blockid={3}", blobContainerUri, blobName, queryString, Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId)));
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
                request.Method = "PUT";
                request.ContentLength = contentLength;
                using (Stream requestStream = request.GetRequestStream())
                {
                    requestStream.Write(Encoding.UTF8.GetBytes(sampleContent), 0, contentLength);
                }
                using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
                {

                }
            }
            string commitBlockListRequestBodyFormat = @"{0}";

            string blockListReuestBodyFormat = "{0}";
            StringBuilder sb = new StringBuilder();
            for (int i = 0; i < blockIds.Count; i++)
            {
                sb.AppendFormat(CultureInfo.InvariantCulture, blockListReuestBodyFormat,
                                Convert.ToBase64String(Encoding.UTF8.GetBytes(blockIds[i])));
            }
            string requestPayLoad = string.Format(CultureInfo.InvariantCulture, commitBlockListRequestBodyFormat,
                                                  sb.ToString());
            int contentLengthForCommitBlockList = Encoding.UTF8.GetByteCount(requestPayLoad);
            string requestUriForCommitBlockList = string.Format(CultureInfo.InvariantCulture, "{0}/{1}{2}&comp=blockList", blobContainerUri, blobName, queryString);
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(requestUriForCommitBlockList);
            req.Method = "PUT";
            req.ContentLength = contentLengthForCommitBlockList;
            using (Stream requestStream = req.GetRequestStream())
            {
                requestStream.Write(Encoding.UTF8.GetBytes(requestPayLoad), 0, contentLengthForCommitBlockList);
            }
            using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
            {

            }
        }

        /// 
        /// Uploads a blob by splitting into blocks in a blob container where SAS permission is defined on a blob using REST API.
        /// 
        /// 
        static void UploadBlobInBlocksWithRestAPISasPermissionOnBlob(string blobSasUri)
        {
            List blockIds = new List();
            for (int i = 0; i < 10; i++)
            {
                string sampleContent = string.Format(CultureInfo.InvariantCulture, "Line {0}: This is sample text.\r\n", i);
                string blockId = i.ToString("d4");
                blockIds.Add(blockId);
                int contentLength = Encoding.UTF8.GetByteCount(sampleContent);
                string requestUri = string.Format(CultureInfo.InvariantCulture, "{0}&comp=block&blockid={1}", blobSasUri, Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId)));
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
                request.Method = "PUT";
                request.ContentLength = contentLength;
                using (Stream requestStream = request.GetRequestStream())
                {
                    requestStream.Write(Encoding.UTF8.GetBytes(sampleContent), 0, contentLength);
                }
                using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
                {

                }
            }
            string commitBlockListRequestBodyFormat = @"{0}";

            string blockListReuestBodyFormat = "{0}";
            StringBuilder sb = new StringBuilder();
            for (int i = 0; i < blockIds.Count; i++)
            {
                sb.AppendFormat(CultureInfo.InvariantCulture, blockListReuestBodyFormat,
                                Convert.ToBase64String(Encoding.UTF8.GetBytes(blockIds[i])));
            }
            string requestPayLoad = string.Format(CultureInfo.InvariantCulture, commitBlockListRequestBodyFormat,
                                                  sb.ToString());
            int contentLengthForCommitBlockList = Encoding.UTF8.GetByteCount(requestPayLoad);
            string requestUriForCommitBlockList = string.Format(CultureInfo.InvariantCulture, "{0}&comp=blockList", blobSasUri);
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(requestUriForCommitBlockList);
            req.Method = "PUT";
            req.ContentLength = contentLengthForCommitBlockList;
            using (Stream requestStream = req.GetRequestStream())
            {
                requestStream.Write(Encoding.UTF8.GetBytes(requestPayLoad), 0, contentLengthForCommitBlockList);
            }
            using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
            {

            }
        }

When working with REST API, for putting blocks you must specify “comp=block” and Base64 encoded block id in the request URI. For putting block list, you must specify “comp=blockList” in the request URI.

Download Blobs

Here’s the code for downloading a blob from a blob container. For downloading blob, “Read” permission must be defined in SAS.

Using Storage Client Library

        /// 
        /// Downloads a blob from a blob cobtainer using storage client library.
        /// 
        /// 
        static void DownloadBlobWithStorageClientLibrary(string blobSasUri)
        {
            CloudBlockBlob blob = new CloudBlockBlob(new Uri(blobSasUri));
            using (MemoryStream ms = new MemoryStream())
            {
                blob.DownloadToStream(ms);
                byte[] data = new byte[ms.Length];
                ms.Position = 0;
                ms.Read(data, 0, data.Length);
                string blobContents = Encoding.UTF8.GetString(data);
                Console.WriteLine(blobContents);
            }
        }

Using REST API

        /// 
        /// Downloads a blob from a blob cobtainer using REST API.
        /// 
        /// 
        static void DownloadBlobWithRestAPI(string blobSasUri)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(blobSasUri);
            request.Method = "GET";
            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
                using (Stream s = resp.GetResponseStream())
                {
                    using (StreamReader reader = new StreamReader(s, true))
                    {
                        string blobData = reader.ReadToEnd();
                        Console.WriteLine(blobData);
                    }
                }
            }
        }

Deleting Blobs

Here’s the code for deleting a blob from a blob container. For delete operation, “Delete” permission must be defined in SAS.

Using Storage Client Library

        /// 
        /// Deletes a blob from a blob cobtainer using storage client library.
        /// 
        /// 
        static void DeleteBlobWithStorageClientLibrary(string blobSasUri)
        {
            CloudBlockBlob blob = new CloudBlockBlob(new Uri(blobSasUri));
            blob.Delete();
        }

Using REST API

        /// 
        /// Deletes a blob from a blob cobtainer using REST API.
        /// 
        /// 
        private static void DeleteBlobWithRestAPI(string blobSasUri)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(blobSasUri);
            request.Method = "DELETE";
            using (HttpWebResponse resp = (HttpWebResponse) request.GetResponse())
            {
                
            }
        }

Summary

That’s it for this post. I could have possibly done more and included more samples but these should give you an idea about how to use SAS to perform various operations. May be in another post, I will show some functionality related to Tables and Queues with SAS. I hope you’ve found this post useful. If you find any issues with this post, please let me know and I’ll try and fix them ASAP. Feel free to provide your comments below.

Happy Coding!!!

Workaround for IIS Express Crashing When Running Windows Azure Cloud Service Web Role with Multiple Instances in Windows Azure SDK 1.8 Compute Emulator

Gaurav — Tue, 05 Feb 2013 05:49:42 +0000

I ran into this weird issue when developing a very simple Windows Azure Cloud Service Web Role using Windows Azure SDK 1.8. I started off with a basic ASP.Net MVC 4 Web Role with no changes whatsoever. When I tried to debug it with just one instance running, everything worked great. The moment I increased the number of instances to 2 (or more than 1) and tried to debug it, IIS Express crashed with the following entries in event logs:

Faulting application name: iisexpress.exe, version: 8.0.8418.0, time stamp: 0x4fbae3d6
Faulting module name: ntdll.dll, version: 6.2.9200.16420, time stamp: 0x505ab405
Exception code: 0xc0000008
Fault offset: 0x0000000000004c39
Faulting process id: 0x54bc
Faulting application start time: 0x01cdfeb70c1e060e
Faulting application path: C:\Program Files\IIS Express\iisexpress.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 4a8f54b2-6aaa-11e2-be81-9cb70d025a13
Faulting package full name:
Faulting package-relative application ID:

Fault bucket 79240842, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: iisexpress.exe
P2: 8.0.8418.0
P3: 4fbae3d6
P4: ntdll.dll
P5: 6.2.9200.16420
P6: 505ab405
P7: c0000008
P8: 0000000000004c39
P9:
P10:

Attached files:
C:\Users\Gaurav.Mantri\AppData\Local\Temp\WER4720.tmp.WERInternalMetadata.xml

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_iisexpress.exe_1263888cd189f3357758cdc998d7afecf3f28_2dae4eb2

Analysis symbol:
Rechecking for solution: 0
Report Id: 4a8f54b2-6aaa-11e2-be81-9cb70d025a13
Report Status: 17
Hashed bucket: b615b24f7933990317ae65713d680dab

This issue does not happen on all machines. I asked a few of my friends to try on their computers and some of them were able to recreate this problem while others couldn’t. My development environment was Windows 8, Visual Studio 2012 and Windows Azure SDK 1.8.

Since I was facing this problem and a few of my friends are also facing the same problem, I thought there may be many more folks who may be facing same problem and hence this blog.

Windows Azure product team helped me find the cause of the problem and gave me a workaround which fixed the problem. I’m also told that this issue will be fixed in the coming versions of SDK but till the time the new SDK comes, the workaround recommended by the team can be used to address this problem.

Cause

From what I have been told, this is related to an issue with Windows Azure SDK 1.8 and it happens because IIS Express is trying to bind same IP address to multiple instances. Here’re the entries from applicationHost.config file for 2 instances in which I was trying to run my application which let the product team to that conclusion:

Instance 0:

Instance 1:

As you can see from above in the “binding” section, both instances were trying to bind to port 82 and hence the error.

Workaround

There were two workarounds recommended by the Windows Azure SDK team:

Install Visual Studio 2010 SP1

If you’re using Visual Studio 2010, it is recommended that you install SP1 for that. Seemingly that takes care of this problem. Since I don’t have VS 2010 on my machine, I did not try it as it was not applicable to me.

Add an Environment Variable

Other workaround recommended was to add an environment variable called “_CSRUN_DISABLE_WORKAROUNDS” and set the value of this variable to “1”. To set an environment varaible, search for “Control Panel” and then go to “System” –> “Advanced system settings” –> “Advanced” tab –> “Environment Variables…” button –> “New…” button.

I’ve also added a screenshot for your convenience as well.

After you do that you would need to restart IIS Express and Compute/Storage Emulator.

I tried this approach and it worked for me.

Hope this helps!

Windows Azure Access Control Service, Windows 8 and Visual Studio 2012: Setting Up Development Environment – Oh My!

Gaurav — Mon, 28 Jan 2013 12:45:02 +0000

Today I started playing with Windows Azure Access Control Service (ACS) and ran into some issues while making a simple application based on the guide here: http://www.windowsazure.com/en-us/develop/net/how-to-guides/access-control/. I had to search for a number of things on the Internet to finally get it working so I thought I would write a simple blog post which kind of summarizes the issues and how to resolve them.

The problems I ran into were because the guide was focused on Windows 7/Visual Studio 2010 and I was using Windows 8/Visual Studio 2012.

It may very well be documented somewhere but I could not find them and there may be many “newbies” like me who are just starting out with ACS on Windows 8/VS 2012 and hopefully this blog post will save some time and frustration implementing this ridiculously easy and useful technology for implementing authentication in their web applications.

With this thought, let’s start

Windows Identity Foundation Setup

When I downloaded and tried to run the “Windows Identity Foundation” from “Windows Identity Foundation” link from above, I got the following message:

I found the solution for this problem on Windows 8 Community Forums: http://answers.microsoft.com/en-us/windows/forum/windows_8-windows_install/error-when-installing-windows-identity-foundation/6e4c65a5-ea24-431c-b4f0-b8740e97cf46?auth=1. Basically, the solution is that you have to enable WIF through “Turn Windows Feature On/Off” functionality. To do so, go to Control Panel –> Programs and Features –> Turn Windows Feature on or off.

Also once WIF is installed, I was able to install Windows Identity Foundation SDK. Prior to that I was not able to do that as well.

Dude, Where’s My “Add STS Reference” Option!

OK, moving on. After following some more steps to a “T”, I ran into the following set of instructions:

So I followed step 1 but could not find “Add STS Reference” option when I right clicked on the solution in Visual Studio. Back to searching and I found a couple of blog posts:

http://blogs.infosupport.com/equivalent-to-add-sts-reference-for-vs-2012/

http://www.stratospher.es/blog/post/add-sts-reference-missing-in-visual-studio-2012-rc

Basically what I needed to do was install “Identity and Access Tool” extension. To do so, just click on Tools –> Extensions and Updates… and search for “Identity” as shown below:

One I installed it, I was a happy camper again

And instead of “Add STS Reference” the options’ name is “Identity and Access”

Configuring Trust between ACS and ASP.NET Web Application

Next step is configuring trust between ACS and ASP.Net Web Application. The steps provided in the guide were for Visual Studio 2010 and clicking on “Identity and Access…” gave me an entirely different window.

Again this blog post from Haishi Bai came to my rescue: http://haishibai.blogspot.in/2012/08/windows-azure-web-site-targeting-net.html. However one thing that was not clear to me (or consider this my complete ignorance ) is from where I’ll get the management key. I logged into the ACS portal (https://.accesscontrol.windows.net) and first thought I would find the key under “Certificates and keys” section there (without reading much above. You can understand, I was desperate ). I copied the key from there and entered that key in the window above but that didn’t work.

As I looked around, I found the “Management service” section under “Administration” section and tried to use the key from there.

Clicked on “Generate” button to create a key and used it and voila it worked . The portal also generates a key for you which you can also use, BTW.

Running The Application

Assuming the worst part was over, I followed remaining steps in the guide and pressed “F5” to run the application. But the application didn’t run . Instead I got this error:

Again I was at the mercy of search engines . I searched for it and found a blog post by Sandrino: http://sandrinodimattia.net/blog/posts/azure-appfabric-acs-unauthorized-logon-failed-due-to-server-configuration/. I applied the fix he suggested but unfortunately that didn’t work . He also mentioned that it worked in an MVC application (this one was a WebForm application) so I tried creating a simple MVC 4 application and guess what, it worked there as Sandrino said .

For now, I’m happy with what I’ve got and will focus on building some applications consuming ACS and see what this service has to offer.

Alternative Approach

While working on this, I realized an alternative approach to get up and running with ACS. Here’s what you would need to do:

Create a new Access Control Service in Windows Azure Portal.
Once the service is active, click on Manage button to manage that service. You will be taken to the ACS portal for your service.
Configure Identity Providers.
Get the management key from “Management service” section.
Head back to Visual Studio project and configure ACS. You would need the name of the service and the management key.
The tooling in Visual Studio takes care of creating “Relaying party applications” for you based on your project settings.

Summary

I spent considerable amount of time setting up this thing which kind of frustrated me. I hope this post will save you some time and frustration. Also this is the first time I looked at this service so it is highly likely I may have made some mistakes or included them here based on my sheer ignorance. Please feel free to correct me if I have provided incorrect information.

Happy Coding!!!