Nuno Filipe Godinho

Lessons Learned: Taking the best out of Windows Azure Virtual Machines

NunoGodinho — Mon, 22 Apr 2013 23:14:51 GMT

Introduction

Now that Windows Azure IaaS offerings are out and made GA a lot of new workloads can be enabled with Windows Azure. Workloads like, SQL Server, SharePoint, System Center 2012, Server Roles like AD, ADFS DNS and so on, and even Team Foundation Server. More of the supported list of server software that is currently supported in Windows Azure Virtual Machines can be found here.

But knowing what we can leverage in the Cloud isn’t enough, every features has its tricks in order to take the best out of it. In this case in order to take the best performance out of the Windows Azure Virtual Machines, I’ll provide you with a list of things you should always do, and so making your life easier and the performance a lot better.

1. Place each data disk in a single storage account to improve IOPS

Last November 2012 Windows Azure Storage had an update which was called “Windows Azure’s Flat Network Storage” which provided some new scalability targets to the blob storage accounts. In this case it went from 5,000 to 20,000, which means that we can actually have something like 20,000 IOPS now.

Having 20,000 IOPS is good but if we have several disks for the same Virtual Machine this means that we’ll need to share those IOPS with all those disks, so if we have 2 disks in the same storage account we’ll have 10,000 IOPS for each one (roughly). This isn’t optimal.

So, in order to achieve optimal we should create each disk in a separate storage account, because that will mean that each disk has it’s 20,000 IOPS just for itself and not sharing with any other disk.

2. Always use Data Disks for Read/Write intensive operations, never the OS Disk

Windows Azure Virtual Machines have two types of disks, which are OS and Data Disks. The OS Disk goal is to have everything that has to do with OS installation or any other product installation information, but isn’t actually a good place to install your highly intensive read/write software. In order to do that, you should actually leverage Data Disks, because their goal is to provide a faster and read/write capability and also separate this from the OS Disk.

So since data disks are better than OS Disks it’s easy to understand why we should always place read/write intensive operations on data disks. Just be careful on the maximum number of data disks you can associate to your virtual machine, since it will differ. 16 Data Disks is the maximum you are allowed but for that you need to have an extra large virtual machine.

3. Use striped disks to achieve better performance

So we told that you should always place your read/write intensive operations software on data disks and in different storage accounts because of the IOPS you can get, and we told it was 20k IOPS, but is that enough? Can we live with only 20k IOPS in a disk?

Sometimes the answer might be yes, but in some cases it won’t because we need more. For example if we think about SQL Server or SharePoint they will require a lot more, and so how can we get more IOPS?

The answer is data disks striped together. What this means is that you’ll need to understand your requirements and know what’s the IOPS you’re going to need and based on that you’ll create several data disks and attach them to the virtual machine and finally stripe them together like they were a single disk. For the user of the virtual machine it will look like a single disk but it’s actually several ones striped together, which means each of the parts of that “large disk presented to the user” has 20k IOPS capability.

For example, imagine we’re building a virtual machine for SQL Server and that the size of the database is 1TB but requires at least 60k IOPS. What can we do?

Option 1, we could create a 1TB Data Disk and place the database files in there but that would max out to 20k IOPS only and not the 60k we need.

Option 2, we will create 4 data disks of 250GB each and place each of them in a single storage account. Then we’ll attach it to the virtual machine and in the Disk Management we’ll choose to stripe them together. Now this means that we have a 1TB disk in the virtual machine that is actually composed by 4 data disks. So this means that we can actually get something like a max of 80k IOPS for this. So a lot better than before.

4. Configure Data Disks HostCache for ReadWrite

By now you already understood that data disks are your friends, and so one of the ways to achieve better performance with them is leveraging the HostCache. Windows Azure provides three options for data disk HostCache, which are None, ReadOnly and ReadWrite. Of course most of the times you would choose the ReadWrite because it will provide you a lot better performance, since now instead of going directly to the data disk in the storage account it will have some cached content making IOPS even better, but that doesn’t work in all cases. For example in SQL Server you should never use it since they don’t play well together, in that case you should use None instead.

5. Always create VMs inside a Affinity Group or VNET to decrease latency

Also another big improvement you can do is to place always de VM inside an affinity group or a VNET, which in turn will live inside the affinity group. This is important because when you’re creating the several different storage accounts that will have data disks, OS disks and so on, you' want to make sure the latency is decreased to the max and so affinity groups will provide you with that.

6. Always leverage Availability Sets to get SLA

Windows Azure Virtual Machines provide a 99,95% SLA but only if you have 2 or more virtual machines running inside an availability set, so leverage it, always create your virtual machines inside an availability set.

7. Always sysprep your machines

One of the important parts of the work when we take on Windows Azure Virtual Machines is to create a generalized machine that we can use later as a base image. Some people ask me, why is this important? why should I care?

The answer is simple, because we need to be able to quickly provision a new machine if it’s required and if we have it syspreped we’ll be able to use it as a base and then reducing the time of installation and provisioning.

Examples of where we would need this would be for Disaster Recovery and Scaling.

8. Never place intensive read/write information on the Windows System Drive for improved performance

As stated before, OS Disks aren’t good for intensive IOPS so avoid leveraging them for read/write intensive work, leverage data disks instead.

9. Never place persistent information on the Temporary Drive (D:)

Careful what you place inside the Temporary Drive (D: ) since that’s temporary and so if the machine recycles it will go away, so only place there something that can be deleted without issues. Things like the IIS Temporary files, ASP.NET Temp files, SQL Server TempDB (this has some challenges but can be achieved like it’s shown here, and it’s actually a best practice).

Summary

So in summary, Windows Azure Virtual Machines are a great addition to Windows Azure but there’s a lot of tricks in order to make it better and these are some of them. If you need any help feel free to contact me and I’ll help you in anyway possible. But best of all, start to take the best out of Windows Azure Virtual Machines today and take your solutions into the next level.

Also if you need some help doing that, please check Aditi’s offerings around Windows Azure IaaS here.

How to make Windows Azure as an Extension of On-Premises Data Center - Windows Azure Virtual Networks - Part 1

NunoGodinho — Thu, 28 Feb 2013 15:01:18 GMT

Now with Windows Azure Virtual Machines and Virtual Networks a lot more capabilities are available to be able to look at Windows Azure not as a 'yet another platform' and not your network, but really think of it as a real extension of your On-Premises Data Center. Of course that this always depends on the type of company we are talking about, since if we talk with Enterprises this is a MUST-HAVE because they have a lot of investments still in the On-Premises world and some that aren't still ready, and might never be, for the Public Cloud, but if we talk to ISV's this isn't that important because they want to reduce as much as possible their On-Premises needs.

In order to achieve this extension there is a component in Windows Azure that is key, which is Windows Azure Virtual Network, since it allows to create a VPN between On-Premises and your Windows Azure resources. But there are some important considerations to have in mind, like:

Windows Azure Virtual Networks is currently still on Preview
In order to use Windows Azure Virtual Network it's required to have a Router device that supports VPN on the On-Premises location.
The On-Premises VPN devices that are currently tested can be found here. This doesn't mean that they are the only ones you can use, it just means that those are a lot simpler to configure because Windows Azure provides a configuration file that is required to import into the device and it's done.
Windows Azure Virtual Networks do not span Regions or Subscriptions, which means that if you have multiple deployments in the same region and within the same subscription you can use the same VNET, if not you're required to create multiple VNET's. Here are some scenarios:
- Scenario A:
  - Description: Subscription A, has Service B deployed into Windows Azure Cloud Services in North Europe region and Service C deployed in Windows Azure Cloud Services in West Europe region
  - Comments: Even though they are in the same subscriptions since they are in different regions you would need to create a VNET for Subscription A for the North Europe region and another for the West Europe region.
- Scenario B:
  - Description: Subscription A , has Service A and B deployed Windows Azure Cloud or Windows Azure Virtual Machines, and it's required that they are in the same VNET
  - Comments: In this case you only need one VNET for both since they do not span either subscriptions or regions.
- Scenario C:
  - Description: Subscription A has Service B and C deployed in Windows Azure Cloud Service within the same region, but it's required to create security when connecting between them.
  - Comments: In order to achieve this it's only required to create one VNET since they are in the same subscription and region, but 2 different subnets one for each service, and then it's the On-Premises VPN/Firewall device that will create the restrictions for each Subnet.
- Scenario D:
  - Description: Subscription A has Service B deployed in Windows Azure on the North Central US region, and Subscription C has Service D deployed in Windows Azure on the North Central US region, but they need to communicate between themselves.
  - Comments: in order to achieve this it's required to create a 2 separate VPN connections, one for Subscription A and another for Subscription C, because VNET's don't span across different subscriptions even if they are in the same region.
Currently there's no ACLing for subnet isolation, so that needs to be done in one of three ways.
- Create different VNET for each Subnet and this way they aren't known
- Perform the ACLing and restrictions between the different subnets on the Windows Firewall level of the instance
- Perform the ACLing in and On-Premises Firewall device.

So by leveraging Windows Azure Virtual Networks we'll be able to connect everything we have deployed in Windows Azure Compute with our On-Premises Data Center. By doing this companies gain the ability of leveraging more of their existing investments and look at Windows Azure in a more "extension of Data Center" way and less as a "Black box" which you don't have a lot of control.

In future posts I'll go through the process of how-to setup a new Windows Azure Virtual Network between On-Premises and Windows Azure.

March 2013 Windows Azure Speaking Engagements

NunoGodinho — Mon, 11 Feb 2013 16:36:25 GMT

been doing a lot of work in Windows Azure both on the PaaS and IaaS world and since we’re entering the typical time where the Cloud Events start to happen, I’d like to share where you guys can see me talking about Windows Azure.

The events I’m currently speaking are:

DevWeek 2013 (http://www.devweek.com/ and http://www.devweek.com/speakers/)

March 5th

- SQL Azure overview – how to develop and manage it (http://www.devweek.com/sessions/conference1.asp)

· In this session we will be looking at an overview of SQL Azure in terms of Architecture, Application Topologies that can be used and its Provisioning Model. We’ll also be looking at how Deployment can be done, and which Security we can expect on it. One of the important parts will be understanding the elements that are currently supported and not supported on the current version of SQL Azure, and what we can expect about the future.

- Advanced SQL Azure – performance and scalability (http://www.devweek.com/sessions/conference1.asp)

· In this session we’ll look at some more important SQL Azure topics like performance and scalability, and how concepts like Sharding, SQL Azure Federation are important in order to achieve scalability improvements. We will also look at how SQL Azure Data Sync is important for these concepts and also to keep a local replica of the SQL Azure Database, SQL Azure Backups, Import and Exports and other new features that are being release by the SQL Azure team.

March 6^th

- Tips & tricks to build Multi-Tenant databases with Windows Azure SQL Databases (http://www.devweek.com/sessions/conference2.asp)

· When we talk about the Cloud it’s very important the we do our solutions in a highly scalable way as well as in a Multi-Tenant way, since this actually helps us lower the costs to the end customer and so grab the long tail. In order to do this a lot of adjustments need to happen on the database side of things, and in this session we’ll look at ways we can achieve Multi-Tenancy with Windows Azure SQL Databases, and how that can be integrated with the work performed while scaling out Windows Azure SQL Databases, using SQL Federations.

TechDays Netherlands 2013 (http://www.microsoft.com/netherlands/techdays/home.aspx and http://www.microsoft.com/netherlands/techdays/SpeakerDetail.aspx?speakerId=1441)

March 7^th

- Crash Course on "Automating deployments in Windows Azure Virtual Machines". How and which tools? (http://www.microsoft.com/netherlands/techdays/SessionDetail.aspx?sessionId=3735)

· Windows Azure Virtual Machines have a very interesting interface in the new Windows Azure management portal and it's very usable when thinking about a small number of Virtual Machines, but when we start building large deployments like 10, 20, 100, 500 virtual machines with VNETs, failover, and so on things start to be different and less doable using the management portal. For this type of work we actually need some form of automation and this is actually what we'll be covering in this session. And so in this session we'll look at tools that can enable us to automate Windows Azure Virtual Machines deployments, like PowerShell, ScaleXtreme, RightScale, and other in order to take the automation issue out of the way and really take advantage of Windows Azure.

March 8^th

- Lessons Learned: Bridging Windows Azure and On-Premises environments with Windows Azure Virtual Network (http://www.microsoft.com/netherlands/techdays/SessionDetail.aspx?sessionId=3736)

· Windows Azure Virtual Network is a very important feature in Windows Azure since it provides a way to extend the existing On-Premises environment into the Cloud, making Windows Azure look as an Extension of the already existing Data Center. Setting up Virtual Network seems simple but sometimes have some important topics that need to be considered. Topics like, Which DNS should I use? How to configure create DMZ's within the Cloud? How to place PaaS components in the same Virtual Network? How do I configure the Gateway between Cloud and On-Premises? How to troubleshoot the connection? These are some of the concerns that you normally handle when configuring and using Windows Azure Virtual Networks, and so in these session we'll cover these topics based on the lessons learned from doing this with a great deal of customer and going through each of those concerns.

- Achieve High Availability with SQL Server on Windows Azure Virtual Machines (http://www.microsoft.com/netherlands/techdays/SessionDetail.aspx?sessionId=3737)

· SQL Server has been massively used by organizations in order to provide a RDBMS and when moving into the Cloud one of the elements that normally caused some concerns was 'THE DATABASE', since SQL Databases are Shared Instances and don't have feature parity with SQL Server, which causes some changes in the existing codebase. Now Windows Azure Virtual Machines opens a new door for the Cloud Migration process, since it allows to completely move our SQL Server into the Cloud and keep feature parity with On-Premises, which is important. But with this great news some new concerns appear also. Concerns like How to achieve High-Availability, Failover, Clustering, Data Security and so on with SQL Server in Windows Azure Virtual Machines. In this session, we'll look at SQL Server in Windows Azure and how we can use it and achieve a Highly Available environment for out RDBMS.

- Lessons Learned: Taking the best performance out of Windows Azure Virtual Machines (http://www.microsoft.com/netherlands/techdays/SessionDetail.aspx?sessionId=3738)

· Windows Azure Virtual Machines are still new in Windows Azure and have a lot of tweaks needed in order to take the best out of their capabilities. After being working with them since the early beginning a lot of lessons were learned in the process in terms of how to achieve the best performance. In this process we'll cover some topics and strategies that will enable us to take the best performance out of Windows Azure Virtual Machines but at the same time also achieve the best SLA possible.

Hope to see you in one of my sessions and if you would like to send any requests for some of the sessions you’re attending just send me a Tweet for @NunoGodinho. This way I’ll be able to create those Windows Azure sessions more related to what all of you really want to see.

Scheduler in Windows Azure

NunoGodinho — Tue, 05 Feb 2013 23:48:25 GMT

Windows Azure has been growing significantly and adding more and more services, but something that has always missed has been a Windows Azure Scheduler service. The windows azure just release one version of it that seems to have potential but still in very early stages.

Recently Aditi Product Services created their first Windows Azure Service available in the Windows Azure Store, and this new service is actually a Cloud Based CRON Scheduler.

This new service basically allows the user to create schedule tasks using a set of HTTP webhooks that run CRON schedules, being either simple or complex.

You can check more about this service and why it was built in this great post by Wade Wegner.

Looking at the features:

Support for CRON statements for reoccurrence scheduling.
Allows simple HTTP GET requests—easily allowing developers to use Scheduler to execute webhooks. With Scheduler, there is no requirement for a Windows Azure machine instance — every endpoint is an HTTP API. Scheduler supports development directly against the API or using our NuGet package.
Fully integrated into the Windows Azure store.
Run up to 5000 jobs in a month (during our Free Trial).
HTTP GET requests to your services and a full Web API for CRUD operations.
Simple and complex CRON job expressions.

For future releases will include HTTP POST, Auth, sending jobs into message queues and more

In terms of the architecture you can check this other post from Ryan Dunn

Now looking at all those features I can remember several different use cases where this might be helpful, one of them is something a lot of DBAs have been asking which is Database Backups. So what do we need to do in order to do this with this Scheduler Service?

1. Add Scheduler to your Subscription by going into the http://manage.windowsazure.com as choosing "PURCHASE ADD-ONS"

2. Choose the Scheduler Add-On

3. Now Create a Visual Studio Project and Add the Aditi.Scheduler Nuget package

4. Open your code where you want to add your Scheduler information and just add the following code:

   1:  var tenantId = "YOUR TENANT ID";

   2:  var secretKey = "YOUR SECRET KEY";

   3:  var scheduledTasks = new ScheduledTasks(tenantId, secretKey);

   4:  var backupTask = new TaskModel()

   5:              {

   6:                  Name = "Import/Export SQL Azure Database",

   7:                  JobType = JobType.Webhook,

   8:                  CronExpression = "0 0 12 1/1 * ? *", // taken from http://cronmaker.com/

   9:                  Start = DateTime.Now,

  10:                  Params = new Dictionary<string, object>()

  11:                  {

  12:                      {"url", "http://myurl/api/BackupDB/TestDB"}

  13:                  }

  14:              };

  15:  scheduledTasks.CreateTask(backupTask);

5. You're done. What you need now is your BackupDB Service to be listening in that URL and executing the requests. For that you can use this project http://sqldacexamples.codeplex.com/ that will help you do that.

There are a lot more scenarios for this but this looked like something a lot of people would be interested, but scenarios like:

Validating is a service or page is available by pinging a url
Performing backups of DB, Storage and so on
Deployments for Windows Azure based on specific conditions
- Example would be a company that has an infrastructure On-Premises and wants to continue to use it but there are certain hours of the day that needs to scale-out into Windows Azure to handle more load. In this case it would do a CRON Task based on a specific time of day to deploy the solution to Windows Azure and change the Load Balancer for example, and doing the same for deleting the deployment when it's not required anymore. This is important because a lot of customer look at Windows Azure in this moment only as a Scale-out platform and need something to help automate the tasks of spinning up new machines.
Perform Archiving processes of data
- Example would be every day a process would be executed that would analyze all the data in Windows Azure SQL Databases and Table Storage and would provide an archiving mechanism passing the data that is older than 1 month into the Table Storage in order to save costs and to maintain only "live/active/most used" data in Windows Azure SQL Databases.

There's a lot of other scenarios but those seemed good to start.

Back to Writing and Experiences Sharing

NunoGodinho — Tue, 05 Feb 2013 23:18:46 GMT

Hello everyone, I’ve completely stopped writing a while back and I really miss sharing my experiences in this b#log, and so I’d like to inform everyone that I’m back on the writing part again and in order to make myself more compliant with writing I’ll make sure I’ll write at least once a week sharing all my crazy experiences and works around Windows Azure.

Lately I’ve been working with a lot of different moving parts of Windows Azure like, IaaS, Media Services, Mobile Service, as well as BigData (HDInsight) and HPC and so a lot to share on those from practices to mistakes I did and will try to help you not to do the same mistakes as I did.

Thanks, and hope my sharing helps you more from now on.

How to choose the right Relational Database Service you should use in Windows Azure

NunoGodinho — Thu, 20 Sep 2012 21:51:19 GMT

Now with Windows Azure SQL Database as well as SQL Server inside a Windows Azure Virtual Machine an important question comes up, and that is, Which Relational Database Service should I use for my solution in Windows Azure?

In order to help answering this question I did a flowchart that should help. (this is a simplification of the process but should answer most of the questions)

What are the features that aren’t supported in SQL Database you might be asking. Here’s a list:

SQL Server Utility
SQL Server PowerShell Provider. PowerShell scripts can be run on an on-premise computer, however, and connect to Windows Azure SQL Database using supported objects (such as System Management Objects or Data-tier Applications Framework).
Master Data Services
Change Data Capture
Data Auditing
Data Compression
Extension of spatial types and methods through Common Language Runtime (CLR)
External Key Management / Extensible Key Management
FILESTREAM Data
Integrated Full-Text Search
Large User-Defined Aggregates (UDAs)
Large User-Defined Types (UDTs)
Performance Data Collection (Data Collector)
Policy-Based Management
Resource Governor
SQL Server Replication
Transparent Data Encryption
Common Language Runtime (CLR) and CLR User-Defined Types
Database Mirroring
Service Broker
Table Partitioning
Typed XML and XML indexing is not supported. The XML data type is supported by Windows Azure SQL Database.
Backup and Restore
Replication
Extended Stored Procedures
SQL Server Agent/Jobs

More about this Windows Azure SQL Database:

More about SQL Server inside a Windows Azure Virtual Machine:

Hope this helps you to make the right choice.

Tips & Tricks to Build Multi-Tenant Databases with SQL Databases

NunoGodinho — Sat, 11 Aug 2012 12:55:17 GMT

Last Wednesday I delivered another session at the Visual Studio Live @ Redmond conference this time about “Tips & Tricks to Build Multi-Tenant Databases with SQL Databases”. The feedback from the session attendees was very good and this is a quick summary of the most important aspects.

First thing in order to be successful is understanding exactly what is Windows Azure SQL Databases (formally SQL Azure) and how it works. If we look at it from a high level, SQL Databases are actually:

SQL Server database technology delivered as a service on Windows Azure

This is actually a Shared Environment where we have SQL Server capabilities and features available in a pay as you go, and scalable mode. This of course doesn’t mean we have all the existing features and capabilities from SQL Server, since some of them would probably create some issues since this is a Shared Environment and also because the goal is High Availability and Scalability.

Ideal for both simple and complex applications

It’s a way for us to have a Relational Database as a Service quickly and powerful which can be used in all types of solutions, but in order to get the best out of it we really need to understand and adjust how it works and avoid things like being throttled, for example.

Enterprise-ready with automatic support for HA

True since it provide a higher level of management since we don’t deal with physical machines anymore but with the actual storing of data this way focusing more on what’s more important for our solution and our business.
Also this provides us an automatic support that enables us to have High Availability without ‘a lot of work’.

Designed to scale out elastically with demand

SQL Databases were created to scale and of course that’s part of it’s DNA, since without the ability to scale elastically it wouldn’t be fit for the Cloud. This will be allowed with SQL Federations.

After a high-level understanding of what is a SQL Database in Windows Azure, it’s important to understand also what are the scaling strategies that we can use, since this way we can better use them whenever needed. And so the strategies are:

Horizontal Partitioning or Sharding

Spreads data across similar nodes

All nodes have the same schema

Allows us to achieve massive scale out both in terms of Data Size and Load
We need to understand that while doing that we aren’t going to be able to get the complete list of data within a single query, and so it’s something we need to understand and consider.

Vertical Partitioning

Spreads data across Dis-Similar nodes

Each node has it’s own schema where the data is stored. Eg. SQL Database, Table Storage, Blob Storage, …

Allows us to place the data where it makes more sense by slitting the data we have for a solution and understanding how it’s used and how it should be stored so we can take the best out of it.
In this case we need to understand that by doing that we a splitting at the row level, so if we want a complete row (if we were to be thinking about a regular database) we won’t be able to get that in 1 only query, since one part can be in a DB, another in Table Storage and another in Blob Storage.

Hybrid Partitioning

This is when we spread our data both in Vertically and Horizontally.

Now let’s have a quick look at SQL Federations:

Integrated database sharding that can scale to hundreds of nodes

Provides the ability to do Horizontal Partitioning or Sharding to data inside SQL Databases in a quick and ‘easy’ way.

Multi-tenancy via flexible repartitioning

Provide the ability to achieve multi-tenancy inside the same Database by providing the ability to split data horizontally.

Online split operations to minimize downtime

Automatic data discovery regardless of changes in how data is partitioned

Finally, before we get to the actual Tips & Tricks we need to understand the multi-tenancy strategies that are typically used, and they are:

Separate Servers

This provides the best isolation possible and it’s regularly done On-Premises, but it’s also the one that doesn’t enable cutting costs, since each tenant has it’s own server, sql, license and so on.

Separate Databases

Very used in order to provide isolation for customer, because we can associate different logins, permissions and so on to each DB. Considered by many the only way to provide isolation for tenants.

Separate Schema

Also a very good way to achieve multi-tenancy but at the same time share some resources since everything it’s inside the same DB but the schemas used are different, one for each Tenant. That allows us to even customize a specific tenant without affecting others.

Row Isolation

Everything is shared in this option, Server, Database and even Schema. The only way they are differentiated is based on a TenantId or some other column that exists on the table level.

So now that we had a high-level view of all this let’s take a look at some of the Tips & Tricks for it, and they are:

Choose the right Multi-Tenancy Strategy

One of the most important steps for delivering a Multi-Tenant solution is understand exactly what should be the approach we should use, and normally the simplest one isn’t actually the best. For example, if we think about Isolation the best might seem Separate Server or Separate Database, but that means that from an economics standpoint we aren’t going to be very competitive, and so for this we need to understand that is we go further for a more shared approach, like Row Isolation, the impact in terms of development might be that at the beginning the investment is bigger, but in the long run that will pay off.
Also important is that if we want Multi-Tenancy and Isolation the only solution is not separating everything, since that is something we can enforce programmatically, through security permissions and so on. It’s just might take a bit more effort but the end result should be that we achieve some other customers that we usual didn’t because of the prices.

Don’t forget Security Patterns

Some time when we start creating Multi-Tenant Databases we start thinking about sharing everything and forget about the Security part of that. This is actually a common mistake that can cost a lot, and in order to make things work we should never forget to:

Filter

Use an intermediate layer that will receive all the requests to the DB and infer the Tenant filter, so that nobody has access to something that shouldn’t. Of course this also means that no one should have direct access to the DB.

Permissions

Very important when we consider Multi-Tenant Databases is the Permissions and how we can affect them. When we use Separate Servers, Databases or either Schemas, we can actually associate different logins, roles and so on to the different Tenants, but when we are in a Row Isolation model that isn’t possible, and that’s why the intermediate layer, that is actually your Data Layer will be very important since not only provides access to the data inferring the filter by Tenant, but also allows us to introduce permissions to access certain parts of the data. For example by leveraging Windows Azure Active Directory Access Control Service.

Encryption

This is of a huge importance. IF YOUR DATA IS SENSITIVE JUST ENCRYPT IT. SQL Databases don’t have the ‘With Encryption’ for Columns but this doesn’t mean I can’t really insert encrypted data in the database, I just need to do it on another layer, again in an intermediate layer.
Also very important when we encrypt our data is to understand the method we’re going to use. Normally one of the best methods, if we don’t want anyone that isn’t part of a Tenant to access the data and have different encryptions per Tenant so that even if someone gets the full DB it won’t access the full data, is to use X.509 Client Certificates. This is a very good approach since the Client is actually the one that has the Certificate that is used, but it also means that we cannot count of doing background calculation with that data in the Cloud since we don’t have the certificate. So it’s a balance.
A quick reminder is that IF YOU SHARE A SECRET IT STOPS BEING A SECRET, it’s just like telling a secret to a child. So for this reason, if you use X.509 Client Certificates to encrypt the data, and then register all those certificates in a Windows Azure Role, that isn’t the best approach because if someone get access to that role it will have access to the KEYS OF THE KINGDOM.

Choose your partitions wisely

When you choose the partitions it should be based on the ‘hotness’ of the data and not on the ‘# of records’. This is a very important premise since we normally see partitions being made in a way that they all have the same amount of records and data, and this doesn’t mean the solution will perform, because if we have one partition that has all the most commonly used data and another one with less common we won’t have any benefit with the partition. So the important part is to partition your data based on how the data is used and how commonly it is used, since in order to get good partitioning sometimes we can have a partition with very few records and another one with thousands.
So it’s also important to before partitioning the data understand how that data will be consumed and used, since that will allow us to better understand what is the most used data or not.

Choose your Partition Keys correctly

We can have several ways of defining partition keys and the most common are:

Natural Keys

Choosing a Natural key is usually one of the most used ways in order to select a partition key, and some samples are:

Tenant
Country
Region
Date
…

But sometimes this isn’t the best approach since if we go for the Tenant that doesn’t partition based on the ‘hotness’ just based on the Tenant and if a tenant is small has less information and so it will be faster than a tenant that is larger because it has more data and all in the same partition. This is exactly what we want to avoid. The same thing happens with Country, since really isn’t the best way since if we use this we might have more customers from a specific customer than from another, and the same with region. When we partition by data, it will mean that everybody would be affected by everyone else data, since everything would be in the same partition.
What this mean is that while Natural Keys are one of the most used partition keys they aren’t actually the best choice because it’s very difficult to find something that allows us to partition it optimally.

Mathematical

This is another option for partition key since what we can do is used things like Hash or Modulo operator and other options to generate a mathematical calculation in order to find the ‘hotness’ point.
Being this a very interesting approach is actually really difficult also since you need to understand your data very well as well as built mathematical formulas which isn’t everyone best hobby and capability.

Lookup Based

Another option is this one, Lookup based, this is actually the best since it really looks at how the data is used and consumed in order to find out the best partition key. In some cases this will mean a concatenation of something like ‘TenantId+Date’ or something like that, because in this case we’ll be saying that every tenant is partitioned independently and even at the same time is partitioning its own data making it faster.

Leverage SQL Federations to handle Multi-Tenancy

SQL Federations is a very good way to leverage Horizontal Partitioning (Sharding) of data for you solutions since allows us a ‘quick and easy’ way to partition our data and provide at the same time Isolation, since each Federation Member is actually a separate DB that is generated, but when we look at it we only see the main DB.
Currently the limitations with this is actually the fact the SQL Federations only allow the partition key to be of types BigInt, Int, UniqueIdentifier and VarBinary. It would be great if it would support also Varchar but we can’t have all, unfortunately, but if we go for a partition key like ‘TenantId+Date’ this can be actually a number and so fall inside the BigInt possibility.

Don’t forget that you’ll continue to need to perform backups

Sometimes when we are in a multi-tenant environment we forget we still need to provide backups and not only for us, but most of the times the customer wants also to have a copy of the data and that is more challenging.
In order to do that backups we can use the Export capability from SQL Databases as well as some third-party tools like RedGate’s SQL Azure Backup.
To leverage a backup of the data for the customer we can leverage SQL Data Sync since it allows us to create filters in the data sets and so we can filter by tenant and get one Data Sync per Tenant this way making everyone happy.

So those are some of the Tips & Tricks you can use in order to be successful building Multi-Tenant Databases in Windows Azure SQL Databases. I hope that helps and would love to have your thoughts about it.

Tips and Tricks on Building Event Driven Architectures with Windows Azure Service Bus

NunoGodinho — Tue, 07 Aug 2012 21:15:45 GMT

Just now I did a presentation at Visual Studio Live @ Redmond under the topic “Tips & Tricks on Building Event Driven Architecture with Service Bus”.

One of the very interesting things to understand is that a lot of us already use Event Driven Architectures even without knowing it. That was one of the parts I explained in the session, since a lot of us currently use messaging mechanisms to decouple our solutions, and that’s already building things in an event driven way.

A summary of the session for the people who didn’t attend is:

What is Event Driven Architecture?
- Architecture Pattern that orchestrates behavior around:
  - Production
  - Detection
  - Consumption of events
- Method for building decoupled enterprise solution
- Suited for Async and unpredictable environment (like the cloud for example)
- Has 3 types of processing:
  - Simple Event Processing
    - Events that are directly related to specific and measurable changes
  - Stream Event Processing
    - Events that can be notable (important) or ordinary (not so important)
  - Complex Event Processing
    - Both Notable and Ordinary events allowing to search for patterns

What is Windows Azure Service Bus?
- Connectivity
  - Service Relay
  - Protocol Tunnel Eventing
- Messaging
  - Queuing Pub/Sub
  - Reliable Transfer
- Svc Management
  - Naming, Discovery
  - Monitoring
- Integration Routing
  - Coordination Transformation

Going now for the Tips & Tricks

Choose your Event Stream wisely
- Messaging (Applicable when you need reliability)
  - Windows Azure Service Bus Queues
  - Windows Azure Service Bus Topics
  - Windows Azure Storage Queues
- Relay (Applicable when you don’t need reliability but only a way to sent a message to all the subscribers)
  - NetEventRelayBinding
- Twitter (Never forget the power of Twitter. For example http://twitter.com/fltadvisor provides you information about flight delays in US. Why not leverage Twitter to do the same for your app?)
- RSS Feed (Also very interesting. For example Windows Azure Service Dashboard uses RSS Feeds.)
- Other sources
Create different types of messages, DON’T GENERALIZE IT
- One of the important parts is never generalize event types.
- You have 4 types of events you need to work on
  - Business
  - Execution
  - Lifecycle
  - Management
- Those are the type of events but you shouldn’t generalize because each event has it’s own properties that are notable for them. For example a DataChangeEvent is different from a NotificationSentEvent. Do the two, don’t generalize into one only with different types.
Choose the right Event processing Type
- Simple Event Processing
  - Use it whenever you need to handle events in a simple way, without having differences between the events. You just need to process all of them.
  - Eg.
    - Coordinating Worker Roles
- Stream Event Processing
  - Use this whenever you need to handle events in a simple way, but you have different types of events with different importance, like Notable and Ordinary events.
  - Eg.
    - Notification System that received events but some are priority notifications and other non-priority
- Complex Event Processing
  - Use this whenever, and only, when you need to search for a pattern inside the event stream. if not, don’t use it.
  - Eg.
    - Stock Exchange. If the Index A goes up 20% and Index B goes up more than 5% in a 5 min period. Do Something
Keep your modules Loosely-Coupled
- Event Driven Architecture is all about decoupling every module of the solution, so continue with that. Do you modules as they don’t need to know who creates the events and why they do it.
‘Shard’ your Event Stream
- It’s important to think from the start how you are going to shard your Event Stream, because if the solution is going to be unpredictable in growth you need to be prepared.
- Some simple way is to think Sharding on the Type (Business, Execution, Lifecycle, Management) but then you can for further.
Consider Windows Azure IaaS with Microsoft StreamInsight for Complex Event Processing
- Very interesting product for Complex Event Processing and that can easily be placed in the Cloud, and of course Windows Azure Virtual Machines
Consider ‘Project Austin’ from Microsoft
- This is really good since it’s Microsoft StreamInsight as a Service in Windows Azure

I presented a sample based on this http://www.codeproject.com/Articles/55620/Event-Driven-Architecture-in-the-Clouds-with-Windo that can’t be found here http://eventdrivensample.cloudapp.net.

Also very interesting to check out is the new CQRS Journey from Microsoft Patterns & Practices here.

Hope this helps you in future developments, and also I’d love to hear you thoughts about this, so please comment and let’s build a conversation about it and generate other posts.

Service Bus 1.0 Beta

NunoGodinho — Fri, 03 Aug 2012 13:43:41 GMT

We all know about Windows Azure Service Bus and how it allows features like Relay, Messaging and even integration. Personally this is one of my favorite features in Windows Azure but all this only worked in the Cloud and sometimes the same features and capabilities would be very good if they were available On-Premises also. That’s exactly what Service Bus 1.0 is, it’s getting those capabilities and features to the On-Premises world also, making it easier to have a complete parity between Cloud and On-Premises.

Of course this version isn’t still the full version of what we see on the Windows Azure version but will help a lot, I’m sure. This Beta is what the Team calls “Service Bus Messaging Engine” release as stated by Clemens Vasters in one of his responses at StackOverflow, and so this version still doesn’t have any features related to the Relay part, but only with the Messaging part.

Another very important part of this release is that by using it we’ll start getting the same (or at least very similar at this moment) API for both Cloud and On-Premises versions, which make things a lot easier when we build solutions that need to be deployed in both.

For more information about Service Bus 1.0 Beta check here.

In terms of Architecture, this version has the following:

(Figure 1 – Service Bus 1.0 Architecture from Windows Azure site)

So we can see that all this uses very well known elements like, SQL Server, PowerShell, .NET Framework, and the 3 most important elements of it are:

Service Bus Gateway

Receives all the incoming requests and provides the necessary Authentication, Authorization as well as address resolution sending then the request for the appropriate block so it can be handled.
Currently only allows communications using NET.TCP or REST over HTTP protocols.

(Figure 2 – Service Bus Gateway High-level Architecture – Windows Azure Site)

Service Bus Message Broker

Responsible for providing the Messaging capabilities, so can be seen as the Service Bus Messaging Engine.
Provides Queues and Topics/Subscriptions features to this release.

(Figure 3 – Service Bus Message Broker High-level Architecture – Windows Azure Site)

Windows Fabric

Provides capabilities like High availability, farm and cluster formation as well as load balancing across farm machines.
We can think about it as being the ‘orchestrator’ of all services since all services will connect to it in order to have those capabilities and be able to work in a success way and with high quality.

So by releasing Service Bus 1.0 (Beta) Microsoft is actually showing that they were really serious when they were saying that they were working on lowering the differences between Cloud and On-Premises and bringing the best solutions to both worlds.

What I believe this will bring to customer is:

Highly-Scalable (and proven) Messaging System
Parity between Cloud and On-Premises making it possible to code once and work on both deployments.

Hope this post helps you better understand what’s Service Bus 1.0 (Beta) is all about and you can expect future post around this topic going deeper in terms of how to use it and benefits of it.

TechEd North America Keynote – Day 1 Rough Notes

NunoGodinho — Mon, 11 Jun 2012 14:37:04 GMT

Satya Nadella, President of Server and Tools Business @ Microsoft

It’s time for Microsoft to reinvent itself. Why?

Modern Data Center: New Hardware abstraction making the OS worry about Resource Management
Modern Apps

Converged Systems

Compute (multi-core, memory density, system density)
Storage (SSD, costs, ..)
Network

Services at internet scale

runs the world biggest apps
feedback loop
global scale, 24/6

Last week Bing was migrated into Windows Server 2012 RC

Modern Data Center

Inside the modern DC

Scalable & Elastic

64 nodes & 4000 VMs

Always up and on

Shared resources

Automated & self service

Windows Server 2012

Scalable & Elastic

64 virtual processes per VM, with up to 4000 VMs with HyperV 3.0

Always up and on

Ability to update the cluster without needing to take it down

Shared resources

Ability to have storage virtualization, network virtualization and isolation of workloads using policies

64TB per virtual disk

Automated & self service

Host

Windows Server 2012

256 GB RAM, 80 logical processors, 5 HBAs,

About 985,000 IOPS from a single machine and it can be faster.

With ODX enabled you can get now around 1GB/sec without any Network utilization

Networking

Windows Azure

Scalable

Windows Azure VMs

Windows Azure Web Sites

Windows Azure Cloud Services

Always on

99.95% SLA

Highly available

Shared Resources

Network Virtualization capability

Automated & Self-Services

Everything available through PowerShell and APIs

Modern Apps

Visual Studio 2012

Visual Studio LightSwitch inside Visual Studio by default

Announcement: LightSwitch has support for OData and also HTML5

New Lifecycle

Define

Develop

Sprint

Operate

Integration between Visual Studio Web Tests and System Center in order to use the same Web Test file to configure the monitoring and at the end debug problems back in Visual Studio with the Historical Debugger (IntelliTrace)

Windows Intune

Integrated with Windows Azure and AD

Self Service

Data-powered Apps

Any data, any size, anywhere

rich capabilities to reason over data. Add intelligence to your data like adding machine learning

Connect this data with other solutions. Capability to connect Excel to BigData

IIS Express support for local development in Windows Azure SDK 1.7

NunoGodinho — Fri, 08 Jun 2012 01:00:17 GMT

Another new feature that was brought by the new Windows Azure SDK 1.7 was the way the Compute Emulator emulates the Web Roles, since it now uses IIS Express instead of Full IIS.

You might be thinking that isn't very important, but it really is since it allows the developer to be more productive even with a less powerful machine, since IIS Express is able to run most of the roles without demanding so much from the machine as IIS does. But on the other hand, if we really have something in our Web Role that requires the use of IIS we are able to change the local development server into Full IIS. By having this we can go from a lightweight development process that doesn't require a huge machine, but changing to a much robust solution only if we require some of the features that only exist in Full IIS.

By default the local development server for the cloud project is set to "Use IIS Express".

(Figure 1 - Windows Azure Cloud Project Properties window)

In order to change to use Full IIS instead of IIS Express for emulating the Web Role locally we just need to change it to "Use IIS Web Server".

(Figure 2 - Windows Azure Cloud Project Properties window)

So, with this new feature we can get a lot more productive since the development and emulation of Windows Azure requires less machine resources, making things go faster.

Side-by-Side support in Windows Azure SDK 1.7

NunoGodinho — Fri, 08 Jun 2012 00:58:10 GMT

As you already seen Windows Azure SDK 1.7 brings a very interesting feature with the development tools that is the ability to run both version 1.6 and 1.7 side-by-side.

You might be thinking about why this is so interesting, so let me help you with that.

First a bit of history, since almost with every release of Windows Azure SDK the recurrent discussion would be about the changes that the new SDK was bringing and how that would affect the existing developed solutions. A lot of time we would hear that some developers couldn't start to use the new version of the SDK immediately because they still had something done in the previously version and would first need to take a look in what was changed and only after that they would be able to start working.
So with this new feature this problem is finally addressed since we can maintain both SDK versions running side by side and so allowing us to get the best out of both worlds, using the new version for new development and the previous version for maintaining the existing ones.

Now that you already know the Why associated with this features, let's take a look at How we can take advantage of that and also what it allows. In order to do this let's start from the beginning and look at the Create new Project windows in Visual Studio.

(Figure 1 - New Windows Azure Cloud Service dialog in Visual Studio 2010)

This dialog allows us to choose the version of the Windows Azure Tools we would like to use in order to start the project, and by doing that adjusting the Role list based on the version we choose.

(Figure 2 - Role Templates for Windows Azure SDK 1.6)

(Figure 3 - Role Templates for Windows Azure SDK 1.7)

So this is a very interesting feature that will enable everyone to be even more productive but first of all, be able to start taking advantage of the new SDK without needing to go through uninstalling the previous version and installing the new version or even needing to ignore the newest version for a period of time because they are supporting some project that is still in the previous version.

What's new in Windows Azure June Release - SDK 1.7

NunoGodinho — Thu, 07 Jun 2012 17:23:00 GMT

Windows Azure just had a very interesting upgrade as it was discussed at the MeetWindowsAzure event. This event basically showed how Windows Azure is moving and becoming even more powerful and at the same time incorporating the community feedback.

In this new release of the SDK the interesting parts are:

1. Side-by-Side installation with SDK 1.7

(Figure 1 - Create New Windows Azure Cloud Service window with SxS 1.6 and 1.7 SDKs)

2. IIS Express support for local development

Now instead of the Windows Azure Compute Emulator using the Full IIS for emulating the Web Role, it uses only the IIS Express version in order to make the process less intensive for the machine.

(Figure 2 - Windows Azure Web Role running in the Compute Emulator under IIS Express)

3. New approach for Caching in Windows Azure called "Windows Azure Cache".

This new approach to Windows Azure Caching is made available in two different scenarios:
- Dedicated Caching Nodes
  - Made available by a new Worker Role called "Cache Worker Role". Used when we want 1 or more dedicated node(s) for Caching purposes only

(Figure 3 - New Windows Azure Cloud Service dialog with the new Cache Worker Role)

Shared Caching Nodes
- Made available through the Role Properties, since now we have a Caching Tab in which we can define how we want to configure our Windows Azure Cache, and so reuse our existing Roles in order to also provide Caching capabilities.

(Figure 4 - Properties Windows for a Windows Azure Role with the Caching Tab Selected)

4. Improved Visual Studio Server Explorer support for Windows Azure

(Figure 5 - New Visual Studio Server Explorer with Services Bus and Windows Azure VMs support)

Service Bus Queues and Topics
- With this new support from inside Visual Studio it makes our life easier to manage and even understand what's happening inside both Queues and Topics, as well as allowing us to Create new Queues, Topics and Subscriptions and also Sending and receiving messages directly from Visual Studio. This is a great improvement in terms of making our life easier since we have an one stop shop for doing everything related to both of these Service Bus features.

(Figure 6 - Managing Service Bus Queues and Topics from the Visual Studio Server Explorer)

Virtual Machines
- New support in Visual Studio Server Explorer to manage the new IaaS offering available in Windows Azure.

RDP
- In this version we'll be able to Connect directly to the Instance through Remote Desktop without ever leaving Visual Studio.

(Figure 7 - Connect using Remote Desktop option from Server Explorer)

(Figure 8 - Remote Desktop Connection for the Windows Azure instance based on the choice in Server Explorer)

5. Improvements in the Publishing Process

New "Delete Deployment on failure" option

(Figure 9 - Windows Azure Advanced Publishing Settings dialog)

Improved "Deployment Update" Options allowing two options:
- Incremental Update.
  - Make the update process a rolling update by performing the update on 1 (one) instance at a time.
- Simultaneous Update
  - Makes the update process a full and complete update, so every instance is update at the same time (or virtually at the same time). This might mean some downtime for you cloud service.
- Note:
  - Also Very important is the new option "If deployment can't be updated, do a full deployment" which makes the update process smart enough to understand if the full deployment is the best option for it and so always being able to have the update being made.

(Figure 10 - Deployment Update Settings window)

New CloudConfigurationManager
- This class provides an automatic cascade for setting lookups from csdef/cscfg first to web/app.config next.

6. Reference improvements (NuGet(s), client libs)

More use of NuGet in order to make the updating of project references easier.

New NuGet Packages for consuming the new Windows Azure Caching
- Windows Azure Caching Preview
  - All client helper classes and configurations to consume the new Windows Azure Caching Role.
  - Package Name: Microsoft.WindowsAzure.Caching.1.7.0.0.nupkg

Windows Azure Caching Memcache Shim Preview
- All client helper classes and configurations to consume the new Windows Azure Caching Role using the memcache protocol.
- Package Name: Microsoft.WindowsAzure.Caching.MemcacheShim.1.7.0.0.nupkg

Windows Azure Shared Caching
- All client classes and configurations to consume the current version of Windows Azure Caching.
- Package Name: WindowsAzure.Caching.1.7.0.0.nupkg

Windows Azure Configuration Manager NuGet Package
- Classes that enable the new Configuration Manager class to work in providing an automatic cascade for setting lookups from Service Configuration to Web.Config/App.Config.
- Package Name: Microsoft.WindowsAzure.ConfigurationManager.1.7.0.0.nupkg

Windows Azure Storage
- Set of helper classes and configurations to make working with Windows Azure Storage simpler.
- Package Name:WindowsAzure.Storage.1.7.0.0.nupkg

Windows Azure Service Bus
- Set of helper classes and configurations that make working with Windows Azure Service Bus simpler.
- Package Name: WindowsAzure.ServiceBus.1.7.0.0.nupkg

(Figure 11 - Contents of the packages folder that is installed with the new Windows Azure SDK 1.7)

7. New Worker Role with Service Bus Queue

New Worker Role Template that has everything needed to consume Messages from Windows Azure Service Bus Queues, as well as also having code blocks on the OnStart, OnStop and Run methods of the RoleEntryPoint class for the Worker role that will handle the configuration, graceful shutdown and of course the receiving and processing of the message

(Figure 12 - Add New Role to Project dialog)

8. UDP support

9. LightSwitch publishing improvements

So those are only 9 of the very interesting things that I come across with the new Windows Azure SDK 1.7 release.

Expect more details about the several elements described here in other blog posts.

Social Meet Up for ‘MEET Windows Azure’ on June 7th

NunoGodinho — Wed, 16 May 2012 22:35:38 GMT

So as you might already know, June 7th will be the ‘Meet Windows Azure’ event online where Scott Guthrie and other technologists from Microsoft will present the Windows Azure platform. This will surely be a great event and in order to make it even better Windows Azure MVP Magnus Mårtensson is thought about an initiative called MEET Windows Azure on Twitter (+ Beer) that will allow us to create a social discussion about all that we hear and create a way to answer questions and at the same time be ‘together’ discussing it. The beer part would be great.

If you would like to participate in this unique event, here’s what you should do:

I think this will be a very nice opportunity to get to know Windows Azure and the great communities supporting this technology.

MEET Windows Azure Blog Relay:

Roger Jennings (@rogerjenn): Social meet up on Twitter for Meet Windows Azure on June 7th
Anton Staykov (@astaykov): MEET Windows Azure on June the 7th
Patriek van Dorp (@pvandorp): Social Meet Up for ‘MEET Windows Azure’ on June 7th
Marcel Meijer (@MarcelMeijer): MEET Windows Azure on June the 7th
Nuno Godinho (@NunoGodinho): Social Meet Up for ‘MEET Windows Azure’ on June 7th
Shaun Xu (@shaunxu) Let's MEET Windows Azure
Maarten Balliauw (@maartenballiauw): Social meet up on Twitter for MEET Windows Azure on June 7th
Brent Stineman (@brentcodemonkey): Meet Windows Azure (aka Learn Windows Azure v2)
Herve Roggero (@hroggero): Social Meet up on Twitter for Meet Windows Azure on June 7th
Paras Doshi (@paras_doshi): Get started on Windows Azure: Attend “Meet Windows Azure” event Online
Simran Jindal (@SimranJindal): Meet Windows Azure – an online and in person event, social meetup #MeetAzure (+ Beer for Beer lovers) on June 7th 2012
Michael Wood (@mikewo): Learn about Windows Azure and Chat with Experts, June 7th
Shiju Varghese (@shijucv): Social meet up on Twitter for MEET Windows Azure on June the 7th
Jeremie Devillard (@jeremiedev): Meet the Cloud–Windows Azure Event 7th June
Kris van der Mast (@KvdM): Get ready to meet Windows Azure
Mike Martin (@TechMike2KX): Don’t miss the online Windows Azure event of the year : MEET Windows Azure on June 7th
Bill Wilder (@codingoutloud): Get ready to “Meet #WindowsAzure” in a live streamed event June 7 at 4:00 PM Boston time
Eric Boyd (@EricDBoyd): Meet Windows Azure – Unveiling the Latest Platform
Magnus Mårtensson (@noopman): Social meet up on Twitter for MEET Windows Azure on June 7th

We’ll definitely meet online and try to answer all your questions and learn also from you.

Visual Studio Live @ Las Vegas Presentations – Tips and Tricks on Architecting Windows Azure for Costs

NunoGodinho — Thu, 29 Mar 2012 17:03:09 GMT

Unfortunately I wasn’t able to go and speak in Visual Studio Live @ Las Vegas as it was scheduled, due to an illness that made it impossible for me to travel, and stay in bed for a few days.

But even if I wasn’t there I would like to share with you some of the points on this topic “Tips and Tricks on Architecting Windows Azure for Costs”.

Tips & Tricks On Architecting Windows Azure For Costs

View more presentations from Nuno Godinho

The Key points to achieve this are:

Cloud pricing isn’t more complex than on-premises, it’s just different
Every component has it’s own characteristics, adjust them to your needs
Always remember that Requirements impact costs, choose the ones that are really important
Always remember that Developers and the way things are developed impact costs, so plan, learn and then code.
Windows Azure pricing model can improve code quality, because you pay what you use and very early can discover where things are going out of plan
But don’t over-analyze! Don’t just block because things have impacts, because even today the same things are impacting you, the difference is that normally you don’t see them that quickly and transparently, so “GO FOR IT”, you’ll find it’s really worth it.

In some next posts I’ll go in-depth into each one of those.

Special thanks for Maarten Balliauw for providing a presentation he did previously that I could work on.

Visual Studio Live @ Las Vegas Presentations – Architecture Best Practices in Windows Azure

NunoGodinho — Thu, 29 Mar 2012 16:50:00 GMT

Unfortunately I wasn’t able to go and speak in Visual Studio Live @ Las Vegas as it was scheduled, due to an illness that made it impossible for me to travel, and stay in bed for a few days.

But even if I wasn’t there I would like to share with you some of the points on this topic “Architecture Best Practices in Windows Azure”.

Architecture Best Practices on Windows Azure

View more presentations from Nuno Godinho.

Here are 10 key Architecture Best Practices in Windows Azure:

Architect for Scale
Plan for Disaster Recovery
Secure your Communications
Pick the right Compute size
Partition your Data
Instrument your Solution
Federate your Identity
Use Asynchronous and Reduce Coupling
Reduce Latency
Make Internal Communication Secure

In some next entries I’ll go in-depth into each one of those.

Lessons Learned from last week's Windows Azure Outage - Data Recovery

NunoGodinho — Tue, 06 Mar 2012 12:46:57 GMT

With last week’s Windows Azure Outage I’ve learned some lessons and I already talked about some in the previous posts, in this one I’ll focus on Data Recovery, since one of the important part of the outage is that we get scared of losing our data. Fortunately this didn’t happened in this one, and why was that? Have you thought about it?

So why wasn’t any data loss in this outage? Let’s dig into this one.

Normally our data is either placed inside Windows Azure Storage or even SQL Azure, and so Windows Azure has in-place for both of them one automatic process that for each content we place in these two options we get 3 replicas and they are placed in different parts of the Data Center, or in the Windows Azure Storage case 1 of the replicas is placed in a different Data Center in the same Region. This was very important to avoid data loss, since what happened was that in this “Leap Year bug outage” we didn’t have the complete Data Centers shutting down and so there were parts that still continued working maintaining our data. Of course this replication strategy doesn’t work for all problems since if all the Data Center crashes at the same time there might be data loss, but that isn’t the most normal outage, and so this way they are solving the biggest problems.

Also the fact that Microsoft has at least 2 Data Centers in the same region reduces a lot the possibilities of having some data loss.

But what if all the Data Center had went completely destroyed for some reason. Would I continue to have my Data?

And the answer is “it depends”. In the case of Windows Azure Storage the answer would be no, because we would have a replica in the other Data Center in the same region so we would be able to get back into action, it would take just a bit more time. If we were talking about the other services the answer would be different because the replicas in SQL Azure is placed inside the same Data Center and so if everything goes down, and also the machines goes down we could lose everything, but what’s the odds of that? Not two high.

If you don’t like your odds with this the best thing you could do was implementing a Data Recovery strategy, like replicating all your data to another Data Center inside your app, like for example with SQL Azure, we could use SQL Azure Data Sync to sync the database to another one in a different Data Center and even Region, or even use the SQL Azure Import/Export capability to have some “backups” (this isn’t a really backup since we don’t have the actual transaction log, but will be enough since it has all the Schema and Data in a particular time providing us a way to “restore” our data to a previous state) being placed in a Windows Azure Blob Storage Container, or even copied to one of our On-Premises machines or any other machine.

Another option would be to had the service available in several different geographies and fallback to the other ones in case of a outage like this, but of course this has costs, and maybe in some cases it would be enough to just point it to a static site inside Windows Azure Storage, or in other cases point into another deployment you have elsewhere in order not to stop working. It always depends on the business requirements we are talking in this case.

So with this I think one thing we should never forget is Data Recovery since thinking about this in the architecture phase will help a lot your business when something like this happens, and not even only in situations like this since there are also some that happen even without outages, like because of some bad update that was pushed, or any other issue.

I hope this helps you also understand how to plan and have measures in place to avoid data loss. Also I’ll continue to blog about some other lessons learned here.

Lessons Learned from last week Windows Azure outage – Redundancy

NunoGodinho — Sun, 04 Mar 2012 20:09:29 GMT

As you might already know, Windows Azure had an Outage last week, and this generated a lot of “fuzz” around it, as well as when the same happened to Amazon last year, or even other providers. Based on these outages, lots of people are now saying that Cloud Computing shouldn’t be an option because it sometimes fails, because it has outages and so on. This isn’t really a very correct approach to the issue, since when we have everything inside our own Data Center, sometimes bad things happen also, from someone doing an update in the network and that crashes it, machines that just “die” from one moment to the other, and a lot more.

What these outages remind us is that even when going to the Cloud, and Windows Azure for example, we need to continue to analyze the impacts that an outage in our solution might have in our business, because Cloud Computing provides us a better platform and a way for us to be more secure, since they already have some Disaster Recovery and Data Replication mechanisms, but they also provide SLA’s and if we need more than those we need to really work on it and architect for it right from the start. And this isn’t a Cloud Computing fault, it’s really a requirement that our business has, and will have inside or outside or our own Data Center.

When we are dealing with something that is inside out own Data Center what we do is Redundancy. Let’s talk on a real-world example not IT related. Airplanes don’t need so many engines to fly, but they have them because if 50% fails, the other 50% will still get the airplane to reach to the desired destination without problems, and of course the level of redundancy depends on the reliability of the engines, and also the impact of that failure. Since airplanes don’t work very well without engines working, this is critical so sometimes you see 50% Redundancy and some other 75% Redundancy, like it happened in the earlier days. So we need to do the same with our solutions when building the on Windows Azure, and that is understand the impacts that an outage has for the business and then plan Redundancy and Disaster Recovery based on those, but we have some things that we can count on already, that is how Windows Azure takes care of Storage, SQL Azure, Compute and so on, since it provides us SLA’s that will provide us a level of security already very good. Also in case of Storage, provide us 3 replicas of everything that is placed inside the Storage account, being it Tables, Queues or even Blobs, and also Geo-Replicates 1 copy into another Data Center in the same region. What this does is that when for example a Data Center goes down, like it happened last week with Windows Azure, normally isn’t all the Data Center, and so some part will continue to be available, and as soon as the platform identifies that some part of the Data Center is down, the same platform will take the primary replica and place it as the original one, and them everything works again, but if for some reason all the Data Center goes down, it will continue to have a replica in the other Data Center of the same Region.

If we talk about SQL Azure, the same thing happen as the Storage, just the Geo-Replication isn’t there, so if all the Data Center goes down, there’s no Geo-Replica fallback process and so we need to plan for it.

So based on all this we should really look at Redundancy and Disaster Recovery as a very important part of our Architecture and System design, but we also need to take into account that this means costs, and so we need to get the right approach for the customer, because there’s no solution “One fits all” for this.

In some next posts I’ll talk about some approaches to designing Windows Azure for Redundancy and Disaster Recovery.

Also you can leave a comment and say what you’d like to hear about and I’ll do my best to write about it.

Importance of Affinity Groups in Windows Azure

NunoGodinho — Sun, 04 Mar 2012 06:28:37 GMT

During this last week some friends asked me more about what are the Affinity Groups in Windows Azure, and their benefits since for some people this is nothing more than a way to logically group both Compute and Storage.

In order to explain this we need to dig a little deep in terms of how Windows Azure Data Centers are created. Basically Windows Azure Data Centers are built using “Containers” that inside are full of clusters and racks. Each of those Containers have specific services, like for example, Compute and Storage, SQL Azure, Service Bus, Access Control Service, and so on. Those containers are spread across the data center and each time we subscribe/deploy a service the Fabric Controller (which chooses based on our solution configuration where the services should be deployed) can place our services spread across the data center.

Now one thing that can happen is we need to be very careful in where we create the several services, because if we place the Hosted Service in North Central US and then the Storage Account in South Central US, this won’t be very good both in terms of Latency or Costs, since we’ll get charged whenever we get out of the Data Center. But even if we choose the same Data Center, nothing tells us that the services will be close together, since one can be placed in one end of the Data Center and the other in the other end, and so this will remote the costs and make the latency better, but it would be great to go a little further like placing them in the same Container, or even in the same Cluster. The answer for this is Affinity Groups.

Basically Affinity Groups is a way to tell the Fabric Controller that those two elements, Compute and Storage, should always be together and close to one another, and what this does is when the Fabric Controller is searching for the best suited Container to deploy those services will be looking for one where it can deploy both in the same Cluster, making them as close as possible, and reducing the latency, and increasing the performance.

So in summary, Affinity Groups provide us:

Aggregation, since it aggregates our Compute and Storage services and provide the Fabric Controller the information needed for them to be kept in the same Data Center, and even more, in the same Cluster.
Reducing the Latency, because by providing information to the Fabric Controller that they should be kept together, allow us to get a lot better latency when accessing the Storage from the Compute Nodes, which makes difference in a highly available environment.
Lowering costs, as by using them we don’t have the possibility of getting one service in one Data Center and the other in another Data Center if for some reason we choose the wrong way, or even because we choose for both one of the “Anywhere” options in the.

Based on this, don’t forget to use Affinity Groups right from the start, since it’s not possible after having deployed both the Compute or Storage to change them into an Affinity Group.

To finalize, and since now you can be thinking that this would be very interesting for other services also, no other services are able to take advantage of this Affinity, since neither of them share the same Container.

Hope this helped and see you in the CLOUD.

ALM Days 2011 Portugal

NunoGodinho — Wed, 07 Dec 2011 11:59:49 GMT

Last week I presented two session both in Lisbon and Porto one about Visual Studio Lightswitch and other about Hosted TFS.

Here are the presentations:

Visual Studio LightSwitch 2011 - ALM Days

Hosted TFS