The Backup Blog

VMware Backup with Avamar and vSphere Part 2

Scott Waterhouse — Thu, 17 Dec 2009 11:53:12 -0700

On the previous post I had a chance to review the major backup and recovery options for VMware. I put the discussion in terms of Avamar, but a lot of that is just for convenience. If you want, you can substitute your favorite backup application in place of Avamar, and you will get more or less the same result (minus the deduplication), albeit perhaps with greater operational pain and expense.

In any event, now that I have covered off the mechanics of the three major options, it is time to take a look at the why and the how of it all. Why would I want to choose one option over the other? How do we differentiate between them? Simply put: what is the best way to back up VMware?

I am going to start by breaking one of the cardinal rules of rhetoric. I am going to state my conclusion before my premises. And my conclusion is this:

Everybody that runs transactional applications in a VM should do guest level backup. You may also do image level backup as an additional method of data protection, but guest level backup is necessary. If you are protecting anything other than a transactional application with a Windows guest OS, then an image level backup (generated by vSphere 4.0 and Avamar 5.0) is sufficient. If you are protecting a Linux guest OS machine that is not running a transactional application, then you will likely find that guest level backups are sufficient.

Now why would I insist that guest level backup is the preferred methodology if you are running a transactional application within your guest OS?

Simply for this reason: no other backup method guarantees (in a strong sense) application consistent recovery. Yes I need a backup agent that ties into the application APIs (RMAN for Oracle, VSS for Microsoft applications, and so on). And yes this entails complexity and additional components in my backup process--although this complexity is no more than what we all have to deal with in an environment that is not virtualized, because it is exactly the same process as we would use in an environment that has not been virtualized. And it may entail additional expense (for licensing, if you are not using Avamar--which has free application agents). But it does have the unique virtue of guaranteeing application consistency.

And if I choose either of my other two methods of backup--meaning the two variations on image level backup that I described in my previous post--you do not get such a guarantee. Not really. I will admit that it is very likely you will get a consistent backup. Almost guaranteed. 99.99% probable, perhaps. Nearly certain. But the technical person in me is not satisfied with that. Nearly? Almost? Probable?

Not good enough.

vSphere 4.0 and Avamar 5.0 will backup changed blocks from the .vmdk associated with a guest. That will result in an image level protection. That image can be restored to the original ESX host or elsewhere. It is portable. It takes relatively few components to generate. The proxy system that mounts the LUNs containing the changed blocks can now be run in a VM too, further reducing the infrastructure commitment. It offloads the CPU effort of backup from the guest VM, which is a very good thing. All these are true. And all these represent distinct advantages over guest level backup.

But it still does not provide the iron clad assurance that I am looking for that my application will be restartable.

Now I have heard from others that they have tried this and never had a problem. If that is good enough for you, then that is fine with me! But I can't help but get nervous, and I would certainly never recommend that somebody adopt the approach of taking (only) image level backups without understanding this issue, and explicitly accepting that image level backups imply that application consistency is not guaranteed.

As a result, I think that there may be some times when we want to do both. We want to do guest level backups to capture an application consistent backup (and perhaps capture application change logs on a more frequent basis). This will be the basis for our operational recovery.

We may also want to capture an image level .vmdk backup for longer term retention, or archival purposes, or even compliance reasons. Perhaps once a month we will capture these images and archive them (likely to tape--as it is not the purpose of deduplicated backup targets--either Avamar or target deduplication appliances--to hold long term archival data as a rule; nor is it cost effective in most cases). Depending on how we use VMware, and whether it is our primary mechanism for site recovery, disaster recovery, and high availability, these images may also take on other purposes. But their primary intent should not be for operational recovery of a transactional application.

For those VMs who guest systems that do not host transactional applications, Avamar 5.0 and vSphere 4.0 provide the best, most efficient, least complex method of providing backup. I do not need agents in each VM. I can mount the backup data (which is much smaller than under a VCB due to containing block level differences only) to a VM proxy. So I no longer need a physical host to act as the proxy backup server. That proxy is significantly more efficient than the VCB proxy server. In the case of Avamar, the proxy can also deduplicate the data prior to sending it to a backup target. And finally, I can do both file level restore for Windows systems, and image level restore, from this one backup process.

This is a very good thing. This is the best of both worlds. The best of guest level backup (file level restore) with the best of image level backup (simplicity, reduced infrastructure, speed, and minimal to no impact on the CPU of the guest VM for backup purposes).

And as soon as we can guarantee application consistency with this approach, I think that guest level backups will no longer be either required or desired. Finally! (And no, that is not a statement or admission that this capability is coming from either EMC or VMware. I have no idea if it is or not. I do know however that this would be such an enormous step forward for data protection that it has to happen eventually. The advantages are just too big, too overwhelming, to ignore.)

Lastly, for those that host Linux as a guest OS within a VM, your decision tree on which backup methodology to choose is a little more complex. Does the VM contain a transactional application? Then you must do, minimally, guest level backups. Do you need file level restore? Again, you need guest level backups to do file level restore. But if you are not running a transactional application, don't care about guaranteed application consistency, or don't care about file level restore, then image level backup is the right approach.

By the way, there are lots of other great points of integration between vSphere 4.0 and Avamar 5.0 There is automated policy protection for VMs (automatically protect new VMs as they are added--which just might mean that I won't find environments that have "forgot" to protect 20% of the their systems any more!). There is a way of browsing the type of protection a VM has. The image level backups are automatically generated (no more or the scripting that accompanied VCBs previously). All kinds of good stuff.

But set all that aside. Before any of that matters you have to decide how to back up a VM. And I hope that this post and the previous one help you to better understand those choices. Choosing Avamar specifically should be secondary to the architectural decision on how to back up.

But what other backup application will reduce the time to backup by 90%, the net CPU load by 95%, and the amount of data transmitted over the network by 99.5% or better? All of which are uniquely beneficial for VMware...

VMware Backup with Avamar 5.0 and vSphere 4.0

Scott Waterhouse — Wed, 16 Dec 2009 12:04:34 -0700

One of the most important, and most frequent questions I get asked is this: what is the best way to back up VMware?

Well, "best" tends to be a bit of a slippery term. Usually the answer to the question should be "it depends." But I thought I would take the time to explore what it really means when we say "it depends." What does it depend on? What are the pros and cons of the different approaches? What are the most important considerations? And what new choices do we have with vSphere 4.0?

Now I am going to explore the answer in the context of Avamar. Mostly I am going to do so because I work for EMC and it is convenient to be able to put a label on these products and processes. Having said that, almost everything I say about the costs and benefits, the value of the various approaches with respect to each other, and the drawbacks of the various approaches is true irrespective of the backup product you are using. Guest backup has certain costs and certain benefits--no matter if you use Avamar or something else. Where benefits are specific to Avamar I will identify them. And certain things that that I identify as unique may only be unique as of the time of this writing; clearly some of the functionality is so desirable that other backup application vendors will be trying very hard to get this capability into their shipping product.

All that said, I think there are two primary methods of protecting a VM: guest level backup and image level backup. There are a couple of ways in which the number of options multiply however--do you use source deduplication or target deduplication? Do you want to use the array to generate a copy for image level backup, or do you want to rely on ESX/vSphere?

But lets take a look at the two primary methods. First: guest level backup (depicted below).

In this case, we really treat each VM as a physical machine. Each VM has a instance of an Avamar agent (backup client) installed. Each agent behaves as it would on a stand-alone physical machine and inspects the file system and data associated with that machine for changes every day. If there is a specific application that requires special backup integration (Exchange or Oracle for example) with a special backup agent, the Avamar agent goes about that activity and talks to VSS or RMAN just as it would on a physical system.

The fact that guest level backup treats a VM as a physical box is both it's great strength, and it's great weakness. Great strength, in that you don't really have to do anything different than you did before: the process and procedure of backup is unchanged. Now Avamar has a huge advantage here because the duration of backup, and the CPU and network load of backup on your ESX server will be vastly smaller with Avamar than with any other backup application. Avamar is as much as 90% faster than a "standard" backup from TSM, or NBU, or CV. And Avamar will reduce network load by as much as 99.5% or more. But those issues aside, guest level backup is much the same as the backup of a physical system.

Great weakness, because I can't leverage any of the unique functionality of VMware/ESX to improve data protection.

The second method--image level backup--really comes in two types. The first is the type you would do if you don't have Avamar 5.0 and vSphere 4.0: it is a VCB backup, depicted below.

In the case of a VCB backup, the ESX server will make a copy of the data associated with each VM in a sequential manner. In the case pictured above, where we have a mere 3 VMs per physical host, this means that the ESX server will copy the files associated with VM1 to a separate LUN (which is mostly the .vmdk file), then the files for VM2, thenn the file for VM3. Those copies are then mounted to a proxy server, on which an Avamar agent runs, and backups are performed from this proxy server.

Practically speaking, not a lot of folks are doing this. And the biggest reason for this is that it takes a long time to generate those VCB copies--about 4 hours per TB of source data (by way of a huge generalization). Secondarily, or the flip side of this coin, is that I am going to need a more than one proxy server--probably at least one for every 5 to 10 physical ESX server. So in a very large environment, this can mean a lot of additional infrastructure, and associated costs.

So, the VMware folks are pretty smart, and realized that traditional VCB backups were not all that attractive to the majority of users. As a result, vSphere 4.0 has implemented a number of improvements through the vStorage APIs to enable better data protection. Chad Sakac refers to this as "son of VCB" but truly it is just another way of generating an image level backup. It is depicted below.

In the case of vSphere 4.0 (and currently unique to Avamar 5.0 of all major backup products--and just keep your hats on my Veeam friends) this means that when I want to do a backup, the ESX host will have an additional VM, which is a proxy server for backups. The Avamar agent will reside only within this VM, and not within the individual VMs hosting guest OS images and applications.

During the course of normal operations, the ESX server tracks what blocks are physically changing within the VMFS file system. When a backup happens the ESX server makes the changed blocks only available to the proxy server via a SCSI hot add command. The storage associated with the changed blocks is mounted by the Avamar agent in the proxy server (one each of Windows and Linux). The agent then inspects the data for commonality (this is source deduplication) and send the unique backup objects to the Avamar server.

This type of image level backup captures the complete .vmdk for backup, but (very different than a VCB) also allows file level restore. So far the file level restore is for Windows only, and is unique to Avamar (single pass backup).

So, guest level backups, and two major types of image level backups. Those are the choices. But why would I do one rather than the other? Which is "better"? All of that in the next post which you can find here.

Data Protection Advisor 5.5

Scott Waterhouse — Mon, 14 Dec 2009 12:56:57 -0700

EMC is releasing the latest version of Data Protection Advisor (DPA) today, version 5.5

DPA has been with EMC for a while now, and it is one of those quiet little overachievers that probably doesn't get the recognition it deserves. One big reason for that is the chronic perception of backup as the most unwanted, and unloved, discipline in the IT organization. And it can be hard enough for backup folks to get funding for infrastructure and software, never mind a piece of software that is (in some senses) secondary to the whole process: software that monitors and reports on backup.

For better or worse however, this stuff is important. Simply put, the native reporting capabilities of most backup applications is not good.

As a result, most backup admins--and their organizations--benefit a lot from software that helps them monitor, report, trouble shoot, do trend analysis and so on. What is maybe not clear is just how much this stuff can help: we have found that it is often the case that DPA is so helpful that it pays for itself in a year or less. DPA can save so much administrative time and effort that it cost justifiable in a very short time frame.

But what has been added in v5.5?

First and foremost, replication support. DPA can now monitor and report replication operations on Symmetrix (V-Max), Clariion, and RecoverPoint appliances. This is an interesting step, as it takes DPA out of its traditional role as purely a backup tool, and towards being true to its name: being a comprehensive data protection monitoring and reporting tool. Replication status is monitored for completeness, and you can report on the level of adherence to SLA agreements that the replication is meeting (or, failing to...). Even better, this can be viewed from an application perspective: it is not just what LUNs are replicating, successfully or not, and in how long, and what the associated link status is, it is what applications are successfully replicating in a consistent state. Which is a pretty critical distinction for data protection; it is the difference between being able to restore a LUN, and being able to restart your application.
Secondly, integration with vCenter. This allows DPA to report on the protection status of VMs.
Third, deduplication reporting with Data Domain appliances. No more ambiguity (for the backup admin) on how much deduplication is actually being achieved. Now, admittedly, this was already very easy to understand with the Data Domain appliances, but this brings the functionality into DPA, and therefore centralizes backup intelligence. Data Domain, Avamar, and NetWorker data are all gathered at a central repository, and can be viewed in a unified reporting environment.
Also new for Data Domain is the capability to monitor and report on the physical components of Data Domain systems, such as power, temperature, fan status and so on.
Finally, we should note that the effort of transitioning to the private cloud will also be eased with DPA as it includes new expanded capabilities around chargeback reporting: DPA can report on the amount of physical resources consumed for various processes, such as replication and deduplication. Meaning that the department or user that consumes 2/3 of the resource can be charged for 2/3 of the resource; the user that consumes but 1/3 only needs to pay for that share.

If you are interested, you can learn more about DPA here.

Do You Need Backup?

Scott Waterhouse — Tue, 24 Nov 2009 11:18:01 -0700

Do you need backup?

And yes that is a serious question. (Given the name of this blog, and my chosen specialization for the last 20 years, it is understandable if you think I was asking it in jest!)

A recent real world situation makes me wonder. It also makes me wonder at what point something becomes a self-evident truth? And are we adequately equipped to answer this question with a logical and defensible answer based on data rather than opinion? Hmmm...

First, a story. One of my favourites in years of IT. And it really happened, about 12 or so years ago.

A certain organization had a badly aging mainframe. It was creaking badly (in the metaphorical sense). The maintenance cost way more than the lease on a newer system. It was bigger than a newer system--taking up too much space. It wouldn't run the most current operating system. And so on. And the only one that noticed was the lead sys admin. When he pointed this out to management--who were apparently surprised by this state of affairs--he asked for funding for a new system. His manager asked him to justify the funding request. The sys admin left, and came in early the next morning to drop off a one page business case for the upgrade. Actually, it was just three words (in 72 point text): "Think about it."

The need for an upgrade was painfully self-evident to the sys admin. But apparently not to the manager.

(And yes, they went ahead with the upgrade. I don't know if a "real" business case was ever prepared.)

So when I think somebody needs backup, I think it is reasonable to also ask: is this a self-evident "truth"? And if so, can that need be justified more thoroughly and specifically than with a simple "think about it"?

Justifying backup is all about understanding risk and cost. Risk of failure, and cost of recovery.

Which is safer: a car or a plane? I know a lot of folks that think driving is a lot safer than flying. And all of them are wrong. Flying is about 50 times safer than driving, when measuring fatalities per passenger mile traveled. For those of you interested in following up that stat, I refer you to the wikipedia page on air safety.

I think that one of the reasons why some people hold cars to be safer than airplanes is because we (people) do a poor job of assessing risk and measuring risk. The car feels safer. It feels unnatural to fly in a giant aluminum tube that weighs hundreds of thousands of pounds.

But if we can't accurately asses risk when it comes to cars and planes, how can we know that it is self evidently true that the mainframe upgrade is required? How can we accurately asses risk of failure, and cost of recovery for IT systems?

Or, at the risk of playing to a stereotype or opening up a cultural divide, why can't non technical managers understand the risk of not doing backups? (Not always, just some of the time. But even some of the time is too much.)

Another organization recently told me that they did not want to do backups for hundreds of TB of data, resident across several hundred hosts. The application was "only" in development, not yet in production, and it was OK if there was no backup. Yikes.

But without an accurate understanding of the cost of recovery (both of an individual system) and the risk of failure, there is really no way to combat the perception that backup isn't needed in this case. Unfortunately, this is not a trivial exercise. The majority of organizations I speak to don't have any good understanding of how much a recovery costs, or what the risk and cost of a failure (data loss event) is. There exists no standard approach to quantifying these metrics; no common understanding; and, many of the elements remain in the realm of soft costs.

In the particular case of the organization with hundreds of TB of data at risk, the metrics and data did not exist to quantify this risk and the costs associated with it. As a result, I had little better response to the problem than saying "well you just need backup!" Why? Because! And it turns out that it is actually a very difficult thing to quantify and justify numerically and logically. (Bearing mind that discussion of ITIL standards, best practices, and so on probably wouldn't carry a lot of weight with an exec running such a project.)

So the question should not be: "do you need backup?" It should be: "how much could it cost you if you don't have backup?"

There is a world of difference between assuming you need to do backup, and justifying it.

Avamar v5.0

Scott Waterhouse — Wed, 18 Nov 2009 11:57:17 -0700

Today EMC formally announced the availability of Avamar 5.0, which is loaded with new features and functionality. I am going to briefly review some of them here, focusing on the changes that I think will be most significant from a customer's point of view.

Having said that, please don't hesitate to ask questions or post feedback in the comments section if any of you are interested in any clarification on the material.

So what is new?

The big news is: bigger nodes, best server side task scheduling, more efficient checkpoints, new clients, desktop and laptop support, major performance enhancements for a variety of clients, and enhanced support for VMware environments.

That is the high level. Let's dive into these in a little more detail.

First off: bigger nodes. The 3rd generation of Avamar hardware is being release concurrently with Avamar 5.0 The new nodes offer 60% more capacity than the old nodes. New nodes are 3.3 TB of useable capacity each. This means that an Avamar grid can now grow to 52 TB of useable capacity. This compares to the 2.0 TB of capacity offered on the older 2nd generation nodes. The real benefit here: lower total cost of ownership. The new nodes are priced very similarly to the old, so the bottom line is simple: more capacity for your dollar. Incidentally, the underlying server hardware has been updated too, this is not just an increase in drive capacity, but an update to CPU, memory, etc. (all the good stuff that comes along with current server architectures).

Desktop and laptop support is also new. Now, truthfully, this could be done before. There was nothing stopping an Avamar user from taking a standard Windows client and sticking it on a laptop or desktop on it. However, this wasn't really the design point of the client, and there were some drawbacks with that approach. So, Avamar has addressed those by offering a web browser UI into the client. Clients can do backups, restores, search through restores, and examine their backup history through a web interface. Which really means that they can directly handle most common activities, rather than having to call the backup administrator. (And yes, Steve Jobs fans--myself included--can rest assured that it works on Mac too.)

Further improvements were made to enable desktop/laptop support: users can be authenticated with the domain service of your choice--either LDAP or Active Directory, or using Avamar user authentication if you prefer. Users can manually initiate backups. And clients can be remotely deployed using push installs to both Mac and Windows, with support for Microsoft SMS 2003.

Avamar has also added new clients: The big ones here are Oracle 11g, MS SQL Server 2008 and VCS support.

And finally we have enhanced our VMware support. I have written many times about Avamar's existing support for VMware: essentially there were two common approaches that users took--VCB backups and guest level backups. In either situation, Avamar offered a clearly better approach than alternative backup applications. Better in that backups were faster, used fewer resources, and were deduplicated at the source. Both of those options remain with the latest release of Avamar.

In addition, Avamar 5.0 offers integration with vSphere 4.0 environments, and leverages some of the native capabilities of vSphere and the vSphere APIs to provide a faster, better, deduplicated backup. Avamar 5.0 can now do image level backups (back up the virtual machine as a complete entitiy) and restores. Avamar can also take advantage of vSphere to do block level incremental backups of an ESX host--with the added capabilities of being able to do a file level restore of data (even if it has been backed up with block level incremental or image level backups).

So this is a big one. To the best of my knowledge, this makes Avamar 5.0 the first major backup product to market with significant vSphere 4.0 integration. Several parties have been asking for this (yes, W. Curtis Preston, I am thinking of you) and with this release, EMC has delivered.

On top of all this, there are numerous other major and minor enhancements, including enhancements to how the Avamar server handles and schedules internal administrative activity, how the Avamar server connects to EMC support, NAT support, dual switch support for high availability, secure deletion of data, improved performance and architecture for Oracle and NetApp filers via NDMP, improved support for multi-core processors to permit multi-threading of backups for faster performance, and system state backup and recovery for Windows Server 2003.

Next time... a more in-depth discussion of the new features in NetWorker 7.6 that I alluded to above.