The Backup Blog

Source Deduplication Issues and Concerns

Scott Waterhouse — Thu, 12 Nov 2009 08:12:30 -0700

The always interesting and informative Preston de Guise has a discussion of source and target deduplication on his blog "The NetWorker Blog". Which is just about as imaginatively named as this blog. Well, he may not be a marketing mad man, but Preston is one of the most informative and most informed people I know writing about backup and recovery.

However, he does raise a couple of concerns about source deduplication that I would like to address. Now source deduplication for EMC means Avamar. Actually, for most everybody it means Avamar, because there really is no other credible source deduplication alternative today. There may be software deduplication solutions--but most of them tend to be target solutions in one way or the other. And not terribly technically or architecturally credible. (Yes, I am thinking of you, TSM.) With vanishingly small market share to boot.

So Avamar it is.

Preston mentions a few points about Avamar:

In regular backups while there may be some benefit to reducing the amount of data transmitted, what you’re often not told is that this reduction comes at a cost – that being increased processor and/or memory load on the clients. Source based deduplication naturally has to shift some of the processing load back across to the client – otherwise the data will be transmitted and thrown away.

This is true, but there is a very significant counter-point. That is: although source deduplication has a higher processing load on the client than traditional backup, it is for a much shorter period of time. More succinctly: your backups use more CPU, but don't take as long.

Experience, annecdotal reports, and testing we have done at EMC suggest that, on average, Avamar backups may consume 25-50% more CPU than a traditional backup. However, the duration of the backup may be 75% less with Avamar than with other backup products. (Even those, like TSM, which utilize a "progressive incremental" approach.) In some cases, Avamar may complete a backup 90% faster than traditional backup. Particularly in large, dense file systems and with filers you will see significant improvements in the time to complete a backup job.

Secondly, Preston writes that:

Onto the second proposed advantage of source based deduplication – faster WAN based backups. Undoubtedly, this is true, since we don’t have to ship anywhere near as much data across the network. However, consider that we backup in order to recover. You may be able to reduce the amount of data you send across the WAN to backup, but unless you plan very carefully you may put yourself into a situation where recoveries aren’t all that useful. That is – you need to be careful to avoid trickle based recoveries. This often means that it’s necessary to put a source based deduplication node in each WAN connected site, with those nodes replicating to a central location. What’s the problem with this? Well, none from a recovery perspective – but it can considerably blow out the cost. Again, informed decisions are very important to counter-balance source based deduplication hyperbole.

Again, Preston is just right. However, there is an alternative with Avamar that can be employed to mitigate this concern: Avamar Virtual Edition. Briefly Avamar Virtual Edition (or AVE) is an Avamar server in a virtual machine. So rather than having to deploy an entire physical node in each WAN connected site, you can simply deploy an AVE. An AVE is typically more modest in size than a physical Avamar node, and hosts anywhere from a very small amount of storage up to 2 TB--normally the capacity is just that required for local recoveries at the remote site. Storage can reside on the same storage system that your ESX server holds its data--it does not need to be dedicated.

So again, the issue Preston raises is true, but there is a practical and easy way to mitigate the financial impact. In practice, both source and target deduplication solutions have small footprint products that are reasonably price and appropriate for placing at a small, remote site.

Finally, Preston says that:

Now let’s look at a couple of other factors of source based deduplication that aren’t always discussed:

Depending on the product you choose, you may get less OS and database support than you’re getting from your current backup product.

The backup processes and clients will change. Sometimes quite considerably, depending on whether your vendor supports integration of deduplication backup with your current backup environment, or whether you need to change the product entirely.

The current support matrix for Avamar can be found on PowerLink (for EMC customers and partners). Without listing it in its entirety here, suffice it to say that Avamar supports a full range of databases, applications, host OS types and versions. Although it may have been true several years ago that support was somewhat more constrained than a typical backup application, that tends to no longer be the case. With the exception of one or two idiosyncrasies in the support matrix, Avamar supports as wide a range of applications and data types as any other backup application.

As for the backup processes changing? Well yes! They do. Thankfully. I will be the first to admit that switching backup applications can be a non-trivial process. Not something that should be lightly undertaken, or that most people will do without a second thought. However, here is the other side of the coin: lots of current backup environments are broken. If not irretrievably so, then very seriously.

VMware backup. Filer backup. Remote backup. Backup of dense file systems. All of these are things which many traditional backup applications simply do a very poor job of. And if your backup application is broken, then changing it may be a very good thing. I would encourage you to weight the costs and the benefits of switching applications very judiciously. Don't just do it because somebody told you it would be a good idea. (Even if that somebody is me!)

But... be honest about the strengths and weakness of your current approach. If the way that NetBackup or TSM does something right now is just really messed up, if it takes way too long to finish a backup, if your failure rate is way too high, if you just can't backup that ESX server in any reasonable time frame, then maybe changing backup applications isn't such a bad idea.

Backup Cost Reduction, Target Deduplication, and TSM

Scott Waterhouse — Wed, 11 Nov 2009 14:36:41 -0700

Two posts previous I was discussing the TCO of backup, the differences between hard and soft costs, and the qualities of a good business case. The general conclusion? Most business cases depend on hard costs (and don't recognize soft costs). If hard costs show an ROI of less than 18 months, most CIO/CFO types will love it. If the business case relies on soft costs to get there, it will probably be rejected.

Looking at the particular example from the previous post, the business case Tony Pearson used as an example showed that 81% of the savings were due to soft costs. This is really bad. In most business environments, such a business case would be rejected out of hand.

I thought it might be instructive to have a look at a better business case: a business case based almost entirely on hard cost (savings). Two of them actually. One for source deduplication, one for target deduplication. In both cases, they are real environments, with real costs, real savings, the proposals were made with the costs and savings exactly as described, and they were accepted by the customer and the projects were funded on that basis. I have only removed names and specifics to the extent necessary to anonymize the customer and any technical or business characteristics unique to that customer. And, as I wrote the TCO tool itself EMC uses for target deduplication, I am more than happy to answer any questions on the data, the assumptions that drive it, or the calculations and formulas that we used to determine the benefits and cost savings of the solution.

In this post, I am going to deal with target deduplication. The business case for a source deduplication solution will be the subject of a future post.

In the case of target deduplication, we were working with an organization that uses TSM. They had a tape infrastructure, with an off-site copy pool (also tape) for their backup and recovery infrastructure. This particular customer also had infrastructure at 3 sites (all more or less local to a single metro area). The customer wanted to refresh the infrastructure, but was unwilling and unable to throw out TSM entirely (due to some dependencies in the mainframe space).

A couple other relevant factors: the business in question employs 30,000 people, manages 140 TB of data, has annual revenues of nearly $500m (CAD), and has a fairly average mix of email, file and print, and database data with typical growth rates and an average number of servers given the amount of data.

The savings that we were able to demonstrate for target deduplication included (all figures in CAD):

Media Savings - $5,012,103
Tape and Library Savings - $2,435,671
Administrative Costs - $690,285
Offsite Tape Storage Costs - $283,177
SAN Port Savings - $251,115
Improved Tape Reliability - Revenue Impact - $147,890

These benefits can be grouped regarding business impact as:

$8,672,352 in IT cost reductions
$147,890 in business strategic advantage benefits (this is a soft cost, but ammounts to less than 2% of the net benefit of the business case)

On the other side of the coin, what were the cost associated with the project?

To implement the proposed project required a 5 year cumulative investment of $4,132,999 including:

$1,542,000 in initial expenses
$0 in capital expenditures
$4,132,999 in operating expenditures (including the initial expense)

Comparing the costs and benefits of the proposed project using discounted cash flow analysis and factoring in a risk-adjusted discount rate of 9.5%, the proposed business case predicts:

Risk Adjusted Return on Investment (RA ROI) of 87%
Return on Investment (ROI) of 113%
Net Present Value (NPV) savings of $2,954,800
Internal Rate of Return (IRR) of 54%
Payback period of 15.0 month(s)

In other words, with an EMC solution, relying almost entirely on hard costs, we were able to demonstrate a payback period of just 15 months, and an ROI of 113%, by augmenting TSM with target deduplication.

As a "C" level executive, this is the kind of cost savings that you simply can't ignore. It isn't funny money. It doesn't rely on voodoo math. It doesn't have anything to do with "soft" costs that are hard (impossible) to justify. It is real money, and real savings.

Is TSM an expensive backup solution relative to other backup applications? Yes. But those costs can be effectively mitigated.

Cloud Backup Event

Scott Waterhouse — Wed, 11 Nov 2009 07:30:38 -0700

I have been running a poll for a few months now that asks the question: would you back up your business data to the cloud? I have to admit that I find the results a little surprising, but they have been consistent over time, and are relatively consistent across geography. Approximately 50% of respondents have indicated that they are willing to backup up to the cloud.

With the latest release of NetWorker, EMC has introduced cloud backup as native functionality. Right out of the box, NetWorker users can clone backup data to a target in the cloud. EMC is hosting a Live Event that discusses the functionality in greater detail.

EMC NetWorker Fast Start and the new Cloud Backup option offer a secure and affordable alternative to the pain and expense of offsite tape vaulting. Attend this webcast and learn how you can:

Reduce personnel expenses by automating offsite data protection
Cut the hard costs associated with tape infrastructure and offsite storage facilities
Choose a flexible “pay-as-you-go” model for offsite storage

You can register for the event here, or by going to http://info.emc.com/mk/get/DBM5396-1950_RAF_LP?reg_src=PA .

The Cost of Backup

Scott Waterhouse — Thu, 29 Oct 2009 12:22:43 -0600

I am going to revisit the issue of backup, cost, and the total cost of ownership of backup in this post. Tony Pearson and I have been discussing this (hopefully in a fairly constructive fashion) over the last little while, and I see he has responded to some of the points that I made in my previous post on the issue. Tony's response can be found here: "The TCO of TSM for Backup".

Tony continues to maintain (I think) that TSM has a "good" TCO and is efficient relative to other backup applications. Although I don't agree with a lot of what he writes, I do think that there are some pretty good lessons to be learned amidst the information he has posted, so I am going to continue to pursue this a bit in this post.

First, Tony claims that I said "only the cost of new TSM servers should be considered in any comparison." Ummm, jeepers Tony, no I didn't! I would never make any such claim! (What I did say is that TSM is unique in the backup world, in that it scales by adding master servers. Once my TSM master server is at capacity, due to I/O requirements, or database scale, or ability to drive disk or tape, I have to buy another master. That stands in sharp contrast to Networker or Netbackup, where I can just buy a Storage Node or Media Server--which tend to be much smaller systems with much less disk than the master servers. So TSM requires many more master servers than competitive solutions--each of which imposes not just a burden of cost, but a management burden at both the server level and the backup application level. Further, based on my experience, the typical TSM customer will end up with more master servers than they would have combined master and media servers if they were using Netbackup or Networker.)

Servers are but one of many cost considerations when it comes to understanding the TCO of a backup environment. Generally speaking, these costs can be divided into hard and soft costs. Hard costs are costs associated with physical, tangible things (servers, network, disk, tape drives, shipping, etc.). Soft costs are those costs associated with stuff that is less tangible. Soft costs can include labor and administration--though truthfully admin costs straddle the line between hard and soft costs, at least in part because they can be difficult to accurately quantify--but also include things like risk, efficiency, and the benefit of systems uptime.

There is one other really important distinction between hard costs and soft costs. Hard costs are easy to justify and include in a business case. Soft costs are notoriously hard to accurately quantify, hard to justify, and very rarely are included in a business case. Let me give an example of this: a business is moving from a tape based backup system to a disk based backup system. Tape is actually quite unreliable; if it has been shipped and handled, it is not unusual to see a reliability of 98 or 99%. Which, from an IT and systems management perspective, is just awful. (How would you like it if your email system would fail to retrieve one out of every fifty emails you receive?) Disk, on the other hand, is considerably more reliable. It shouldn't be too much of a stretch to say that we can achieve 99.999% availability/reliability from disk backup systems. So we have a gap of roughly 2% in the reliability of the two systems.

Now imagine the restore of a critical business system. If I am restoring from tape, I can imagine the restore is 2% likely to fail. And a failure may cost several million dollars--lets say $5m for the sake of discussion. That cost may come from any one of a number of things: are people sitting around unproductive during the restore process? Am I losing revenue because my retail system is not processing transactions? Do I have to re-enter data? How about potential damages to my reputation or to other business partners if the system is unavailable? When I put it that way, $5m starts to look like a very small number indeed. (Realistic estimates depend on industry type, size of business, revenue, etc., but could rapidly stretch to the tens of millions.) Other similar factors may be considered in some industries too.

Finally, we have to understand the likelihood of a restore being required. Generally, we do this by understanding how many systems we have, and what the chances are of a full recovery being required. That usually gives us a metric like: for every 1000 systems, 3 will experience a catastrophic failure per year, and require recovery from backups.

So we multiply the chances of a catastrophic failure per system by the number of systems, then multiply by the increased reliability of disk over tape, and we have a way of quantifying the decrease in risk we can achieve by moving from tape to disk for backup. (We might also factor in the duration of recoveries, and we can further refine this by modifying the model to include tiers of servers, with different values for the cost of a recovery, length of a recovery, and so on, for the various tiers).

So at the end of the day, I might say that I am reducing my risk by $7.5m over 3 years (stipulating the math comes out at .75 critical recoveries per year at a cost of $5m each) by acquiring a new backup system that uses disk rather than tape. So as long as the new system costs less than $7.5m it has a positive ROI, pays for itself, and should be acquired right away! Right?

Well, maybe not.

How come?

Isn't all this legitimate, defensible, and accurate? Absolutely yes. No question.

Will it be considered in a business case to justify the acquisition of a new backup infrastructure? No way. Almost never will a business include these considerations in their business case. Admittedly some do, but it is rare. Anecdotally, I would suggest that less than 5% of the business cases I have worked on have included such soft costs.

Business case justifications live and die on the basis of hard costs. Not soft costs.

So, when Tony goes on to discuss a case of a customer switching to TSM from EMC Networker as an example of cost efficiency, I was a little surprised to see that a large number of soft costs appeared to be included in the justification. (As an aside, I am making a few assumptions about what is going on with this particular business case. Tony simply doesn't provide sufficient data to make a really good critical analysis of it--nor would I expect him to necessarily--so I am just doing the best I can with the data that is given.) Lets look at the numbers Tony provides:

Reduce Business Risks $6,749,796
Consolidate and Standardize IT Infrastructure $4,975,667
Reduce IT Infrastructure Costs $2,057,107
Improve IT System Availability / Service Levels $1,409,431
Improve IT Staff Efficiency / Productivity $982,919

If we look at these numbers, we can see that the first and second numbers are soft costs, as is the fourth. The fifth is equivocal. Without knowing where the benefit comes from, it is tough to say if it would normally be included in a business case or not. Are the gains from using newer systems? Different systems (i.e. TSM rather than Networker)? BIgger systems (if the infrastructure has not been updated in 5 years, then we would well expect new systems to be bigger)?

Being extraordinarily generous, and admitting this is a hard cost, we can see that a staggering 81% of the costs in the business case appear to be soft costs. Costs, in short, that would never be included in the vast majority of business cases.

Further, if we exclude these soft costs from this business case, we can see that there is simply no reasonable financial justification for moving from Networker to TSM. The benefits over 3 years would be about $3m, and the costs would be $5.76m.

Interesting.

Lets take this further, however, and ask what portion of the infrastructure cost reduction and staff efficiency gains would be achieved by any technology infrastructure upgrade? Absolutely irrespective of the underlying backup application itself? If I moved from LTO-1 to LTO-4 or a modern disk deduplication technology for example, I would fully expect to achieve some pretty considerable savings.

In other words, Tony's conclusion that "IBM Tivoli Storage Manager uses less bandwidth, fewer disk and tape storage resources than EMC Legato. For even a large deployment of this kind, payback period is only NINE MONTHS" is not only unsupported, but actually contradicted by the evidence he himself provides. Without soft costs, TSM appears to be more expensive than the alternative (EMC Networker).

So one of the points that I try to make every time I discuss TCO is that numbers are meaningless unless they are your numbers. How much do you pay for administration? For tapes? Do you include some soft costs? If so, which ones? I think that point is doubly true here. Without some understanding and justification of the numbers, it is very hard to attach any significance or relevance to them.

The next point that is particularly important here is regarding my earlier observation (also quoted by Tony):

Final point: there is actually a really important secondary point here--what is the TCO of your backup infrastructure. In some ways, TSM is one of the most expensive (number of servers and tape drives, for example), relative to other backup applications. However, I think it would be a really interesting exercise to critically examine the TCO of the various backup applications at different scales to evaluate if there is any genuine cost differentiation between them.

Based on the data provided by Tony, all we can say is that we cannot make any reasonable inference or conclusion with respect to the question. It remains unanswered.

However, I do have lots of anecdotal evidence to suggest that Tony's answers are pretty deeply flawed. For example, I have ample evidence that shows Networker's deduplication option can reduce bandwidth consumption by over 95% when compared to a TSM deployment. (Yet Tony suggests there are bandwidth savings when comparing TSM to Networker.) I have further evidence in the form of business cases that deal only with hard costs that Networker tends to have a considerably lower TCO than TSM. And, contrary to Tony's suppositions, these business cases were built for organizations with 5,000 to 50,000 employees (not the rather trivial "10 employees" that Tony attempts to ascribe to the typical Networker installation).

So what would I take away from this it it was my business case? I would want to see hard and soft costs clearly differentiated. I would want to see a return on investment of less than 24 months (perhaps less than 36 in some cases) based solely on hard costs. I would want the business case to be built with numbers that are the actual numbers for my business--not based on industry averages or general assumptions. I would want to understand what benefits accrue as a result of doing a technology refresh, and what benefits result from any proposed change in application. And I would want to see all the calculations, assumptions, and reasoning clearly described and justified--no voodoo economics please.

The Backup Blog Returns

Scott Waterhouse — Thu, 29 Oct 2009 10:51:28 -0600

Just a quick note to say that I am back, and once again obsessing about all things backup.

Here is a small photo of some of the scenery I saw on my trip. Bonus points for anybody that knows where this is! I will say one thing: when the landscape looks like this, you know you are a long way from home!