Progressive incremental model
TSM’s incremental only approach presents the biggest challenge to deduplication. By sending only changed data, TSM limits the amount of redundant information stored and hence deduplication benefits. This is in stark contrast to most other backup applications that rely on frequent full backups; it is these jobs that provide the best data reduction. However TSM is often configured to perform nightly full backups of data types such as Exchange, Oracle or SQL. These backups will provide similar deduplication benefits as seen in other applications, but typically represent a fraction of total data protected.

Data movement
Deduplication algorithms use various methods to recognize redundant information. They analyze data as it is written to the system to find the redundancies. In traditional backup applications, data is typically written to a disk or tape device and remains there until expiration. In contrast, TSM writes data to a given pool and then will run multiple processes which move the data such as reclamation. These processes create deduplication challenges because they force the appliance to constantly re-hydrate data and then re-deduplicate data. The frequent data movement creates inconsistent data patterns that can be difficult for deduplication mechanisms to recognize resulting in decreased data reduction.

Data fragmentation
TSM is designed to maximize performance by using multi-streaming. This means that a given job could backup to almost any disk or tape device and its data could be multiplexed or mixed with data from any other server. This can be a challenge because data becomes fragmented and will be written in many different physical tape or disk locations. Deduplication struggles because these segments can be small and the algorithm must effectively recognize the varying block sizes to achieve the best possible data reduction. In practice this reduces data reduction.

Reclamation/Overwrites
Since TSM typically spreads a backup job across multiple pieces of media, data becomes fragmented. TSM has a process called reclamation which reduces fragmentation by moving data onto new cartridges and expiring the previous cartridges. Reclamation is designed to minimize the number of cartridges required in a TSM environment, but it can be I/O intensive. As the process it runs, it moves unexpired data onto new cartridges and expires the old cartridges. This can be a challenge for deduplication systems that utilize a batch cleaning process. These systems run a weekly or more frequent process to deleted expired cartridges; free space is not realized until the cleaning is completed. Deduplication ratios will decline since the expired cartridges will consume disk space. The other challenge is that the clean process can be I/O intensive and could negatively impact other TSM processes.

TSM is a very powerful application with a unique data protection model. Unfortunately, the same technology that reduces backup windows also creates challenges for deduplication. The result is that deduplication ratios decrease in TSM environments. However, even with smaller ratios, deduplication still provides a strong benefit for TSM environments particularly with deduplicated replication.

Related posts:

1. TSM Deduplication
2. Defragmentation, rehydration and deduplication
3. DeltaStor Deduplication

It was in the context of this situation that I saw a tweet last night about COPAN being acquired. The first questions were who and for how much and the tweet suggested that the answers were SGI and $2 million dollars respectively. Wow, what an amazing decline. COPAN raised $124 million dollars in multiple financing rounds and they exit the market at a $2 million valuation.

COPAN focused on MAID (massive array of idle disks). The technology allowed them to spin down unused disks to reduce the power and cooling requirements. The design included proprietary highly dense disk packaging that provided the densest storage in the industry, and actually required some datacenters to specially reinforce their flooring. They focused on $/GB and said that they offered the lowest in the industry both from an acquisition and operational cost standpoint. All of this sounded compelling from a marketing perspective, but the reality was different.

COPAN’s technology was optimized for storage density and green-ness, and not for performance. Their initial route to market was to sell a VTL appliance through an OEM agreement with FalconStor. The difficulty is that the limiting factor for many backup environments is speed. Companies must ensure that they are meeting their backup and recovery requirements and performance is typically the key element. COPAN’s positioning of “very green, but not very fast,” did not resonate. Of course, customers with specific power and cooling limitations were interested, but this represented a fraction of the market. In my opinion, this positioning error and architecture limitation limited initial adoption.

The advent of deduplication became another huge problem for COPAN. Deduplication can be highly I/O intensive and often requires high performance disk systems. In fact, some vendors even require Fibre Channel disk drives for performance. COPAN’s architecture did not align with this technology trend. They did not have the I/O performance for deduplication and their message of space, power and cooling savings was being echoed by deduplication vendors. They tried to remedy the situation by including non-COPAN disk as storage for the FalconStor deduplication landing zone, but that clearly was not a winning solution.

The other area COPAN tried to address was the archival space. They partnered with Quantum on the StorNext file system and positioned their platform as an archival target. Unfortunately they were unsuccessful in this endeavor as well.

The core problem with COPAN is that they made a bet on a technology that did not address a primary customer market pain. End users appreciated the power and cooling benefits of the solution, but that need was secondary to the real requirement of backup and recovery performance. Deduplication technology further eroded their position by bringing substantial power and cooling benefits to traditional disk platforms. COPAN was stuck with a proprietary MAID technology with a limited market opportunity. The result is that COPAN’s technology became irrelevant due to deduplication and more recently disk spin-down technologies, and this was reflected in the company’s dwindling financial performance. SGI is now the owner of COPAN’s technology and it will be interesting to see how they bring it to market.

Lessons learned

The key lesson is that success depends on addressing real customer pains. COPAN tried to build a business on a feature, power and cooling efficiency, while customers wanted a solution to their real backup and recovery pains. This disconnect resulted in a solution that was not well aligned with the market and hindered COPAN’s success. If you are an emerging vendor, you must thoroughly understand the markets that you address and focus your product on solving customer pains. Anything less will inevitably lead to failure.

Related posts:

1. Lessons from the Sidekick debacle
2. NetApp and Quantum: Why an acquisition would be difficult
3. Data protection and natural disasters – Part 2

Via Beth Pariseau from TechTarget.

No related posts.

There are many different approaches to protecting personal data. The two key concerns to consider are:

1. What happens if I lose the hard drive where my data is stored or experience a software problem such as a virus?
2. What happens if I suffer a more extreme data loss such as my house burning down?

Each question is critical, and the answer will vary depending on the data. For example, digital pictures of your family might have a different priority than your MP3 library. The former is irreplaceable and the latter is not. These priorities will impact the chosen data protection medium and methodology.

Protecting from loss of a hard drive or a software problem:

The most basic method to protect from hard drive failure is RAID. RAID essentially writes your data across multiple hard drives and in most instances will protect you from a loss of a single drive. Of course, this only protects you from a hardware problem. If you get a virus that erases your data, RAID will not help and will provide an efficient way for the virus to erase multiple drives simultaneously!

Another option is an external USB drive. External USB storage provides a cost effective target that you can copy data to. It often makes sense to unplug the USB drive when finished to provide some protection from the virus problem mentioned above. However, remember that someone or some piece of software must run a process to copy data to the drive; if the process runs inconsistently or unreliably, your data will be at risk.

Protecting from extreme data loss:

This is the worst kind of disaster with much larger implications than data loss. However, it is important to design a strategy that enables restoration of data in this scenario. The best solution to this problem is to maintain a complete copy of your data at an offsite location. The simplest and easiest method is to use a backup service like Crashplan or Mozy. These companies sell a data protection service that includes software that encrypts your data and sends it over the Internet to their location. The process of encrypting, compressing and sending data is automated by the software agents provided by the vendors making these solutions entirely automated. The challenge is that it can take days or even weeks to initially send your data over your Internet connection and a complete recovery will take the same amount of time. Crashplan offers an added cost solution (~$130) where they will send you a USB drive which would help speed recovery in cases of total loss.

In summary, there are multiple different ways to protect from data loss. RAID and local USB drives provide reasonable solutions to hardware failure, but limited or no protection for OS or software failures. Remote services provide excellent disaster protection with the downside of a long initial backup and long complete recovery. There is no reason that you cannot use multiple approaches simultaneously. For example, I store my data on a RAID protected device and keep a local copy on a USB drive. I also use Crashplan to backup the data nightly and for extra redundancy, I post my images to a dedicated image hosting service. My approach may be overkill for many, but every computer user must think about how they are protecting their data.

Related posts:

1. Data protection and natural disasters – Part 1
2. Lessons from the Sidekick debacle
3. The Cloud, Company Size and Data Protection

I watched the first half of the game with my friends with no thought of social media. The half time show was headlined by The Who, and as the band started, my first thought was, “these guys are unimpressive; I must tweet about it.” I retrieved my Blackberry, starting tweeting with SocialScope and engaged in a lively discussion with my twitter comrades about the merits of the band. As the second half started, the topic moved to the game and the performance of the two teams. It was great seeing a diverse range comments and perspectives.

In summary, my Twitter interactions improved my Super Bowl viewing experience. Some people are skeptical about the value of social media and this scenario is a microcosm of the benefits of the new medium. Where else can you create relationships with people with a common interest that transcends geographic boundaries? I continue to be impressed with the many smart people I meet on Twitter and suggest that my readers try it out. You can find my full Twitter profile here.

Related posts:

1. Now on Twitter

NetApp acquired VTL technology with their purchase of Alacritus for $11 million back in 2005. Alacritus provided a software only VTL solution that ran on a Linux platform. Their product specifications appeared impressive, but they had limited success in the US. Our partners in Asia saw them more frequently. For NetApp, the acquisition made sense because it represented a relatively cost-effective entry into the rapidly growing VTL market. However, as in most things, the difficulties were in the details.

NetApp’s core intellectual property is their ONTAP operating system and associated WAFL filesystem. These components provide the intelligence and value-added features of their arrays. The challenge for NetApp after acquiring Alacritus was the integration of the two technologies.

NetApp’s short term strategy after the purchase was to build a VTL using Linux, Alacritus, x86 servers and existing disk shelves, and they released their first system in 2006. This leveraged their hardware design and configuration, but bypassed their core software IP. Customers could reuse VTL disk shelves in their filers if they decommissioned the VTL, but there was no ability to share the same storage between filers and VTLs. This is your classic island of storage problem.

The next trend in the market was deduplication. NetApp recognized the importance of the technology and announced A-SIS deduplication for their filers in May of 2007. However, the Alacritus VTL did not support ONTAP/WAFL and so the technology could not be reused. NetApp then embarked on a project to develop a second deduplication engine (after A-SIS) for use on their VTL.  They said that the new deduplication system would be available in June of the same year, and I believe that they underestimated the difficulty of creating a new engine. The development took far longer than expected and they announced the product in October of 2008. To this day, we have seen few implementations of NetApp VTLs with deduplication, and I often wonder if this is because their new deduplication engine does not work effectively.  Given the limited success, it is not surprising that NetApp has officially ceased development.

In my opinion, NetApp’s VTL failed due to a lack of commitment. To be successful, they needed to port the Alacritus solution to ONTAP/WAFL to allow it to co-exist transparently with existing technology. I believe that NetApp was focused on snapshots and replication for backup and recovery and so the NearStore VTL was not given the attention it deserved. Had they integrated Alacritus and ONTAP, they had the potential to create a unified filer, VTL and deduplication appliance which would have been very competitive.  Instead, they chose to leave Alacritus as a separate product, and NetApp’s aggressive pursuit of Data Domain marked the end of the NearStore VTL.

The irony is that I believe NetApp had an opportunity to be an early player in the VTL space. They had the relationships and footprint to aggressively sell the solutions. Unfortunately, their critical missteps and lack of focus limited their success.

Related posts:

1. NetApp and Data Domain: ‘Til death or a better offer from EMC do we part
2. NetApp Dedupe: The Worst of Inline and Post-process Deduplication
3. EMC one-ups NetApp

Engadget

FUJIFILM Announcement

The Register Article

A blog post by Robin Harris at ZDnet

My thoughts

It is interesting to see IBM/Fuji driving tape development. With this announcement they have increased native tape capacity over 21x from LTO-5, the newest LTO offering. The dramatic density improvement will drive a continued decrease specification-based $/GB. However it also raises some new questions:

Performance

If performance is unchanged from LTO-5 and you could stream the drive to maximum specifications and achieve 2:1 compression, it would take about 35 hours of continuous backup to fill the cartridge. Wow, that is a long backup window! Chris Mellor in his Register article linked above suggests some alternative approaches to performance including striping multiple tape drives and adding more write heads to drives. Regardless of the method used, it appears that these new tapes will have to be streamed much faster than any previous tape technology. This will create infrastructure challenges; the IT team must ensure that they have enough bandwidth through the backup process to stream the drive. The concept of shoe-shining becomes particularly painful here since the minimum data transfer rate will also increase.

$/GB

This new high density tape can create $/GB economic challenges both for DR and in the model Chris suggests of striping across drives. The acquisition cost for these tapes will be substantial. If you assume the same $/GB as LTO-4, each of these new tapes will cost about $1,400! Clearly the real world price will be much less, but the cost will increase. The challenge is that the best $/GB is achieved when the tape is filled and compressed. Thus, the best ROI is achieved when you have stored 70 TB of data on the tape (assuming 2:1 compression). If you are using tape for DR, it becomes even more challenging because you want to fill your tapes and your backup jobs will not typically fit in consistent 70TB increments. If you do not manage this effectively you will end up sending hundreds of partially filled tapes offsite which will destroy the ROI.

The situation above gets worse if you look at Chris’s model of striping data across multiple drives. Now your maximum capacity increases linearly with the number of drives. If you assume four drives then your ROI is only maximized when you store 280 TB of data. These numbers get maddeningly large very fast.

Risk of data loss

Now imagine that you have resolved the performance and $/GB issues. You are comfortably filling each tape to its max of 70TB. The challenge now becomes one of data protection. With so much data on one tape, what happens if the tape is lost? What happens if it is unreadable for whatever reason? You are now placing a much larger bet on the reliability and consistency of these tapes than ever before. Are you really comfortable storing so much data on a single piece of relatively unreliable medium? Of course, you place the same bet on tape today but the lower capacity means that an individual tape failure will have a reduced impact.

Clearly the development and launch of massively dense tape can in the right circumstances provide business value. However, I believe that customers must evaluate these technologies in the context of their business SLAs and budgetary requirements and that the new technology will be hard to justify in many instances. Of course, the final specifications have not been announced and so it is possible that the actual product released will be different from the one announced today. I am curious to see the final product and how/if they address these concerns.

Related posts:

1. Musings on the Spectra Logic T-Finity Announcement
2. The Fallacy of Faster Tape
3. W. Curtis Preston on physical tape

1. How is metadata stored?

It is important to understand how the storage system is configured to protect this data. Simple questions like “how much storage is allocated to metadata?” can be vital since running out of metadata space is essentially the equivalent of running out of object space. (e,g, if you run out of metadata capacity then you cannot add any more items to object storage since this creates metadata.) I have heard of numerous scenarios with competing systems where customers run out of metadata space while still having ample object space available. This results in inefficient space utilization.

2. How is metadata protected?

Metadata is a critical part of the deduplication process and its loss or corruption can be very problematic. You should understand how metadata is protected. Are backup copies made? If so, how are they created, stored and recovered.

3. How does metadata storage requirements grow over time?

As mentioned above, running out of metadata space results in poor capacity utilization. You should understand how metadata is created and its growth over time. In many implementations, the size depends on deduplication efficiency and so the more data you backup that cannot be deduplicated (e.g. compressed or encrypted data), the more metadata space you will use. Replication may also require meta space and you should ascertain this impact as well.

4. What happens if the metadata it is corrupted?

Since this data is critical for the operation of deduplication, you should understand what happens if you permanently lose the metadata repository due to corruption or other problem. In extreme cases this could result in a complete system loss while in other cases the impact may be a reduction in deduplication ratios. The impact will vary by deduplication algorithm.

In summary, the creation management and handling of metadata is a key component of every deduplication algorithm. Many vendors prefer to gloss over the importance of metadata and focus on other areas such as deduplication ratios or post-process vs inline. The reality is that the management and creation of metadata is critical and each end user should evaluate how this effects various solutions and their business SLAs. In a future blog post, I will discuss how SEPATON addresses each of these four questions.

Related posts:

1. TSM and Deduplication: 4 Reasons Why TSM Deduplication Ratios Suffer
2. Deduplication and restore performance
3. Inline Deduplication: What Your Mother Never Told You

As data backup and recovery SLAs have become more stringent, end users have migrated rapidly to disk-based technologies.  Deduplication also adds value by reducing $/GB and required disk capacity although the technology can negatively impact backup and recovery performance.  These two trends have combined to reduce the requirements for physical tape and many tape vendors are seeing declining revenues.  This is not to say that tape is dead, it is very much in use and will be for the foreseeable future, but the use model has changed.  Physical tape is typically used for very long-term data archival where multi-year retention is not uncommon.

It is in the context of this changing market that Spectra Logic released the T-Finity.  It looks like an impressive piece of machinery, but it makes me wonder whether the market really needs it.  If anything, it appears that the market is more interested in mid-range libraries versus these massive units.  However Spectra Logic deserves credit for continuing to innovate in the world of tape.  Many vendors have limited their tape development efforts in order to focus on disk and Spectra Logic has not.

The feature set of the T-Finity appeals to my inner geek.  I have never seen a library with this level of technology although some might consider it gimmicky.  Here are some of the fun features:

• Multiple remotely controlled webcams inside the frame to allow for panning and zooming of the interior.
• LED light on the top that moves with the robot to show where it is at any given moment.
• Dual robots – if one breaks, the other shoves it out of the way and takes over.
• Complete remote management that allows for remote control and management of all features.

It certainly appears that the T-Finity ranks highly on the gadget factor, but what does that mean in actual operation?  Once the geek coolness recedes, the real question is how valuable are these features in real world use.

I believe that Spectra Logic has an opportunity with T-Finity even though the solution addresses a slow growing or even shrinking market.  Prior to the T-Finity, Spectra Logic primarily sold solutions that addressed small enterprises and below.  The T-Finity with its massive scalability targets a segment that Spectra Logic could not effectively address in the past and creates the potential for incremental revenue.  I wonder about the size of the market, but for Spectra Logic this represents a new revenue opportunity.

In summary, customers have been changing their tape use model.  Disk is now the primary target for backup and recovery in most environments and tape is used for long-term archival.  Tape is not dead, but the use model is different.  Spectra Logic has made a bold statement by releasing the T-Finity.  They are one of the few remaining innovators in a shrinking market, and have the potential to take market share from other vendors and improve revenue and profits.

Related posts:

1. Tale of the Tape: Musings on IBM’s 35TB Tape Announcement

Click for larger image

I find it amusing that just over a year ago, EMC was promoting the Quantum technology and said that “These offerings compete directly with Data Domain and are sharply differentiated by their superior, policy-based de-duplication capabilities, including de-duplication checks between local and remote sites, better scalability, Five-9s availability, and unmatched service and support.” (Local copy of the source, original source.)  EMC has since completed the acquisition of Data Domain and Quantum is now the pariah and Data Domain is the best thing ever.  Frank Slootman, VP of EMC’s backup group, said in the interview linked above, “We’re swapping a lot of those [Quantum-based] boxes out at zero revenue. We’ve taken out about a dozen and we’ll continue to take out a similar number this quarter. Customers don’t want it.” Wow, so much for sharply differentiated products with superior technology!  I feel sorry for end users who bought into EMC’s DL3D story and now have to deal with a forklift upgrade.

Related posts:

1. DL3D Discussion
2. Data Domain keynote at SNW – Slootman’s surprising response
3. EMC and Data Domain: It was the best of times, it was the worst of times