FalconStor Blog

Optimizing Deduplication Performance

Darrell.Riddle@falconstor.com (Darrell Riddle) — Fri, 27 Apr 2012 15:49:33 +0000

Regarding Gary Parker’s recent blog, Deduplication - The Power of Flexibility, Gary discusses the importance of data deduplication and the trade-offs among the various deduplication options that are available in the market. An interesting point was the comment that “for the highest performance levels, a recommended best practice is to use flexible deduplication policies to leverage post-process deduplication for the initial backup (for speed), and then switch to inline deduplication for subsequent backups.” I would like to expand on that because it is an important element of a good deduplication implementation.

Every deduplication method has tradeoffs, no matter which vendor is selected. Inline deduplication saves storage space because it does not require a staging area; but it usually is not as fast as post process or concurrent, because inline is a CPU intensive process that performs parsing, hashing, compression, and unique block look-up on the fly for incoming data.

While the CPU is performing all those calculations, some delay is added to the flow of transferred data. Thanks to new-generation processors, the impact can be limited by multiplying the number of cores available. As an open solution, FalconStor VTL has the benefit of letting you choose the server that best fits your performance needs.

On the other hand, post process (and concurrent) are faster because they are stand-alone operations. Post process just compresses and writes the data to disk as soon as they are received. Deduplication is performed after the backup operation is completed and does not impact the ongoing backup. Post is then the most basic and fastest way to back up data to disk.

So what is the problem with first backup and inline deduplication?

One thing to know about inline is that the speed is dependent on the deduplication ratio. The higher the deduplication ratio is, the less data has to be written on disk. Initial backups usually have a low deduplication ratio because most items are unique; however, the ratio improves as the subsequent backups contain much data that did not change since the last backup. Since the deduplication ratio of the first backup will be lower, the backup speed will be reduced.

This is why we recommend performing the first backup with post process for initial best performance, leveraging the optimized performance of post and avoiding inline initial performance impact, and then switching to inline. Thankfully we have the flexibility to do that without having to recreate the job and start all over again. That would be a nightmare!

Each of the deduplication processes are tools that can be used based on your needs. When you decide on which vendor’s deduplication solution to implement, you want to be sure that they include the full range of options (as FalconStor does); otherwise, your flexibility to craft a specific solution for your business and technical needs will be severely restricted.

Deduplication - The Power of Flexibility

Gary.Parker@falconstor.com (Gary Parker) — Thu, 26 Apr 2012 15:57:41 +0000

Today’s experienced IT staff with responsibility for storage know the importance of data deduplication and understand that there are trade-offs among the various deduplication options that are available in the market. It is important to select the type of deduplication that best meets their unique and changing requirements, so I will discuss these. It is important to note that these options should be available at the job level, so that IT staff can easily make adjustments for the many types of data they manage. To keep this blog reasonably short, we won’t attempt to define the types, but please drop us a note if you have questions on definitions.Inline deduplication has the primary benefit of minimizing storage requirements, reducing them by as much as 40 percent. It is ideal for small storage configurations or environments where immediate replication is desired. For the highest performance levels, a recommended best practice is…

Concurrent deduplication runs in a similar fashion to post-processing, but it starts as soon as the first set of records has been written and runs concurrently with backup. Deduplication engines start working immediately, making full use of available CPU resources. This is highly suitable for clustered VTL environments. Because replication starts sooner, data can be quickly recovered from a remote site.

FalconStor Turbo deduplication improves the performance of post-process or concurrent deduplication through a unique hybrid approach that leverages inline deduplication processes from within the VTL itself rather than the deduplication engine. FalconStor Turbo deduplication works best in large and clustered environments with sufficient CPU power to support inline pre-processing. FalconStor Turbo deduplication reduces the I/O required for deduplication, improving the performance of post and concurrent processing by as much as 300 percent.

No deduplication can be used on data that does not deduplicate effectively or data that is exported to physical tape. Examples include image files and data that is pre-compressed and encrypted. By selectively “turning off” deduplication, you can save deduplication cycles and focus on deduplicating the data that will yield the highest overall ratios and value.

As with most things in life, “one size does not fit all.” This is especially true for storage management and data deduplication. Take the time to understand your needs and then pick a product like FalconStor VTL Deduplication that does not handcuff you to only one type of deduplication.

FalconStor breaks the deduplication speed barrier

Darrell.Riddle@falconstor.com (Darrell Riddle) — Thu, 26 Apr 2012 14:25:06 +0000

Today FalconStor announced the latest version of its industry-leading deduplication solution, FalconStor Virtual Tape Library (VTL) 7.5. In performance tests with a four-node cluster, FalconStor VTL 7.5 demonstrated the fastest sustained deduplication speeds in the industry: more than 28 terabytes per hour with inline deduplication and more than 40 terabytes per hour with post processing deduplication. And an array of options, including inline, concurrent, and post-processing as well as Turbo capabilities, let you optimize performance for your environment. All this makes FalconStor VTL not only the fastest but also the most flexible deduplication solution on the planet! Beyond speed and flexibility, FalconStor VTL 7.5 also delivers greater scalability, efficiency, and security – all the right functionality to optimize IT infrastructure, effectively manage data growth, and align data protection processes with business goals. Bottom line: If you take the features that FalconStor provides – a broad selection of libraries and tape…

Data Deduplication – the Storage Industry’s Waste Management System

fadi.albatal@falconstor.com (Fadi Albatal) — Tue, 17 Apr 2012 20:05:32 +0000

Let’s call it what it is: data deduplication is the waste management system of the storage industry, and just as with any other waste management process, you really need your system to be very efficient. But to start, and just as with any other pandemic, let’s take a look at the symptoms of data duplication! The biggest duplicate producer in today’s IT world is the traditional backup process. Yes, I’m talking about the antiquated, passé, and totally broken batch backup process that produces more data than you can ever get any use for and way less than what you’d really need.

Traditional backup, not to name any, has few challenges in our modern IT environments. First is that addiction to copying data over and over again, leaving behind a very long trail of tape that can become very challenging to manage, or even to make sense of, even with the best catalogues.

Second, it’s the reason why we have a “backup window,” due to its thirst for CPU and network resources. It will very simply bring all systems to a halt during the backup process. An old friend use to say, if you want to know whether you have a solid network and IT infrastructure or not, start your backups and watch what happens.

By looking at the above symptoms, data deduplication seems to be a good remedy, but all deduplication solutions are not created equal. The choice should be based on the type of IT operations that you have. The performance of the solution will directly impact your backup window. Common sense would tell you the smaller the backup window the higher performance you need, but there is a bit more to that. Your deduplication solution should be able to support different types and techniques of backup operations. An example would be how effective the solution is at deduplicating multiplexed backup streams!

Also, since the deduplication process itself is a CPU-intensive operation, you should check whether you could exclude some data types from deduplication and only apply the process where it matters. It doesn’t make sense to try to deduplicate encrypted data, for example, as it’s pretty much all unique data. You also want to exclude some other data streams, such as archiving medical imagery or microscopic or telescopic data, that are not very friendly to the deduplication process.

And since we are talking about backups, despite the fact that data deduplication allows you retain data longer on disk resources, many organizations still have a requirement to backup data to tape. In that case, you’d want a solution that streamlines the process and integrates with tape infrastructure seamlessly. Having two separate infrastructures and performing backups twice, once to the deduplication target and another to a tape library, defeats the whole purpose of deduplication.

And to get back to the waste management analogy, in the greatest borough of all, Manhattan, during curbside collection days you see only garbage bags and no containers, and there is definitely a logistical reason for that. So as with any waste management system, while choosing your data deduplication solution, try to align it to your business and IT goals and operations.

Check out FalconStor’s next data deduplication announcement on April 26^th. You may find your solution there!

New Video - Don’t Wait. Automate.

webmaster@falconstor.com (FalconStor Marketing Team) — Wed, 04 Apr 2012 18:44:00 +0000

FalconStor has released a new video about the benefits of incorporating disaster recovery automation into DR plans. The video shows how DR automation is able to simplify complex recovery processes, ensuring a smooth recovery of operations after data loss or corruption, equipment failure, or even a complete site outage.In the event of disaster, businesses must perform dozens, if not hundreds, of steps to bring back their systems and continue with their essential business practices. A misstep in the recovery process caused by human error, a process flaw, a routing issue, etc., can delay recovery time objectives (RTO) by hours and cost exorbitant amounts in lost revenue.

That's where automation comes in. By investing in DR automation, businesses reduce the risk and uncertainty that traditional recovery processes carry. Not only does automation technology provide peace of mind, it reduces costs and saves resources by minimizing the time it takes to recover.

To learn more about DR automation, check out this white paper. It explains in detail how disaster recovery automation quickly works to preserve critical data – saving valuable time and money in the process.

Backup is Old School

Gary.Parker@falconstor.com (Gary Parker) — Wed, 28 Mar 2012 20:14:11 +0000

As Wikipedia notes, full backups had been the traditional approach to protecting large data sets, but the problem is that, in today’s high data growth and demanding 24x7 environment, full or even incremental backups take time that is just not available. Multi-tasking or multi-user systems will constantly be trying to send writes to data that is being backed up. The traditional approach to this problem is to temporarily disable write access to data during the backup, by quiescing the application orby having the operating system enforce exclusive read access. This works when regular downtime is acceptable, but 24/7 systems cannot bear service stoppages. To avoid downtime, high-availability systems may instead perform the backup on a snapshot—a read-only copy of the data set frozen at a point in time—and allow applications to continue writing to their data. In some systems once the initial snapshot is taken of a data set, subsequent snapshots…

These are often called delta snapshots and contain only changes from the original source data, and therefore do not consume a
significant amount of disk storage space relative to the original volume. Systems have now advanced to provide greater flexibility by maintaining up to 1,000 snapshots per volume for highly granular recovery points and enhanced business continuity. These snapshots can be maintained on any storage array in addition to the array where the primary data is found. Plus, backups can complete in a fraction of the time with no production impact. In fact, the notion of the “backup window” disappears as backups can be run at any time, even during the busiest production hours. The impact is diminished even further when heterogeneous devices can be treated as a common pool instead of needing to be backed up one by one.

For full application support, application-aware snapshot agents provide complete data protection for databases such as Microsoft SQL Server, Oracle, Sybase, and DB2, and messaging applications such as Microsoft Exchange and Lotus Notes, to ensure that snapshots are taken with full transactional integrity. Transactional integrity means that the database copy can be brought online without going through lengthy database rebuild processes or file-system checks, saving hours of valuable recovery time in the event of a disaster. Comprehensive transactional data
and integrity is attained through an automated process that takes snapshots of databases for point-in-time granularity and disaster recovery. The agents allow instant recovery to any point in time without requiring scripting. They also utilize a group snapshot feature to ensure transactional integrity of databases across multiple storage volumes.

Data protection has come a long way since the early days of full backup tapes!

CRN Designates FalconStor PartnerChoice a 2012 5-Star Partner Program

webmaster@falconstor.com (FalconStor Marketing Team) — Fri, 23 Mar 2012 21:26:01 +0000

FalconStor has been named to CRN’s 2012 Partner Programs Guide, and the FalconStor PartnerChoice program was awarded a 5-Star Partner Program rating. The 2012 CRN Partner Programs Guide is the definitive listing of manufacturers and software publishers that service solution providers or provide products through the IT Channel. CRN’s 5-Star rating recognizes an elite subset of vendors who give solution providers the best partnering elements in their channel programs.

The Power of Collective Intelligence

jim.mcniel@falconstor.com (Jim McNiel) — Wed, 14 Mar 2012 14:49:32 +0000

Last month, we conducted FalconStor’s annual sales kickoff meeting, an event in which we invest a considerable amount of time and resources at the beginning of each year. Why do we take the time and money to gather all our sales professionals in one place for a week? In two words: collective intelligence.One of our company’s key values is teamwork. The reason we believe in teamwork is the simple fact that the collective experience and knowledge of the group is far superior to the smartest human on the planet, let alone in the room. We gather to get smart. We gather to learn from each other and solve problems using a massively parallel system – our collective brainpower

This year we included some important new input channels: our key partners and our customer advisory board. We all understand garbage in, garbage out; but what happens when you feed the hopper with insightful gems from industry leaders? You get diamonds, or at least a collection of precious stones.

We had an amazing exchange of ideas. We continued to propagate new processes and best practices. America learned from Europe, Asia learned from America, we all learned from Asia and we all left the meeting smarter and more energized. I believe we are a much tighter, more globally aware company as a result.

Was it fun? Yes. Was it grueling? Yes. We delivered nearly 100 hours of content to 140 professionals. That adds up to about seven man years of work. I believe now we are all better armed and prepared to educate and support our global customer base.

Thanks to all who attended and special thanks to the key sales and marketing people who put together such a positive event. Well done.

Is now the time for massive adoption of storage virtualization?

fadi.albatal@falconstor.com (Fadi Albatal) — Mon, 12 Mar 2012 19:48:06 +0000

As a storage virtualization company “literally” at its core, FalconStor has been one of the pioneers of this technology since early 2000. Very few of those who first took on that challenge are still standing, and while server virtualization has taken the data center over like a wild fire, storage virtualization hasn’t seen massive adoption in IT organizations. Reflecting on 12 years of our company’s history may provide some answers, but I’ll save this for another blog. The fact is through these 12 years FalconStor grew to be one of the most respected and recognized names in the storage industry by staying faithful to it’s primary mission: simplifying very complex storage management processes while containing IT costs And our storage virtualization technology has been at the heart of everything we do.

Now the term storage virtualization has been abused by all storage vendors, many of them delivering some form of virtualization within the same storage array, which diluted the meaning of term. Lately the term “storage hypervisor” is being used to distinguish the high-level storage abstraction layer that spans across systems and storage protocols from inter-array storage virtualization.

The benefits of a storage hypervisor map to those of a server hypervisor: high utilization rate, storage-platform and vendor independence, enabling data mobility at the storage layer, simplifying overall storage management, etc. But this year we are seeing a resurgence of demand for a storage virtualization. This demand is motivated by three main factors, SSD adoption, Thailand, and finally an all-too-familiar but equally vague term, “big data.”

Solid state disk (SSD)– or as some people prefer to call it, solid state memory, flash, NAND, NAND flash, or any other term for super high- performance, extremely expensive storage – is all the rage. It seems everyone wants to have some flash in their data center, but the big boys, or the storage cartel as some refer to the big 5 (not me of course) are charging an exorbitant amount for SSD. Therefore people are looking around and buying the technology from new SSD startups that are popping up like mushrooms these days. The challenge remains that once you buy this expensive toy how do you fit it into your storage infrastructure? It’s great, but it doesn’t talk to anything around it! And this when the storage hypervisor becomes the knight on the white horse.

Thailand is where a lot of hard disk is manufactured. The humanly devastating floods in Thailand this year have had far-reaching effects on every data center around the globe. The disk shortage and the significant disk price hikes that resulted have had IT managers worldwide scrambling to keep up with capacity demand while trying to manage their already tight budgets. Being fully accustomed to the initial value proposition of server virtualization, higher resources utilization rate, they are starting to look for the same on the storage side. And again, the storage hypervisor is the answer.

And finally big data. Yeah, what about that? Well, the cartel has been singing the big data song very loudly; and you can bet that when the big sopranos, in the musical sense of the term, sing in a choir, the CIOs will listen. Big data it is, and big data machines we will buy. But that machine is now running live applications and running heavy load analytical tools and applications, which are bringing business applications to a halt. And here is where the storage hypervisor excels, (at least the one from FalconStor), allowing organizations to move or copy data to secondary resources and run all the analytical tools you can dream of in total isolation from production environments. And, by the way, we’ve been doing this for a quite a while for all the data junkies who have been running big data analytics for years before we called it big data.

Storage hypervisor! Thank you, again!

Dell to Create New Services Offering with FalconStor Data Migration Technology

webmaster@falconstor.com (FalconStor Marketing Team) — Thu, 08 Mar 2012 16:14:44 +0000

Dell customers are facing unprecedented data growth that is straining existing storage resources, and they need an efficient way to expand and upgrade legacy storage installations without disrupting operations. To address this challenge, Dell has entered into an agreement with FalconStor that allows Dell Services to employ FalconStor® Network Storage Server (NSS) technology for a new data migration service. The new service will allow Dell to effectively transition customers from legacy EMC storage systems to the latest EqualLogic and Compellent storage arrays. The comprehensive data migration capabilities of FalconStor NSS will allow Dell’s Data Center Deployment team to provide a safe, effective, and flexible tool to virtualize and migrate data in any host environment and between any storage systems.