StorageWonk

Buying A Storage Array? 5 Questions You Should Ask!

2010-09-13T10:51:00.008-06:00

Buying a storage array can often feel like buying a mattress. The product is the same but the labels are different so you're not certain you are comparing the same thing in the same way. There are areas in life you can scrimp on, but SAN storage and mattresses aren't among those. The choice you make will impact your daily life for years to come so buy the best you can afford or suffer the consequences. To help with that there are at least 5 questions, in no particular order, you should ask of each storage vendor you interview:

1. How far does the array scale and what happens after that? Any array that shares a back-plane, or has in-band controllers is limited in its ability to scale. Some will grow further than others but all have limits. Will you outgrow your array during its 3-5 year lifetime? If so, what happens when it is maxed out? If it means you will have to purchase another system have you factored that into the total cost of the solution? Too, when a vendor tells you how large a system will scale, do they have actual instances of this configuration in the field. More importantly, do they actually have a fully scaled system running in their test labs? It is surprising how many storage vendors publish maximum configurations for their arrays but have, in fact, never physically tested the configuration. If your vendor of choice has never physically tested their maximum configuration you have no reliable assurance it will scale as promised. Weird things happen on large scale implementations.

2. Is it future-proof? Storage arrays are subject to all the same obsolescence issues as any other computer technology. The problem is storage arrays are more expensive than your typical server so there is more at stake. Does the architecture of your chosen system lend itself to future-proofing? It's probably safe to say that no technology is completely future proof but there are architectures that are far closer than others. A storage array with a tightly coupled architecture such as a back-plane do not lend themselves to future-proofing in that their hardware is at the mercy of the oldest technological component. Modular systems that are only dependent upon their connectivity path are the most future-proof since connectivity (Ethernet, Fibre Channel, etc) change far less radically than hardware components such as bus speeds, processors, etc., and since modular architectures are loosely coupled you don't have to change out everything to take advantage of newer technologies, usually you can simply plug them into the architecture next to the legacy equipment. Oh yeah, one other thing in this regard, is there some hidden reason why you are getting such a screaming deal on the storage array under review? Could it be that the array is officially (or more likely unofficially) end of life? When is the next hardware refresh coming out? What is the upgrade path and price? Are you being sold old technology?

3. What does the system cost in total, including all licensing and maintenance fees for the 3-5 years of its life expectancy? Have you ever price-compared systems and found one vendor was substantially less expensive than the others? Or, perhaps one vendor was substantially more expensive than the others. Why the discrepancy? There can be several reasons. When a vendor is substantially less expensive than its competitors there is usually something missing from the configuration. The missing piece is often tied up in licensing fees and version charges that will be needed in the future but not at the time of purchase. This makes the system appear less expensive but in actuality it is a smoke-screen. Storage vendors, like the rest of us, have found it is easier to ask for forgiveness than permission and so they leave these items out to win the deal and hit you up for them later when you are stuck. The cure for this? Get total cost of ownership in writing for the expected life of the system. Have it include any and all future software licensing fees, all capacity fees, all maintenance fees. Stipulate that any omissions are the sole responsibility of the vendor to care for. Will the next hardware refresh require fork-lifting out your array? What is the cost to upgrade? Who does the work, you or the storage vendor/integrator? Does your quote include these cost projections? Can you opt out of the upgrade? Will the storage vendor sock it to you on maintenance in order to get you to upgrade or will your maintenance costs remain the same?

4. Of the raw capacity you purchase, how much data can you actually store and stay within vendor best practices? Capacity is a tricky area with storage arrays. Hard drive vendors calculate storage using the decimal system, operating systems calculate capacity using a binary method. The resulting difference is about 9% meaning that a 1TB hard drive will actually store about 9% less than 1TB (931.32GB); subtract another 20% for a typical RAID 5, 50% for RAID 1. Adding to the capacity pressure is feature, performance, and hot-spare requirements which further limit the usable capacity. To avoid the tricks played in this area, customers should absolutely insist on storage vendors supplying enough capacity to store the actual data the customer intends; after RAID; after feature hold backs; after hot-sparing; and with the level of performance expected despite the capacity being full (best practices “full” not necessarily physically full. Most vendors cannot perform adequately beyond 75%-80% of capacity after RAID and they will have a "best practice" target you should abide by). Stipulate to the storage vendor that they are responsible for supplying any additional hardware at no charge should the system not perform as promised and expected for its intended design and workload. You will find your quotes will become much more realistic and accurate minimizing the risk of your falling victim to a storage vendor's competitive pricing games.

5. What are the "Gotchas" that can come back to haunt me? Usually this falls into the category of "questions you should have asked but didn't know you needed to." Storage vendors know the most common mistakes made by customers that leave them upset and scrambling to rectify. Tell your vendor-of-choice you want to avoid this even if it means paying more up-front and ask them for guidance in avoiding the "gotchas." Storage vendors who speak often of forming a partnership, a trusted adviser relationship, should be willing to honestly help you avoid the pitfalls of not knowing what you don't know. They should tell you about the extra cost incurred with different rev's of software, the licensing costs beyond what you initially purchase. If you are getting a "sweet deal" from a storage vendor you should be aware of how that storage vendor intends on "getting well" financially in the future. Nobody gets something for nothing, not even you. Sales teams who are allowed to give you a "sweet deal" by their management have to have some plan for making up lost revenue. Find out how they are going to do it and make sure you are okay with the plan.

Ginormous Hard Drives Part 2--A (DIF)ferent Storage Array Required!

2010-07-25T21:42:00.001-06:00

My last post addressed issues encountered when hard drives, and especially ultra-large hard drives, vibrate too much. Another issue particularly concerning with ultra-large hard drives is something called an unrecoverable bit error (UBE). Although rarely discussed with potential customers and definitely a topic "down in the weeds" UBE is a problem anyone working with ultra-large hard drives and RAID sets will eventually have to deal with. Why? Because as drives get bigger and bigger, RAID striping puts the end user at risk for having to read more blocks of data than the hard drive UBE can accommodate reliably and the rebuild from parity won't complete thus compromising the entire RAID set.

So what is UBE? All hard drives have unrecoverable bit error ratings (UBER) which indicate how many bits a drive can likely transfer before encountering an UBE. Normally a drive can cope with bit errors and life moves on as usual. But when a drive encounters an unrecoverable bit error, bad things happen. SATA desktop drives have a UBER of 1 in 10^14 or 12.5TB. A 2TB drive equals roughly 1/6 of this UBER. A 5 drive RAID set would equal 5/6 of the UBER. Should a single drive fail in this RAID configuration there is an 83% probability that the RAID set will experience an UBE during rebuild. Although these drives are not the standard in higher-end drive arrays, some are making their way into the datacenter in an effort to get prices lower.

Most array vendors will push "cheap and deep" near-line SATA or SAS drives with a UBER of 1 in 10^15 or 125TB. This increased UBER would mean that our 5 drive RAID set mentioned above would have an 8.3% probability of experiencing an UBE during rebuild thus preventing successful RAID recovery. Even though the relative risk is lower, imagine telling your management (or regulator) that the corporate data has a 91.7% chance of surviving a single drive failure on that new, expensive RAID array you just convinced them to buy? Probably wouldn't have the effect you were looking for (unless, of course, you were wanting another job and then it might have just the effect you were looking for!)

So what is the answer? Well, for one thing, quit thinking that all hard drives are equal and the only thing that matters is the cost. They aren't and it isn't. Make sure the drives your vendor is quoting meet the criticality of data they will be storing. If you insist on using desktop class drives use RAID 10 instead of RAID 5 (although this added capacity overhead may negate the cost savings of the lower-end drives). Second, make sure your storage vendor subscribes to the ANSI T10 Data Integrity Field (DIF). DIF is a standard that provides end-to-end data integrity. As one writer put it, "T10-DIF provides a logical block guard to compare the data actually written to the disk with what is supposed to be written, a logical block application tag to ensure that the data is written to the correct logical unit and a logical block reference tag to see that the data is written to the proper virtual block."*

In other words, DIF first utilizes cyclic redundancy check (CRC) to make sure that raw data is mathematically the same between the source and destination. CRC has been used for years and as most people in this business know, if the binary sequence (CRC Code) calculated at the destination calculates differently than the source an error is deemed to have occurred and the data is usually written again. Where CRC falls short is that it only guarantees the data was written correctly, it doesn't guarantee it can ever find the data again. DIF addresses this weakness. DIF ensures that not only has data been accurately written through CRC, but that it has been written to the correct folder and to the correct file. DIF makes certain data will always be retrievable thus going a long way towards mitigating the UBE problems of hard drive technology.

As one of the original companies to sponsor the Data Integrity Initiative (DII), Seagate incorporated ANSI T10 DIF into their Intelligent Storage Element (now exclusively offered by Xiotech as the ISE). The ISE is one example of an intelligent array that has been built to make sure the world's "love affair" with "cheap and deep" hard drives doesn't end in a nasty divorce (and there are others). Make sure your vendor is prepared for and has incorporated this standard into their drive array 'cuz the 3TB drives are coming and the problem will continue to compound.

*(RAID 6 or double parity RAID can also help but it suffers from a double write penalty that can seriously impact performance. It's not a bad option but it addresses the symptom of the problem versus fixing the cause of the problem).

--StorageWonk

Ginormous Hard Drives Part 1-Why Drives Should be Frozen in a Block of ISE

2010-07-20T08:12:00.005-06:00

For the customers that buy them and the people that sell them, large hard drives are a thing of beauty. "Cheap and Deep" is the mantra that calms the nerves of those trying to accommodate an era of surging data and dwindling budgets. On the other hand, for those technical and engineering types that actually make their living working with large hard drives (or more accurately, make their living making large hard drives work) the impression can be quite different.

For those of us who, in 1980, marveled at the immense size and elegance of the first microcomputer hard drive (Seagate ST506; yes, I know...IBM...1950's...1973...3340...etc) and couldn't figure out how we were going to ever fill all those 5 glorious MB of spinning magnificence, the world and hard drives have become a very interesting and different place.

Who would have guessed that over the next 30 years that 5MB hard drive would grow to the 1TB and 2TB drives commonly used today? Reason to celebrate? Perhaps, but it is also a reason to take a moment for a sanity check. As with everything else, technological advances also have their pitfalls. Fortunately if we are aware of them we can take steps to protect ourselves and those who employ us.

VIBRATION

A current 2TB Constellation ES drive from Seagate stores data to the tune of 240,000 tracks per inch. As a point of measure, your typical 20# copy paper measures out at a whopping .0038". That works out to be about 900 tracks sitting on a space the size of the edge of a piece of paper. A hard drive head reads these tracks one at a time so in order to read or write your precious data it has to accurately target 1/900th the thickness of a piece of paper every time it goes to work. To make matters worse, your typical hard drive, encased in a drive shuttle and strapped in next to other drives turning in the same direction can vibrate these heads like a 1980's disco (relatively speaking). It doesn't take much movement to knock a head off 1/900th's the thickness of a piece of paper making it miss its target and forcing it to wait for the next pass. Miss the target enough and seek times go up, performance dies as illustrated by the above graph.

But is this a real problem or just a mechanism for spreading fear, uncertainty, and doubt (FUD)? Check out the graph below of 33 drive enclosures on the market today and their relative rotational vibration (RV) measured in rads/sec². Of the 33 drive enclosures tested, 11 were unacceptable for any drive, 21 were unacceptable for Nearline drives (the cheap and deep drives you usually buy), and 28 were unacceptable for desktop class drives. Which enclosure are you using and where did it fall in this test? Since the manufacturers weren't listed you'll have to ask.

Enterprise drives tolerate the most vibration because of special technology that helps counteract head drift. Works pretty well too (but you've got to pay for it). Nearline drives are cheaper but they don't handle vibration very well (that's part of the pain you feel long after the joy of saving some money has dissipated).

As drives continue to inflate in capacity and increase in areal density enclosure vibration will become more of a problem to drive and data reliability. Architectural changes to how drives are placed can help reduce RV interference from other drives. Avoiding loose fitting drive shuttles also helps. Self-Healing drive enclosures like Xiotech's Intelligent Storage Element (ISE) utilize these and other techniques developed by Seagate to minimize drive vibration well below conventional drive enclosures (in the neighborhood of 2 rads/sec²).

Why not ask your storage vendor how many rads/sec² their drives vibrate? The dumb look on their faces might just be worth the extra money you'll pay for better drives.

Next blog--Ginormous Hard Drives Part 2-A (DIF)ferent Storage Array Required!

--StorageWonk

Block-based Automatic Data (BAD) "Tearing"-Why Automatic Data Tiering from a Block Storage Device is a BAD Idea.

2010-05-24T11:57:00.009-06:00

Hey, I've got an idea-Let's move all the highly utilized data to the fast, expensive storage tier, and we'll move all the infrequently used data to a lower, cheaper storage. You'll save money through efficiency and its all automatic? Sounds great right? Who wouldn't jump at a deal like that? But as with everything, the devil is in the details.

First, How Does Automatic Data Tiering work on a Block Storage Device?

The premise is fairly simple. A block storage device checks once per day to see if a block has been updated. If it has been updated two or three days in a row the block is deemed "important" and is moved to a higher tier of storage and/or a higher performing RAID type. If a block is not updated once within a 12 day time frame, the block is deemed "less important" and moved down to a lower performing tier of storage and/or a lower performing RAID type. Sounds pretty effective.

So where is the rub?

Block level devices have no awareness of data and data-value. They simply see all things as blocks of storage. Those blocks either have data on them or they don't. The only thing a block storage device can tell is if the block has recently been updated, nothing more. Block-based Automatic Data (BAD) "tearing" has no method for determining the inherent value of the data. Why is this a problem? Well, consider those unauthorized .mp3's of the Beatle's "White Album" (or Don Ho's Greatest Hits if you prefer ;-)) that have been secretly uploaded to your SAN for the enjoyment of the listening employee base. These files, although unauthorized, are accessed every day. Where do they end up? On expensive Tier 1 RAID 10 storage. What about your Oracle payroll application? Well, assuming the really important parts are accessed once every two weeks for payroll, most, if not all of this application will find itself on Tier 2 or Tier 3 storage, RAID 5 or worse. Now, you may think "that's not a big deal because my storage vendor told me it will move back up to a faster tier and RAID type as it is being used. Right?" Well, yes, but you have to use the blocks three days in a row (systems only check the block once per 24 hours) before it will move up again. How long does your payroll last? By the time it your data has moved up in Tier/RAID your payroll is done (or actually, since your database is on such slow storage, maybe it isn't done and you have a line of people at your door looking for their paycheck?).

Another problem with this approach is that nobody's data is perfectly balanced between "often used" and "not often used". The net result is one of two scenarios: 1) you use more of your data blocks than you think, in which case it all gets crammed into your Tier 1 storage (well there goes your thin provisioning benefits), and leaves your tier 2 and 3 storage relatively unused (so now you have too much Tier 2 and 3 and not enough Tier 1, guess you've got to go buy more-too bad licensing cost is based on disk count); or, 2) you don't use very much of your total block pool at all, and so everything gets crammed into your slow, tier 2 and 3 storage, and leaves your expensive Tier 1 storage spinning in the wind (now your system is slow for at least three days in a row and heaven help you if your usage is up for two days and low the third. Got to start all over). Now admittedly these settings can be changed from their defaults and it is possible, with a little effort, to tune your environment. But this is a manual process and does not automatically re-tune itself to match your changing business demands over the months and years.

Clearly this is a worst case scenario and storage vendors will tell you there is an easy solution to this problem. You simply turn BAD "tearing" off for your critical applications. Yes, that valuable software that was going to solve all your efficiency problems needs to be turned off (not a big deal for the vendor since he already has your check for the software). But if you have to turn this expensive piece of software off then why have it in the first place? Doesn't that really highlight the simple truth that BAD "tearing" is a great marketing tool and may even work fine for small shops, but it is not the correct solution for anything approaching the enterprise? Agreed, the concept is good, it is the execution that is limited-in this case, by the block storage device itself.

Now before I get all kinds of comments from champions of Compellent's
Data Progression", or EMC's "FAST", or NTAP, or 3PAR (among others) let me say this is not meant as a slam to those companies. Although I compete with them every day I've also come to respect them for what they do. This paper is simply to point out that, although these companies offer BAD "tearing" in the only way they can given the architectural limitations of their systems, BAD "tearing" alone simply isn't the proper way to accomplish the task. Rather, software that has the ability to see data as the filesystem sees it and allows company management the power to arbitrarily assign value to data regardless of its daily access is, by definition, true Information Lifecycle Management (ILM) as defined by the Storage Network Industry Association (SNIA).

Policy-based Automatic Data (PAD) tiering is the correct way of moving data from one tier to another because it allows your company decision makers to tag business critical applications as, well, business critical, regardless of how much or little they are used. It also allows you to tag certain data types as low priority (or unauthorized) regardless of how often they are accessed. Thus, the "White Album" and "Don Ho's Greatest Hits" won't find a happy place on your expensive SAN storage no matter how popular (or not for you Wayne Newton fans) they are.

So do yourself a favor and look at automatic data tiering the way Xiotech and many others see it (using "best of breed" PAD tiering solutions) and forget the BAD "tearing" approach. You'll save yourself money, time, and headaches as you mange your data with tiers and not "tears".

--StorageWonk

The Real Skinny on Thin Provisioning

2010-05-18T08:33:00.001-06:00

Clearly one of the most popular storage features to-date, thin provisioning (provisioning what an application or server "thinks it needs" while physically allocating only what it actually needs) seems for many to be the answer to saving money by limiting over-allocation of storage. Over-allocated storage is "stranded storage" and is a waste of money and resources. Thin provisioning promises to free up this "stranded storage" making it available to other applications, servers, or even delaying the purchase of the storage until a later date; all attractive options in our frugal economy. But along with its benefits, there are some important considerations to keep in mind so that the "Thin" in Thin provisioning ends up on your data storage and not in your wallet.

Arguably there are two key factors driving the over-allocation of storage. First there is the lead time it takes for an application owner to acquire additional storage when he runs out (it may be as simple as making an internal request for more capacity but all too often it necessitates the approval and purchase of additional storage resources). The second key driver for over-allocation is the difficulty in actually sizing the amount of storage an application will need, especially with rapid growth environments such as email, database, and server (VM)/desktop (VDI) virtualization. Sometimes we calculate (or guess) wrong and end up stranding precious resources behind servers or applications that are not using them. So we reason, "if I can just pull back the stranded storage through a mechanism like Thin provisioning I can save money and cost-justify the acquisition." The theory is appealing and certainly makes sense but beware, pulling back over-allocated storage is not as straightforward as you might think.

There are some things to consider:

1. How will you migrate a "thick" LUN to a "Thin" LUN? Migration tools and techniques differ. Most simply copy blocks in a volume from one system to another without making a distinction as to whether those blocks contain data. Some Thin systems allocate storage whenever they detect a block has been updated so that 500GB of data sitting on a 1TB volume ends up migrating as a 1TB volume. It is true that there are systems out there that can detect blocks that have no data and may be able to reclaim or not allocate storage to these blocks during migration which would certainly put you in a better position to achieve a net savings. What about the system you are looking at? How does it migrate data?

2. Will you see the cost savings you are expecting? Even though, as mentioned, some Thin systems can reclaim blocks that are empty, a bigger and more serious problem to the bottom line occurs when a Thin storage system has to deal with blocks that were previously used. Why? Because the physical allocation occurs when the file system needs to store data. It does not become "unallocated" again just because the file system deletes the file. The reason for this is because when a file system deletes a file it doesn't "null-out" the block used for that file (as this would be prohibitive from a performance standpoint), instead it marks the block as "no longer used" and it ignores the data on it until it is eventually overwritten.

Unfortunately the block storage device still sees it as full of data (because it is) and it has no clue that the block is now viewed by the file system as "empty". How effective your Thin storage system is at saving you money depends on what percentage of previously used blocks your volumes contain. In addition, some file systems are more efficient at reclaiming previously used blocks than others and thus are considered "Thin friendly." Unfortunately one of the most widely used (Microsoft's NTFS) is not among these. To make matters worse, NTFS likes to use fresh new blocks before recycling previously used blocks. It is true that some Thin storage vendors do provide applications that will bring the previously used storage "to the front" so to speak, thus making NTFS use it sooner than later, but from a Thin perspective the damage is done. The block, though no longer used, is still allocated just like it was before you bought your Thin storage system. If your volumes are made up of a high percentage of previously used blocks and your file system is not "Thin friendly" your savings will not be what you are expecting or hoping for. This doesn't mean Thin is bad, it just means you have to factor this in to your cost justification model when considering a Thin storage purchase-something most do not do.

3. What will your data migration experience really be like? If your data migration tool is not "Thin friendly" and it copies all the blocks it sees regardless of whether or not they are empty, or if your file system is not "Thin friendly" and splurges on how it allocates and maintains blocks to applications, your migration experience will likely be difficult, especially if you are intent on seeing any real cost savings.

4. What about "Zero Reclamation" technology? This technology is offered by most Thin storage vendors and potentially may help. The concept is that any contiguous span of storage that is all zeros will be reclaimed. So if the volume you are migrating is 40% full and the remaining 60% contains all zeros you can reclaim that 60% of data to the unallocated storage pool. However, like anything else, there are caveats:

1. Although Thin storage vendors claim high recovery rates (30%-40% or more) the recovery rate is entirely dependent on how much of the allocated space is actually zeros. This would typically be over-allocated space that was carved out of the storage pool but not made part of the volume or formatted with zeros and not overwritten. It should also be noted that this reclamation is typically a "one-time only" event since it is highly uncommon for a database or file system to overwrite used blocks with zeros (again for performance penalty reasons) so the zeros you see at migration are usually the only ones you get. Your savings disappear as time goes on.

2. Zero reclamation technology will only reclaim space IF it is marked with zeros, a situation that is not typical. In most cases, unused storage space either has old data from previous use, or random bits of data that float around on most hard drives. The point is, no zero, no savings.

3. To make matters worse, Zero reclamation requires an entire "chunk" or "segment" (size of a piece of a RAID stripe on each disk x the number of disks) be zeros. If even one bit is not a zero it's a no-go. Chunk size varies among array vendors and application (HDS USP-V is reported to utilize a 42MB chunk, EMC is at 768KB based upon a 64KB Block).The larger the chunk the less likely it is to be all zeros.

So even though most Thin storage vendors try to be efficient in the capacity they reclaim there is only so much they can do with widely varying end-user environments. You will likely still have to resort to methods such as re-loading existing applications and data from your backup environment or, if this is too risky, you may have to be satisfied with targeting only new applications to your Thin storage environment. Either way your cost goes up considerably due to management overhead, risk, and time. A pretty disappointing reality check in the presence of so much enthusiastic marketing hype.

So what is the answer? When considering the purchase of a Thin storage system you should discuss the options for solving the problems above with your storage vendor. Fortunately, there are software solutions that can help mitigate the problems of block-based Thin provisioning, but most Thin storage vendors don't broach the subject since the software can add significant additional CAPEX cost to an already expensive hardware solution. So if your storage vendor doesn't bring it up, do yourself a favor and ask. If the storage vendor you are speaking to cannot (or will not) adequately advise you then look somewhere else. If they say it's not necessary ask them to prove it (they have to have a file system view of storage rather than just a block level, to establish and maintain Thin savings efficiency). The simple truth is, without specialized software to mitigate these problems you will not see the returns you expect to see in utilizing Thin storage. Without factoring in the cost of this software, you cannot accurately assess your savings, if any, by utilizing Thin storage for your production environment.

Thin provisioning can be an excellent tool for solving some of your over-allocation problems. It can be especially useful in development and test environments which are temporary in nature. But switching from a "thick" to a "Thin" environment for general, long-term production is not as straightforward as some would have you believe. Without additional software to bridge the gap between block and file your cost savings may not be what you expect it to be nor what your storage vendor is telling you.

Sources:
http://www.symantec.com/connect/articles/thin-provisioning-without-symantec-not-good-idea "Read this before buying a thin provisioning system 1.0"
http://www.xyratex.mobi/pdfs/whitepapers/Xyratex_White_Paper_RAID_Chunk_Size_1-0.pdf
http://www.zdnet.com/blog/storage/chunks-the-hidden-key-to-raid-performance/130

http://www.symantec.com/connect/blogs/difference-between-storage-foundation-smartmove-and-hardware-based-zero-reclaim-solutions

http://thestorageanarchist.typepad.com/weblog/2009/12/2033-v-max-is-much-more-than-fast.html

http://thestoragearchitect.com/2009/12/11/enterprise-computing-has-emc-slipped-zero-block-reclaim-into-v-max/

https://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/entry/does_size_really_matter_for?lang=en

--StorageWonk

Removing the Performance Stranglehold from Virtual Servers

2010-05-14T21:38:00.003-06:00

It is ironic that in the 21st century, despite having the fastest processors, the largest cache pools, and the most powerful applications, server performance, both virtualized and physical, would still be constrained in our data centers by the storage foundation they have been built upon.

It has been discussed in past blogs about how processor performance increased 30-fold between 1996 and 2005, while hard drive performance increased only 1.3x during the same period. And it is true that mechanical hard drives will never match the performance gains of a microprocessor; but it is equally true that inefficiencies in RAID, caching, controller firmware, and error detection have formed an unnecessary stranglehold on storage causing it to remain far slower relative to performance than it has to be.

This stranglehold has resulted in the proliferation of Performance Starved Applications (PSA) in the areas of databases, email, high performance computing (HPC), and server virtualization. Even now application and OS developers are struggling to find ways to improve the way they lay data onto physical disk so as to enhance performance but there is only so much that can be done. Until the storage industry does some serious "out of the box" thinking storage will continue to be the "boat anchor" dragging the rest of IT down.

Xiotech's ISE, developed by the Advanced Storage Architecture Group formerly at Seagate Technologies and now a part of Xiotech, is a major effort to "think out of the box" and find ways to remove these inefficiencies freeing up the full potential of today's fast spinning hard drives.

ISE’s “Redundancy Allocated Grid System” (RAGS) striping methodology eliminates complex algorithms of traditional RAID controller technology in favor of creating a three-dimensional grid across all heads inside all hard drives contained within a DataPac drive assembly. This allows for more efficient data movement via a flat lookup table versus the traditional way of mathematically tracking data required by conventional RAID devices. Combined with its patented cache algorithms RAGS makes the ISE “wicked fast” in both IOPS and MBps and it does so even at 100% full without losing performance, a trick which simply cannot be duplicated by conventional storage arrays which lose performance steadily as a hard drive fills, nor solid state drive (SSD) manufacturers who typically limit capacity to no more than 50% full due to wear leveling.

The Xiotech ISE architecture can have a major impact in the world of server virtualization. Virtual machine applications such as VMWare, Citrix Xen Server, and Microsoft HyperV all have unique performance requirements that conventional storage struggles to fulfill. All of these companies are working feverously to mitigate this bottleneck through better software mechanisms. But by providing, in many cases 3x the performance per drive even at 100% full, Xiotech ISE provides unmatched performance and usable capacity through smarter hardware technology. Not only will it offer today’s hypervisor applications immediate relief from the boat-anchor storage they are accustomed to running on, but it provides an enhanced platform to leverage all the secret performance mojo and pixie dust hypervisor companies are creating to speed up their present and future applications.

Want some real numbers to chew on? Check out the test results at www.storagehacker.com and see for yourself.

--StorageWonk

"Why Can't Storage Just Simply Plug into the Cloud?"

2010-05-09T14:51:00.006-06:00

The lament above came from a potential customer, frustrated that storage continues to be one of the most difficult parts of datacenter planning. I thought it would be a good topic to consider especially since "cloud" seems to be the newest important buzzword of the day.

The simple answer to the question above is that storage has always been proprietary and monolithic in its design. Although storage vendors have made doing things within the confines of their storage easier as time has gone by the truth remains that the storage silo architecture is not conducive to the "Cloud" concept.

The proprietary nature of how storage is managed is another reason why storage can't "simply plug into the cloud". Not to say there haven't been attempts to remedy this fact. The Storage Network Industry Association's (SNIA) Storage Management Initiative-Specification or SMI-S was just such an effort. It was meant to "define a method for the interoperable management of a heterogeneous Storage Area Network (SAN)". Unfortunately for the end user having a method for interoperable management of heterogeneous storage is not in the business and financial interests of the storage industry. Why not? Because the only thing that differentiates one chunk of storage from another is its wiz-bang software and ease of management. The hardware is almost identical and so a storage company can't really differentiate its product (read, make you want to buy its product) based on hardware alone. So, it has to come up with ever new and exciting software features to attract you to its side of the table and keep you there. Having a method for interoperable management of heterogeneous storage nullifies this one differentiator and so the concept, though well intended, really had no serious chance of success. Oh there has been lip service to it from some vendors but nobody truly supports SMI-S in its complete form. Bet that SMI-S will quietly go away as another good idea that failed.

Does this mean there is no way we'll ever have a storage array that can simply "plug into the cloud"? Not necessarily, but it won't happen until storage vendors come up with a better way to differentiate their products apart from their management and software features. Some vendors are already trying to move in this direction but they are still too reliant on old technology and a proprietary interface.

So far, there is only one player in the market today that offers a radically different hardware architecture based upon spinning hard drives. That architecture, designed by the world's leading hard drive manufacturer Seagate Technologies, differentiates the company enough to allow it to stay in business without focusing all its efforts on proprietary software. In fact, it is designed to allow other companies to control the hardware if they wish through API's called "CorteX". CorteX can facilitate a symbiotic relationship with other vendors' software thus allowing the end user to build a cloud using ISE storage and manage it with just about any software that tickles their fancy. It even allows you to leverage the features already built into most enterprise class software (Thin Provisioning, Snapshots, hot backups, Data Migration, Deduplication, and others found in Symantec, Oracle, Microsoft, VMWare, Citrix, etc.)

That company is Xiotech and the storage product is the Intelligent Storage Element (ISE). You've never seen anything like it. Typically three times faster than conventional disk (even at 100% full, no short stroking required), capable of re-manufacturing drives in-situ and placing them back into service without human intervention, a five-year, four-hour, 24x7x365 hardware warranty included, and managed entirely through representation state transfer (RESTful) web services. REST is the language of HTTP and the cloud. Storage that understands the language of the cloud simply plugs right in. No muss, no fuss. No other storage vendor does it this way (yet). Why? Because they don't have the ISE to differentiate their product line. Want to learn more? Talk to your favorite storage integrator or contact Xiotech direct at www.xiotech.com.

--StorageWonk

What Makes Storage "Cloud Ready"? Take 2

2010-04-28T09:57:00.005-06:00

A couple weeks ago I posed a question of what makes storage "cloud ready"? It was met with a resounding silence. At first I found that a bit odd since I considered it to be a reasonably neutral question that would generate a bit of discussion on the value of one architecture over another. Given there were a few hundred visitors to the page I just wasn't expecting "crickets" but, alas, that's what I got. However, as I considered the question anew it occurred to me that it would probably be a tough question to answer if I was championing a traditional storage silo, because for storage to be "cloud ready" it has to be flexible, fluid, and dynamic, like the cloud itself. Now there are storage vendors out there that pitch their silos as flexible, fluid, and dynamic, but you quickly find that they are only flexible, fluid, and dynamic as long as you stay within the confines of their silo. The minute you step outside the capacity of one of their silos you find their systems to be anything but "cloud ready."

An architecture that requires you to purchase an entirely new storage silo when the first one has reached its functional limit is not a practical design for a dynamic architecture such as the cloud (both internal and external clouds). But neither is moving backwards to an earlier time of direct attached storage (DAS) the answer. The truth is, we need a hybrid system, one that provides the benefits of centralized management that has made SAN storage popular, with the lower cost of the next terabyte of storage that made DAS an economical choice. However, please keep in mind the cloud demands more than modularity because there are several designs on the market today that are modular but not necessarily "cloud ready." To be "cloud ready" storage has to be nearly infinite in its ability to scale (both in terms of capacity and performance), automatic, and invisible to the end-user. "Cloud ready" implies a certain level of automation and dynamic growth that requires hardware to once again become the servant of the application because the cloud is all about applications and virtual server environments (VSE). This requires a cloud ready storage array to speak the language of the application and of the Internet through non-proprietary, open standards such as Web services and Representation State Transfer (REST).

A storage subsystem architecture based on this premise would allow practically limitless scale starting from a system as small as a terabyte or two, and it would allow the cloud to control the provisioning and de-provisioning of storage without requiring human intervention or huge expenditures to provide the next terabyte of storage beyond that which is currently owned. It would allow applications and servers to self-tune based upon actual needs such as a server automatically and temporarily provisioning additional storage for "end-of-month runs" or an Oracle or SQL database automatically moving a LUN to a group of storage modules that had more resources to meet the moments demands, and then resettling the LUN to lesser performing storage as demands dropped. And before you say we already have this feature, understand this goes far beyond the crude dynamic tiering offered by some arrays because it looks at real-time need and adjusts immediately rather than simply looking at whether a block was accessed during the past 24 hours. It differs from auto-raid types of technology in that the resources for this movement are multiplied across the entire spectrum of storage rather than relying on two controllers to effect the move (which they might be too busy to do at any given moment). Because it is application driven, it addresses data movement based upon data value rather than last access. Last access doesn't always add up to data value as any storage admin with unauthorized .mp3's on his network will attest.

So what makes storage "cloud ready"? It has to be modular, it has to be intelligent, and it has to be seamlessly controlled by the applications and VSE's. Until storage vendors figure out a way to build these three things into their array architecture their systems will never truly be "cloud ready" and the end-user will continue to pay the price.

--StorageWonk

What Makes Storage "Cloud Ready"?

2010-04-10T06:37:00.002-06:00

If your day is like my day, you probably spend a good bit of your time visiting with end-users about "the Cloud" and how this architectural concept will impact their datacenters going forward. Now software as a service (SaaS) seems to intuitively make sense in that being able to use (and pay for) a license for an application only when you need it has a certain charm about it. Virtualizing servers according to demand also makes sense since this is essentially a software function. But how do you do the same thing with hardware, namely, storage? How do you make hardware as a service (HaaS) as viable as SaaS? Is it realistic or not?

In an effort to generate some healthy discussion on this subject, I'd like to get your feedback on what it is that makes a storage array "Cloud ready"? Perhaps this just another buzzword that doesn't mean anything? As competitive as the storage industry is let's try to keep the "religious debate" to a minimum and discuss it from an architectural perspective. For example, if you work for, or use, a storage manufacturer that sells a more traditional "storage silo," how is this approach better than another for HaaS? What is it that your chosen storage vendor does to make you feel their approach is the logical choice in the context of "the Cloud"? If your preferred storage vendor is more modular in design and architecture, what do you feel gives this architecture the edge in "the Cloud" over a more conventional approach? And if you're big on direct attached storage (DAS) for the Cloud, why is going back to this approach preferred over SAN/NAS? Finally, perhaps you feel they are all valid architectures for the Cloud and it doesn't really matter which approach is used. Why do you feel this way?

All opinions are valid and welcome. Care to weigh in?

StorageWonk

Storage Forecast-Clouds and SUN with WIND, RAIN, ISE, HAIL, and SNOW Likely!

2010-03-30T22:26:00.003-06:00

I was reflecting on the recent announcement from Xiotech releasing their new Intelligent Application Storage (IAS) initiative and how it would play into the Cloud design being bantered about by the industry of late. If you are not familiar with the Cloud concept it has to do with designing IT infrastructure where everything (software, hardware, storage, etc.) becomes a service rather than an island unto itself. The concept behind the Cloud makes sense--to design our infrastructures so that user and organizational demands control resources rather than letting resources control demand as they often do. But what changes will the Cloud mean for the industry in general and storage in particular? The forecast calls for stormy weather and that's not a bad thing!

Up to this point, storage networks have been fairly conservative in their design, with Fibre Channel, iSCSI, NFS/CIFS/NAS, Infiniband being the mainstay's and Fibre Channel Over Ethernet (FCoE) on the horizon. The Cloud concept blows the conservative design book out of the datacenter by allowing the WIND (Wildly Independent Network Designs) to blow (don't groan yet, it's going to get worse). The Cloud concept minimizes the importance of how we connect to storage because storage simply becomes a resource that is provided as demand swells. Storage providers who embrace the SUN (Storage as a Unified Network) will shine in the cloud environment (OK, start groaning but save some for later) because they won't care which protocol or cables you use to tie into their storage.

But the real power of the Cloud will come from sharing and utilizing the intelligence found at the application and virtualization layer. This will require a big change for most storage providers since they have a long history of building proprietary interfaces and feature sets that have not utilized the language of the application such as Web Services and REST (Representation State Transfer) to provision and control storage. Their monolithic storage silo mentality has led to a slew of proprietary features that can't be shared by other vendors and applications and they end up adding complexity and cost and in the worst case, provide little to no value. For example, if you have a SAN, a NAS, VMWare/Citrix Xen/Microsoft HyperV, and a backup solution, you likely have four versions of thin provisioning, four versions of snapshot, and three versions of deduplication. Which one will you use? It typically doesn't make sense to use them all together so you end up picking one or two and shelving the rest, not exactly money well spent.

The storage industry desperately needs to embrace the storm and refocus on the future. You can't keep adding application features to storage arrays forever and you can't put MUD (Monolithic Unwieldy Dinosaurs) in the Clouds. The long-term solution is to add ISE (Intelligent Storage Elements) and where there's ISE in the Clouds can HAIL (Hardware Application Intelligence Layer) be far behind? HAIL would make it possible for storage vendors to plug their hardware directly into the application and virtual world of the Cloud (ok, let the tears begin). With HAIL in place and hardware to cooperate with it, you can buy a version of the features you want from the vendor of your choice, and then all the other applications and hardware in your Cloud can make use of this one version of your feature without requiring you to purchase the same feature from every provider. This refreshing RAIN (Real Application Intelligence Networking) makes the concept of the Cloud a reality instead of just a tool to make an audience glaze over. This is the concept behind Xiotech's IAS initiative.

Storage providers are in an especially good position to embrace the concepts of RAIN and HAIL since they make SNOW (Storage Networking Over the WAN) a heck of a lot easier to do than it is now. Although HAIL and RAIN may wipe out the current crop of proprietary storage systems (now it's even getting to me!) it promises a new generation of storage that is greener (grown), leaner, faster, more flexible, and more cost effective for everybody. HAIL and RAIN return storage to its rightful place as the servant (and beneficiary) of all the features the application and virtualization Cloud has to offer without wasting money and resources. This is the reason why Xiotech, CommVault, Oracle, Microsoft, Adobe, VMWare, Citrix, and many others utilize Web Services and REST as a communication model. It is a storm that has been building for quite some time but it is also a storm that promises healthy changes to an industry that truly needs a change in weather.

StorageWonk

"Wicked Fast!"-How Xiotech ISE Is Changing The Storage Performance Curve

2010-03-11T17:48:00.011-07:00

I had a speaking engagement early this week involving a number of Systems Integrators who specialize in virtualization (which these days is probably redundant). One of the issues they are all facing is in the realm of performance. Ironically, they have the fastest processors, a gob of cache, fibre channel connections, and yet performance on their virtual machines simply isn't fast enough. Why? Because of the "boat anchors" they have been using for storage.

Interestingly, these "boat anchors" weren't considered boat anchors just a few years ago. But the up-tick in processor power on the server side, as well as the server virtualization market has really put the hurt on conventional storage performance. There simply aren't enough IOPS coming out of the hard drives in traditional, monolithic arrays these days. The only recourse has been to buy more storage silos, more hard disk drives, and try to leverage the power of more hardware. Unfortunately, storage silos aren't cheap, they suck a lot of power, put out a lot of heat, and they take up a lot of data center real estate. Not exactly the prescription for a down economy and a world strangled by its over-consumption of energy and other resources.

Like it or not, the reality of IT is changing. It's no longer fashionable (or practical; possible) to spend a few hundred thousand dollars to provide users with the "next terabyte" of storage. Like software and information, storage must accommodate a "cloud" mentality by allowing the provisioning of capacity and/or performance easily, and at a small incremental cost regardless of how large the SAN becomes. The technology exists to accomplish this, the demand from the customers is there, where are the storage vendors? Pushing their new software features and trying to maintain the status quo.

Xiotech's Intelligent Storage Element (ISE) is an effort to change the status quo. Its ability to utilize Seagate technology to actually repair hard drives (rather than simply failing in place as other so-called "self-healing" arrays do) is remarkable enough, but the ISE is also amazing at actually reading and writing data too! (I believe the technical term used by a test engineer to describe the ISE was "wicked fast!"). I recently had a customer who was running some performance tests on their new ISE and spent quite a bit of time trying to figure out why their tests weren't running. Finally they discovered, they were running, in fact, they were finishing so fast they thought they were broken! To say they were impressed is an understatement. Man I LOVE getting those kinds of calls!

Since each ISE comes complete with two active-active controllers and 1GB of read/write cache, every time you add storage you add performance. So the "next terabyte" of storage has all the functionality and manageability of a large array without the large cost. It's also not constrained by limited controllers, cache, and static backplanes of monolithic array architectures. In effect, the ISE is "cloud" ready and brings real benefits to the pain now being experienced by the virtualization market. Indeed, any Performance Starved Application (PSA) would benefit from the raw power of the technology.

If you'd like to see some performance numbers check out this link from an openBench Labs report. On page 14 they noted: "To test random access throughput for database-driven applications, openBench Labs used Iometer on two 25GB test volumes. I/O streams involved 80% reads and 20% writes using 4KB and 8KB requests. With IOPS rates doubling for both 4KB and 8KB I/O in lock step, the performance profile of our Emprise 5000 was more in line with that of RAM disk arrays [italics mine]." I did mention ISE was fast didn't I?

So if you are struggling with performance in a virtual environment, or any other PSA, check out the ISE, a wicked fast technology that is changing the storage performance curve.

Addendum: Check out the new results of 10,000 Heavy Exchange Users on a 3u, 20 drive ISE; 100,000 heavy users in a 42u rack!!!

Now that's Wicked Fast!!!

Cheers,
StorageWonk

Store, Move, Protect-The Fundamentals of Storage

2010-02-26T17:34:00.005-07:00

It's no secret that technology has had a major impact on the world of business. The impact has been significant in areas of automation and robotics. But technology has also greatly impacted the very core of how companies transact business. The Internet has made the adoption of technology a necessary component for even the smallest commercial undertaking.

Few would argue that the foundation of this techno-boom has been the data storage hardware. Whether this hardware amounts to an EPROM, flash, CD/DVD, or hard disk drive, all the cool applications in the world are pretty much worthless without some mechanism to store and deploy them.

Yet, it is quite common to see data storage hardware purchased as an after-thought rather than as the result of a targeted decision process. Now understand, when I speak of data storage hardware, I'm not speaking of any particular SAN or NAS vendor as choosing one of these vendors often involves A LOT of thought and consideration. But for all of the effort that goes into selecting a storage vendor, how much do you really know about the hardware on which your precious data will be living? Does your storage hardware vendor spend as much time talking about his hardware as his software? If your experience is like most, the answer is no.

The reason for that little oxymoron is due to the fact all storage array vendors utilize similar (or in some cases identical) hardware. So why buy from Vendor A when Vendor B has exactly the same equipment? Enter the software discussion. The problem with this reality is that storage hardware vendors aren't always the best at writing software (but they try) so you end up buying features and hardware that are often (not always) substandard to those offered by the application vendors (who tend to be good at writing software).

What's more, this penchant for differentiating their products through software features causes storage vendors to focus on the “need to have” and “nice to have” of storage with little effort expended in addressing the all important "must have" of storage such as storing, moving, and protecting data. This is one reason we now have drives in the 1-2TB size category that, when they fail, can literally take days to rebuild from parity. Most hardware vendors have not taken the time to address the fact that drive capacities have simply outpaced current RAID technologies and have exposed data to an unacceptable level of risk (This is not to say there haven't been any innovations designed to minimize hardware risk; dual parity RAID is available which helps mitigate a double fault, but it has its downside in the area of performance. Dual parity, while helpful, is still just a band-aid on a technique that is fundamentally broken).

RAID technologies were envisioned when hard disk drives were many orders of magnitude smaller than they are today. The industry's focus on the "need to have" and the "nice to have" features has caused them to basically ignore the development of newer ways of striping data. They also tend to focus on ways of surviving a drive failure instead of figuring out ways of preventing them from happening in the first place. The result is our businesses (and our lives) are essentially running on a storage foundation made of sand. The sad part is it is totally unnecessary!

So what’s the fix? Both the IT and business community need to fundamentally rethink the hardware side of the house. Drive enclosures and RAID technologies need to get with program (no pun intended) and fix their longstanding weaknesses in the areas of redundancy, parity, vibration, and heat, all of which contribute to premature drive failures and data risk. The technology exists to build drive enclosures with drive intelligence; enclosures that can monitor drive telemetry and intervene when a drive begins to have difficulty (or that can preventively show a drive “some love” so it doesn’t fail in the first place). The technology exists to clear anomalies from drives, and if necessary, to refurbish them in-situ (even if there is a hardware failure) to a factory-fresh state without the need to send them back to the factory or even to remove them from the enclosure. Instead of creating a RAID stripe at the drive level, the technology exists to create a RAID stripe at the drive-head level. This alone would offer a couple advantages over what we now do. First it would reduce our failure granularity from the entire drive to a portion of the drive. If a typical hard drive has 8 heads and 8 writing surfaces (platter surfaces) and one of them failed, utilizing this technology you would only have to wait for a parity rebuild of 1/8th of the drive instead of the whole drive. This would serve to dramatically reduce drive rebuild times over conventional RAID. Second, because this type of RAID sees the head as the smallest unit of a RAID set we wouldn't need hot-spare drives sitting there doing nothing. The sparing could happen at the head level elsewhere in the enclosure and we wouldn't have to worry about how many hotspares we have or what size they were. All drives would participate in I/O delivery and protection, in effect, earning their keep instead of requiring us to power and cool them while they sat there doing nothing.

And what’s the deal with hard drives losing performance as they fill up? There are certainly ways of mitigating this effect if you have the time and expertise to do it and maintain it, but for most implementations, performance degradation is a reality as a hard drive fills preventing you from utilizing all the capacity you pay for without suffering a penalty. If you have to limit the amount of data you store on a drive or face significant performance degradation, then why do you still have to pay for all the capacity and licensing as though it could realistically be used? In addition to striping for data protection, the technology exists to stripe data in such a way so as to gain significant performance over conventional methods (often 3x-5x) even when the drive is 100% full. Sound good? Then why isn't the storage industry using these technologies more? Not surprisingly, the prospect of selling 100 drives for capacity when 75 drives would do, or selling 300 drives for performance when 100 drives would do might be a pretty compelling reason not to use it (that is, if you are a storage manufacturer, not so much if you are the one footing the bill).

The truth is, end-users will start seeing more of this type of technology offered when they quit allowing storage vendors to use the software features of their storage array as a smoke screen to hide the underlying weakness of foundationally unsound hardware.

So the next time you are looking for enterprise storage, ask your hardware vendor how they provide you the "must have's" of storage (store, move, and protect) on the hardware they're trying to sell you. When they answer by pitching you on their cool new software tell them you get that from your software vendor and show them the door.

Trimming the High Cost of the "Next" Terabyte of Storage

2010-02-18T14:33:00.014-07:00

Architecturally, storage silos are an expensive proposition. True, by centralizing management of data storage they can make it easier to provision storage in a large data center. They can also maximize the ability of a single admin to manage more storage capacity than he could if the storage were distributed among dozens or hundreds of servers. In these two ways alone storage silos can add value and may even justify their cost. But it is equally true that the cost of a buying a storage silo can leave you feeling substantial "sticker shock" because of all the things you have to purchase in addition to the hard drives.

The good news is that once you've "bitten the bullet" and bought your shiny new storage silo you can relax somewhat because adding capacity to a storage silo tends to be relatively inexpensive (at least as compared to the cost of the initial acquisition). Unfortunately, one of the inherent weakness of storage silo architecture is its limited ability to scale. Once your silo is "maxed out" from a hardware perspective, the only way to expand it is to buy another silo and the cycle begins anew. This is not necessarily always a bad thing; many companies grow at a relatively slow pace and so they can spread out these CAPEX acquisition cost over several years. For this type of company this may not be a big deal.

But what happens if your company is growing at an accelerated rate? Adding another storage silo to accommodate ever increasing storage requirements becomes an extremely pricey proposition. In this circumstance, the last terabyte you own is "cheap" but the "next" terabyte you don't own is expensive because you have to buy an entirely new storage silo to get it.

In a perfect world, we would have the option of taking the time to decide whether or not that "next" terabyte was worth the price of a new silo. Unfortunately, we don't operate in a perfect world and the demands of business force us into the acquisition whether we like it or not. So what is a fast growing business to do? How do you trim the cost of the "next" terabyte you buy after the last terabyte is gone?

The simple answer is to get away from traditional, monolithic storage silos and standardize on a distributed model for your storage needs. This distributed architecture is made up of intelligent storage elements. This type of architecture scales because it is not tied to a traditional physical or logical back plane, but instead, plugs directly into a fibre channel fabric

Intelligent storage elements are designed to be highly reliable, enterprise class, and performance oriented (having their own cache, controllers, and a higher than usual spindle density; up to 40 drives in a 3u enclosure). Since each 3u element has all the features and computing power of large silos they give you that "next" terabyte of storage without breaking the bank. The fact that they also communicate with each other forming a collective intelligence means you can utilize a single user interface (UI) to manage them much as you would a storage silo but without the limitations of scale. Xiotech's ISE is an example of this type of architecture.

The Intelligent Alternative to Thin Provisioning and Over-Subscription

2010-02-11T13:07:00.006-07:00

One of the most celebrated technologies to hit the storage array market in recent years has been “thin provisioning.” Thin provisioning (not to be confused with over-subscription) is the ability to seemingly create a LUN on the storage array, but in actuality, the physical storage for that LUN is provisioned at the time the server writes to a given block.

Why thin provision a LUN? Well, historically it has been a pain to grow traditional “thick” LUNs due to the requirements (and limitations) of the operating system. Being able to create a LUN at its largest expected capacity without having to provision the actual capacity up front is an appealing notion and the industry has gone after this technology with a vengeance.

Unfortunately, the strength of this technology is also its weakness. Thin provisioning allows you to make your servers think they have more physical capacity than they actually do. When this allocated capacity exceeds the physical capacity of the storage array you venture into the high stakes world of “Over-Subscription.”

Over-subscription is a tantalizing concept because it allows you to “borrow” or allocate more capacity than you actually own and potentially gain the advantages of a larger infrastructure without having to pay for it up front. This is similar to what investors do when they buy stocks “on margin.” Buying stock "on margin" is essentially the act of “borrowing” funds from your stock broker in order to buy more stock. As long as the markets and your chosen stocks do what you expect them to do you can make good money leveraging your broker's assets.

The problem comes when the markets and your stocks lose value unexpectedly. This can result in your broker's issuing a “margin call” where you must immediately pay the "loan" back, either with cash, or with the broker-imposed sale of your stocks. If the market fall is precipitous, these "margin calls" can literally wipe out your investments leaving you penniless, or even worse, deep in debt. Buying stocks on margin is not a strategy for sissies.

Over-subscribing your storage is analogous to buying stocks on margin. When your server and storage environment (your stocks) are running as expected there can be great advantages to your company's bottom line. But when the unexpected occurs (as commonly happens in the modern data center) and we miscalculate the actual amount of storage needed to cover our needs, we can be on the receiving end of a storage “margin call” that can leave all SAN-connected servers COMPLETELY OUT OF STORAGE CAPACITY! Can you say, “blue screen of death?” Just like buying stocks on margin, over-subscribing storage is not for sissies either.

What conclusions should we draw from this? Well, just as most investors should avoid buying stocks on margin, most IT departments should avoid over-subscribing their storage as well. If you are going to use thin provisioning it should not exceed the maximum usable capacity of your storage array. Admittedly, this is easier said than done. Budgetary pressures can sorely tempt us into venturing into this risky form of storage allocation in order to meet business demands. Thin provisioning is like having a credit card with no limit and then being told to only spend what you have in the bank despite the mounting pressure of your daily bills. Few people could do this with their own money; fewer still can do it with their company’s money.

So how do you gain the benefits of thin provisioning without incurring the risks (and temptations) of over-subscription? Xiotech has developed a hybrid system, called “intelligent provisioning” that allows for the automatic expansion of small “thick” LUNs based upon the actual allocation needs of server data. Instead of creating an artificially large “thin” LUN and then provisioning capacity to fill it over time, Intelligent Provisioning allows you to create a small “thick” LUN and then automatically grow it as the capacity is used up. As with “thin” LUNs, intelligently provisioned “thick” LUNs grow seamlessly from the server's perspective but have the inherent advantage of making it impossible to over-subscribe your storage array. Utilizing this technology you can eliminate the risk of storage “margin calls,” and “credit cards with no limits,” while gaining the advantages of easier LUN management.

Addendum:
Evidently a bit more on Intelligent Provisioning (IP) is in order. IP is a feature that allows you to create a physical LUN of actual capacity (say 100GB), you then set a threshold of how full you want that LUN to get before it expands (say 80%). You then indicate how much you want the LUN to expand by when it reaches that 80% threshold (say 100GB), indicate how big (maximum) you want the LUN to grow (say 800GB). You're done. Now, when your physical 100GB LUN reaches 80% full, the system automatically enlarges it by 100GB, communicates this to the operating system, and voila, the 100GB LUN is now 200GB. Best of all, this 200GB LUN is actual physical capacity. If there is insufficient capacity to enlarge the LUN as you've stipulated, notification is sent indicating this. Setting your threshold to the appropriate percentage full gives you time to negotiate pricing with the storage vendor. Utilizing this technology you have all the LUN management ease of thin provisioning without the risk of accidentally running out of capacity through an unplanned or unexpected event. Since it is using "thick" LUNs at all times the capacity is always real and in-flight IO always has a place to land.

Bridging The Gap Between Servers That "Can" And Storage That "Can't"

2010-02-04T21:00:00.005-07:00

An interesting whitepaper was published in December, 2009, from Microsoft's Global Foundation Services on server rightsizing as a means to save on server cost. What is meant by "server rightsizing"? MS Data Centers blog from Dileep Bhandarkar, Ph. D., Distinguished Engineer, Global Foundation Services, Microsoft who co-authored the paper summed up the answer: “A lot of people don’t realize the huge performance disparity between the Central Processing Unit (CPU) and disk input/output for instance, and the opportunity that this disparity provides to achieve significant savings by balancing the system.”

The white paper points out that one of the key mistakes IT makes is buying servers with exceptional CPU and RAM capabilities while giving little regard to the realities of storage performance limitations and actual workloads experienced in their environment. It is particularly interesting to see reference made to an Intel study describing how CPU performance scaled 30x between 1996 and 2005 whereas hard drive performance scaled only 1.3x during the same period.

So what conclusions are drawn from this? "Our analysis and experience shows that it usually makes more sense to use fewer and less expensive processors because the bottleneck in performance is almost invariably the disk I/O of the platform, not the CPU" notes the white paper. With comparatively low disc performance preventing CPU's from maximum performance the analogy of "why opt for a V8 engine when you are only able to drive in the city" makes a good argument for reducing server costs to more closely match the constraints of the hardware environment they will be operating in.

Although this white paper is about server rightsizing, it aptly describes the real problem making server rightsizing the necessity it is. Conventional disc performance is too slow and we, as an industry, need to rethink ways we can improve disc I/O in a substantive way. Although it is probably safe to say hard drive performance will never match CPU performance we certainly can do better than continuing to use technology that hasn't fundamentally changed its performance capabilities over the last 25 or 30 years!

Invented by Seagate's Advanced Storage Architecture group, Xiotech's Intelligent Storage Element (ISE) is an effort to do just that. Whereas storage vendors in general don't really talk much about the hardware backing their storage arrays (not surprising since there isn't a whole lot of difference among them), Xiotech spends a significant amount of time on just this topic. Why? Because an array utilizing ISE technology can provide greater disc I/O, and greater throughput than an array utilizing conventional disc and these performance efficiencies result in lower costs per watt, btu, I/O, and MBps. The fact that it can perform optimally at nearly 100% of capacity also drives down cost per usable TB.

In the case of server rightsizing, the net result of a faster disc element is it allows practical utilization of more powerful servers giving greater overall system performance and helping justify the increased costs of buying these expensive and powerful CPU's (as opposed to buying them only to waste their potential due to the abysmal performance of conventional disc).

For the typical performance starved applications such as Exchange, SQL, Oracle, and VMWare (among others), being able to fully capitalize on the new multicore processors can mean the difference between success and failure in business. Companies that are serious about improving their overall application performance would do well to match their powerful servers with a next generation disc storage element that is in a position to keep up with server demand. ISE is one example of a disc system uniquely created to do just that.

ISE and Storage Efficiency-A Tree-Hugger's Dream!

2010-02-03T09:37:00.002-07:00

We've all been inundated with marketing hype about how "green" storage manufacturer's are making their products. To hear it told, "gone are the days of the 'gas guzzling' storage silos" welcome to the new era of "environmentally friendly" disc arrays. What has really changed? Marketing buzzwords mostly.

Oh to be sure there have been some improvements; some storage arrays can spin down individual discs, other arrays tout "thin provisioning" or over-subscription as a mechanism for achieving "green", drive manufacturers have created "reduced power" hard drives that have helped. But for the most part, storage arrays spec out nearly the same in the heat and power consumption area due to the fact that they all use similar or identical hardware.

So what does it take to really achieve a "green" storage array? In a word, efficiency. For example, if the performance and capacity potential of an existing array could be matched by a competing array using half as much power and heat producing hardware, the competing array could then be considered "green" in comparison to the array it replaced.

This is the concept behind Xiotech's Intelligent Storage Element (ISE). The ISE is a technology developed by Seagate, the worlds largest manufacturer of hard drives. It was built from the ground up to take advantage of technology improvements discovered over the past 30 years. A couple of the key improvements included in the technology was the ability to utilize nearly 100% of the available capacity, after RAID, without the typical performance hit experienced by conventional storage arrays. Each ISE also includes its own active-active controllers and read/write cache giving it a significant performance boost over conventional enclosures.

What has been the effect? Well, storage array vendors differ greatly in the amount of usable capacity, after RAID, they recommend a user consume. Feature "hold-backs", and performance penalties are among the limiting factors and each vendor is different but the range for conventional storage is anywhere from 30% to 80% usable. What about the 70%-20% that's left? Wasted, plain and simple. Fortunately for the manufacturer, the end user has to pay for it anyway and since it cannot be used it simply drives-up the cost per usable terabyte, cost per IOPS, cost per watt, cost per btu, and cost per rack unit, which all serves to drive down efficiency making the array less "green" than it should be.

The ISE technology is different. With a usable capacity of nearly 100%, after RAID, all capacity is usable to the end-user without performance penalty. This allows the user to reclaim the 70%-20% overhead of conventional storage driving down the cost per terabyte, cost per watt, cost per btu, and cost per rack unit, which all serves to improve the efficiency making the ISE one of the only true "green" enclosures on the market today. In practical terms, using this metric, if you needed 100 hard drives from a conventional enclosure to meet your capacity needs, the ISE could do it with 30-80 drives depending on the vendor being compared against.

The ISE has also been given significant performance enhancement. The currently published SPC-1 benchmarks (www.storageperformance.org) show the ISE hard drives averaging 294.6 IOPS per drive at 96.6% full. That's three times the speed of a 15k hard drive running in a conventional enclosure at 96% full. Using this metric, if you needed 100 drives from a conventional array to meet your performance needs the ISE could do it with 30. That's 1/3 the drives, 1/3 the rack space, 1/3 the power, 1/3 the btu's and 100% of the performance.

As icing on the cake, the ISE comes with a feature called PowerNap which allows, not just a few of the drives, but the entire enclosure to "sleep on LAN" in 40 seconds, and "wake on LAN" in 60 seconds ready to accept IO. This drops the power consumption from 500w per 20-drive enclosure to 23w while the system is asleep. As a backup target it can't be beat.

So, if you have the urge, or the mandate, to "hug a tree", check out Xiotech's ISE. It's one of the only true "green" stories in storage today.

Comparing Storage Arrays Apples-to-Apples

2010-01-26T20:30:00.011-07:00

One of the biggest challenges facing the enterprise IT professional is to vet out the best data storage solution (SAN, NAS, etc) for their business. Often a review committee of five or six individuals are tasked with comparing and recommending a particular storage array from the many vying for their business. These, in turn, build a relatively complicated flow sheet and process for giving weighted scores to different features from those invited to quote and commonly ask each vendor to supply a given amount of capacity, often defined by drive spindle count or raw space.

Compounding the difficulty is that all storage array vendors have different capacity utilization levels due to feature holdbacks (snapshot reserve, threshold sets, performance limitations, software upgrades, data movement and optimization requirements, thin provisioning, etc.). If a team asks for quotes based upon the raw capacity of a fixed number of drives they'll find a significant variation from vendor to vendor in the amount of capacity that is ultimately available for data storage (typically the primary reason for the purchase in the first place).

Those who buck the trend and stipulate usable capacity instead of raw capacity often have no idea of how to verify whether the stipulated capacity is contained in the quote and they almost never clarify to the vendor that he/she will be required to provide any capacity deficiency free of charge if the implementation comes up short. Instead the team relies on the vendors' understanding the concept of usable capacity in the same way as the customer. This is almost always off because it's easier for the storage array vendor to ask for forgiveness after the deal has been awarded than clarification before the deal is done and risk having to raise his/her price. This method works because no IT professional wants to admit to their management that they didn't verify the vendors' claims and so the whole thing is quietly covered over to the benefit of the storage array vendor who unfairly won the deal and to the detriment of the end-user who thought they knew but didn't.

So how do you boil it down and make a fair comparison based upon the merits rather than the marketing? The answer is to compare all storage arrays from the standpoint of true usable capacity. Because of the issues mentioned above, it is critical to define what usable capacity means to you. Stipulate that usable capacity means capacity that exists after all feature holdbacks, hot spares, Base10/Base2 conversion, RAID penalties, and best practice usage levels are subtracted from the total capacity. It means that if you had a single volume of data that would consume all the capacity you asked for, the storage array would not only accommodate it but the vendor would approve it as meeting its own published best practices for utilization in the presence of included feature sets. When all storage arrays are compared in this way they are reduced to a common denominator which removes marketing fluff from the equation and allows you to compare them on a dollar-for-dollar basis.

Another key to comparing storage arrays is to compare performance analytics from unbiased sources such as the SPC-1 and SPC-2 tests found at www.storageperformance.org. Although no test is a perfect duplication of a company's unique environment, these tests are unbiased, audited, and all storage arrays are subjected to the same series of tests based upon average response time giving the end user a standard means of comparison.

But be aware that tested storage arrays will vary in size and subsequent total IOPS. So how do you compare a $2,000,000 with a $20,000 system? Take the total amount of IOPS or MBps and divide it by the number of hard drives it took to accomplish the task. That will give you an average IOPS per drive or an average MBps per drive number. Compare each vendor by this number and you'll have another common denominator you need for your apples-to-apples comparison of the cost per IO or cost per MB. This test works well in comparing all of the major storage array vendors in the market today with the exception of EMC and Compellent as these two vendors apparently prefer not to submit their array performance to public scrutiny (they must be really, really, fast!!!). ;0)

Agree? Disagree? I would love to hear from you either way. Cheers!

Embracing My Inner Nerd!

2010-01-24T22:44:00.008-07:00

I had the opportunity to attend the annual Xiotech "Nerdfest" in Colorado Springs, Colorado last week learning to embrace my inner "Nerd". After spending the past 10 years working with (and competing against) "traditional" storage array architectures it was a fascinating journey into a completely new way of looking at storage array technology through the eyes of the "Nerds" who helped invent the concept years ago. One of the reasons Xiotech was so appealing to me was that other storage array technologies were fundamentally difficult to use as opposed to the Xiotech way of storage-virtualization which meant adding all disk drives into a giant capacity pool and then carving out an ANSI T10 LUN/Virtual Disk which could be presented to the host without requiring server-side software. Xiotech not only made capacity management possible---they made it easy.

Since that time, the industry has done much to make all other storage arrays easier to manage narrowing the advantage Xiotech held for many years. Unfortunately, storage array vendors didn't stop there. Whereas they originally seemed to understand that storage was the servant of the application, they've added so many features that today's storage arrays have now become the peer or even the master of the applications they once served.

Why has there been such an explosion of storage-based software features like thin provisioning, snapshots, deduplication, and replication (to mention a few)? One reason is that, at the most basic level, all storage arrays have been locked into using what is essentially the same hardware architecture (same hard drives, SBODs, Fibre Channel or iSCSI, etc.) and so up-til-now, the only way we have been able to differentiate our products in the market has been to compete at the feature level. Since we've all had to use "the same pig" we had to apply a "different lipstick" as a means of keeping and growing market share. This emphasis on storage array features has caused our industry to continue relying on storage array hardware that was first envisioned and architected 30 years ago when hard drives were measured in megabytes. Thus storage array hardware and RAID technologies have been outpaced by massively large hard drives, multi-core processors, hypervisor server deployments, and 24x7 data center operations.

So where does that leave us? Hard lessons from the past have been learned. With Seagate/Xiotech’s Intelligent Storage Element (ISE) the "Nerds" have, for the first time, re-architected the hardware and RAID from the ground up to accommodate the performance and reliability demands of technology as it is today instead of as it was 30 years ago! They have also re-focused on the fundamental reality that a storage array should be the servant of the application and for the first time, have created a storage element that utilizes the language of the application (Web Services) to manage that relationship.

Sound like a bunch of hype? Don't think the difference in performance and reliability can be that great? Try embracing your inner "Nerd" and come see for yourself. I guarantee you'll never look at data storage in the same way again.