As I worked on the design of my upcoming server virtualization infrastructure, obviously storage was one of my primary decisions. I already had my primary storage platform in place with various SAN arrays sitting behind a storage virtualization appliance, and I pretty much knew that I would just be expanding on that to support the new virtual machines. Plus, that fit in perfectly with the HA design that was already decided for the VM host machines.  I rolled out two quad proc quad core hosts with 64 GB of RAM each, and then 1 smaller 2 proc quad core machine with some DASD and a decent amount of RAM for a staging box for my P to V migrations.

Next I had to decide on the arrays. I pretty much knew that any physical box that was currently RAID 10, would be RAID 10 as a VM. What I had to decide was if I would just go RAID 10 across the board. Many physical machines were RAID 5, but virtualization adds its own bit of overhead, and there’s never anything wrong with boosting performance on anything. It really wasn’t cost prohibitive in buying the physical disks, but it did present a problem in maxing the physical disk count per controller a little quicker than I liked. Also, my IOPS per $$$ looked better with going all 10, but again, it was going to lead to bringing in additional controllers to support the number of physical disks that were going to be required. So in the end, I did a server by server worksheet, and ended up about half and half. I simply have TIER 1 and TIER 2 disk groups in my storage arrays to support the 2 groups of servers I decided on. Everything is very manageable and scalable, and I felt like I got the best bang for my buck.

We started our P to V migrations, and quickly had our first dozen or so servers up and going. We monitored performance, tested failover, and felt comfortable with what we had put in place. We were ready to proceed aggressively at this point. We knew we couldn’t virtualize everything we had in the data center (about 130 servers), as we needed to stay under 50% utilization on the cluster to maintain high availability, but we wanted to get a good number of physical boxes out of there. Soon we were near 40 VM’s, and all was running great. Disk performance was more than satisfactory, including the VM’s on the large shared RAID 5 arrays.

Next came the part that I’m sure many admins of virtualization run in to. Besides virtualizing production boxes, our developers (around 100 or so) had little test servers all over the place. Some were retired production boxes, some were just PC’s running Server, etc. Time to clean house, get all these things virtualized, and give them the benefit of snap shots and so on. It all went great as we piled up the carcasses. But all of a sudden, our utilization was spiking up near 60% on CPU and memory. While no single one of those was a hog, the group of them took a significant amount of disk, processor, and memory. Now I was stuck in the position of not being able to virtualize anything the rest of this year.

Then I thought, while it was great to virtualize all those little test machines, why am I chewing up the resources on my HA cluster, and throwing high end SAN disk at them. What to do. Then I thought, now that the bulk of the P to V migrations are done, and we are totally comfortable doing them going forward, that staging server sure does just sit idle now. With all I push for centralized SAN disk, moving those low end test machines on to that staging server and its DASD array, sure looked attractive. And that’s exactly what I did. 8 cores, 24 GB of RAM, and over ½ TB of 15K DASD. A perfect home for those machines, and now my HA cluster has breathing room. I worried a little about losing HA on those, but then again, they’re test (no SLA for test machines!). So, test machines, snapshots, clones etc., are all happily existing on that box. A small risk, and it would be minimal pain if we lose it.

So, in the end, my virtual infrastructure ended up a combination of large RAID 5 arrays, pooled up smaller RAID 10 arrays for IO intensive machines, and some good old DASD for Dev/Test. I nearly went into this with the “what’s good for one is good for all” mentality, but am very happy with the mixed environment I ended up with.

You virtualized what?

Anything is possible.  We can abstract the power from the hardware and circuits, which might allow us to redistribute power as needed to various parts of the data center and ease use of products with heterogeneous power requirements, including the power hungry 10 gigE coming our way.  We would only trip virtual circuits.  Virtual power, virtual cooling, virtual phones, virtually anything is possible.  Virtual IT staff?  Well, that’s called outsourcing the staff and is as useful as virtual money, right?  In the world of IT storage, providing a layer of abstraction between the file system and the operating system should be focused on.  We also need to improve the underlying resilience technology.  With SSDs being fragmented by nature and most of the time being slower on writes than spinning disk, a RAID 1+0 looks pretty good, especially since it’s cheaper per gigabyte too.  A RAID 1+0 could lose half of its disks and still run without issue.

Back to Reality

I understand there may not be a perceived need for some of these technologies yet.  There still are several opportunities for more abstraction.  Too much abstraction can make management cumbersome.  The key is abstraction with less complication.  Let’s take a look at the needs of SMB/SMEs, the Small-to-Medium Enterprises.  Many of these companies have enterprise-class needs for storage, network security, and messaging solutions.  These needs don’t necessarily mean they have the money or space to fill these roles and house this technology.  How do you get network monitoring around the clock, and a warehouse full of information in a small office?  Bring in the appliances and the outsource companies.

Where Are The Abstract Appliances?

Appliances serve many purposes, and do a great job fulfilling various needs.  The problem with appliances is that the vendors that make them seem to be removing layers of abstraction instead of adding them.  Specifically I’m thinking of deduplication appliances.  There are too few deduplication appliances available for SMEs that do not have their own proprietary and locally attached storage.  EMC bought Avamar and rather than leaving the option of it being available as software or offering an appliance gateway, they now force the consumer to buy it with locally attached storage.  There is a virtual edition, but it comes with many restrictions and Avamar in general requires giving up on existing backup software investments and using the Avamar interface.  Data Domain does have a gateway product, but it is their flagship model and its price point punts it out of reach from the SME market.   Maybe an abstract appliance is an oxymoron, but I know I’m still looking for one to buy for our datacenter.

Proprietary file system you say?  Profit margin prohibitive?  Give us abstraction, and we’ll reward you with our business.

Storage Management: Scaling Up

As I discussed in my last piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is reducing complexity and increasing standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share, \fireflybackups, would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level - all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 Acopia, NetApp, BlueArc, and ONStor have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files.  As I pointed out in my piece, We Need a Storage Revolution, the ideal storage platform for data would be one in which each individual record or object included custom metadata and was managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware recently announced that Update 2 of Virtual Infrastructure will enable Microsoft Volume Shadow Copy Service (VSS) integration for shared storage. So a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment.

As virtualization technology matures, expect this type of integration to improve. More and more APIs will be exposed, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

When budgeting for your new storage system or changes to an existing SAN, you’ll want to factor in not only hardware costs but also the likely consulting costs to fully deploy the new systems.   Even for a fully in-house solution there will likely be costs outside of just the new hardware, and these will vary depending on vendors, solutions, and sales negotiating ability.

One new option is to use Dell services SAN consulting, a new offering from the company that now provides more ICSI installs than any other thanks to the aquisition of major SAN market leader Equalogic.    Obviously a Dell consultant will steer you in the direction of Dell|Equalogic servers and products, but unless you have a highly specialized application or prefer other hardware you are likely considering Dell solutions already.   You’ll want to estimate the cost savings of going this route over private consulting with more company choices for hardware to see if you are likely to make up differences in hardware costs by what is likely going to be a cheaper consulting and hardware combination package.

Over at the ARS technica blog an experience SAN IT manager recently suggested:

… get someone that can safely guide you through this minefield and get you a solution that you can understand. We buy all our hardware under the Dell Gold Support contract. It’s a live-saver to be able to call someone that can understand all those systems and get you back on track ASAP.

More important than the initail cost and choice of hardware, it is critical to make sure your consultant understands your storage needs very clearly and is capable of delivering a solution very consistent with these needs, especially if you need to integrate legacy storage systems - for example Fiber based - with newer deployments which are often iSCSI.  Before choosing a consultant take some time to clearly inventory your existing network and your current and future storage needs, and create a brief interview for your prospective consultants that will determine if they appear highly proficient with solutions that address all your storage needs.

So, do you need a network storage consultant?   If you are not sure about it the answer is probably yes, and choosing one wisely should be a top priority.

[EMC] The recent CLARiiON CX4 overhaul clearly paved the way to support FCoE target controller sometime in 2009.

[HP] The data storage blog (http://www.communities.hp.com/online/blogs/datastorage/archive/2008/03/04/HPPost5865.aspx) certainly concludes they embrace FCoE.

[IBM] One of the T11 committee members driving hard behind FCoE standard. I don’t see why they wouldn’t jump on and work out native FCoE target.

[HDS] “Moving to FCoE for storage arrays would only require the replacement of the front end port modules….it is expected to be much easier for HDS to convert to FCoE.” from Hu Yoshida’s latest blog (http://blogs.hds.com/hu/2008/08/storage_network_unification.html).

A year ago, I had doubts whether or not these storage vendors would take native FCoE seriously. Now I have no doubt that they, at least some of them, will start to support FCoE within the next two years.

It seems very clear they decide to move down FCoE path.

The value of HBAs

In this plan, your network adapters and drivers (HBAs) must respond to these flow control messages to stop or slow data transmission so as to avoid congestion in the network and subsequent packet loss. Thus HBAs will becomes critical to the success of iSCSI or FCoE because they will need to have large and smart buffering schemes to be able to handle the data pumping. I have talked in previous articles about the value of HBAs in your servers and this is another example of their value.

The value of a single Network fabric

The current storage network strategy means that there are two networks in our data centre, Fibrechannel and Ethernet. Many people are co-opting this strategy for their iSCSI backbones by creating a second network dedicated to IP Storage.

Unlike using TCP/IP to perform QoS, where the QoS mechanisms are not will suited to storage traffic, CEE will deliver a zero to low loss switching backplane to iSCSI and yet offer support to a number of other applications. The option for single, unified Ethernet fabric in the Data Centre is very real.

Further, we will still have native IP support for WAN for inter-Data Centre connections when needed without extra actions or work. (Unlike Fibrechannel, which will still need FCIP plus FCOE / FC to perform inter-Data Centre connections).

Lets merge these together

Not so long ago, Telephony was migrated to an IP network. It was widely agreed that voice could never travel across a packet network because delivery was not guaranteed, delay / jitter was a problem, the sky would fall down etc etc.

The argument against merging storage and data networks is a reasonable statement with the Ethernet technology we have today. However, Converged Enhanced Ethernet is expected to deliver (in time) a single Ethernet fabric for our entire data centre.

Consider the Brocade approach “FCoE is Fibre Channel, not legacy Ethernet.” - Page 62 - Brocade Tech Day briefing here. This means buying super special sauce switches from Brocade, that support FCoE / FC. It hard to perceive that we can build a unified fabric using these products. At best, they are a “transition” or short term fixit for the Fibrechannel problem.

Building a Unified Fabric

So there are several things that have to happen. Lets be real, and think about what to look for.

First the CEE standard needs to be completed by the IEEE. Second, the vendors need to announce Ethernet switches that support CEE. Look for Cisco and Brocade to be early to market (because they want to force customers to FCoE), but keep an eye on Juniper, Force 10 and Woven who can all use this disruption in the market to take a new position in the market and attack Cisco’s dominance.

Third, look for quality HBA’s and their software drivers. Fourth, look at the integration required between the Network and Server Teams to seamlessly deploy this new network strategy.

When do you think it will be ready

I would _guess_ that CEE will be done by 2010 (OK so there will be pre-standard stuff around like the Cisco Nexus 7000 / 5000, but not many people will buy that). Because this is a market disruption point, I would expect the vendors to have products out quickly, particularly Juniper. So you should be thinkinh about deployment in 2011. Thats only _three_ years away, or a single investment cycle.

Conclusion

So this sounds like a long way off, but we are already getting the marketing hype from Cisco, and I guess Brocade will not be far behind. We haven’t heard much from NetApp / EMC / HP / IBM so its not clear what might happen if they decide to move down a different path.

In the mean time, the focus should be to stick with iSCSI and that will be a viable, long term, technology. When you need to scale to muligigabit performance, new technologies will be there to make it happen. Reliable, Scalable, and Easy to Use. Thats what we want, right ?

What are we trying to solve ?

Existing applications are tolerant to packet loss in the network, and QoS is managed (but not solved) by classifying traffic and managing queues in the network (where queues exit). However, Storage is very sensitive to loss, both FC and iSCSI can retransmit lost data, but this causes significant bottlenecks and a performance hit to the application.

In a modern data centre, attempting to QoS storage AND voice AND other high value traffic is a practical impossibility. The configuration and maintenance of a converged backbone with competing requirements would be unsuccessful in enough cases that the market would reject any such attempt.

The obvious thing is to stop packet loss. The only practical way to achieve this is to create some signalling that notifies the sender to stop sending data before the loss or congestion occurs. Congestion is always possible in ANY NETWORK no matter how much bandwidth you provision.

IEEE are working on 802.1Qau Congestion Notification which is an end to end message. When combined with 802.3x Pause control (which operates at link level), we can guarantee zero loss in Ethernet backbone because the source can be signaled to slow down the data transmission.

Note that 802.3x needs to be modified from a start/stop for the entire link, to be able to Pause certain traffic priorities, so when combined with 802.1Qaz you are able to overcome this limitation. 802.1Qaz describes Enhanced Transmission Selection which support allocation of bandwidth amongst traffic classes.

Another standard  under development is Discovery and Capability Exchange where servers and network equipment are able to signal their capabilities to each other, so that strategies for traffic management can be agreed between all elements.

So now we have the ability to configure our network with at least three major types of buffering or congestion strategies. 1) zero loss suitable for voice traffic 2) buffer overflow causes signal to be sent back to source or upstream network device such traffic is controlled and not dropped 3) discard traffic if needed and let application protocols retransmit. (of course, you should be able to have less, none, or variations in the same way that we do today.

Unlike using TCP/IP to perform QoS, where the QoS mechanisms are not will suited to storage traffic, CEE will deliver a zero to low loss switching backplane to iSCSI and yet offer support to a number of other applications. The option for single, unified Ethernet fabric in the Data Centre is very real.

Further, we will still have native IP support for WAN for inter-Data Centre connections when needed without extra actions or work. (Unlike Fibrechannel, which will still need FCIP plus FCOE / FC to perform inter-Data Centre connections).

Conclusion

The interesting outcome of this is the impact on iSCSI. Storage network technologies are very sensitive to packet loss, and if the network can assert mechanisms for assuring low latency and very low (or no) loss, then our Converged Enhanced Ethernet Networks will take over the data centre for our storage.

Most importantly, iSCSI will now be able to reach it’s full potential. Most people find that iSCSI works fine in Small to Medium server farms, but in large scale IT the impact of packet loss is too great a risk. By comparison Fibrechannel networks deliver a lossless medium and that is a key enabler for the technology.

Look for my next post which discusses more aspects of Converged Enhanced Ethernet and its impact on Storage networks.

References

“Storage convergence over Ethernet: iSCSI for new SAN installations FC tunneled over Ethernet (FCoE) for expanding FC SAN installation” Data Center Bridging, IEEE 802 Tutorial - Page 11 - 12th November 2007.

http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf

The cost benefits of virtualization can be exceptional but the process also provides for superior availability, disaster recovery, and load balancing.  It also facilitates faster software deployment and development as those processes are disconnected from physical hardware limitations.   Virtualization also reduces downtime and the complications associated with hardware failure.

In a paper for BMC software, IT consultancy Enterprise Management Associates took a look at best practices in the virtualization space and identified areas where virtualization brings unique challenges to the enterprise infrastructure.

EMA recommend adoption of the best practices as defined in the IT Infrastructure Library “ITIL“, an IT community approach to bringing workable, quality standards to IT practices.

The study suggested that despite its many advantages, virtualization does bring a host of complications to the table relating to the need to manage, allocated, and scale the virtual server environment effectively.

EMA:

Application performance monitoring and availability management become much more complex, as virtualization makes it harder to map virtual resources to physical systems…

It is harder to accurately detect, measure, and plan capacity, because virtual services are so rapidly deployed, consumed, and destroyed….

Cost accounting and financial management becomes much more intricate and time-consuming, as tools must measure rapidly changing virtual environments, and license management and compliance are much harder to measure and enforce …

Ensuring service levels becomes harder, because the new virtualization layer increases the potential for errors and chokepointd…

The solution, says EMA, is to implement a quality virtualization management software system, consistent with the IT Infrastructure Library Guidelines, such as the solutions provided by BMC.  EMA suggests BMC is a superior environment and more consistent with best practices and the Infrastructure Library than much of the competition, though we should note that this white paper was written for BMC and thus is probably not to be taken as an unbiased evaluation of all the options - rather just good basic insight into the issue of best practices with respect to virtualization of the enterprise.

We have been carrying this message for quite some time and Stephen drills it. We have a couple of short videos on our site that help people deal with this today, not waiting for tomorrow:

http://www.davenportgroup.com/proof/san_101/

Thank you Stephen for a “spot on” article.

Paul Clifford
Davenport Group
www.davenportgroup.com

Now, a lot of this data is sitting on people’s desktops, laptops, ipods, phones, digital cameras, etc. right now. However, things like cloud storage will change all of that. Heck, we are seeing some of the change right now with things like social networking sites, photo sharing sites, etc.  IDC says that for 85% of that data a corporate entity will be responsible for the protection and security of the data.

So, in the future, we are going to have to store a lot more data than we do today, a LOT more data. How are we going to do that? Just the physical aspect of getting exabytes of data on the floor is going to be a challenge. I don’t even want to talk about protecting and managing that much data. But for now, I want to talk about the density of the hard disk drive since that’s going to soon become the physical limit of what we can store on the floor of our data centers.

The bits are getting too small!

Enterprise disk drive capacity has obeyed Moore’s Law and doubled every 18 months for quite a few years.  However, this growth has appears to be slowing down over the last 5 years, and it is now taking approximately 29-30 months to double the capacity of an Enterprise disk Drive.

This shows that we are nearing the maximum areal density (max capacity) of current disk drive technology called the superparamagnetic  limit. Areal density as it refers to disk drives is measured by the number of bits per inch (bpi) times the number of tracks per inch (tpi).

The areal density of disk storage devices has increased dramatically since IBM introduced the RAMAC in 1956. RAMAC had an areal density of two thousand bits per square inch, while current-day disks have reached 100 billion bits (100 gigabits per square inch). Perpendicular recording is expected to increase storage capacity even more over time, but we do appear to be approaching the limit.

As the magnetic bits get smaller, at some point they no longer hold their charge. Thermal fluctuations reduce the signal strength and render the bits unstable. However, this ultimate areal density keeps changing as researchers find new techniques for recording and sensing the bit. Years ago the limit was thought to be 20 gigabits per square inch. Today, the limit is several hundred gigabits per square inch, and more than a terabit is expected soon.  But that’s about all you can get out of the technology.

Denser is faster.

Increasing the density of hard disk drives has a side benefit. It makes the drives faster as well. This is really quite logical when you think about it. Since the closer things are together on the drive, the more data passes by a read/write head in the same period of time thus making the drive faster.

Shorter term solution.

So, if the disk drive is not going to be able to continue to provide us with the kinds of capacities we are going to need in the future, what will?  Well, there are a number of things that are being looked at by a lot of folks who are a lot smarter than me! But in the short term, things like SSD look promising once we work out some of the kinks.  Specifically, the write speed issue. Until we can get that up I’m not sure how much general acceptance SSD technology is going to get. Price, I am convinced, will take care of itself as the scales of economy kick in. Holographic storage, some people have been working on this for a very long time and it seems like such a promising technology, but it has yet to come to fruition. There is one company out there that’s trying to ship a product, but they recently pushed off their release date until the end of this year. Still, if they can work out the kinks, it definitely has promise, especially for media applications. But what about beyond that? What technologies are the researchers looking at that sound really cool? I look at some of those next.

Sci-Fi data storage.

So, this is where it gets fun. Some of the technologies that researches are currently looking into really do sound like something out of a Sci-Fi movie.  Here are some examples of the stuff I’m talking about:

• Nanodots - A nanodot has north and south poles like a tiny bar magnet and switches back and forth (or between 0 and 1) in response to a strong magnetic field. Generally, the smaller the dot, the stronger the field required to induce the switch. Until now researchers have been unable to understand and control a wide variation in nanodot switching response. A NIST team significantly reduced the variation to less than 5 percent of the average switching field and also identified what is believed to be the key cause of variability. Nanodots, as small as 50 nanometers (nm) wide could be used to storage data.
• Array’s of magnetic snakes - According to a weekly digest from the American Physical Society (APS), physicists at Argonne National Laboratory (ANL) have found that under certain conditions, magnetic particles could form magnetic ‘snakes’ able to control fluids. According to the researchers, this magnetic self-assembly phenomena may be used to make the next generation of magnetic recording media or transparent conductors based on self-assembled conducting networks of magnetic micro-particles.
• Nanowires - Switchable fluorescent proteins, able to move reversibly between two optical states, have been known from some years. But now, German researchers have discovered the mechanism behind this optical switch in a protein found on the tentacles of a sea anemone. According to the researchers from the University of Pennsylvania, Drexel University and Harvard University, barium titanium oxide nanowires suspended in water could hold 12.8 million GB per square centimeter. If the memory density can be realized commercially, “a device the size of an iPod Nano could hold enough MP3 music to play for 300,000 years without repeating a song or enough DVD-quality video to play movies for 10,000 years without repetition,” the University of Pennsylvania researchers said.

Is the disk drive dead?

So, does this mean that the disk drive is dead in the future?  I don’t think so. I believe that the disk drive we know and love will simply move from one tier of storage to another. We are already seeing some of this movement with the implementation is backup to disk. Technologies such as data deduplication will continue to accelerate this process, and the addition of new primary data storage technologies will simply end the process by pushing hard disk drives from on-line primary storage to what will be considered near-line storage in the future. Long live the disk drive!