The Problem

Storage tiering is not a new idea. Back in 2003, EMC started a big campaign to help customers save money by tiering their data. It was called ILM - Information Lifecycle Management. The idea was to help customers put the right data on the right storage at the right time, saving them money. There were only two problems: the tiering placements were manual (like this process example), and to place it correctly you needed to understand what the profile for the data was currently and what it would be in the future.

Many customers decided that the manpower was more expensive than just buying additional hardware 'to be safe.' Sure, you could buy a mix of drive types, but buying all the same drives and spreading the workload across all of them is much easier to plan for. And if you did it right, the workload would push the drives hard (to about 70% busy) about the same time you 'ran out' of space (at 75-90% full). Of course, if the workload turns out to have much different needs, you either waste space (the drives get too busy) or waste money (you bought more performance than you need).

The Solution

In December of 2010, EMC introduced Symmetrix VMAX Fully Automated Storage Tiering for Virtual Pools (FAST VP). This new software does what ILM set out to do 7 years before - place data on the right tier of storage in a timely manner. The difference is that now all of that storage fits in the same array, and the user gets to manage the movement by setting simple policies. Details on how this works appear in Barry's blog here and here, and the Enterprise Stratey Group did a nice white paper covering their testing of FAST VP with Oracle. And I have included some additional notes in a section below.

In general, the plan is to replace an all high speed disk (10/15K drives) configuration with a tiered solution that combines Enterprise Flash, high speed, and bulk drives. The Flash drives are quite expensive for the capacity, so they can drive costs up. However, with FAST VP, the hottest data is moved to the small Flash capacity. By using about 3% Flash capacity, most workloads can move over 50% of their disk I/O to Flash (assuming 7.5 MB movements). Not only does this drive down the I/O requirements for the remaining drives, it also means that the top 50% of the disk I/O is happening at 1 ms, decreasing the response time for the most active space used by the applications.

So now that half of the workload is off of the remaining drives, using larger high speed drives presents less of a risk. After all, if the workload becomes hotter than was planned, additional Flash can be added to the array to take off more of the heavy I/O areas. Managing changes in I/O density is much easier with these automated policies.

And there is a lot of data in most systems that is just rarely accessed. In general, about 70% of the capacity does less than 10% of the overall I/O workload (assuming 7.5 MB movements). The bulk drives hold a lot of data and use less power, so the savings in purchase and operational costs per TB are both wonderful. And EMC prices all VMAX software 50% less for bulk capacity drives, providing even more savings. Many customer workloads will see improved performance, a smaller power and floor space configuration, and lower costs (both initial and ongoing) by moving to such a tiered design.

To map this out, EMC has Tier Advisor, which can analyze current workloads and model them on various storage configurations. When the array arrives, Symmetrix Management Console (SMC) makes it easy to define tiers and policies, and to apply them to Virtually Provisioned devices. Once FAST VP is running, Symmetrix Performance Advisor (SPA) gathers performance details on the array, metrics on data moving between tiers, and capacity levels of each tier.

The Results

I got a note last week from one of our Technical Consultants who had helped his customer to implement a FAST VP solution. They were very pleased with the implementation and the results - Oracle was running faster, and they could see how this was going to make storage management easier AND make storage cheaper for them going forward.

The customer decided to build two policies, one for high priority production and the other for standard priority production. Since they were just getting started on growing into the array, they decided to start fairly conservatively. They set the high priority policy at 20% Flash and 100% high performance. With this policy, nothing gets demoted to bulk drives, and up to 20% can move up to Flash if the I/O density warrants it. They set the standard priority policy at 5%/65%/40%, which will force at least 30% of the space managed by this policy onto bulk storage. It also allows up to 5% to move up to Flash if it is justified. And note that these policies share the same 3 tiers of storage on the same pools and drives. The applications were running pretty well on the Fiber Channel space they were originally assigned to. Then FAST VP was activated, and response times for reads went down nicely. The blue line here is a day before FAST VP was live, and the green line is a week later. It was clear that the Flash drives were having the desired impact. (This discussion focuses on read time since the writes all go to cache, so the performance of the drives make little difference for writes as long as they can destage in time.)

The layout that resulted in the storage pools also validated the percentages they had planned. The Flash tier was holding 2.7% of the allocated capacity and driving over 50% of the workload. The ATA tier was holding 32% of the allocated capacity and getting less than 1% of the workload. And the FC tier was taking the data in the middle. Over time, as they chose to increase the percentage of capacity in the array that is ATA, and allow the policies to move more data down, the effective costs will continue to improve.

Additional Information

There are several products in the market that are talking about automated tiering within or between arrays. And the fundamental concepts of Hierarchical Storage Management (HSM) are nothing new. But there are several things that make FAST VP different.

Engineered for Smart Efficiency

VMAX was designed with FAST VP in mind. EMC did extensive reseach, collecting I/O profiles from millions of workloads on thousands of arrays. Based on this research, the VP extents (allocation units) were set at 768 KB. Looking at how to isolate the hot data from the warm data from the cold data, we worked to use the smallest size without too much overhead. A general goal was to get 50% of the workload onto something near 3% of the overall capacity (which we would place on Enterprise Flash Drives - EFDs) for over 90% of the workloads we reviewed. Our research showed that managing data at the sub-1 MB level, we could hit the 50% workload with about 2.5% of the capacity. With 7.5 MB (10 VMAX VP extents), the number was 3% - a small increase, and it cut the management overhead by a factor of 10. Managed at the 40 MB level, the number jumped to around 6%. And at 1 GB, the number jumped again to around 12%. So we went with the 10 VP extents per unit we would move with FAST VP (called an extent group).

Fast Reactions

FAST VP policies set the percentage of data that can live on each of 3 tiers. Based on the percentages in the policy and the historical activity patterns, FAST VP sets performance thresholds for promotion and demotion. These thresholds are adjusted for each device in the array every 10 minutes.

For each disk I/O that happens from the device, a counter is updated to note the activity. The counters decay over time, so while there is historical data, there is also a bias toward more recent activity. This activity level is continuously being checked against the thresholds, with promotions being even more heavily biased toward recent data (move hot things up quickly, move things that were hot for a while down slowly). A huge jump in activity on an area of a LUN can get that space promoted in a few minutes.

And FAST VP is ready to make lots of moves. If the performance profile changes justify it, a VMAX may move over 10 TBs of data each day even on the default relocation rate (priority of reacting to change). And since the change is ongoing, this presents a background load, not a huge burst to get in the way of normal operations. Since the research was done before VMAX was designed, the bandwidth to keep up with any needed moves without getting in the way of production workloads was part of the orginal specifications.

Simple Tier Management

FAST VP supports 3 tiers of storage, and it does this without mixing the drive types into mixed pools. The virtual pools are constructed from protect space on a given technology (flash, high speed 10/15K drives, or bulk drives) and RAID type. FAST VP Tiers are built from up to 4 pools that are based on the same technology. FAST VP Policies are used to combine 2 or 3 tiers based on different technologies.

What makes this really different is the granularity of control this gives the storage manager over how a given device will be treated. Assume that a customer has an array with 3% of the capacity on flash, 27% on high speed drives, and 70% on bulk storage. Now they want to support 2 business needs: production needs all 3 tiers, but test does not need any flash, and does not need much high speed. With FAST VP, we can build policies of 5%/100%/100% for production and 0%/25%/100% for test. They can all share the same drives, so that when test is not running, production has the full use of every drive in the array. And no matter what changes the business may need to support changing I/O levels, a simple policy adjustment will reallocate the storage based on the best use given the historical performance data. If a few production devices need special attention, such as a desire to keep them on flash even when their performace may not warrant that, adding a new policy can make this happen rapidly.

Now consider what happens when tiering is built from RAID groups and combining drives into mixed pools. To allow test to have access to high speed capacity, I have to dedicate that capacity to the test pool in RAID group increments. To allow production to move data down to bulk storage, I again have to dedicate space for that in RAID group increments. And if I need to move space between different pools, that also has to be done in RAID group increments. And when customers are using 2+ TB drives, even a single RAID group can get pretty big. In the end, each of the mixed pools has to have the right physical drives in it to meet the current need, plus unused space to allow for things to move between tiers. This does not make for a pretty management picture.

EMC World 2011

For those attending EMC World 2011 in Las Vegas next week, I will be presenting "Virtual Storage: Overview and Direction" on Tuesday and Wednesday at 10 AM, which includes a review of the value of FAST VP. I would welcome your attendance at the session and a chance to discuss any questions you may have.

Conclusion

VMAX FAST VP makes it simple for customers to take advantage of the performance of Flash drives in the most cost effective way - by only putting the hottest data on them. It also makes it easy to save money by pushing inactive data to cheap bulk storage drives. The VMAX has the power and the smarts to move the right data between tiers while improving production throughputs and decreasing response times. Now the trusted Symmetrix brand is a smarter choice than ever.

The Problem

Customers are constantly adding more capacity to their systems. Application retain more data. New applications are added to servers. Upgrades need more space. It seems that every few months each server is getting some additional storage capacity. So how hard should it be? After all, the servers and storage are already connected.

So let's make this a little more complicated. Many customers are now rolling out VMware ESX clusters of 8 or more servers. Some of these servers have more than 2 HBA ports. So when adding a device to the cluster, there may be 30 or more HBAs that all need to get their access updated.

The Solution

The EMC Symmetrix VMAX has a unique way of managing host access to LUNs. Most arrays fit into one of two storage methods:

1. Manage LUNs to each HBA, or
2. Manage LUNs to groups of HBAs (one one or more servers)

The EMC Symmetrix DMX fits into the first camp. Each LUN to HBA relationship is managed independently. For large cluster environments (ESX or otherwise), manging access to each HBA is both time consuming and risky. It takes careful processes to ensure that the new LUNs are always matched with all of the right HBAs when it is added. Are the processes always updated when an HBA is replaced or a server is added to/removed from the cluster? This was a very workable design when SANs were smaller (and so were the clusters). A better option is needed.

The EMC CLARiiON line has been a great example in the second camp for years. Host groups are defined, and a new host can fairly easily be added to or removed from the group. HBA changes are made on the host and reflected in the group, so that the group view stays up to date. Adding a new LUN to the group takes seconds, and is reflected across all of the hosts in the group.

The EMC Symmetrix VMAX with Auto-provisioning Groups (AP) takes this one step further. AP adds the idea of cascaded initiator groups - like nested host groups. So ESX servers can boot from the SAN, and each set of HBAs will be stored in an initiator group for that server. These are paired with private LUN(s) to create a storage view. Then a cluster-wide initiator group is created which contains the initiator group of each server - not by WWN, but using the (cascaded) names of the server initiator groups. The cluster-wide LUNs are then associated with the cluster-wide initiator group to expand the storage 'view' for each host in the cluster. In this way, it remains easy to manage both the host access to private LUNs and the cluster access to shared LUNs.

A nice description of all the details of such an implementation with VMware can be found in the EMC White Paper HERE.

The Results

VMAX customers can easily manage large amounts of storage even in complex clustering environments. For cluster hosts that need private LUNs, whether for boot information, local configuration storage, logging, etc., each server can easily have a private storage view. Adding a private LUN just means adding it to the private view for that host. And changing an HBA is a simple update to the initiator group for that host.

For the shared LUNs, adding capacity is just as easy. Add the LUN to the shared view, and all servers in the cluster get access on all of their defined HBAs. And since the initiator group is cascaded, updates to the HBA information for a given server automatically flow up to the cluster view.

Additional Information

I am confident that some folks reading this will say that the answer is easy: use 'Thin Provisioning' (for VMAX, Virtual Provisioning) to give the servers huge LUNs, and let them consume the space as they need it. And for some environments this works very well. However, it also presents a new risk: storage bankruptcy. That is, an application (like an Oracle database) goes to write to some of the 'free' space it has on its drive, only to have the write fail because the array (or at least that thin pool) is out of space. Hopefully, the application can crash cleanly and be recovered when new storage is added. Maybe not. The point is, storage management under Thin Provisioning changes from something that can slow down the implementation of a project to something that needs to be monitored 24x7 or it can create catastrophic data loss.

The point is, some customers have a very tough time ordering 'just in time' storage. They need a purchasing process that allows them to purchase at a later time the storage they did not buy when the applications were implemented. If the project was justified based on the initial costs, then subsequent growth costs may be left on the budget of IT, rather than being paid by the business unit that is using the application. This presents a major process and accounting challenge for some businesses, which they will need to adjust to as IT moves into a service delivery model.

There is also the challenge that handing out huge LUNs invites the users of those systems to use the space. After all, who would not like to have space for an extra database export, or room to rebuild a set of image files, or.... Unused disk space has a way of filling up, and unless there is a reason to be careful about what it fills up with, there can be a lot of waste.

Hopefully this won't be taken the wrong way. I believe that Thin Provisioning is a great thing for customers (would we make if if EMC did not see the value?). I also think that it cannot live up to the industry hype ("all systems at 100% utilization with less administrative work and no risk" - if your thin pools are at 100% full, there is a lot of risk...). I recommend that customers not over-subscribe their production 'thin' space until they have run out of space twice in a non-production setting. Once that has happened, they will understand what to watch for, how serious it is, and what needs to be done to restore normal operations (even if that means deleting some devices so that others have the space to function).

Better storage management tools is another option. There are some applications, such as EMC Ionix ControlCenter, that can mask the complexity of storage management from the users. By creating cluster entities and allocating storage to those entities rather than the individual servers, customers are able to achieve a simplified view of the ongoing operations. The actual array views of the access, however, are still those of the native array tools (where VMAX excels). 

Conclusion

VMAX customers can more easily manage both private and shared LUN access to clustered servers thanks to Auto-provisioning Groups and the cascaded initiator groups they support. And simpler storage management is always helpful.

The Problem

This customer deals with continuous data feeds, both coming in and going out.  They need to have the data integrated into their overall information framework within very specific time windows.  They know how much data they need to be able to move - after all, they are in the information business.  But how does that translate into a storage solution?

They mapped their performance needs to a set of hardware requirements.  They took these requirements to multiple vendors, and they got very different solutions from each.  They have enough relationship with their vendors to know that all of the solutions would eventually meet the need.  However, it was possible that some were not going to support the full (planned) production workload with only the gear being proposed.  Price was the major factor in the purchase decision between workable solutions, and the vendors were well aware of this.  Might some be planning to fix things later with upgrades, or hoping that the planned workload would never appear?  (Customers have to consider this possibility, even if only causes them to do enough checking to 'trust but verify' - thus the pictured fake news clipping)

The customer originally started out by asking for a collection of specific storage components - drives, cache, etc.  The vendor responses either had all of those components, or had substitutes and explanations for why the substitutes were suitable.  The differences in the architecture of the systems that they were looking at purchasing turned this into comparing apples, oranges, and grapefruit (as a general metaphor for the problem, not a specific comparison of these products).  So how were they really supposed to compare the proposed solutions?

Sure, the solutions all had to have the right usable capacity.  But their application had serious performance needs.  If they got into production and then found a problem with the system, would they be asking questions about the ability of the storage to keep up with the rest of the environment?  They needed a way to ensure that the solution was mapped back to their true business requirements.

This was also a challenge for the vendors.  Each had their own solution which they believed was well suited to the need as it was described to them.  (I assume that the competition has the same goal of a satisfied customer at the end of the day.)  So how could they convince the customer that their solution was complete enough to meet the business need?

As a part of the discussion with the customer, we noticed that they were making a leap from the performance that they needed for their application to the storage components that they were looking to buy.  This is a necessary step, but one we were convinced they should not be handling (or at least not alone).

The Solution

EMC proposed that the customer let the storage vendors do the translation from the application performance needs to the solution.  Since the solutions were not going to be simple to compare at the hardware level, and the customer did not have specific hardware need (it was a business need), why not make the purchase based directly on the business need?

We worked with the customer to detail their application performance profile.  They not only had their older production system as a model, but they had large-scale testing of the new version well enough developed to have a really good handle on their I/O profile.  We worked to build a model.

Now the customer had a new set of requirements:  T iops at 8 KB that are 20% write and 95% random, L iops at 8 KB that are 60% write and 100% sequential, and W iops at 32 KB that are 15% write and 80% random.  (The model here is a rough set of requirements, not the actual requirements of the customer.)  They could then ask the vendors to propose both a solution to meet these requirements and a method to demonstrate that the requirements were met.

The Results

The change in the discussion was immediate.  Now the vendors had a specific performance target to meet.  EMC believed that we met the new performance requirements with the configuration that we had been proposing.  We offered guarantee language that put a performance test in as a part of the acceptance criteria, which included the use of the iorate performance tool and detailed the performance model translated into the iorate input files.

This was also much simpler on the customer.  Rather than trying to reason out all of the vendor claims about how the work was done, they were able to work from their specification of what they needed the array to handle.  It was then up to the vendors to propose configurations to meet the need, and to discuss how to document that to the customer.

Additional Information

It turns out that there was additional value in actually performing the storage performance test.  There were some host settings that were not optimized for multi-path I/O.  As a result, the array was not able to meet the performance needs during the initial test - we were waiting on one HBA to do the work of 4.  By finding this early, the customer eliminated a problem that might not have otherwise been found until production was live, when it would be both harder to diagnose and more complex to change (as changes to a live production system always are).  This gave the team confidence in what they could expect out of the array and from an individual server to the array.  When questions came up later in the pre-production testing, the team knew if the storage was getting close to their tested values - and if not, then there was probably not a problem in the storage area.

As a proper disclaimer, those who read through the details of iorate, or read my about page or LinkedIn profile, will find that I am the author of the iorate performace tool discussed here.  That has limited impact (beyond being familiar with it) on the choice of this tool for general storage testing.  The tool is array independent, and was developed to help another customer do storage performance profile testing back in the late 1990s.  Since then it has become fairly widely used across the industry (I have seen IBM propose it as a tool to test their storage, among others) and around the world (I get e-mail questions from all over).  I don't ask for anyone to trust that the code is independent - feel free to read through the source code, or even compile your own version.

One additional note on storage performance testing is that not all arrays test the same.  For example, the IBM XIV array does not save blocks of all zeros to disk (they log it as empty tracks of data).  So a performance tool that was only writing zeros (and then reading them back) would be very skewed if testing XIV against other platforms, since the XIV would not be doing any I/O to disk.  Since the real data will be non-zero, the testing data needs to be as well (iorate uses non-zero data by default).  Similarly, the NetApp arrays with WAFL work to turn all writes into sequential writes (much easier on the back-end disk drives).  As a result, the write performance seen from a 'fresh' NetApp file system can be dramatically different from one that is very full/fragmented (so that the array has to find smaller places to put writes, forcing the RAID read/modify/write workload onto the drives).  Customers need to understand such attributes of their arrays to be able to model the performance that will be sustained for their actual production workloads.

This customer has a need to provide replicated data for disaster recovery.  Their current choice for this system is to use the database tools to perform the replication.  However, they have a complex (3 site) recovery model, and made sure that they understood the storage-based options that were available should the database replication not met their needs in the future.

Conclusion

Customers can benefit from documenting the business/application requirements for their systems and leaning on their vendors to meet those needs.  The vendors understand the abilities of their systems much better than most customers ever will.  And, of course, the customer needs to 'trust but verify.'  In the process, everyone will learn more about the final solution before it goes into production.

The Problem

I have a corporate laptop.  I also have a home desktop system.  I need to be able to easily move files between them – mostly personal documents and media.  Some of the files are rather large.  Thumb drives are inconvenient.  And since I have a home network, using anything other than the network to move these files just feels wrong.

I also need to share things with my family.  My wife has a home desktop system and a laptop as well.  The kids each have a desktop system (hand-me-downs).  And when someone creates a document they want the others to have access to, we need a way to do it easily.

Then there are the friends and family that come by.  Many use the WiFi network.  Then we want to share pictures (without cutting them down to nibs or posting them to a public area like Facebook).  And again, we want it to be easy.

Of course, I also have tax records and other documents that, while I would like to store them centrally for easy access, I have a real need to secure.  Sorry, I don’t trust the WiFi security for much.

After trying and failing with a Windows security solution (more below), I decided it was time for a simple NAS solution.  After all, I helped roll out one of the first NFS networks for CAD design while I worked at Lockeed, so I was not shy about the technology.  And EMC had recently purchased Iomega, which had just released a new low-end NAS device.  So I went online and bought a 2 TB ix2.

 The Results

I have to say that the Iomega team got the idea of simple down well for this little unit.  The ix2 connected to the network with DHCP and was ready to go in minutes.  The GUI allowed me to quickly set the configuration items I wanted (mirror the 1 TB SATA drives for data protection, lock the box down to a hard IP address so that it would always be easy to find, etc.).  Within 10 minutes I was sharing data on my network.

Now that I had a shared repository for data, I could place things like the software distributions in a central place, and easily be able to reload them on systems that were wiped or replaced.  We can easily share data, like robot and project information for our FIRST LEGO League team, between systems.  And friends and family that come to visit can push pictures and more to and from the shared space simply and easily.

Then we had our first Mac come to visit.  I was not sure exactly how well this was going to work.  The answer turned out to be better than expected.  The Mac was able to quickly find and attach to the ix2.  And it could access all of the same files that the PCs were sharing.  Then we created a directory from the Mac with some new files.  The PCs could not read the directory.  The default permissions on the Mac had the file system locked down.  So we went to the Mac and opened up the permissions, and bingo the PCs were able to work with the files from the Mac.  Really easy sharing, and nicely secure. 

Finally, I was ready to place my financial data onto the device.  So I created a new share that was password protected.  The security worked great – I had easy access, and everyone else was unable to open the share at all.  The only challenge was the Windows mount code – it seems that I have to mount the secure share before mounting the others.  Windows will not mount multiple devices from the same server with different authentication credentials, so the secure device has to be mounted first.  Figuring that out from the Microsoft error message was a challenge!

We have all of our digital photos shared on the ix2.  And a copy of the music repository for iTunes.  I noticed that TiVo can see the ix2 as a media source, and it has offered to play music straight from the device.  More useful sharing.  For now, I am using iTunes, but that sharing may come in handy later.

Additional Information 

My first solution was to use the Microsoft tools at hand.  So I put Windows 2003 Server in a VM on ESX 3.5 and created a real home Windows security domain.  I wound up making it the DHCP server (taking over from the WiFi box) to make the hierarchy work right.  Then I upgraded to XP Pro on the desktops to get the domain logins – which meant converting the file permissions and settings for everyone on their systems.  In the end, the security was overly restrictive and the systems were all more difficult to use.  I had a home rebellion.  These are great tools for the enterprise, but they did not work for my home environment.

At one point I had my own small CLARiiON CX3-20 in the basement, complete with Brocade switches and a pair of old 1 Gb FC HBAs for my server.  The power draw was significant, and the HVAC was working overtime.  Since I was not trying to run an OLTP system from home, I passed the solution to someone else for their lab and went back to the drawing board.

As for the purchase of the ix2, there is an EMC Employee discount.  After everything it said and done, that is about $5 cheaper than the online deals you can get on the box.  I would be glad to have it even if I were not working for EMC, or if EMC did not own them.  This is a solid, simple solution to a real problem I had.

Once I installed the ix2 and decided that I liked it, I added an Gb switch to the home network.  It just connects the ix2 and a few high-use devices (my desktop Dell and my Mac Mini, primarily).  The fact that the ix2 and most of the new systems all come with 1 Gb Ethernet, and the new switches being under $100, it seemed too cheap to pass up.

Conclusion

Data sharing has become a part of life in this digital era.  Having an easy way to share data at home between devices and platform types without a lot of effort makes the digital life a bit easier to enjoy.

The Problem

A retail customer purchased a set of 8 EFDs with a new array.  They had a few systems that were critical to their overall operations that they were hoping to improve.  The business would get the most value from an improvement in the PeopleSoft HR batch runs, ensuring that payroll computations were completed on time.  They had upgraded to new HP-UX servers, and found that they were now usually waiting for disk I/O.

The systems had been running on a CLARiiON CX3 array with dedicated groups of 15K FC drives.  They were to be moved to a new Symmetrix DMX-4 which had the EFDs.  The natural discussion was what data to place on this valuable EFD space.

The initial idea from the customer was that the Oracle logs would be best served by this faster space, since Oracle transactions cannot complete until the log writes are on disk.  And the logs do tend to be very busy with lots of small writes.  However, the nature of the logs is that they are all small and sequential writes.  Disk array cache eats small sequential writes for breakfast.  Not only does the cache get to reduce the number of physical disk I/Os by coalescing them, but it also gets to perform optimized RAID writes.  So while RAID 1 protection may be desired for data availability, RAID 5 will work great for the overall performance need.  And EFD would be wasted on this workload, where the only reads are sequential reads for archiving (or during recovery).

Since this is an OLTP system, most of the random I/O will be directed at the index files.  It happens that this database was already separated by index and data.  The total of the index space was about 1 TB, which fit well since they had a 7+1 RAID 5 group of 200 GB EFD, which provided more than the needed 1 TB.  New devices were created, and one weekend the file systems were moved over.

The Results

As expected, the results were magical.  The weekend batch runs were not just faster, they flew.  Overall wall clock time was cut roughly in half.  The maximum host device busy went from 100% to around 25%, clearly showing that the disks were no longer the bottleneck.  The combination of being able to have more I/Os on these devices and having the response times drop was clear.

Then Monday morning came, and the phone lines on the IT Help Desk lit up.  Everyone using the PeopleSoft system was wondering what was going on.  They system had never been this fast.  The comments were effectively, "whatever you did, do that again!"  That is always a nice way for the IT team to start a week.

Getting both the batch performance improvement and the tremendous user feedback was like lightning striking twice.  Moving the rest of the database onto the DMX-4 also went well.  Performance was slightly better than it had been on the older CLARiiON, though the difference from this array move was nothing like the change to EFD.

Additional Information 

For those not familiar with EFD, this is the high end of solid state storage.  These devices use Single Layer Cell (SLC) technology, which provides faster access, lower error rates, and a much higher numbers of erases (re-writes) before the cells die than MLC.  EMC provides these drives with a similar maintenance cost to other hardware items as a percentage of purchase price - so having them wear out in much less than 5-6 years could get really expensive for EMC.  To help this, about 20% of the raw space is reserved for spare tracks (as opposed to a couple of cylinders on the average hard drive), so that the drive continues to operate normally even after some of the cells have failed.  And EMC holds a lot less space in reserve than other vendors shipping these same drives, perhaps because of the additional experience we got testing them for 2 years before we released them.

EFD Magic

The first magic of the EFD drives is that they can do a lot more I/O – think at least 30 times more iops per drive than a 15K FC drive before the response time starts to head up because of queuing.  The additional magic is that all of these I/Os are handled in less than 1ms.  This compares to around 6-7ms for random reads on a 15K FC drive.  And the response time and throughput for EFD are about the same whether the workload is random or sequential, and fairly independent of block size.  This is definitely a big change.

Data Placement

For those looking for general Oracle LUN suggestions, the index files should be placed on the fastest random read drives, since that will deliver the most value.  This is what makes index data such a great target for EFD.  A large SGA can cut the amount of physical I/O dramatically, making the performance demands on the drives much lower – the fastest I/O is the one you never do.  If a large portion of the data can be served from the SGA, it is likely that EFD drives will have limited impact and may not be worth the investment.  Data files should also be placed on fast drives, though they are less important for most OLTP systems than the index files.  Log files can be placed on almost any cached RAID type, since the cache will handle the writes.  Logs should be placed on their own host devices (LUNs), so that their I/O does not wait for any other devices.  The archive logs are fine to place on large ATA space, especially since they are only used in a sequential manner and ATA drives have good sequential performance.  Systems will generally perform better if there are at least 10 active devices per HBA port driving I/O, so that queuing does not add latency.  These recommendations are in place for this customer, and have served them well.

Array Cache

The Oracle SGA can do a fine job of caching recently used data.  Customers need the array cache to work on holding onto data that the SGA was not able to retain.  Or reading ahead to get data into cache that the database is likely to use in the near future.  And, of course, the array can cache writes in a fault tolerant design that is not available in the server memory.  If there are multiple workloads sharing a single array, and there usually are these days, then the ability to partition the cache by workload may be necessary to ensure that high bandwidth applications (like backups) don't swamp out the data from more response time sensitive applications.  The Symmetrix cache is very good in these areas.  When selecting an array to use for Oracle databases, customers should consider the cache management abilities of the array.

Database Logs

One final note on Oracle logs.  They should always be placed on separate physical drives (separate RAID groups) from the data in the database.  This is a consistent recommendation from Oracle, and applies to Microsoft SQL Server and most other databases as well.

Conclusion

Enterprise Flash Drives can make a huge difference in system and application performance.  By placing the most heavily used random read data on these devices, response times will drop and the number of I/Os will increase dramatically.  This can be a huge political win for an IT organization by driving increased satisfaction for internal and external customers.  It can also drive competitive advantage for the business.

The Problem

The starting size of the data warehouse was 25 TB, which would increase by about 10% per year.  This was for 2 years of retained data.  Some data was available to load throughout the day, while other sources would provide bulk feeds during the night.  With 2 years of retention, the change rate for the database was under 2% per week.  In general, backups every few days were OK, since there was a separate record of all data imported into the system and the ‘current’ image could be recreated from even a few day old copy within a matter of hours.

The business required that the data be available, and that backups be able to restore the system to a point in time up to 1 year old (there had been times of some small feeds of errant data that were corrected weeks later, so prudence dictated that extended recovery be possible).  Given the value of the operational system, there needed to be space to perform a full restore while continuing production operations.  This would also provide for space to conduct quarterly testing of the restore processes.

Given the value of the data, the business was interested in the option of having two full images of the data.  This would allow for quick resumption of production operations should there be an environmental or other event at the primary location.  This was, however, a desire with limited funding, and would only be included if it could be accomplished without greatly increasing the cost of the solution.

Looking to the future, there would be business value in being able to extend the data retention period from 2 years to 5 or even 7 years.  The more the solution is able to scale at reasonable cost and without driving up complexity, the sooner it would be cost effective to keep the data active in the warehouse for a longer period of time.

We met this challenge by focusing on snap technology.  This type of data warehouse that has large capacity and a low change rate is a perfect candidate for storage array snaps.  We delivered a pair of Symmetrix arrays with TimeFinder/Snap and SRDF, each with 35 TB of usable capacity (protected with 7+1 RAID 5 on 15K 300G drives).  This provided room for the current capacity plus another year’s growth.  By including 5 TB of capacity for the save area (Symm SAV devices), multiple snaps were possible.

In order to be able to have the space to do full restores, building a clone in the same array was considered.  However, this would also mean that the array would need to be sized for performing a full restore and testing on that restore without significant impact to production.

Given that there was also business value in being able to have a separate image of the data, the solution included a second Symmetrix array connected with SRDF.  By placing the second image in a separate array, we were able to have a completely separate platform to conduct testing of restores or other operations on if needed.  And when that was not needed, it was a business continuity solution with minimal additional cost.  The two arrays were within the same campus, providing plenty of bandwidth at minimal latency.

The Results

The customer chose to make snaps 3 times per week, and they could do more if they found value in that later.  The customer makes database snaps on Tuesday and Thursday evenings in the production (source) array, each kept for one week.  They make alternating Saturday snaps from the target system, each kept for 2 weeks.

Backups are run weekly from the Saturday snap copy.  After the backup, the Saturday snap is often mounted on a failover server for further data integrity testing, just to be sure.  Quarterly restore testing is conducted against the target array, validating the backup and restore processes.  Of course, this also invalidates the snaps on the target, as well as the business continuity options while this testing is conducted.

In the event of data problems, the various snaps are mounted to standby systems to validate the most recent alternate image that does not have the problem.  Then a decision can be made to either repair production, or to update the snap with all of the most recent data and a repaired copy of the damaged data feed.  If the repairs are done on the local snaps, the production database is shut down.  The disk restore process is almost instant, and the production database is restarted on the restored image (with the data copy completing in the background).

In the event that the repairs are done on the target array, the process is only a bit more involved.  The chosen snap copy is updated just like the local snap would be.  SRDF is then suspended, and snap is locally restored.  Once that restore is in process, the production database is shut down.  An SRDF restore is initiated with the ‘invalidate local updates’ option, which is required since both the source and target have changed.  SRDF then begins the differential resynchronization of the local data to match the desired target image.  Once this resynchronization has started, the production database is restarted on the restored image (and as with the local snap, the data copy continues in the background).

In the event that a tape restore of an even older copy is needed, the target array is used as the restore location, and then updates are applied just like they would be to a target snap copy.  Since SRDF is tracking changes between the source and target as a bit flag on each track, the restore will still be differential.  However, since all of the data will be read back from the backup copy (even if it has not changed), the SRDF restore will effectively do a full array copy.  The process remains the same, though with so much more data the copy will take longer to complete.  And production can still be restarted while the copy is in process.

Note that for major database changes, SRDF is suspended between the arrays.  In cases where there is a problem, SRDF can undo the updates at the storage level, ensuring that the change window can always be met even if something goes very wrong.

All of the devices are collected into device groups on the arrays.  The Symmetrix Group Name Service (GNS) ensures that device group updates are kept consistent between the arrays, and are available to all hosts that may be used to control the replication operations.  The group operations make management of the various copies simple.

Additional Options

The customer could run more snaps on the source or target.  However, most data corruption is fixed by making changes to production, not by restoring from a copy.  Since the daily change rate is so small and there are so few restores, there is no justification yet for performing the snaps more frequently.  Since it can take days to discover the details behind a bad data import, doing daily snaps but only keeping them a couple of days was thought to be much less useful.

The customer could add ATA space for a clone at the target site, allowing for testing of restores without disrupting the disaster recovery copy.

The backups are currently being done to actual tape for vaulting offsite.  A Data Domain backup target deduplication solution would both improve the speed of the backup and be able to replicate the changes to a remote location with minimal bandwidth.  This sort of low change rate database will get the most benefit from backup deduplication technology.  It should be very cost effective to keep 3 months of weekly backups on such a system.

Conclusion

This customer was able to meet all of their business requirements without buying an excess of infrastructure.  The snaps give them quick recovery from multiple points in time with very little additional disk capacity.  All the data recovery options can be executed while production continues with the best available data.  SRDF provides a target system for backups and restore testing, and also provides a target for system recovery should the source site be compromised.  With the proper tools, large warehouses are easy to store and protect.

The Problem

This customer has over 1,000 separate systems that are critical to ongoing operations and are located in a core data center.  The systems are a combination of Mainframe, UNIX variations (AIX, HP-UX, LINUX, etc.), and Windows.  They use SAN and NAS storage (the Mainframe CKD over FICON I include in the SAN), with over half a Petabyte of unique data and various additional copies.  They have SAP and other systems that have data flowing between different them, often with different operating systems, with no easy way to perform data isolation (picture an interaction of web services, shipping, billing, and warehouse data and you get the general idea).  The customer wanted to implement business restart, with clear business requirements.

Recovery Point Objective (RPO)

There is value in having the data during any site recovery be as current as possible.  These are not public finance systems, so the value of a minute of data was not measured in millions of dollars, but it has real corporate value.  The goal was set to lose less than 5 minutes of data.

These systems are critical to ongoing operation of customer-facing systems, including web sites.  In the event of a true site disaster, there was a desire to have the systems recover quickly.  Some systems are more customer-facing than others, so there were RTO objectives from 2-12 hours.  Of course, the 'commutative property of Disaster Recovery' (any system that I depend on needs to be recovered as fast as I am so that I can use it) brought many systems down toward the 2 hour target.  Since this short time window was not much longer than the time required to restart most of these systems, the data needed to be ready to go when the restart decision was made.  Restoring from an alternate source was not going to be an option.

Recovery Geography Objective (RGO)

These systems support customers across the US and, with some functions, internationally as well.  The recovery needed to be able to handle geographic issues such as the northeast power outage and still be available.  The plan had to deal with the loss of power or communications not just for a site, but for a 500 mile radius.

The Solution

We proposed a solution of Symmetrix arrays with SRDF/A and Multi-Session Consistency (MSC).  To handle this scale of unique data, there were 8 arrays at each site.  SRDF/A provided the data replication, connected with Cisco FCIP technology over 4 OC-12 links.  SRDF moves the data from the (outsourced) source site to the target.

SRDF MSC has the unique ability not only to create multiple groups within a Symmetrix system, but also to define groups acrosssystems, and then take actions on a group as a whole without sacrificing consistency.  SRDF/A sends the differences between one point in time and the next (the 'delta set') from the source to the target with each cycle switch.  By default, the cycles operate within a given array and are independent of each other.  MSC has the ability to synchronize the cycle switches, so that multiple arrays will make the switch at the same point in time.  This creates a single image of all of the data that is consistent from a write-dependency point of view.  The delta sets are then transmitted to the remote site, and are either all applied (if they all make it completely) or all discarded.  In this way, the target site always has a valid, consistent view of the data on disk across the arrays.

The big magic of MSC is that we are not trying to get all of the writes to the target in order.  Given the level of I/O on these systems, the complexity of putting a precise enough time on each I/O without some common clock to reference would be massive.   The common Mainframe solution for this problem is to use the clock from MVS to provide the timestamp on the I/Os, so that they can be replayed at the target site in the right order even of there are multiple arrays involved.   In this case, there are many unrelated systems without the level of clock synchronization to make such a solution possible.  By managing the cycle switch times across the arrays, SRDF MSC eliminates this problem.

Servers are located at both the source and target sites.  People and processes are in place to deliver a quick restart after the declaration of a disaster at the source site.

The sites are over 600 miles apart, with separate power and telecommunications grids.

The Results

The customer has a single, consistent replica of over 500 TBs of data at over 600 miles that is within 2 minutes of the state of the source at any given time.  While MVS is in control, the data also supports UNIX and Windows systems running a wide variety of applications.  With this design, recovery is greatly simplified since all of the interconnecting systems will recover with data from the same point in time.

All of the business objectives have been met, and there are no near-term limits on scale or other items that would impact the future usefulness of this solution.  They are free to add or change anything above the storage layer and maintain the needed business consistency.

Additional Observations

Most large businesses have a DR plan.  However, many of these plans overlook critical items or assuming things that may not be accurate.  It is difficult for many businesses to understand the value of the last X minutes of data.  For a manufacturing concern, the value of the last hour of production and shipping, or even the last day, may not make a significant difference in the value of the business.  For an online business, the value of the last hour may be very significant.  For an investment or banking firm, losing the last hour of records may mean the end of the business.  Figuring out where each part of a business lives on the recovery value spectrum can be an arduous task, and shortcuts can lead to bad decisions.

This customer realized that they did not want to own two data centers.  So they found a good partner, and only have ownership of one.

They also considered the fact that the restart of the business after a disaster can actually be more difficult than ongoing operations.  And if things are bad enough that you have to shut down a data center, there are probably real problems in the primary data center geography.  Even if air travel or roads are available, employees may have family or other concerns that could prevent them from traveling to the alternate site.  So how do you get all that expertise to the target site, and make it OK for them to stay there for what may be months?

They decided that they needed to staff both sites.  They have an operational team for each.  And the big difference from what many customers have done is that they have the engineering team at the target site.  The primary data center is the outsourced one, supporting ongoing operations with a staff trained for that.  The alternate data center is where the ongoing changes are designed, and where all the expertise lives to handle any challenges that come up in the event of a site failover.  This is a very innovative solution to the problem of staffing the target site, and I believe more customers will follow this in the future.

Conclusion

This customer has a long-standing relationship with EMC.  The companies have been doing business together for years, as EMC provides them with solutions that help them to meet their business needs.  The customer takes the time to meet with EMC and discuss their needs in detail, including trips to meet with our management and engineering staff to help us better understand where they are going with IT and what we might do to help.  By building a real partnership, we are able to work together to solve their ongoing challenges.  SRDF MSC is an example of a technology that EMC has built to help our customers meet their business needs.

The Problem

The customer is a transportation company. They perform large nightly database updates. These updates have critical timelines for completion, so that the results are ready for use when other processes kick off in the morning. The updated data relates directly to the profitability of ongoing operations. 

The application had been running OK, though it used a lot of hardware (Sun E10K servers and a pair of DMX-4 arrays, duplicated for DR and linked with SRDF). Since the application was purchased from an outside vendor, the tuning options were limited. The customer had worked with the vendor on several occasions to optimize the nightly batch processing. Benchmarks were conducted. Extensive examination was done of the data model. It appeared that the database and application were about as tuned as they could be, and more hardware was going to be the only path to keep up with growth.

EMC was engaged to perform an Oracle assessment. The proposed assessment provides a complete analysis of a single production database instance. It looks at everything from network traffic to CPU and memory usage to semaphores and buffer usage to file systems and disk I/O. This particular assessment is conducted in a 4-day format: 

• Day 1: Install the tools
• Day 2: Gather the data
• Day 3: Process the data
• Day 4: Discuss the results

When a system generates this level of I/O, the reports coming out can get rather large. There is also an index noting which areas are the highest priority to fix, and the risk level associated with each change.

The Results

There were several high priority suggestions that came out of this report, but one was critical. There were a few SQL statements that were shown to be generating a huge volume of I/O that was flagged as unnecessary. 

APPLICATION Observation: Some SQL statements are performing an excessive amount of I/O. This can slow performance, since this might cause other cached data buffers to be quickly aged out of the SGA, thereby causing subsequent queries to perform physical I/O rather than logical I/O. This can be caused by over-indexing tables, joining tables in the wrong order, or using an inappropriate index, or if there is a missing index that Oracle could have otherwise used to reduce the total number of I/O's. This can also be caused by SQL statements which use numeric group functions (such as AVG, COUNT, MIN, MAX, and SUM). Queries which perform many Logical I/O's consume an excessive amount of CPU cycles and can cause latch contention problems, due to the serialization of latches. The highest number of I/O's per execution measured was 36,406,850. The total number of excessive I/O's measured was 2,505,090,174.

With the additional data around the execution plan for these statements that was included in the detailed report, it was easy to identify what should be changed. As a result, the customer was able to take a clear set of directions back to the application vendor for a fairly simple adjustment.

Conclusion 

The vendor was able to quickly integrate the changes and eliminate over 1 billion I/Os per day. The assessment cost less than the value generated by not missing the nightly batch window for a single night. And it may have eliminated the next hardware upgrade cycle, or at least allowed it to be completed with a much lower scale of system and storage. This is an example of the results that are possible with a custom Oracle engagement from EMC. Obviously, most systems will not get this level of performance boost, as most systems never get near this level of I/O. So how many I/Os might your systems be able to avoid?