Let’s jump right to the answer…

In this test, enabling jumbo frames improves iSCSI performance by nearly 10% and NFS performance by almost 15%. Improving performance by 10-15% is a significant win when there is no cost required to do it. The simple change of enabling jumbo frames on the ESX servers, the network switch, and the storage array made the existing infrastructure faster. Much like block alignment, there is no downside.

Infrastructure details: The test infrastructure consists of 4 dual socket Intel Nehalem-EP servers with 48GB of RAM each. Each server is connected to a 10GbE switch. A FAS3170 is connected to the same 10GbE switch with a single 10GbE link. There are 200 virtual machines: 50 Microsoft Windows 2003, 50 Microsoft Vista, 50 Microsoft Windows 2008, and 50 linux. Each operating system type is installed in a separate NetApp FlexVol for a total of 4 volumes. The guests were separated into multiple datastores to allow the VMware ESX systems to use 4 different NFS mount points on each ESX system. Each physical server mounts all 4 NFS datastores and the guests were split evenly across the 4 physical servers. The timing listed in the table is the time from the start of the 200 systems booting until the time the last system acquired an IP address.

The F20 card is a SAS controller with 4 x 24GB flash modules attached to it. You can find more info on the flash modules on Adam Leventhal’s blog and the official Oracle product page has the F20 details.

All of my tests used 100% random 4KB blocks. I focused on random operations, because in most cases it is not cost effective to use SSD for sequential operations. These tests were run with a variety of different thread counts to give an idea of how the card scales with multiple threads. The first test compared the performance of a single 24GB flash module to the performance of all 4 modules.

4KB Random Operations

At lower thread counts the 4 module test is roughly 4x the operations per second of the single module test. As the thread count rises, the single module test tops out at 35,411 ops and 4 modules can deliver 97,850 ops, or 2.76x the single module test. It would be great if the card was able to drive the 4 modules at full speed, but 97K+ ops is not too shabby. What is more impressive to me is that those 97K+ ops are delivered at roughly 1ms of latency.

The next round of testing included three different workloads. The three phases were 100% read, 80% read, and 100% write and they were run against all 4 flash modules. Again, all tests used 4KB random operations. Here are the operations per second results.

And a throughput version in MB/s for anyone that is interested.

Flash and solid state disk (SSD) technologies are developing at an incredibly fast pace. They are a great answer, but I think we are still figuring out what the question is. At some point down the line, they may replace spinning disk drives, but I do not think that is going to happen in the short term. There are some applications that can leverage this low latency capacity inside the servers today, but this is not the majority of applications.

Where flash and SSD make more sense to me is as a large cache. NetApp and Sun are using flash this way today in their storage array product lines. DRAM is very expensive, but flash can provide a very large and very low latency cache. I expect we will see more vendors adopting this “flash for cache” approach moving forward. The economics just make sense. Disks are too slow and DRAM is too expensive.

It would also be great to see operating systems that were intelligent enough to use technologies like the F20 card and the Fusion IO card as extended filesystem read cache. Solaris can do it for zfs filesystems using the L2ARC. As far as I know, there are no filesystems that have this feature in the other major operating systems. What about using as a client side NFS cache? At one point, Solaris offered CacheFS for NFS caching, but I do not believe it is still being actively developed. While CacheFS had its challenges, I believe the idea was a very good one. It costs a lot more to buy a storage array capable of delivering 97K ops than it does to put more cache into the server.

Here is a chart showing the difference between aligned and misaligned random read operations to a Sun F20 card. I guess it is officially an Oracle F20 card.

Oracle F20 - Aligned vs. Misaligned

With only a couple threads, the flash module can deliver about 50% more random 4KB read operations. As the thread count increases, the module is able to deliver over 9x the number of operations if properly aligned. It is worth noting that the card is delivering those aligned reads at less that 1ms while the misaligned operations average over 7ms of latency. 9x the operations at 85% less latency makes this an issue worth paying attention to. (My test was done on Solaris and here is an article about how to solve the block alignment issue for Solaris x64 volumes.)

I have seen a significant increase in block alignment issues with clients recently. Some arrays and some operating systems make is easier to align filesystems than others, but a new variable has crept in over the last few years. VMware on block devices means that VMFS adds another layer of abstraction to the process. Now it is important to be sure the virtual machine filesystems are aligned in addition to the root operating system/hypervisor filesystem.

Server virtualization has been the catalyst for many IT organizations to centralize more of their storage. Unfortunately, centralized storage does not come at the same $/GB as the mirrored drives in the server. It is much more expensive. Block misalignment can make the new storage even more expensive by making it less efficient. If the filesystems are misaligned, then it makes the array cache far less efficient. When that misaligned data is read from or written to disk, the drives are forced to do additional operations that would not be required for an aligned operation. It can quickly turn a fast storage array into a very average system. Most of the storage manufacturers can provide you with a best practices doc to help you avoid these issues. Ask them for a whitepaper about block alignment issues with virtual machines.

Last week we had an executive briefing for folks interested in Exadata V2. My colleagues Kurt Rosenfeld and John Laferrier presented information on business intelligence and the Exadata, as well as the business case and use cases for considering buying one. Joe LaFlamme from Oracle presented some reference customer examples.

I presented the Exadata V2 technical overview, traveling through the architecture details, migration strategies, and component details. Along the way there were a few points I made that seemed a bit surprising to the audience, and that led to a lively discussion. I summarize those points here, as they do not seem to be well known within the industry.

• Existing Oracle licenses are transferable to Exadata (including Oracle DB, RAC, and Partitioning). That can greatly reduce the cost of an Exadata that is being used for database consolidation, for example.

• The Exadata looks to be an excellent consolidation engine. Included with the Exadata software are resource management tools that can, for example, give some databases resource priority over others. These tools also allow the use of the flash storage to be fine tuned, pinning specific tables into flash or letting Oracle use the flash as an extended cache.

• The Exadata V2 is designed to be able to perform OLTP and Data Warehouse transactions concurrently. If a single system can be used both ways, consider the implications compared to stand-alone, separate Data Warehouse solutions. Normally data must be extracted from the OLTP system, copied to the DW system, imported there, and then processed. The extraction and copying are overhead, on both the OLTP and DW systems. And, any reports or queries on the DW system are performed against “stale data” – data from the time the extraction started. Now consider being able to do DW operations against live, current OLTP data. And according to the performance numbers published by Oracle, those operations could run much faster than on most DW systems. That speed could result in completing more complex reports, the allowing of more ad hoc queries, and so on. Such a change could be a fundamental advantage to DW consumers (finance and senior management, for example).
• Consider the cost of Oracle database software licenses. Now consider the hardware on which they run. Increasing the performance of that software gains your site more database performance at the same database license cost. The Exadata V2 is optimized to run OLTP and Data Warehouses, very quickly. The resource management software included with the Exadata, and its use as a consolidation engine, probably leads to the appliance running with more databases using more resources and with less reserved headroom than having a non-Exadata database environment. That means that, for a given number of Oracle database licenses, your site would get more database performance.

• Customization of the pre-dedefined Exadata V2 configuration is allowed. For example, if your business need required fewer database engines and more storage it is possible to get such a configuration from Oracle. Also, some sites might want to use the included Infiniband interconnect for fast backup of the data. However, the support model for custom configurations is likely to be different than the pre-defined ones. At the moment, even splitting a full rack of Exadata V2 into two racks (to prevent the rack from being a single point of failure) is a custom configuration.

• You can’t build your own Exadata V2 system. Even though the hardware components of Exadata V2 are off-the-shelf Sun servers and networking, there is “magic sauce” in the Exadata. The Exadata storage software manages the storage nodes; the Exadata servers off-load storage-centric operations to the storage nodes (again increasing the database performance you get with those Oracle licenses); and “Hybrid Columnar Compression”, a new method for compressing columns of data while still making them available for OLTP access are Exadata V2-only features. Following the Oracle / Sun best practices and blueprints, and using the same hardware components, could lead to something similar to the Exadata V2 in terms of features and performance, but the lack of those features means that it will not match the features and performance of Exadata V2.

Unfortunately, the slide deck contains some proprietary and confidential information, and so cannot be posted here. But feel free to get in touch if you would like to know more about any of these aspects (or the Exadata V2 in general). Here are some links to further reading.

Larry Ellison introducing the Exadata V2
Exadata V2
Exadata V2 Datasheet
Exadata Storage Server
Exadata Support
Oracle Exadata Blog
Kevin Closson’s Blog

When:
Burlington MA Sun Campus – Feb 2, 2010 6:00PM to 9:00 PM
Boston MA – Boston University – Feb 3, 2010 6:00PM to 9:00 PM
(Note: The same content will be presented twice – once in Burlington and once in Boston. Pick the best location and date as convenient.)

Where:
Feb 2 – Sun Microsystems Burlington Campus; 1 Network Drive, Burlington, MA
Feb 3 – Boston University, Electrical and Computer Engineering Department Photonics Center Building – Room PHO 339 (3rd floor), 8 Saint Mary’s Street Boston, MA 02215
BU Parking: Street parking available on St. Mary’s Street and Bay State Road. Metered parking spots do not require a fee after 6pm.

RSVP: To Linda Wendlandt: lwendlandt@cptech.com

Registration Required! – so we can plan food and drink

Join Jim Mauro and Shannon Sylvia for how-to DTrace, and how to use LDOMs with ZFS.

AGENDA:

6:00-6:20: Registration, Pizza and Beverages

6:20-6:30: Introductions: Peter Galvin, CTO, Corporate Technologies

6:30-8:30: Solaris Dynamic Tracing – DTrace – Jim Mauro, Principle Engineer, Sun Microsystems

8:30-9:00: LDOM Domains and ZFS: An example of creating a ZFS bootable root LDOM domain using jumpstart – Shannon Sylvia, Sysadmin, Northeastern University

9:00 Q&A and Discussion

Also we’ll be giving out official NEOSUG T-Shirts and other trinkets, and copies of the OpenSolaris CD and instruction manual.

For more information see the NEOSUG discussion forum.

What type of disk were the virtual machines stored on?

• The virtual machines were stored on a SATA RAID-DP aggregate.

What was the rate of data reduction through deduplication?

• The VMDK files were all fully provisioned at the time of creation. Each operating system type was placed on a different NFS datastore. This resulted in 50 virtual machines on each of 4 shares. The deduplication reduced the physical footprint of the data by 97%

A few interesting stats gathered during the testing. These numbers are not exact and due to the somewhat imprecise nature of starting and stopping statit in synchronization with the start and end of each test.

• The CPU utilization moved inversely with the boot time. The shorter the boot time, the higher the CPU utilization. This is not surprising as during the faster boots, the CPUs were not waiting around for disk drives to respond. More data was served from cache the the CPU could stay more utilized.
• The total NFS operations required for each test was 2.8 million.
• The total GB read by the VMware physical servers from the NetApp was roughly 49GB.
• The total GB read from disk trended down between cold and warm cache boots. This is what I expected and would be somewhat concerned if it was not true.
• The total GB read from disk trended down with the addition of each PAM. Again, I would be somewhat concerned if this was not the case.
• The total GB read from disk took a significant drop when the data was deduplicated. This helps to prove out the theory that NetApp is no longer going to disk for every read of a different logical block that points to the same physical block.

How much disk load was eliminated by the combination of dedup and PAM?

• The cold boots with no dedup and no PAM read about 67GB of data from disk. The cold boot with dedup and no PAM dropped that down to around 16GB. Adding 2 PAM (or 32GB of extended dedup aware cache) dropped the amount of data read from disk to less that 4GB.

For details about the ZFS deduplication feature, what it does, and how it does it, have a look at Jeff Bonwick’s blog post on the topic. He was the lead engineer on the project so you can take his word on it.

Deduplication was integrated into OpenSolaris build 128. That takes a little explanation. Solaris is Sun’s current commercial operating system. OpenSolaris has two flavors – the semiannual support-able release, and the frequently-updated developer release. The current supportable release is called 2009.06 and is available for download here. Also at that location is the “SXCE” latest build. That distribution is more like Solaris 10 – a big ol’ DVD including all the bits of all the packages. OpenSolaris is the acknowledged future of Solaris, including a new package manager (more like Linux) and a live-CD image that can be booted for exploration, and installed as the core release. To that core more packages can be added via the package manager.

For this example I started by downloading the 2009.06 OpenSolaris distribution. I then clicked on the desktop “install” icon to install OpenSolaris to my hard drive (in this case inside of VMware Fusion on Mac OS X, but it can be installed anywhere good OSes can live). At the conclusion of that step, my system is now rebooted into 2009.06. The good news is that 2009.06 is a valid release to run for production use. You can pay for support on it, and important security fixes and patches are made available to those with a support contract. The bad news is that deduplication is not available in that release. Rather, we need to point OpenSolaris at a package repository that contains the latest OpenSolaris developer release. Note that the developer release is not supported, and performing these next steps on Opensolaris 2009.06 makes your system unsupported by Sun. But until an official OpenSolaris distribution ships that includes the deduplication code, this is the only current way to get ZFS deduplication.

pbg@opensolaris:~$ pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
Refreshing catalog
Refreshing catalog 1/1 opensolaris.org
Caching catalogs ...

Now we tell OpenSolaris to update itself, creating a new boot environment in which the current packages are replaced by any newer packages:

pbg@opensolaris:~$ pfexec pkg image-update
Refreshing catalog
Refreshing catalog 1/1 opensolaris.org
Creating Plan . . .
DOWNLOAD                                  PKGS       FILES    XFER (MB)
entire                                   0/690     0/21250    0.0/449.4
SUNW1394                                 1/690     1/21250    0.0/449.4
. . .

A few-hundred megabytes of downloads later, OpenSolaris adds a new grub (on x86) boot entry as the default boot environment, pointing at the updated version.

A clone of opensolaris-1 exists and has been updated and activated.
On the next boot the Boot Environment opensolaris-2 will be mounted on '/'.
Reboot when ready to switch to this updated BE.

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://opensolaris.org/os/project/indiana/resources/relnotes/200906/x86/
---------------------------------------------------------------------------

A reboot to that new environment brings up the latest OpenSolaris developer distribution, in this case build 129:

pbg@opensolaris:~$ cat /etc/release
                       OpenSolaris Development snv_129 X86
           Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 04 December 2009

Finally, ZFS deduplication is available in this system.

pbg@opensolaris:~$ zfs get dedup rpool
NAME   PROPERTY  VALUE          SOURCE
rpool  dedup     on             default

Let’s try using it:

pbg@opensolaris:~$ pfexec zfs set dedup=on rpool/export/home/pbg
cannot set property for 'rpool/export/home/pbg':
pool and or dataset must be upgraded to set this property or value

Hmm, the on-disk ZFS format is from the 2009.06 release. We need up upgrade it to gain access to the deduplication feature.

pbg@opensolaris:~$ zpool upgrade
This system is currently running ZFS pool version 22.

The following pools are out of date, and can be upgraded.  After being
upgraded, these pools will no longer be accessible by older software versions.

VER  POOL
---  ------------
14   rpool

Use 'zpool upgrade -v' for a list of available versions and their associated features.

pbg@opensolaris:~$ zpool upgrade -v
This system is currently running ZFS pool version 22.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties

For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.

pbg@opensolaris:~$ pfexec zpool upgrade -a
This system is currently running ZFS pool version 22.

Successfully upgraded 'rpool'

Now we are ready to start using deduplication.

pbg@opensolaris:~$ zfs get dedup rpool
NAME   PROPERTY  VALUE          SOURCE
rpool  dedup     off            default
pbg@opensolaris:~$ pfexec zfs set dedup=on rpool
pbg@opensolaris:~$ zfs get dedup rpool
NAME   PROPERTY  VALUE          SOURCE
rpool  dedup     on             local
pbg@opensolaris:~$ zpool list rpool
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
rpool  19.9G  10.7G  9.19G    53%  1.00x  ONLINE  -

Ben Rockwood provides a nice blog entry discussing its use, so I refer you to that site rather than repeating the information.

Also, according to this “PSARC” (architecture plan), deduplication also applies to replication, so in essence a deduplicated stream is used when replicating data. Let’s take a look:

pbg@opensolaris:~$ pfexec zfs snapshot rpool/export/home/pbg@friday
pbg@opensolaris:~$ pfexec zfs send -D rpool/export/home/pbg@friday > /var/tmp/pbg-friday-dedupe
pbg@opensolaris:~$

Unfortunately, the current “zfs send -D” functionality is only a subset of what is really needed. With -D, within that “send”, a given block is only sent once (and thus deduplicated). However, if additional duplicate blocks are written, executing the same “zfs send -D” again would send the same set of blocks again. There is no knowledge by ZFS of whether a block already exists at the destination of the send. If there was such knowledge, then “zfs send” would only transmit a given block once to a given target. In that case ZFS could become an even better replacement for backup tape: a ZFS system in production replicating to a ZFS system at a DR site, only sending blocks that the DR site has not seen before. Hopefully such functionality is in the ZFS development pipeline.

As it stands, ZFS deduplication is a powerful new feature. Once integrated into production-ready operating system releases and appliances, it could provide a breakthrough in low cost data reduction and management. We plan to track that progress here, so stay tuned.