The ZFS Revolution, Circa 2006

In my posts on FreeNAS, I emphatically state that “ZFS is the best filesystem”, but if you follow me on social media, it’s clear that I don’t really love it. I figured this needs some explanation and context, so at the risk of agitating the ZFS fanatics, let’s do it.

When ZFS first appeared in 2005, it was absolutely with the times, but it’s remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10:

• ZFS achieves the kind of scalability every modern filesystem should have, with few limits in terms of data or metadata count and volume or file size.
• ZFS includes checksumming of all data and metadata to detect corruption, an absolutely essential feature for long-term large-scale storage.
• When ZFS detects an error, it can automatically reconstruct data from mirrors, parity, or alternate locations.
• Mirroring and multiple-parity “RAID Z” are built in, combining multiple physical media devices seamlessly into a logical volume.
• ZFS includes robust snapshot and mirror capabilities, including the ability to update the data on other volumes incrementally.
• Data can be compressed on the fly and deduplication is supported as well.

When ZFS appeared, it was a revolution compared to older volume managers and filesystems. And Sun open-sourced most of ZFS, allowing it to be ported to other operating systems. The darling of the industry, ZFS quickly appeared on Linux and FreeBSD and Apple even began work to incorporate it as the next-generation filesystem for Mac OS X! The future seemed bright indeed!

Checksums for user data are essential or you will lose data: Why Big Disk Drives Require Data Integrity Checking and The Prime Directive of Storage: Do Not Lose Data

2007 to 2010: ZFS is Derailed

But something terrible happened to ZFS on the way to its coronation: Lawsuits, licensing issues, and FUD.

The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. Although ZFS definitely did not copy code from NetApp, the copy-on-write approach to snapshots was similar to WAFL, and those of us in the industry grew concerned that the NetApp suit could impact the future availability of open-source ZFS. And this appears to have been concerning enough to Apple that they dropped ZFS support from Mac OS X 10.6 “Snow Leopard” just before it was released.

Here’s a great blog about ZFS and Apple from Adam Leventhal, who worked on it: ZFS: Apple’s New Filesystem That Wasn’t

By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. And the CDDL license Sun applied to the ZFS code was judged incompatible with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world’s server operating system.

Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and GreenBytes helped push ZFS forward in the enterprise, but Oracle’s lackluster commitment to Sun in the datacenter started having an impact.

What’s Wrong With ZFS Today

OpenZFS remains little-changed from what we had a decade ago.

Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today’s prices.

And ZFS never really adapted to today’s world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It’s laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it’s everywhere in performance PC’s.

Then there’s the question of flexibility, or lack thereof. Once you build a ZFS volume, it’s pretty much fixed for life. There are only three ways to expand a storage pool:

1. Replace each and every drive in the pool with a larger one (which is great but limiting and expensive)
2. Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff)
3. Build a new pool and “zfs send” your datasets to it (which is what I do, even though it’s kind of tricky)

Apart from option 3 above, you can’t shrink a ZFS pool. Worse, you can’t change the data protection type without rebuilding the pool, and this includes adding a second or third parity drive. The FreeNAS faithful spend an inordinate amount of time trying to talk new users out of using RAID-Z1 ¹ and moaning when they choose to use it anyway.

These may sound like little, niggling concerns but they combine to make ZFS feel like something from the dark ages after using Drobo, Synology, or today’s cloud storage systems. With ZFS, it’s “buy some disks and a lot of RAM, build a RAID set, and never touch it again”, which is not exactly in line with how storage is used these days.²

Where Are the Options?

I’ve probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it’s startlingly limiting and out of touch with the present solid-state-dominated storage world. So what are your other choices?

Linux has a few decent volume managers and filesystems, and most folks use a combination of LVM or MD and ext4. Btrfs really got storage nerds excited, appearing to be a ZFS-like combination of volume manager and filesystem with added flexibility, picking up where ReiserFS flopped. And Btrfs might just become “the ZFS of Linux” but development has faltered lately, with a scary data loss bug derailing RAID 5 and 6 last year and not much heard since. Still, I suspect that I’ll be recommending Btrfs for Linux users five years from now, especially with strong potential in containerized systems.³

On the Windows side, Microsoft is busy rolling out their own next-generation filesystem. ReFS uses B+ trees (similar to Btrfs), scales like crazy, and has built-in resilience and data protection features⁴. When combined with Storage Spaces, Microsoft has a viable next-generation storage layer for Windows Server that can even use SSD and 3D-XPoint as a tier or cache.

Then there’s Apple, which reportedly rebooted their next-generation storage layer a few times before coming up with APFS, launched this year in macOS High Sierra. APFS looks a lot like Btrfs and ReFS, though implemented completely differently with more of a client focus. Although lacking in a few areas (user data is not checksummed and compression is not supported), APFS is the filesystem iOS and macOS need. And APFS is the final nail in the coffin for the “ZFS on Mac OS X” crowd.

Each major operating system now has a next-generation filesystem (and volume manager): Linux has Btrfs, Windows has ReFS and Storage Spaces, and macOS has APFS. FreeBSD seems content with ZFS, but that’s a small corner of the datacenter. And every enterprise system has already moved way past what ZFS can do, including enterprise-class offerings based on ZFS from Sun, Nexenta, and iXsystems.

Still, ZFS is way better than legacy storage SOHO filesystems. The lack of integrity checking, redundancy, and error recovery makes NTFS (Windows), HFS+ (macOS), and ext3/4 (Linux) wholly inappropriate for use as a long-term storage platform. And even ReFS and APFS, lacking data integrity checking, aren’t appropriate where data loss cannot be tolerated.

Stephen’s Stance: Use ZFS (For Now)

Sad as it makes me, as of 2017, ZFS is the best filesystem for long-term, large-scale data storage. Although it can be a pain to use (except in FreeBSD, Solaris, and purpose-built appliances), the robust and proven ZFS filesystem is the only trustworthy place for data outside enterprise storage systems. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It’s going to be a long time before I trust anything other than ZFS!

© Stephen Foskett for Stephen Foskett, Pack Rat: ZFS Is the Best Filesystem (For Now…)

Zero to Hero: Thunderbolt

I was an early enthusiast for Thunderbolt (originally called Light Peak), since it opened up a whole new world of I/O performance. And, being a storage nerd, I was particularly keen on the combination of fast PCIe connectivity and flash storage peripherals. Although the original Thunderbolt fell short of expectations (being mainly Mac and copper, not a universal PCIe-over-fiber dream), later revisions have matured nicely.

Today’s Thunderbolt 3 boasts 40 Gbps of PCIe 3.0 bandwidth along with up to 100 Watts of power delivery and support for USB, DisplayPort, and HDMI protocols besides. Although interoperability is more of a nightmare than a dream, Thunderbolt 3 works and is supported on many platforms and peripherals. With Intel and Apple pushing solidly to promote the interface, and leveraging popularity of USB-C, Thunderbolt 3 looks to become a real multi-platform standard.

We are seeing generic motherboards shipping with Thunderbolt 3 onboard, not to mention high-end PC’s and just about every Apple Macintosh computer. It wouldn’t be surprising to start seeing Thunderbolt 3 showing up in server hardware as an interconnect for external PCIe chassis full of NVMe storage or co-processors for machine learning. And Intel is talking about moving Thunderbolt support inside the CPU for the next generation.

So What’s OCuLink?

But this wasn’t always a given. And apparently the PCI-SIG (standards body for the PCI interface) thought so too. So in 2012, word started spreading that PCI-SIG was developing a standard cabled protocol for PCIe devices off the motherboard. And this standard would be free and unencumbered by corporate overlords, Apple and Intel.

Thunderbolt had already reached the market by then, but it used Mini DisplayPort connectors and only supported PCIe 2.0, for aggregate throughput of 20 Gbps. In contrast, OCuLink would start with four lanes of PCIe 3.0, good for 32 Gbps of throughput.

Like Thunderbolt, the initial plan was for both optical and copper cables to be used. But both standards hit roadblocks as optical cables failed to materialize in volume. Although there are optical Thunderbolt cables, it’s a far cry from the silicon photonics wonderland promised by Light Peak. And although the “O” in OCuLink stands for “optical”, every implementation I could find uses the “Cu” for copper!

Early reports gushed about OCuLink for laptops, allowing external GPU’s to be connected to thin-and-light PC’s. But this has been a rare application even in the Thunderbolt space (Apple just introduced official support in 2017), and I couldn’t find any OCuLink laptops or GPU enclosures on the market.

Instead, OCuLink was picked up by server developers looking for an in-box PCIe interconnect for storage or I/O virtualization connectivity. And the standard 48-pin connector and cable was sometimes re-purposed to carry multiple 12 Gbps SAS 3.0 channels as an alternative to HDminiSAS, replacing the old 4-channel SFF-8087 mini-SAS connectors we in storage have learned to love.

SFF-8639, SATA Express, and U.2

OCuLink has a kinship to another PCIe interface that might just be more popular. At the same time that the OCuLink cable was introduced, the PCI-SIG also pushed for a drive-attachment interface with PCIe using a connector called SFF-8639. This was initially used for SATA Express drives, a 2-lane PCIe storage interface.

However, in 2015 SFF-8639 was officially renamed “U.2″ for four-lane PCIe storage applications. This has become somewhat more popular. So, in a way, U.2 is a cousin of OCuLink and some devices might even use OCuLink protocol over the U.2 connector!

U.2 drives look like conventional 2.5” SSD’s. So they might take off in servers and datacenter applications. But lately, most PCIe storage implementations are leaning towards the compact M.2 interface instead. And on the pro-sumer side, it’s almost as difficult to find a U.2 motherboard or drive as it was to find SATA Express!

OCuLink-2

In 2016, PCI-SIG announced OCuLink-2, bringing PCIe 4.0 bandwidth and a new connector. Ironically, the OCuLink-2 external connector and cable bears a strong resemblance to full-size DisplayPort, bringing us back to the days of Thunderbolt’s Mini-DisplayPort cable.

So far, there has been little discussion of OCuLink-2, but perhaps that’s changing. The aforementioned Tyan server chassis actually appears to use OCuLink-2, supporting either PCIe 3.0 x8 or 16 6 Gbps SAS/SATA drives.

Stephen’s Stance

With the advent of AMD Threadripper and Epyc, we are about to see an explosion of PCIe lanes in the pro-sumer and datacenter market. Although many of those lanes will be taken up by conventional PCIe cards, some will be used for SSD’s (M.2 and U.2) or for external connectivity. This is where OCuLink might finally take off: As an AMD alternative to Thunderbolt for external PCIe peripheral connectivity.

© Stephen Foskett for Stephen Foskett, Pack Rat: What is OCuLink?

Storage in the Cloud

First up is NetApp Heads Into Microsoft’s Azure Cloud, wherein I discuss my impressions of an overlooked announcement at the recent NetApp Analysts Day in Boulder, CO. In short, NetApp is putting their enterprise storage expertise behind a new NFS storage offering inside Microsoft Azure.

This is major news since it extends NetApp’s Data Fabric into the cloud for real, completely integrated with Azure. It’s a great fit with Microsoft’s world of enterprise cloud solutions and could really help pull old NetApp out of the datacenter.

Next, I posited that 2017 is shaping up as The Year of Cloud Extension, as more and more storage systems truly connect to the cloud. See a trend here? It’s not just a gateway to the cloud, it’s real integration from datacenter to cloud.

If you’re not reading Gestalt IT regularly, you’re really missing out. Along with content from me, you’ll find some from Tom Hollingsworth and lots more by Rich Stroffolino just about every day! Subscribe to the newsletter and check out the new On-Premise IT podcast too!

Stephen’s Stance

The Big Story here isn’t that some storage is connected to the cloud. It’s that we’re fundamentally changing how we look at storage locality. We’ve always thought of storage as being “here”, whether it’s in the datacenter or on our own devices. But we’re starting to see storage as being fundamentally “there”, in the cloud, with a local representation wherever it’s needed.

From iCloud Photos to Google Drive to NetApp and Primary Data, we’re putting storage wherever it needs to be. And this is a major shift for computing, from the iPhone to the datacenter. Watch this space! And read about it on Gestalt IT.

© Stephen Foskett for Stephen Foskett, Pack Rat: Storage is Getting Cloudier!

Error Recovery Basics

Hard disk drives have more points of failure than most other modern computer components: They are physical devices that rely on magnetism and mechanical precision, not just solid state electronics. And ever-increasing drive density magnifies the challenge of always returning valid data. In fact, magnetic disk media is surprisingly unreliable, with hard drives often relying on error recovery technologies to cover for read and write errors.

The most basic form of error recovery on hard disk drives is CRC32C, a simple error-detecting code that reliably uncovers read and write errors. In most cases, disk drives can re-try a read, adjusting the heads slightly to detect the correct value. Once an error is detected and the correct data is uncovered, the disk drive will either re-write the data in place or mark that spot as bad and re-map it to another physical location.

All this should happen very quickly, but the application must wait for it to complete. Under light load, this process is barely noticeable. But systems with heavy I/O can escalate this wait time to unacceptable levels. In busy systems, an error recovery can take many seconds or even minutes to complete.

RAID and Error Recovery

Multi-drive systems, including RAID and similar solutions, can’t tolerate long waits for error recovery. Most RAID controllers assume that a drive that hasn’t completed an I/O request within a few seconds has failed. The controller will then mark the entire disk drive as “offline” and attempt to rebuild using an available spare disk or simply take the entire RAID set offline to avoid data loss. This can prove problematic, since a RAID rebuild can take hour or days to complete!

It’s not the fault of the RAID system, either. There has to be some threshold where a disk is declared to have failed. It wouldn’t be practical (or even desirable) to escalate the I/O wait “up the stack” and pause all operations until a disk recovers (if ever). So most RAID solutions or controllers set a threshold of a few seconds.

The rule of thumb for RAID controllers is 8 seconds, though this can vary. Some controllers wait for 10, 20, or 30 seconds, for example, and this can be configured on many. ZFS will generally wait as long as needed for error recovery, and this can dramatically impact performance.

Time-Limited Error Recovery

Disk drives intended for RAID use typically implement some form of time limiting for error recovery. Western Digital calls this Time Limited Error Recovery (TLER), while Seagate calls it Error Recovery Control (ERC) and Samsung and Hitachi call it Command Completion Time Limit (CCTL).

Regardless of what it’s called, the drive will limit the wait time on any error recovery command to a settable value, typically 7 seconds by default. The drive will usually report a failed I/O up the stack and attempt to re-try the error recovery at a later time. Meanwhile, the RAID controller will likely recover the data from parity or erasure code and continue operation.

ZFS, and other software RAID systems, will typically “react” the same way when TLER is enabled, recovering data and remapping that block.

Note that most desktop hard disk drives to not have this capability. Error recovery is always turned on and recovery will take as long as necessary. This is one reason that conventional desktop disk drives are not appropriate for use in RAID solutions.

Checking and Setting TLER

If a hard drive is to be used in a RAID or similar setup, it is desirable to have TLER or ERC enabled and set to a value under 8 seconds.

Most UNIX-like systems have the “smartmon” tools package, including the command, smartctl. This can be used to query TLER and similar settings. For example, here is the result of that command in FreeNAS (FreeBSD) for a Western Digital Red NAS drive:

# smartctl -l scterc /dev/da2

smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
 Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

SCT Error Recovery Control:
 Read: 70 (7.0 seconds)
 Write: 70 (7.0 seconds)

This tool can also set TLER on a drive as follows:

smartctl -l scterc,80,80 /dev/da2

Western Digital provides a DOS utility, WDTLER.EXE, with similar functionality.

Stephen’s Stance

One reason to use enterprise or NAS hard disk drives is the capability to limit error recovery for smoother performance. I strongly recommend only using such drives with RAID systems, especially ZFS (as in FreeNAS)!

© Stephen Foskett for Stephen Foskett, Pack Rat: Turn Off Error Recovery in RAID Drives: TLER, ERC, and CCTL

The book, “Essential Enterprise Storage Concepts“, is intended as an introduction to the field of enterprise storage for technical audiences. It’s not some grand discourse on how storage ought to be done. Rather, it’s an overview of what you need to know about enterprise storage.

I start with some storage basics: The difference between memory and storage, a discussion of storage media, and the perennial topic of block vs. file storage. Then I talk about scaling storage arrays and the capabilities that enterprise arrays bring to the table. In a dozen pages, you’ll know enough to have meaningful conversations with storage vendors.

Then I shift gears, talking about performance and capacity. I begin with throughput vs. latency and the implications of parallel and serial storage access. Then I go into queueing, caching, and streaming and the difference between asynchronous and synchronous access. This will help you to understand how storage works in production.

The next section focuses on understanding storage usage. Storage administrators spend much of their time measuring and analyzing capacity, and this question of utilization is essential. We then talk about optimization of capacity and the trade-off between capacity and performance. If you’re looking at storage management software, this section is for you.

Finally, I wrap up with a discussion of storage best practices: Managing and monitoring storage, storage system sizing and planning, storage design considerations, and data protection fundamentals. This is a very-brief summary of the rules of thumb I’ve come to rely on after decades in enterprise storage.

The whole book is under 30 pages long, and I am very proud to have produced this concise treatment of enterprise storage. I urge you to download the book and give it a read! And I welcome your feedback!

Note: Somehow, the sidebar entitled “Flash Storage” has the wrong text. Here’s the correct sidebar text: “We’ll be talking a lot about flash memory later. Although this is dynamic, solid-state, and read/write, it has special characteristics that make it ill-suited as primary system memory. Plus, it’s non-volatile. Therefore, even though it’s technically memory, we typically use it as storage!”

This e-book is sponsored by SolarWinds, who also handled the formatting, illustrations, and design. I encourage you to learn more about SolarWinds products and check out their thwack technical community, and especially the Geek Speak blog!

© Stephen Foskett for Stephen Foskett, Pack Rat: Download My Free E-Book, “Essential Enterprise Storage Concepts”!

Investors Are Happy with Magic Beans?

I have been skeptical about the Dell and EMC merger since the announcement. The price seemed too low, the included “tracking stock” too flimsy, and the questions about VMware, RSA, and other Dell and EMC businesses too great. The purported value of the deal ($33.15 per share) was based on a cash price of $24.05 per share of EMC plus 1/9 of a share in a tracking stock to value the company’s ownership of VMware. As discussed at the time, this offer dramatically under-values not just EMC itself but also its ownership stake in VMware, Pivotal, and the new SecureWorks company.

Essentially, Dell bought all the $28 EMC shares for $24 plus some magic beans.

But it seems that Wall Street was willing to go along with the deal to rid itself of EMC. There has long been a cloud hanging over the EMC’s enterprise value due to a perception that the traditional enterprise storage world is on the verge of massive disruption. Analysts and investors apparently do not believe that EMC can adapt to the new world of IT and maintain earnings and profitability. Investors appear to be willing to accept this compromise, letting EMC go to Dell to get it off their hands.

I’m betting the VMware tracking stock ($DVMT) will look pretty good for a while, rising to between $45 and $60 per share before collapsing for good. Why $45? That’s the difference between EMC’s share price the day before the merger ($29.05) and Dell’s cash ($24.05) times 9 (remember that each shareholder gets 1/9 of a share of $DVMT). Owners of $EMC shares will sell out somewhere in there and that will be that.

In the mean time, I also expect lots of “pump the price” stories to be published, suggesting that $DVMT is a good way to get shares of VMware on the cheap. But tracking shares have no intrinsic value and no voting power. Unless Dell adds a dividend to $DVMT, it’s likely to follow the trajectory of similar tracking stock offerings (and magic beans): Right down the drain.

The Future of Dell EMC is Brighter

But the Dell/EMC deal is fundamentally all about brushing aside Wall Street’s demands and focusing on corporate success. And that aspect of the deal is a whole lot more positive. Dell has handled the integration as well as could be hoped, and it is likely that the newly-formed Dell Infrastructure Solutions business (selling as Dell EMC and managed out of Hopkinton) has a strong few years ahead of it.

Although Dell has traditionally been a weak spender when it comes to R&D, EMC is awash in solid technologies. Just about every product line is competitive in its market segment and some (DSSD, ScaleIO) are trend-setters. Dell EMC ought to be able to make hay with these products, selling to traditional EMC buyers as well as Dell’s massive customer base. And now they have the Dell Enterprise Solutions products to push as well.

Certainly Dell EMC has too many product lines now, but the correction won’t hurt customers. One can only hope that management will have the wisdom to cut products based on long-term viability rather than short-term sales. EMC has been content to allow competitive product lines to continue until a clear winner emerges. Perhaps Dell will continue this practice, especially now that Wall Street isn’t watching as closely.

Personally, I remain concerned for my many friends inside this new company. Dell would be wise to be graceful and kind to employees, allowing them to move to other areas instead of letting them go. But many have already given notice, casting a cloud on the future of the company. Dell needs to retain these talented people.

Dell also needs more clearly to explain what happens with the other parts of the combined business. Security has a home, but what about Dell’s networking assets? I’m not really worried about VMware under this new management regime, but what was the deal with that SecureWorks transaction? There are so many questions and still not many answers.

Impact Beyond Dell

The biggest question is the impact of this combination on the rest of the industry. HP clearly doesn’t agree with the Dell “embiggening” strategy: They split into HP InkInc and HPE and seem to be doing well so far. But Cisco has yet to react.

Then there’s the changed acquisition landscape for all the startups in the industry. Dell and EMC aren’t going to be bidding on startups any time soon, and HPE and Oracle seem fairly sated right now. This leaves Cisco as the only hungry company at the buffet. I expect some feasting to be done, building a new full-line competitor for Dell and HPE!

We must also consider the new world of IT, with a resurgent Microsoft joining Amazon, Google, and the cloud providers in attacking the IT industry. This is the reason investors were willing to take Dell’s magic beans in the first place, and it’s the real deal. Many are saying that sales of enterprise IT products have peaked, leaving companies like Dell to fight it out in a narrowing ring.

But this transaction is also a vote of “no confidence” in Wall Street. Dell went private and has, by all accounts, done very well since. Many other companies are following suit. Clearly, corporate management is fed up with the quarterly mindset so prevalent among investors and analysts. Will we see fewer IPO’s, as management realizes that being a public company isn’t a positive for a growing company? Watch for Nutanix and SimpliVity IPO’s as bellweathers.

Stephen’s Stance

EMC’s management clearly knew that it was time to make a move, and were willing to take a little less for the company than folks like me thought. Dell management clearly thought they could leverage EMC to solidify their position at the top of the IT market, and I bet they’ll do a good job with it at least for a while. Investors clearly are happy to take what they got (some cash and some magic beans), and who am I to question their logic? As for me, I’m just sad to see one more pillar of the enterprise storage industry wiped away.

© Stephen Foskett for Stephen Foskett, Pack Rat: Dell, Wall Street, Magic Beans, and the End of EMC

ZFS Send and Receive

ZFS is a volume manager, a file system, and a set of data management tools all bundled together.¹ One of the interesting features of ZFS is the ability to backup and restore data natively using the “send” and “receive” (or “recv”) commands.

Like rsync, tar, and other UNIX backup utilities, ZFS Send and Receive is a two-part process:

1. “zfs send” reads snapshot data and outputs a datastream to stdout
2. “zfs recv” reads a stream and restores data to a snapshot

You’ll note that I said “snapshot” there: In the interest of consistency, ZFS Send and Receive mainly operates on ZFS snapshots. This is a huge advantage over other utilities since it gives a point-in-time copy rather than reading some data at one time and other at a different time.

It’s also important to see that ZFS Send and Receive respectively output or input a stream of data. You can redirect this to a file or send it over an ssh connection, but you can also use it locally with a simple pipe. This means that ZFS Send and Receive can be used to move data internally to a system as well as for backup or remote replication.

Copying a ZFS Dataset

Let’s say you have a dataset called gang/scooby and wanted to create a perfect copy, gang/scrappy. ZFS Send and Receive can do this quickly and easily, guaranteeing a perfect and consistent copy! Here’s how:

1. Try to quiet gang/scooby (turn off Samba, stop applications, etc)
2. Make a snapshot: zfs snap gang/scooby@migrate
3. Send that snapshot to overwrite gang/scrappy: zfs send -R gang/scooby@migrate | zfs recv -F gang/scrappy²
4. Check out the results: zfs list -t snapshot -r gang
5. Promote gang/scrappy’s new snapshot to become the dataset’s data: zfs rollback gang/scrappy@migrate

Note that this will totally blow away any existing data in the target dataset. If this is not what you want to do, you have been warned! But no one likes Scrappy Doo anyway…

Wait For It…

You can also redirect ZFS Send to a file and tell ZFS Receive to read from a file. This is handy when you need to rebuild a pool as well as for backup and replication.

In this example, we will send gang/scooby to a file and then restore that file later.

1. Try to quiet gang/scooby
2. Make a snapshot: zfs snap gang/scooby@ghost
3. Send that snapshot to a file: zfs send -R gang/scooby@ghost > gzip /tmp/ghost.gz
4. Do what you need to gang/scooby
5. Restore the data to gang/scooby: gzcat /tmp/ghost.gz | zfs recv -F gang/scooby
6. Promote gang/scooby’s new snapshot to become the dataset’s data: zfs rollback gang/scooby@ghost

Incremental Improvements

You can also send incremental updates of ZFS datasets. This is handy for more-active data that can’t be offline for the hours it might take to move data from place to place.

In this example, we’ll send an initial snapshot using ZFS Send and Receive, followed by an incremental snapshot of the data that changed.

1. Make a snapshot: zfs snap gang/scooby@initial
2. Send that snapshot to overwrite gang/scrappy: zfs send -R gang/scooby@initial | zfs recv -F gang/scrappy
3. Check out the results: zfs list -t snapshot -r gang
4. Try to quiet gang/scooby
5. Make another snapshot: zfs snap gang/scooby@incremental
6. Send the incremental snapshot to gang/scrappy: zfs send -i -R initial gang/scooby@incremental | zfs recv gang/scrappy
7. Note that both gang/scooby and gang/scrappy now have both @initial and @incremental snapshots: zfs list -t snapshot -r gang
8. Promote the latest gang/scrappy snapshot to become the dataset’s data: zfs rollback gang/scrappy@incremental

You could script this to make sure it happens as quickly as possible. Of course it’s best not to try to do something like this on an active dataset, but it could still be useful on an active server: You only need to truly quiet the source application before the incremental snapshot is taken, and this will hopefully be much quicker than the initial data transfer.

Stephen’s Stance

I like ZFS Send and Receive, but I’m not totally sold on it. I’ve used rsync for decades, so I’m not giving it up anytime soon. Even so, I can see the value of ZFS Send and Receive for local migration and data management tasks as well as the backup and replication tasks that are typically talked about.

Update: Fixed a few typos. You need to add -R to zfs send! Thanks!

Here’s my entire FreeNAS series so far:

1. Hello FreeNAS! Goodbye Drobo and Iomega…
2. My FreeNAS Build: Supermicro X10SL7, Intel Haswell Xeon, ECC RAM
3. 14 Drives For 14 Ports: A Case For FreeNAS
4. It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down
5. Migrating Data With ZFS Send and Receive
6. FreeNAS First Impressions

Image credit: “Shipping” by Constance Abram, CC-by-NC-ND

© Stephen Foskett for Stephen Foskett, Pack Rat: Migrating Data With ZFS Send and Receive

My new server case, the NZXT H440, has five drive sleds with mounting holes for one drive on top and another on the bottom. Although this makes power and data cable routing a little more interesting, it’s perfectly fine to mount drives like this. Especially with the H440, since there are three very large fans blowing air across them at all times!

Just to make sure, here’s what each drive manufacturer says on the subject of drive mounting:¹

• Seagate: “All Seagate and Maxtor-brand hard drives can be fitted sideways or upside down. As long as they are not moved during use and get enough cooling, it is irrelevant in which direction they are mounted.”
• Western Digital: “The drive can be mounted sideways, on end, or even upside down as long as the mounting screws are used properly.”
• HGST: “Hitachi Deskstar drive can be mounted with any side or end vertical or horizontal. Do not mount the drive in a tilted position.”

So it’s perfectly fine to mount a hard disk drive upside down, vertical, or on either end as long as it’s secure from vibration and shock, not tilted at an angle, and gets enough cooling. This is no surprise since every vendor sells external USB drives with vertical or end-on mounting.²

Here’s my entire FreeNAS series so far:

1. Hello FreeNAS! Goodbye Drobo and Iomega…
2. My FreeNAS Build: Supermicro X10SL7, Intel Haswell Xeon, ECC RAM
3. 14 Drives For 14 Ports: A Case For FreeNAS
4. It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down
5. Migrating Data With ZFS Send and Receive
6. FreeNAS First Impressions

© Stephen Foskett for Stephen Foskett, Pack Rat: It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down

Let’s start with “free”. As “the Internet” is quick to point out, this critical word has multiple meanings, and corporate interests have a tendency to divert everything toward profit. OpenStack is a free and open-source software platform for cloud computing, but it’s also a community of innovators ranging from enthusiasts to lone hackers to the world’s largest corporations.

OpenStack is free, but like all open source projects, someone has to pick up the tab. Who’s paying for development and supporting users? Certainly many individual contributors are involved, but the vast majority of code commits and new project components are coming from vendors trying to leverage OpenStack to sell their products and services.

OpenStack may be “free as in freedom” and “free as in coffee”, but it sure as heck costs a lot to deploy and productize. This is why so many companies came to Austin to be part of the Summit. Some, like Mirantis, see a tremendous opportunity to profit from growing OpenStack deployments. Others want to plug their existing products into these massive installs. Yet more seem to be hedging their bets or even just happy to be part of “the new thing”.

One popular refrain in the vendor booths at the Summit is that “no one but Mirantis is making any money here at this point”. This was refuted by a few companies who have managed to land big service provider customers for hardware and software, but the overall sense is that there’s a lot of free coffee being given out without a lot of sales coming in return. “Elephant hunting” was a familiar phrase.

As with any large open source project, OpenStack is developed mainly by large corporations. Companies like IBM, HP, Intel, and NetApp see it as an entrance to the new cloud/service provider market. Red Hat, Mirantis, and Canonical want to sell services and support. Rackspace needs a functioning cloud. All of these companies are pulling the project in different directions, and all are doing the right thing as far as they are concerned.

The risk, however, is that each of these agendas might disrupt or derail the whole project. Ultimately, customers will decide which to adopt, but this isn’t as democratic as it seems. Their decisions are affected by development resources and the availability of quality integration components more than the inherent technical appropriateness of a protocol or product. This is why NetApp is smart to be pouring resources into storage integration projects like Cinder: They enable storage integration generally while reassuring customers that NetApp products will work really well!

The happy outcome isn’t necessarily a cynical product push. Rather, by contributing to the code base, all these companies improve the entire product. But what about the companies that aren’t contributing much base code but are instead focused only on integrating their own products? Although not helping build OpenStack, these companies still contribute to the feeling of longevity for the project as a whole.

What will OpenStack be outside the enterprise? Although most assume it’s just for building “your own Amazon”, customers have been clamoring for an overarching data center management framework. OpenStack might just become that, too. This is certainly how Platform9 sees OpenStack: They can leverage the common OpenStack management framework across heterogeneous enterprise data centers, not just for the cloud. VMware too is quick to point out that OpenStack isn’t tied to a single hypervisor and works just as well with ESXi inside.

OpenStack might prove to be the ultimate unifier not just of cloud but of conventional IT systems as well. One could envision a future where OpenStack is both the unified management layer for the entire data center but also the framework on which next-generation applications rest. That’s the optimistic projection.

The pessimist might spot more clouds than blue skies. OpenStack could become just what it is today: The cloud platform for service providers not named Amazon. This is an acceptable outcome, but not exactly a transformative one for the enterprise data center.

Disclaimer: Tech Field Day came to OpenStack Summit, and NetApp was a sponsor. We also interviewed a dozen non-sponsors as part of the Tech Field Day Forum. See our Day 1 and Day 2 interviews. About half of the other companies mentioned here have sponsored Tech Field Day events in the past.

© Stephen Foskett for Stephen Foskett, Pack Rat: Free as in Coffee – Thoughts on the State of OpenStack

Not Just Another All-Flash Array

Many of us in the storage industry wondered if the day would ever come that Nimble would introduce an all-flash array. What took them so long? Was there some legal impediment to them introducing an all-flash storage array? Was it some kind of internal resistance from this hybrid array standard-bearer? Or was there a technical issue?

Even after spending hours with the executives and founders of Nimble, I can’t answer those questions. But I can say that none of this matters anymore. Nimble finally has an all-flash array to sell.

Every time a company comes out with a new all-flash storage array, I fear I’ll have to point out that it’s just a regular disk array with SSDs swapped in. This criticism is inevitable when an all-flash array shares the same codebase as a disk-based storage array. Just ask NetApp (AFF), HP (3PAR), and Dell (Compellent) how often they hear this kind of jab. Then ask them to explain why it’s not fair in their case.

I must point out that Nimble’s AF series retains much of the same code from their existing hybrid arrays, so we all know what EMC (XtremIO) and Pure Storage sales reps are going to be saying. But Nimble will certainly counter that this is a proven platform with mature enterprise features and that their all-flash solution integrates seamlessly with existing disk-based arrays. Unlike the feature-light all-flash arrays from some competitors.

If they’re smart, Nimble will also point out a few novel elements of the AF series:

• The Nimble InfoSight analytics platform allowed the company to design their all-flash array based on real-world customer usage patterns
• Nimble added in-line data deduplication to the array but implemented it in a clever way that doesn’t use as much memory as competing products (more on this below)
• The familiar Nimble write sequentialization technique has been tuned for SSD, allowing them to use bigger/cheaper SSD’s without losing performance or longevity
• The Nimble approach to clustering arrays (which I’d rather call “pooling”) means you can seamlessly migrate data between all-flash and hybrid arrays

Is this just another “rip out the disks” array? I don’t think so. Nimble deserves credit for some serious engineering.

Some Cool Details About the AF Series

I spent some time with Nimble founder and VP of Engineering, Varun Mehta in an attempt to figure out what’s cool about this array. Our wide-ranging and honest discussion brought up quite a few interesting details, which I’ll try to summarize here.

It is clear that Nimble was concerned on locating the areas of technological inefficiency in storage array design. Varun pointed out that the amount of RAM available on a controller tends to limit overall array scalability, especially when data deduplication is used, and that the cost of these controllers has remained high even as the cost of NAND flash capacity has dropped dramatically. He also noted that most of the price drop for NAND is in low-end MLC, not the enterprise flash some arrays use. As Nimble set about developing an all-flash offering, they focused on optimizing the system around these limits.

Let’s start with the controllers. When designing an all-flash array, there is a temptation to cut out the SSD and try directly to integrate flash chips. After all, flash is not disk and ought to work better when treated as such. But every flash implementation needs specialized code and processors to function, not to mention physical carriers and connectors. Why not leverage the vast economies of scale by using SSD’s rather than chips directly? This is what most mainstream flash-based storage arrays do, and it’s what Nimble chose to do too.

Since flash remains fairly expensive relative to disk capacity, Nimble decided to use cheaper 3D NAND in their array. But cheaper NAND has less longevity and different performance characteristics. In an attempt to “protect” the chips in these SSD’s from excessive wear, Nimble applied a write cache and sequentialization strategy similar to the CASL concept from their hybrid arrays. Incoming writes are committed to NVRAM and then written sequentially to the SSD’s in whole blocks. Nimble believes that this will allow their 3D NAND drives to last many years longer than their competition.

One challenge for implementing data deduplication is the amount of memory it requires. The “map” of block “fingerprints” takes up quite a bit of space on every controller in the cluster. This is the main reason many all-flash arrays with inline deduplication can only support tens of terabytes of SSD. Nimble’s solution to this problem is to “page out” parts of the deduplication map to SSD and aggressively pre-fetch what is likely to be needed based on proprietary algorithms.

This is a novel approach, but it remains to be seen if it will impact performance. I can already hear the XtremIO competitive analysis folks thinking up ways to explain deterministic latency! But this trick allows Nimble to support five times the amount of storage capacity per array compared to Pure or XtremIO. For the sake of Nimble’s AF customers, I hope it works!

Nimble’s clustering approach really works to their advantage in environments with both all-flash and hybrid arrays. Unlike most symmetrical clusters, Nimble’s act as a virtualized pool. A LUN can be moved transparently between the arrays participating in the cluster, regardless of whether they are all-flash or hybrid. Clones and snapshots can also be stored on a hybrid array, where capacity costs 1/3 as much. This is a tremendous differentiator for customers, and I expect to hear a lot about it from Nimble in the coming year.

See also:

• Nimble Storage Announces Predictive Flash Platform by Dan Frith
• Nimble Storage All-Flash, late but right by Enrico Signoretti
• Nimble Storage announced their Predictive All Flash Arrays by Vipin V.K

Stephen’s Stance

It took longer than I expected for Nimble Storage to introduce an all-flash array, but their AF series looks to be a very credible offering. They’re targeting XtremIO and Pure with their marketing, but I expect HP, Dell, and especially NetApp to be cross-shopped more frequently. In that fight, I expect the Nimble AF series to be very attractive indeed!

Disclaimer: Nimble Storage flew me to San Francisco for this product launch and they’re a frequent Tech Field Day presenter. In fact, they launched the company at Tech Field Day 3 back in 2010! But no one can force me to write about them on my blog.

© Stephen Foskett for Stephen Foskett, Pack Rat: Nimble Storage Rolls Out an All-Flash Array