The Wall Street Journal gave Brocade free advertising with the article Network Specialist Brocade Up for Sale back in October. 5 months later Brocade is still for sale – and despite the HP/Cisco network wars it still isn’t clear why anyone might buy them.

Brocade’s acquisition of 10GigE developer Foundry Networks was supposed to make them a purer networking company, but Cisco’s market share doesn’t leave a lot of room for anyone else. Brocade is losing share in the Ethernet market and stock analysts are cutting their forecasts.

A weak foundation
Brocade’s troubles have been years in the making. The company put itself in a box with its go-to-market strategy and hasn’t found a way out.

Brocade’s troubles reflect the dangers of an OEM strategy when your partner’s strategic interests are different than yours. None of them wanted Brocade to succeed as a networking company.

Hope
A key reason for the excitement around storage networks in the late ’90s was the hope that the network effect would drive storage costs and commoditization
the way it had with Ethernet. But that didn’t happen.

Single-vendor networks
The major vendors paid lip service to interoperability – plug feasts at SNW – but production-quality interoperability never happened. With all FC switches coming from either Brocade or McData it should have been easy – but since the storage vendors realized that there was no business advantage in openness, it never did.

Brocade was powerless to press the issue since the OEMs controlled the product support. No enterprise customer would buy a switch that his array vendors wouldn’t support.

So Brocade was stuck with a profitable business that wasn’t going anywhere. Except south. They needed better strategy advice than they were buying.

None of Brocade’s customers want the company either. Other customers won’t want to buy from them – EMC buying switches from HP? not for long – so the company valuation looks rich from a 5 year cash flow perspective.

The StorageMojo take
The OEM channel is popular in tech because it removes the expense and exasperation of selling to end-users. You work with fellow engineers at the OEM to qualify the product and their sales people do the work.

You don’t think you need much marketing – really, you do, but few realize that – and very few sales people. And the sales people you do need are the cool, savvy relationship cultivators, not the high-pressure closers. It’s almost all good.

The bad is the loss of control. Other people position you, test you, support you and ultimately use you for their gain. That can work well if, like DEC early on, your widget is buried inside another product and you’re free to market to end-users.

But as Brocade illustrates, you can’t rely on OEMs to build your brand. You have to do it yourself.

The WSJ article suggested HP and Oracle might buy Brocade. But the only valuable part is Foundry’s 10Gig products. And that piece is heading south too.

Courteous comments welcome, of course.

Related posts:

1. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
2. Enter Exagrid With all the kerfluffle, one might think that Data Domain...

Related posts brought to you by Yet Another Related Posts Plugin.

1. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
2. Enter Exagrid With all the kerfluffle, one might think that Data Domain...

The Wall Street Journal recently published a ranking of the top 50 venture-backed companies – and storage got its share.

#2 on the list: Fusion-io, a company StorageMojo has followed from the early days. They’ve closed major deals with IBM, HP, Dell and Samsung. From the WSJ profile:

Fusion-io has a weight-loss program for data-fat Web businesses, and its “storage accelerator” has been embraced by Facebook and MySpace — and a thousand other companies looking to do more with less in a bad economy, co-founder David Flynn says. “Our datacenter uses half the power [and] half the floor space.” . . . It aims to displace storage arrays from the likes of EMC Corp. and competes with flash makers from Micron Technology Inc. to Intel Corp.

#20 is Silver Peak Systems whose appliances accelerate data transfer. From the WSJ profile:

Using proprietary algorithms and the power of multi-core chips, it’s helping customers like Google Inc., AT&T Corp. and Visa Inc. shuttle data between datacenters at a rate of one gig per second, often to backup locations. That’s “a thousand times faster than what was done historically,” says founder David Hughes. . . . He took techniques of “wide-area-network optimization,” used mainly to move data between branch offices, and super-sized for the datacenter. Today, Silver Peak, which has raised $60 million over four rounds, is growing and profitable.

#26 is Metaweb Technologies, whose product, Freebase, is an open database of online information. The big problem with massive data is search – and we’re just at the beginning of the massive data era. From the WSJ profile:

Metaweb . . . rejects freewheeling keywords (Google’s currency), because their multiple meanings can sometimes baffle computers. . . . It’s better, Metaweb believes, to organize information into people, places and things, or what it calls “entities.” . . . After five years of toil, Metaweb and its collaborators have created 12 million entities and mapped how they relate to each other.

#34 is Schooner Information Technology, who builds acceleration appliances for popular software like MySQL. From the WSJ profile:

The machines are also optimized to run the popular free middleware programs MySQL, an open-source database program, and Memcached, a Web site memory-caching system. The approach boosts speed by 10 times, which lets datacenters cut costs in half by dropping server numbers, power use and real estate needs. Appliances for other applications glisten in Schooner’s future. . . . Early customers include Flixster and Plaxo Inc.

#35 is Vidyo, who produces HD-quality video-conferencing apps and services via personal computers. From the WSJ profile:

With a new video-conferencing architecture built on an emerging standard called scalable coding, it can deliver high-quality video without annoying pauses to multiple users via their computers for just cents per minute.

#41 is Force10 Networks, if you can call a company with $200 million in revenue a startup. Their 10Gig Ethernet switches and routers are popular with Internet datacenter folks.

#43 is Akorri Networks whose focus is software for managing virtualized servers and the entire network environment inside large datacenters. Founded by Rich Corley, formerly of Pirus. From the WSJ profile:

Akorri . . . built a management system that both understands virtual machines and helps pros manage datacenters as a single system, rather than as individual components. The tools pinpoint problems quickly so datacenters can better utilize the equipment they have, get competing parts of the system to play better together, and boost overall performance. . . .

The StorageMojo take
Why are those networking companies on a StorageMojo list? Because bandwidth and storage are fungible at about the 80% level. If you have lots of bandwidth you need less local storage – and the reverse is true.

Faster and more abundant data means more storage, local and cloud. What makes this fun is that what “lots” means keeps growing.

When I got my first personal hard drive – 30 whopping MB – I couldn’t imagine ever filling it. Now I ignore 2 GB thumb drives because they’re too small. I routinely generate 5-10 GB files.

I’m ahead of the curve, but not by much. People like moving pictures and sharing. Businesses like understanding their customers. Massive data storage helps us do both.

Courteous comments welcome, of course. I’ve done work for Fusio-io. Learn more with this Video White Paper, my own contribution to our growing storage and bandwidth needs.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

StorageMojo’s Global HQ will move to Las Vegas for a couple of days next month to visit NAB 2010. Since NAB is the same week as SNW in Orlando, I won’t be attending SNW for the first time in years.

Why?

StorageMojo’s focus is the future: emerging technologies, markets and products – that’s why I don’t often discuss, say, backup or tape silos. Those are mature markets that someone like Gartner is much better equipped to handle.

While SNW has its share of future-oriented companies – I talked to 3 stealth-mode firms at the last one – the large-file-workflows that dominate the digital visual media space represent the future of data storage. The people and companies attending NAB have the kinds of applications that will drive the technology for the next decade.

That’s where I want to be.

The StorageMojo take
For my first NAB show a few years ago, I got up early, drove 5 hours to Las Vegas, spent 7 hours at NAB, and then drove another 6 hours to San Diego to attend SNW. Knowing what I know now I would have spent more time at NAB. So that’s what I’m doing now.

Sorry I’m going to miss SNW, but when they are scheduled on top of each other a choice must be made.

Courteous comments welcome, of course. I’m looking forward to the completion of Hoover Dam Bypass, a spectacular bridge over Black Canyon that should knock 35 minutes of the drive to Las Vegas. It is due to open this year.

Related posts:

1. StorageMojo at NAB next week Consumerization is a major driving force in 21st century IT....
2. StorageMojo @ Interop I’ll be checking out Interop tomorrow in Las Vegas. If...
3. StorageMojo at SNW Orlando next week StorageMojo’s global HQ is packing up for Orlando on Monday...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo at NAB next week Consumerization is a major driving force in 21st century IT....
2. StorageMojo @ Interop I’ll be checking out Interop tomorrow in Las Vegas. If...
3. StorageMojo at SNW Orlando next week StorageMojo’s global HQ is packing up for Orlando on Monday...

Patrick Eaton, the Berkeley PhD. who architected EMC’s Atmos cloud storage product, left EMC about 3 months ago. He joined a Boston-area search firm, Endeca, where he is a software architect working on a team to scale Endeca’s core MDEX search engine.

Dr. Eaton co-authored a couple of key papers on the computer science underpinning Atmos.

The StorageMojo take
The Atmos project is having a rocky time of it inside EMC. The sales force, which traditionally has had a lot of autonomy as long as they deliver the numbers, isn’t excited about selling much lower-cost/GB storage. Conservative enterprise IT buyers are as leery of unproven technology from EMC as they are from anyone else. And EMC hasn’t been crowing about Atmos either.

I recall a quote from Patrick where he commented on the all the resources Atmos was getting – dozens of people, exec attention – and I thought that might be a mixed blessing. Trying to solve hard problems to meet the CEO’s deadline is no one’s idea of a good time – especially a newly minted PhD.

Kudos to EMC for the Atmos effort and I wish them success. Yet several former EMC’ers have told me that EMC is not the most congenial place to do software. Given the enterprise sale force’s focus on hardware I find that easy to believe: the temptation for sales to discount “free” software to close a big hardware deal is almost irresistible.

And Atmos is a far step beyond EMC’s traditional management and data protection software. It is a whole new product family whose economics and implications are not well understood.

The bottom line: it is rarely a good sign when a company can’t keep the architect of a new product on board. Yes, there are other architects out there, but it usually means that some decisions were made that people now wish were made differently. Only time will tell what the case is with Atmos.

Courteous comments welcome, of course. EMC doesn’t brief me on Atmos or anything else since I won’t sign the non-disclosure agreements (NDA) they require of all analysts. Why brief people who talk and write for a living and then require them not to talk or write about the briefing?

Related posts:

1. Atmos gets no love from EMC sales A couple of reliable informants tell me the same story:...
2. Ask StorageMojo: is Atmos the only choice? A regular reader writes: We’re facing a customer problem that...
3. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Atmos gets no love from EMC sales A couple of reliable informants tell me the same story:...
2. Ask StorageMojo: is Atmos the only choice? A regular reader writes: We’re facing a customer problem that...
3. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...

StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector Errors and How to Protect Against Them (pdf) by Bianca Schroeder, Sotirios Damouras, and Phillipa Gill, University of Toronto.

The paper builds on research and a dataset that StorageMojo reviewed 2 years ago in Latent sector errors in disk drives. That research analyzed the error logs of 50,000 NetApp arrays with 1.53 million enterprise and consumer drives disks.

Understanding
Understanding LSEs does a statistical deep dive on the disk LSE dataset and then evaluates scrubbing and intra-disk redundancy strategies against the field data.

Latent sector errors are important for 3 reasons:

• 1 LSE can cause a RAID reconstruction failure in a single parity RAID system (RAID 5).
• Ever-tinier disk storage geometries make LSEs more likely.
• The insidious failure mode: no detection until access is attempted.

Schroeder et. al. used a subset of the LSE dataset that included only drives that had LSEs. This covered 29,615 nearline (presumably SATA) drives and 17,513 enterprise drives that had been in the field at least 12 months.

LSE metrics
Some of the papers conclusions:

• For most drives almost all LSEs are a single error. Multiple contiguous logical block errors are less than 2.5% of all LSEs.
• If there is a 2nd error, most are within 100 sectors of the 1st error.
• Depending on the model, between 20% and 50% of errors are in the first 10% of the drive’s logical sector space. Some drives have a higher concentration of errors at the end of the drive as well.
• LSEs are highly concentrated in a few short time intervals, not randomly spread out over a drive’s life.
• It appears that events that are close in space are also close in time.

The rest of the paper
The paper also goes into 2 interesting topics – intra-disk redundancy and scrubbing strategies – that deserve posts of their own. For the latter the research found that changing the order in which sectors are scrubbed can improve mean time to error detection by 40% – with no increase in overhead or scrub frequency.

Conclusions
Key quote:

We observe that many of the statistical aspects of LSEs are well modeled by power-laws, including the length of error bursts (i.e. a series of contiguous sectors affected by LSEs), the number of good sectors that separate error bursts, and the number of LSEs observed per time. We find that these properties are poorly modeled by the most commonly used distributions, geometric and Poisson. Instead we observe that a Pareto distribution fits the data very well and report the parameters that provide the best fit. . . . We find no significant difference in the statistical properties of LSEs in nearline drives versus enterprise class drives.

[bolding added -ed. However, nearline drives are about 4x more likely get an error.]

The StorageMojo take
Disk-based storage arrays are facing a real challenge from flash and possibly PCM technology. Disks win the $/GB race, but piling double and triple parity on arrays increases costs and firmware complexity.

Understanding the nature of the enemy – in this case latent sector errors – helps array designers develop more reliable and cost-effective arrays. Yet one has to wonder if the RAID paradigm is reaching the end of the line.

Parallel and object-based systems from Isilon and Panasas, for example, are very fast at disk rebuilds because they can draw data from many disk drives in parallel – without the performance-killing overhead that RAID rebuilds impose.

But those are larger systems. Putting these techniques together may give us reliable and economical RAID 5 systems for the SMB market for another decade or more.

Courteous comments welcome, of course. I’ve done work for Isilon – who also advertises on StorageMojo – and Panasas. The official best paper of FAST ‘10 was quFiles which I blogged about last week.

If you spot a typo please let me know. Thanks!

Related posts:

1. FAST ‘10 papers: wow. Just checked out the papers for this year’s FAST ‘10...
2. Why we need 4k drives WD has started shipping drives that drop the ancient 512...
3. Does RAID 6 stop working in 2019? Late last year Sun engineer, DTrace co-inventor, flash architect and...

Related posts brought to you by Yet Another Related Posts Plugin.

1. FAST ‘10 papers: wow. Just checked out the papers for this year’s FAST ‘10...
2. Why we need 4k drives WD has started shipping drives that drop the ancient 512...
3. Does RAID 6 stop working in 2019? Late last year Sun engineer, DTrace co-inventor, flash architect and...

Late last year Sun engineer, DTrace co-inventor, flash architect and ZFS developer Adam Leventhal, analyzed RAID 6 as a viable data protection strategy. He lays it out in the Association of Computing Machinery’s Queue magazine, in the article Triple-Parity RAID and Beyond, which I draw from for much of this post.

The good news: Mr. Leventhal found that RAID 6 protection levels will be as good as RAID 5 was until 2019.

The bad news: Mr. Leventhal focussed on enterprise drives whose unrecoverable read error (URE) spec has improved faster than the more common SATA drives. SATA RAID 6 will stop being reliable sooner unless drive vendors get their game on. More good news: one of them already has.

The crux of the problem
SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 200,000,000 sectors, the disk will not be able to read a sector.

2 hundred million sectors is about 12 terabytes. When a drive fails in a 7 drive, 2 TB SATA disk RAID 5, you’ll have 6 remaining 2 TB drives. As the RAID controller is reconstructing the data it is very likely it will see an URE. At that point the RAID reconstruction stops.

Here’s the math:
(1 – 1 /(2.4 x 10^10)) ^ (2.3 x 10^10) = 0.3835

You have a 62% chance of data loss due to an uncorrectable read error on a 7 drive (2 TB each) RAID 5 with one failed disk, assuming a 10^14 read error rate and ~23 billion sectors in 12 TB. Feeling lucky?

When 4 TB drives ship later this year only 3 drives will equal 12 TB. If they don’t up the spec, this will be a mess.

RAID 6
RAID 6 creates enough parity data to handle 2 failures. You can lose a disk and have a URE and still reconstruct your data.

NetApp noted several years ago that you can have dual parity without increasing the percentage of disk devoted to parity. Doubling the size of RAID 5 stripe gives you dual disk protection with the same capacity.

Instead of a 7 drive RAID 5 stripe with 1 parity disk, build a 14 drive stripe with 2 parity disks: no more capacity for parity and protection against 2 failures. Of course, every rebuild will require twice as many I/Os since each disk in the stripe must be read. Larger stripes aren’t cost free.

Grit in the gears
The chance that a single sector rebuild will encounter 2 read errors is tiny, so what is the problem?

Mr. Leventhal says a confluence of factors are leading to a time when even dual parity will not suffice to protect enterprise data.

These include:

• Long rebuild times. As disk capacity grows, so do rebuild times. 7200 RPM full drive writes average about 115 MB/sec – they slow down as they fill up – which means about 2.5 hours per TB minimum to rebuild a failed drive. Most arrays can’t afford the overhead of a top speed rebuild, so rebuild times are usually 2-5x that.
• More latent errors. Enterprise arrays employ background disk-scrubbing to find and correct disk errors before they bite. But as disk capacities increase scrubbing takes longer. In a large array a disk might go for months between scrubs, meaning more errors on rebuild.
• Disk failure correlation. RAID proponents assumed that disk failures are independent events, but long experience has shown this is not the case: 1 drive failure means another is much more likely.

On the last point: in a corridor conversation at FAST ‘10 I was told that at a large HPC installation they found that with drives from the same manufacturing lot that 1 drive failure made a 2nd 10x more likely – while a 2nd made a 3rd 100x more likely. Not clear how manufacturing or environmental issues – or interaction between the 2 – led to the result. YMMV.

Simplifying: bigger drives = longer rebuilds + more latent errors -> greater chance of RAID 6 failure.

Mr. Leventhal graphs the outcome:

Courtesy ACM Queue

The StorageMojo take
For enterprise users this conclusion is a Big Deal. While triple parity will solve the protection problem, there are significant trade-offs.

21 drive stripes? Week long rebuilds that mean arrays are always operating in a degraded rebuild mode? Wholesale move to 2.5″ drives to reduce drive and stripe capacities? Functional obsolescence of billions of dollars worth of current arrays?

What is scarier is that Mr. Leventhal assumes disk drive error rates of 1 in 10^16. That is true of the small, fast and costly enterprise drives, but most SATA drives are 2 orders of magnitude less: 1 in 10^14.

With one exception: Western Digital’s Caviar Green, model WD20EADS, is spec’d at 10^15, unlike Seagate’s 2 TB ST32000542AS or Hitachi’s Deskstar 7K2000 (pdf).

Before entering full panic mode though it would be good to see more detailed modeling of RAID 6 data loss probabilities. Perhaps a reader would like to take a whack at it.

Comments welcome, of course. I worked at Sun years ago and admire what they’ve been doing with ZFS, flash, DTrace and the great marketing job the ZFS team did without any “help” from Sun marketing. An earlier version of this post appeared on Storage Bits. Looking for a scientific calculator program? PCalc – Mac & Windows – is the best I’ve found.

Related posts:

1. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
2. Why we need 4k drives WD has started shipping drives that drop the ancient 512...
3. Nightmare on DIMM street A 2½ year study of DRAM on 10s of thousands...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
2. Why we need 4k drives WD has started shipping drives that drop the ancient 512...
3. Nightmare on DIMM street A 2½ year study of DRAM on 10s of thousands...

The official best paper winner at FAST ‘10 isn’t one of the several I excerpted. I’m listening to the presentation as I write – trying live blogging – while following a fast talking presenter.

The winning paper is quFiles: The right file at the right time by Kaushik Veeraraghavan, Jason Flinn and Brian Noble of the University of Michigan and Edmund B. Nightingale of Microsoft Research.
From the abstract:

A quFile is a unifying abstraction that simplifies data management by encapsulating different physical representations of the same logical data. Similar to a quBit (quantum bit), the particular representation of the logical data displayed by a quFile is not determined until the moment it is needed. The representation returned by a quFile is specified by a data-specific policy that can take into account context such as the application requesting the data, the device on which data is accessed, screen size, and battery status.

One application is video files that may be played back on a variety of devices with differing resolutions, compute and graphics engines, codecs, editing capability and storage. There is one quFile that encapsulates several versions of the file – even different versions of the same file – and which is returned depends on the device requesting the file.

The key is that every device asks for the same file name, simplifying file management on the server and file distribution. quFiles are space efficient, adding little to file size, while their compute overhead is in the single-digit percents. And no application changes are required.

The StorageMojo take
FAST’s Best Paper isn’t necessarily StorageMojo’s Best Paper, yet this is a worthy candidate. Hiding the gory details of file types and network requirements from users is a Good Thing. I particularly like the support for file versioning, a feature I grew to love on the VMS operating system, but not widely appreciated today.

Congratulations to the team on their win.

Courteous comments welcome, of course. I’ll provide a link to the conference papers once the USENIX folks make them public.

Related posts:

1. Google File System v2, part 2 Bigtable to the rescue (sort of) In Part 1, Sean...
2. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
3. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Google File System v2, part 2 Bigtable to the rescue (sort of) In Part 1, Sean...
2. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
3. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Just checked out the papers for this year’s FAST ‘10 in San Jose this week. They are impressive.

Here are a few that caught my eye with quotes, mostly from the abstracts, but sometimes from the results.

SRCMap: Energy Proportional Storage using Dynamic Consolidation

During a given consolidation interval, SRCMap activates a minimal set of physical volumes to serve the workload and spins down the remaining volumes, redirecting their workload to replicas on active volumes. We present both theoretical and experimental evidence to establish the effectiveness of SRCMap in minimizing the power consumption of enterprise storage systems.

Evaluating Performance and Energy in File System Server Workloads

We concluded that default file system types and options are often suboptimal: simple changes within a file system, like mount options, can improve power/performance from 5% to 149%; and chang-ing format options can boost the efficiency from 6% to 136%. Switching to a different file system can result in improvements ranging from 2 to 9 times.

Provenance for the Cloud

While it is feasible to provide provenance as a layer on top of today’s cloud offerings, we conclude by presenting the case for incorporating provenance as a core cloud feature, discussing the issues in doing so.

Discovery of Application Workloads from Network File Traces

Our method is successful at discovering the application-level behavioral characteristics from NFS traces. We have also shown that given a long sequence of NFS trace headers, it is able to annotate regions of the sequence as belonging to the applications that it has been trained with. It can identify and annotate both sequential and concurrent execution of different workloads. Finally, we demonstrate that small snippets of traces are sufficient for identifying many workloads.

BASIL: Automated IO Load Balancing Across Storage Devices

In this paper, we introduce BASIL, a novel software system that automatically manages virtual disk placement and performs load balancing across devices without assuming any support from the storage arrays. BASIL uses IO latency as a primary metric for modeling.

Panache: A Parallel File System Cache for Global File Access

Panache is a scalable, high-performance, clustered file system cache for parallel data-intensive applications that require wide area file access. Panache is the first file system cache to exploit parallelism in every aspect of its design—parallel applications can access and update the cache from multiple nodes while data and metadata is pulled into and pushed out of the cache in parallel. Data is cached and updated using pNFS, which performs parallel I/O between clients and servers, eliminating the single-server bottleneck of vanilla client-server file access protocols. Furthermore, Panache shields applications from fluctuating WAN latencies and outages and is easy to deploy as it relies on open standards for high-performance file serving and does not require any proprietary hardware or software to be installed at the remote cluster.

Accelerating Parallel Analysis of Scientific Simulation Data via Zazen

We have implemented our methodology in a parallel disk cache system called Zazen. By avoiding the overhead associated with querying metadata servers and by reading data in parallel from local disks, Zazen is able to deliver a sustained read bandwidth of over 20 gigabytes per second on a commodity Linux cluster with 100 nodes, approaching the optimal aggregated I/O bandwidth attainable on these nodes. Compared with conventional NFS, PVFS2, and Hadoop/HDFS, respectively, Zazen is 75, 18, and 6 times faster for accessing large (1-GB) files, and 25, 13, and 85 times faster for accessing small (2-MB) files.

Write Endurance in Flash Drives: Measurements and Analysis

Our chip-level measurements show endurance far in excess of nominal values quoted by manufacturers, by a factor of as much as 100. We reverse engineer specifics of the Flash Translation Layers (FTLs) used by several devices, and find a close correlation between measured whole-device endurance and predictions from reverse-engineered FTL parameters and measured chip endurance values.

Extending SSD Lifetimes with Disk-Based Write Caches

In this paper, we propose Griffin, a hybrid storage design that, somewhat contrary to intuition, uses a hard disk drive to cache writes to an SSD. Writes to Griffin are logged sequentially to the HDD write cache and later migrated to the SSD. Reads are usually served from the SSD and occasionally from the slower HDD. Griffin’s goal is to minimize the writes sent to the SSD without significantly impacting its read performance; by doing so, it conserves erase cycles and extends SSD lifetime.

DFS: A File System for Virtualized Flash Storage

Instead of using traditional layers of abstraction, our layers of abstraction are designed for directly accessing flash memory devices. DFS has two main novel features. First, it lays out its files directly in a very large virtual storage address space provided by FusionIO’s virtual flash storage layer. Second, it leverages the virtual flash storage layer to perform block allocations and atomic updates. As a result, DFS performs better and it is much simpler than a traditional Unix file system with similar functionalities.

Understanding latent sector errors and how to protect against them

The statistical analysis revealed some interesting properties. We observe that many of the statistical aspects of LSEs are well modeled by power-laws, including the length of error bursts (i.e. a series of contiguous sectors affected by LSEs), the number of good sectors that separate error bursts, and the number of LSEs observed per time.

We find that these properties are poorly modeled by the most commonly used distributions, geometric and Poisson. Instead we observe that a Pareto distribution fits the data very well and report the parameters that provide the best fit. We hope this data will be useful for other researchers who do not have access to field data. We find no significant difference in the statistical properties of LSEs in nearline drives versus enterprise class drives.

Some of our statistical observations might also hold some clues as to what mechanisms cause LSEs. For example, we observe that nearly all drives with LSEs, experience all LSEs in their lifetime within the same 2-week period, indicating that for most drives most errors have been caused by the same event (e.g. one scratch), rather than a slow and continuous wear-out of the media.

A Clean-Slate Look at Disk Scrubbing

Our work is a first step in the exploration of more intelligent scrubbing strategies for hard drives. It shows that single drive reliability can be greatly improved by expanding the design space for scrubbing strategies be- yond naive sequential and constant-rate approaches.

The StorageMojo take
Wow. Power, flash, errors and error handling, advanced file systems, virtualization, there’s something for everyone. It reads like tomorrow’s news.

The competition for best paper is intense this year – and that’s a good thing. I’ll be attending the conference and look forward to writing more about some of these papers.

Courteous comments welcome, of course. IIRC, the papers will be available on the FAST web site on the 24th.

Related posts:

1. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
2. TPC-C: comparing SSD & disk Steve Jones of BT sent in the following, which I...
3. StorageMojo @FAST ‘10 next week The USENIX conference on File And Storage Technologies is in...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo’s best paper of FAST ‘10 StorageMojo’s best paper of FAST ‘10 is Understanding Latent Sector...
2. TPC-C: comparing SSD & disk Steve Jones of BT sent in the following, which I...
3. StorageMojo @FAST ‘10 next week The USENIX conference on File And Storage Technologies is in...

After 9 years and $100,000,000, holographic storage pioneer InPhase Technologies has shut down without ever shipping a product. Their office building was also seized for non-payment of back taxes.

They assured me that the product would ship in May, 2008. It didn’t. Reportedly many employees took pay cuts – or no pay at all – to help keep the company going.

It is a sad and ignominious end to a brave technology experiment. And a warning to anyone trying to replace disk drives as random access storage.

The 40% problem
At a 40% annual capacity growth rate hard drives are difficult to catch. When InPhase started showing their initial prototype, 300 GB wasn’t much less than hard drives. But 3 years later 300 GB is less than 1/6th the capacity.

Nor was it very speedy: 20 MB/sec. You can do almost as well with a USB thumb drive.

InPhase planned to take the drives to 1.6 TB and 120 MB/sec. If they could ship that today, they’d have a competitive product.

In the meantime, cheap hard drives and cheaper hard drive docks make it easy to use bare drives for backup and data transfer. The market for 300 GB removable drives withered before it had a chance to grow.

As I wrote 4 years ago:

I love holographic technology and wish InPhase the best, but I don’t believe they have a viable business with their technology – yet. The problem: 3.5″ disk drives will reach 750GB by the end of this year with much faster transfer rates. InPhase’s 20 Mbps is only 2.5 million bytes per second or only 9GB per hour. It will take over 30 hours just to fill one disk! I predict that hard drives will still be more convenient and fairly cost-competitive than this promising new technology.

But keep at it guys. Lightning will strike if your investors are patient enough.

They weren’t.

The StorageMojo take
The disk industry spends over $1B a year improving hard drives. Thousands of PhD scientists and engineers are busy researching drive problems.

That kind of momentum is hard for a startup to overcome. NAND flash did so only because it built a large business in mobile applications where disk drives couldn’t compete.

For a startup to succeed with optical storage they’ll need to:

• a) build a multi-billion dollar business where disks and now flash don’t compete, such as Blu-ray’s movie distribution, or
• b) start with a product that is 10x – 5 years – ahead of current disk drive capacity, and
• c) have a clear grasp of what continued 40% annual growth means for disk drive capacity pricing – and product delays.

There are several firms pushing optical storage forward. Blu-ray is the least ambitious – now that HD DVD is (almost) gone – and that huge investment is almost certain to have a negative ROI, even if 3D content succeeds in making it the preferred consumer physical medium.

Nor is the outlook for other optical drives promising. Like removable magnetic drives before them, they are being crushed by substitute technologies like USB flash drives, 2.5″ drives & drive docks and downloading and wireless networks.

With the InPhase demise we may never see holographic storage commercialized. Especially if disk vendors start building archive-quality disks.

Comments welcome, of course. I was rooting for InPhase’s success, to no avail. Another version of this post was published on my ZDnet blog.

Related posts:

1. Optical nearing the end of the line TDK recently demo’d an impressive technical achievement: a 10 layer...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Optical nearing the end of the line TDK recently demo’d an impressive technical achievement: a 10 layer...

Isilon reached an important milestone this month: their first profitable quarter and an annual run rate of about $150 million a year — in only 10 years. But what if I told you there was another Seattle-area storage company on target to do $500 million in only 5 years?

That would be Amazon. Since the August 4, 2006 startup of the Elastic Compute Cloud, Amazon Web services has seen staggering growth.

Amazon doesn’t break out AWS results, so all numbers are speculative. But I went through five years of Amazons annual reports and it is clear that a 2009 $300M run rate — most of it storage — is in the ballpark.

Amazon has until August 2011 to reach $500M. Will you bet against them?

Running with numbers
Here’s a sketch of the numbers.

In 2005 Amazon had sales of $8.5 billion, $230M in “other” revenues (“Includes non-retail activities, such as our co-branded credit card agreements, Amazon Enterprise Solutions, and miscellaneous marketing and promotional activities”), fixed asset purchases including internal-use software and web site development of $204M (up from $89M in 2004) and fixed assets of $223M.

In 2009, sales of $24.5B, “Other” revenue of $653 (“Includes non-retail activities, such as marketing and promotional activities, Amazon Web Services, other seller sites, and our co-branded credit card agreements”), fixed asset purchases of $373M, $1.3B of fixed assets.

What do these numbers tell us? Not as much as we’d like, but . . . .

It is obvious that the non-AWS revenue has not kept pace with Amazon’s 3x growth in the last 5 years – otherwise AWS revenues would be a pathetic $40M.

Also, the fixed assets – which don’t include real estate, as Amazon doesn’t own any – have risen 6x. Factor in Moore’s Law and disk capacity increases over the last 5 years and Amazon has 30x more compute cycles and terabytes than it did 5 years ago – assuming a 3 year equipment life cycle. Yes, there are other fixed assets – no telling what the percentage is – but that growth rate is extreme.

All that to support 3x revenue growth? Hardly. They have to put hundreds of thousands of AWS customers somewhere.

Other indicators
Amazon is also promoting AWS much more aggressively than in the past. From James Hamilton’s Private clouds are not the future to the new AWS Promo Economics Center white papers, references, vertical solutions and reseller catalog, Amazon is pushing hard.

Courtesy Amazon

Courtesy Amazon

The StorageMojo take
Amazon and Google proved the economics of cloud infrastructures several years ago. Corporate America is beginning to catch on.

This is reminiscent of the 1985, when it was apparent that PCs were going to bulldoze the minicomputer market, but DG, Wang, Prime and DEC were all making brave noises. In the next 10 years they all failed – billions of dollars of revenue and capital gone.

AWS isn’t the PC and today’s storage companies aren’t Route 128 minicomputer companies, but the implications are still sobering.

As I noted in Why private clouds are part of the future

. . . public clouds will claim the majority of the market whether measured in dollars or exabytes, but private clouds will remain significant contributors to our data infrastructure for decades, if not centuries, to come.

Thus the question: will today’s storage vendors adapt better than DEC did 25 years ago? They can, if they have the vision and the will.

Courteous comments welcome, of course.

Related posts:

1. Why private clouds are part of the future James Hamilton, Amazon architect and a very smart guy,...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
3. 1 million IOPS in 1 RU Sun announced the, or removed from stealth mode, the F5100,...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Why private clouds are part of the future James Hamilton, Amazon architect and a very smart guy,...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
3. 1 million IOPS in 1 RU Sun announced the, or removed from stealth mode, the F5100,...

The USENIX conference on File And Storage Technologies is in San Jose, Feb. 23-26. Here’s the blurb:

FAST ‘10 brings together storage system researchers and practitioners to explore new directions in the design, implementation, evaluation, and deployment of storage systems.

And it does. This is the cutting edge of storage.

The StorageMojo take
I’m looking to meet new companies and players. If you are in the Valley – or will be – let’s set up a time to meet.

Courteous comments welcome, of course.

Related posts:

1. StorageMojo@SNW next week Finally, SNW is back in Phoenix, an easy 2 hour...
2. StorageMojo @ Storage Visions & CES this week 2010 is off to a fast start: I’ll be at...
3. StorageMojo at NAB next week Consumerization is a major driving force in 21st century IT....

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo@SNW next week Finally, SNW is back in Phoenix, an easy 2 hour...
2. StorageMojo @ Storage Visions & CES this week 2010 is off to a fast start: I’ll be at...
3. StorageMojo at NAB next week Consumerization is a major driving force in 21st century IT....

MapReduce and its open source version, Hadoop, are parallel data analysis tools. A few lines of code can drive massive data reductions across thousands of nodes.

Cool.

Powerful though it is, Hadoop isn’t a database. Classic structured data analysis of the model/load/process type isn’t what it was designed for.

That’s where the paper HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads (pdf) comes in. Written by Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Avi Silberschatz and Alexander Rasin (the former 4 @Yale, and the latter @Brown) the paper proposes a method for building an open-source, commodity hardware-based massively scalable, shared-nothing, analytical parallel database.

What it is
HadoopDB coordinates SQL queries across multiple independent database nodes using Hadoop as the task coordinator and network communication layer. It uses the scheduling and job tracking of Hadoop while it intelligently pushes much of the query processing into the individual database nodes.

There are four components to HadoopDB.

• Database Connector. Each node has its own independent database. The connector is the interface between the database and Hadoop’s task trackers. A MapReduce jobs supplies the Connector with an SQL query and other parameters. The Connector executes a SQL query on the database and returns results as key value pairs. It can implemented to support a variety of databases.
• Catalog. The information needed to access the databases and metadata such as cluster data sets, replica locations and data partitions is kept in the catalog.
• Data loader. The data loader is responsible for two jobs. First executing a MapReduce job over Hadoop that reads the raw data files and partitions them into as many parts as the number of nodes in the cluster. Second, the partitions are loaded into the local file system of each node and chunked according the system-wide parameter.
• SQL to MapReduce to SQL planner. The planner provides a parallel database front end to enable SQL queries. The planner transforms the queries into map reduce jobs and optimizes the query plans for efficiency. This is where scratch that this is the secret sauce of HodoopDB.

HadoopDB complements the Hadoop infrastructure and does not replace it. Analysts have both available as needed.

Heterogeneity
A key issue for Internet-scale systems is the ability to run in a heterogenous environment where multi-year build-outs and rolling node replacement are the norm. That means that some nodes will be faster than others. HadoopDB breaks the work down into small tasks and moves them from slow to fast nodes automagically.

Results
The authors ran some benchmarks on Amazon’s EC to to test performance. The HadoopDB load times were about 10x that of Hadoop, but the higher performance of HadoopDB usually justified the longer set up time.

The authors found that HadoopDB was able to approach the performance of parallel database systems on much lower cost hardware and free software. Given the gift of the projects one can expect higher performance as improvements are made.

The killer app for private clouds?
MapReduce and Hadoop are already in wide use among Internet-scale datacenters. As companies begin to understand and correlate social media, web activity and ad response rates, the demand for large-scale parallel database processing will grow. But will they want to ship it out to Amazon?

Depending on the quantity and sensitivity of the data many organizations may prefer to keep the processing in-house. Private scale out Hadoop clusters may become the poor companies data warehouse of choice.

The StorageMojo take
HadoopDB is more science project than commercial tool today. Yet the project demonstrates the feasibility of using scale out compute/storage clusters for work that day typically requires proprietary high-end scale up system architectures.

If capital costs are reduced by two thirds with a commodity/FOSS architecture, companies could afford to hire the expertise required to make it work. The free software/paid support model will prove quite successful in this space.

Courteous comments welcome, of course.

Related posts:

1. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
3. Not a filesystem, not a database. Jeff Darcy has a good post on key data stores,...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
3. Not a filesystem, not a database. Jeff Darcy has a good post on key data stores,...

James Hamilton, Amazon architect and a very smart guy, recently blogged about private clouds. In Private Clouds Are Not The Future he argues that economies of scale make public clouds much more efficient than private clouds.

I think we agree that several effects make web scale public clouds more efficient:

• Higher quality services. Large clouds can economically employ experts to design and optimize their services and infrastructure. Security and server/storage design are two areas where deep expertise can provide more reliable and efficient service.
• Utilization. Power systems and power cost are optimized when data centers are run at 100% utilization. As utilization rises across the board so does the capital efficiency, i.e. work per invested dollar.
• Cost. Large-scale investments create their own lower-cost dynamic. Public cloud providers save money on infrastructure acquisition through volume buys. In addition, their volume enables them to acquire optimized components, such as high-efficiency power supplies or custom cost-reduced motherboards, that offer little economic advantage to small volume buyers.
• Portfolio advantages. With a mix of customers and jobs web-scale clouds have a more stable aggregate load. Some customers are growing, some are shrinking, but the net demand becomes more stable with size. This, in turn, enables public cloud managers to drive utilization higher with less risk of pegging the system.

With all these advantages it is obvious that private clouds are not the future. Or is it?

It isn’t all about the Benjamins
Economics is not the driver many assume. Individuals and companies often select less economic choices. Some people buy cars that cost $200,000 and get 12 miles to the gallon. Some companies buy $6/GB storage and then utilize just 1/3rd of that costly capacity.

Often perceived benefits are not well measured in dollars. Convenience, availability, consistency and control often relate to emotional needs and wants that are rarely quantified or questioned.

But we don’t have to invoke those to understand why private clouds will be part of the computing landscape. Just a quick look at one of the large Internet data centers will tell us what we need to know.

Show me the power
All the advantages of public clouds have analogs in the world of power generation and distribution. Power generation is cheapest when centralized and large-scale distribution systems move power at the lowest cost per watt.

Electrical power generation and distribution is over 125 years old. The technology is well understood, the industry is mature, and a massive infrastructure — including mile-long coal-hauling trains — supports production and distribution.

And yet, Google’s massive Dalles, Oregon data centers, built next to a substation a few miles from the nation’s largest hydropower system – one of the world’s most reliable power sources – flanks each data center with generators. I expect Amazon does the same.

Access
Clearly, access to data is at least as important as access to power or why would data centers spend the money on uninterruptible power supplies?

Despite the maturity of the power industry people realize it cannot be relied upon 100%. Therefore they maintain their own power storage, generation and distribution systems.

Is the Internet that different?

We cannot rely 100% on Internet access to our data. If the application is important enough, as judged by often subjective human criteria, we will keep our data as close as Google keeps its generators.

Even if it isn’t the most economic choice.

The StorageMojo take
My thanks to James Hamilton and his post for a lucid justification for an all cloud IT infrastructure future. He helped me see why that isn’t going to happen and for that I thank him.

I’ve grappled with the question of private clouds for the last couple of years. The advantages of web scale systems became more obvious, but the human desire for reliable data access and control has not receded.

Public and private will not displace each other: they will coexist just as public and private power sources coexist today. No doubt public clouds will claim the majority of the market whether measured in dollars or exabytes, but private clouds will remain significant contributors to our data infrastructure for decades, if not centuries, to come.

Courteous comments welcome, of course.

Related posts:

1. Private clouds won’t fly Massive economies of scale make cloud computing and storage inevitable....
2. A RisingTide lifts all clouds Check out their homepage and, as of today, “This page...
3. 2009’s big STORies 2009 has been an eventful year: the Great Recession has...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Private clouds won’t fly Massive economies of scale make cloud computing and storage inevitable....
2. A RisingTide lifts all clouds Check out their homepage and, as of today, “This page...
3. 2009’s big STORies 2009 has been an eventful year: the Great Recession has...

While everyone else was watching the Apple iPad intro I was watching Oracle’s John Fowler talk about their systems and storage strategy. I like the iPad, but the O+S strategy could reshape the storage industry.

More details will emerge and many decisions still remain but the basic elements are clear:

• Focus on direct sales. In the mid-1990s, when I joined Sun, the tenacity and aggressiveness of their direct sales force was a welcome change. Direct sales forces are expensive, but losing touch with your customers is even costlier. The combo’s unique value propositions can’t be sold by channels today. In 5 years – maybe.
• A dedicated storage sales force. Generalist salespeople with millimeter deep storage product and application knowledge can’t compete with EMC and NetApp. Storage specialists aren’t easy to develop, so they’ll hire them – and they promise top commissions.
• Deep integration of ZFS into storage systems. A software company should like a software solution to many of the biggest storage problems? Putting real muscle behind ZFS will help thousands of enterprise customers to rethink their high-performance data protection strategies.
• Flash everywhere. Sun has done some creative things with flash already, such as Logzilla, and Oracle sees that much more can be done.

Not mentioned – not that it should have been – is the fate of ZFS on Mac OS X. That would be a boost for all concerned.

The StorageMojo take
Sun’s primary storage business has been a black smoking crater of disaster for over a decade. And it didn’t help StorageTek to have them answer to know-nothings.

Despite that Sun engineers outside the storage group developed innovative and game-changing technologies that the company couldn’t capitalize on. With Oracle’s investment now they can.

No database/systems company can be successful without a healthy and very competitive storage team — and the high gross margins don’t hurt. With a hard-nosed focus on application performance, marketing competence and continued innovation, the O+S storage group could be a fun place to work. They are hiring!

It will take Oracle 12 to 18 months to develop the kind of customer traction that will make other storage vendors set up and take notice. But Larry Ellison isn’t planning to lose and there is no reason he should.

Storage competition in the enterprise is about to get cranked up several notches. And that is a good thing for all customers.

Courteous comments welcome, of course.

Related posts:

1. Btrfs vs ZFS – OMG! Am @ Interop today – a nice, relaxing 250 mile...
2. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
3. The Sun 4 TB flash array F5100 The 4 TB Sun F5100 Flash Array product launch is...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Btrfs vs ZFS – OMG! Am @ Interop today – a nice, relaxing 250 mile...
2. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
3. The Sun 4 TB flash array F5100 The 4 TB Sun F5100 Flash Array product launch is...