A commenter recently asked

Archivas was focused on archive, do you expect the new solution to sustain performance for primary storage as well?

Which is a good question, if you know what “primary” means. Do we?

Tiers of a clown
10 years ago we all agreed on 1st tier or primary storage: block-based; RAID 5; enterprise FC or SCSI drives; SCSI, FC or ESCON host connects; optimized for transactional workloads; and large mirrored (with 1 notable exception) caches. When SANS took off we stuck FC switches in front of the boxes and called it good.

But something happened to that consensus: iSCSI; NFS; CIFS; SSD; MEMcache; Internet scale-out; Infiniband; 10GigE; storage & processor virtualization; CDNs; web-serving; pNFS; and lower-cost out-sourced high-scale infrastructure (i.e. cloud). And more – such as non-SQL data management – is coming.

Will the real primary storage please stand up?
Amazon runs a high-growth $25B/yr business on scale-out storage, servicing millions of customers, taking real money and shipping real goods, 7x24x365. Smells like enterprise spirit.

Is Amazon’s storage “primary” and, if so, what makes it primary?

Yes, it is primary storage. No, it isn’t the logo that makes it so.

Workload & service level
It’s tempting to consider workload, but what workload? IOPS? Bandwidth?

How about parallelism? Web service is highly parallel. ACID database updates less so.

And what about files vs blocks? Blocks don’t require as much processing as files, as the host is handling the file system.

It is clear that most files aren’t often accessed. Does primary storage for files mean availability and reasonable performance? Or is there little difference between archive and primary for files?

NetApp is deduping primary storage. Others will follow, whether it makes sense or not, at least in messaging. Skeptics ask “If it is deduped, is it really primary?”

The StorageMojo take
We do a disservice to customers if we talk about “primary” storage as a class of equipment. It isn’t.

Primary storage is whatever works as primary storage for your application. Bare SATA drives Velcro’d to motherboards to a big cluster of DMXs. Both are in use in major enterprises for mission critical applications – and they both work.

The 60 year secular trend to cooler data is the cause – an inverse of Moore’s Law. As the average accesses of data declines, technologies that meet the need at a lower cost become attractive, find a market, and grow. Niche products become mainstream – and perhaps “primary” – for their markets.

At the same time Moore’s Law is working its magic: creaky slow 10Mbit Ethernet becomes 10GigE. Board level controllers become chips. Storage software migrates from firmware to a stack running on commodity processors. Yesterday’s “archive” storage is tomorrows “primary” storage for the right apps.

Even the term “enterprise” is losing its meaning. As firms begin the 10 year migration to private clouds for cooler data, commodity hardware – servers, unmanaged switches, SATA drives – will be knit by cluster software that may even be open source. It is “enterprise” because an enterprise is using it.

This why all the big iron vendors are migrating their software from embedded firmware to stacks running on commodity processors and operating systems. For the mainstream market the commodities are fast enough and the economics are compelling.

If if works for you, it’s primary.

Courteous comments welcome, of course. BTW, I’m getting a briefing from HDS on the old Archivas product, so maybe I’ll have more to say RSN.

Related posts:

1. Cisco’s UCS limited scale Should a Unified Computing System be able to scale to...
2. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
3. A 1 petabyte science project But not that kind of science project. This is the...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Cisco’s UCS limited scale Should a Unified Computing System be able to scale to...
2. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
3. A 1 petabyte science project But not that kind of science project. This is the...

Winding up the week – it is Friday here – in Japan as a guest of Hitachi Data Systems. Fine hospitality from my American and Japanese hosts in steamy mid-summer Tokyo. Looking forward to Arizona.

The practitioners in the group – one who loves XIV, others with EMC and NetApp kit – were surprised by what the HDS stuff does. Such as virtualizing and managing your current storage platforms, regardless of vendor.

Seems like the big guys have been promising that for years. HDS delivered? Whoa.

A couple of things impressed me:

• The senior Japanese execs weren’t the starchy, face-saving guys I’d expected. The Chairman of Hitachi made a speech to about 10,000 people without a tie, and all the other execs I spoke to followed suit. Even giving careful non-answers they came across as relaxed and realistic. Are they also decisive? We’ll see.
• HDS has a clustered object store. I hope to get briefed on it next month.
• The parent company has a vision for using massive amounts of data to improve our quality of life. Since they also produce power systems and high-speed trains they have a direct line into some critical issues.

The StorageMojo take
HDS is a multi-billion dollar company with some leading edge products and technologies. They’re about the size of NetApp – and I know you’ve heard of them.

As their OEM relationship with Sun winds down – or at least I expect it to – they’ll have more direct contact with a new group of customers. Now is the time for HDS to sharpen their messaging and turn up the volume.

Sadly that isn’t likely. The internal dynamics of the company seem to lead to generic messaging that fails to plant a hook. Maybe it is a consensus thing. But they aren’t doing customers any favors.

Courteous comments welcome, of course. Any recent experience with HDS?

Related posts:

1. Oracle+Sun storage: wiser & brighter While everyone else was watching the Apple iPad intro I...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Oracle+Sun storage: wiser & brighter While everyone else was watching the Apple iPad intro I...

The friendly folks at Hitachi are flying me and a number of other analysts and bloggers to Tokyo. They want to tell us about their plans for – well, I don’t know what – and it’s under NDA.

Normally I don’t sign those, but between Tokyo – which I like – and the promise of seeing Hitachi’s strategy, I was reminded of Emerson’s comment:

Foolish consistency is the hobgoblin of small minds.

The StorageMojo take
I’ll be arriving at Narita about 4am PDT, so don’t expect crisp comment moderation. I will try to post this week though.

Courteous comments welcome, of course. Anything you want me to ask Hitachi, even though I won’t be able to tell you what they said?

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Cloud computing gets a bad rap because it can’t replace corporate data centers for mission critical apps. But new computing paradigms never do that: it is the new capabilities they enable that drive adoption. Case in point: transcoding.

Why?
Anyone who shoots video soon discovers that changing from, say, AVCHD to an editing-friendly codec and then to H.264 for distribution takes a lot of compute cycles. Conversion from one codec to another is called transcoding. It is the price we pay for high quality compressed content.

Compression and format conversion are necessary because highly compressed video – the kind most camcorders shoot – isn’t easy to edit. And the stuff that’s easy to edit has large files that chew up bandwidth and storage.

So we transcode. Add to that the number of formats we use – ranging from iPhones to flash to SD and 1080p – and transcoding is a major CPU cycle sink.

Fortunately, transcoding can be a highly parallel operation. A frame – or a series of frames – can be divided and split among multiple cores and CPUs.

Where?
Where can you find a lot of CPUs for a quick job? Right, the cloud. Which is why there are a number of online services that front-end Amazon Web Services to provide transcoding.

I spoke to the CEO of startup Zencoder, Jon Dahl to learn more.

Zencoder
Zencoder is a transcoding service provider that uses Amazon as a cloud provider. The Zencoder team has developed transcoding infrastructure for several startups and finally decided to build a general-purpose service.

While they use open source software in their stack – as do most transcoding providers – their major value-add is in a high-performance scalable interface. Handling 100,000 concurrent transcodes is non-trivial.

They also look out for problems common in transcoding such as audio/video getting out of sync and aspect ratio distortion. They can transcode 1080p faster than real time. And they’ve licensed the proprietary formats as well.

Amazon offers Linux as a service and a file service. S3′s files are limited to 5 GB, but that isn’t a problem for Zencoder: customers can specify input and output locations, bypassing Amazon storage.

Also they don’t transcode Mac ProRes – Final Cut Pro’s preferred editing format – today. But they do handle QuickTime movies.

The StorageMojo take
So the glass house doesn’t want to outsource cloud infrastructure. Who cares? They’re the last to adopt new technology anyway.

It is apps like transcoding that drive the business. In 5 years much, perhaps most, transcoding will be cloud-based.

Before the digital video craze in the last 5 years there wasn’t much demand for transcoding. But today, with HD video smartphones, millions are producing videos that they want to share and save.

Your smartphone won’t have the cycles to do it, but the cloud does. Expect transcoding vendors to add new features, such as noise-reduction or sharpening.

Business units are discovering the power of short videos to inform, train, persuade and excite. All at a fraction of the cost of 4-color brochures.

The outlook for storage vendors is mixed. Yes, much more storage will be sold – but cost-conscious cloud managers will be buying it. And as more new services develop on the cloud, consumers will be as hazy about “local” and “cloud” as they are about “memory” and “disk” today. Branding nightmare, but that’s where those petabytes will be.

Courteous comments welcome, of course.

Related posts:

1. Bringing cloud to the masses – of service providers If cloud computing or cloud storage is a gold rush,...
2. Cloud at Storage Visions 2010 I moderated a panel on cloud storage at Tom Coughlin’s...
3. The Cloud Quadrant Thinking about cloud Amid the hype and glitz on cloud...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Bringing cloud to the masses – of service providers If cloud computing or cloud storage is a gold rush,...
2. Cloud at Storage Visions 2010 I moderated a panel on cloud storage at Tom Coughlin’s...
3. The Cloud Quadrant Thinking about cloud Amid the hype and glitz on cloud...

Given how hard it is to save data you want (see The Universe hates your data) to keep, losing data on the web should be easy. It isn’t, because it gets stored so many places in its travels.

Problem
But the power of the web means that silliness can now be stored and found with the speed of a Google search. You don’t want sexy love notes – or pictures – to a former flame posted after infatuation ends.

Or maybe you want to discuss relationship, health or work problems with a friend over email – and don’t want your musings to be later shared with others. Wouldn’t it be nice to know that such messages will become unreadable even if your friend is unreliable?

Researchers built a prototype service – Vanish – that seeks to:

. . . ensure that all copies of certain data become unreadable after a user-specified time, without any specific action on the part of a user, without needing to trust any single third party to perform the deletion, and even if an attacker obtains both a cached copy of that data and the user’s cryptographic keys and passwords.

That’s a tall order. Their 1st proof-of-concept failed. But they are continuing the fight.

Vanish
In Vanish: Increasing Data Privacy with Self-Destructing Data Roxana Geambasu, Tadayoshi Kohno, Amit A. Levy and Henry M. Levy of the University of Washington computer science department present an architecture and a prototype to do just that.

Ironically, the project utilizes the same P2P infrastructures that preserves and distribute data: BitTorrent’s VUZE distributed hash table (DHT) client.

The basic idea is this: Vanish encrypts your data with a random key, destroys the key, and then sprinkles pieces of the key across random nodes of the DHT. You tell the system when to destroy the key and your data goes poof!

They developed a data structure called a Vanishing Data Object (VDO) that encapsulates user data and prevents the content from persisting. And the data becomes unreadable even if the attacker gets a pristine copy of the VDO from before its expiration and all the associated keys and passwords.

Here’s a timeline for that attack:

DHT overview

A DHT is a distributed, peer-to-peer (P2P) storage network. . . . DHTs like Vuze generally exhibit a put/get interface for reading and storing data, which is implemented internally by three operations: lookup, get, and store. The data itself consists of an (index, value) pair. Each node in the DHT manages a part of an astronomically large index name space (e.g., 2¹⁶⁰ values for Vuze).

DHTs are available, scalable, broadly distributed and decentralized with rapid node churn. All these properties are ideal for an infrastructure that has to withstand a wide variety of attacks.

Vanish architecture

Data (D) is encrypted (E) with key (K) to deliver cyphertext (C). Then K is split into N shares – K₁,…,K_N – and distributed across the DHT using a random access key (L) and a secure pseudo-random number generator. The K split uses a redundant erasure code so that a user definable subset of N shares can reconstruct the key.

The erasure codes are needed because DHTs lose data due to node churn. It is a bug that is also a feature for secure destruction of data.

Prototype
They built a Firefox plug-in for Gmail to create self-destructing emails and another – FireVanish – for making any text in a web input box self-destructing. They also built a file app, so you can make any file self-destructing. Handy for Word backup files that you don’t want to keep around.

The major change to the Vuze BitTorrent client was less than 50 lines of code to prevent lookup sniffing attacks. Those changes only affect the client, not the DHT.

The Vanish proto was cracked by a group of researchers at UT Austin, Princeton, and U of Michigan. They found that an eavesdropper could collect the key shards from the DHT and reassemble the “vanished” content.

Who is going to collect all the shard-like pieces on DHTs? Other than the NSA and other major intelligence services, probably no one. For extra security the data can be encrypted before VDO encapsulation.

The StorageMojo take
The Internet is paid for with our loss of privacy. Young people may think it no great loss, check back in 20 years and we’ll see what you think then.

It is slowly dawning on the public that their lives are an open book on the Internet. Expect a growing market for private communication and storage if ease-of-use and trust issues can be resolved.

You don’t have to be Tiger Woods to want to keep your private life private. I hope the Vanish team succeeds.

Courteous comments welcome, of course. Figures courtesy of the Vanish team.

Related posts:

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. Avere Systems: dynamic tiering for the masses – of data? Last week RisingTide Systems, a stealth startup with no web...
3. Why private clouds are part of the future James Hamilton, Amazon architect and a very smart guy, recently...

Related posts brought to you by Yet Another Related Posts Plugin.

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. Avere Systems: dynamic tiering for the masses – of data? Last week RisingTide Systems, a stealth startup with no web...
3. Why private clouds are part of the future James Hamilton, Amazon architect and a very smart guy, recently...

Greg Reyes, former CEO of Brocade, received a sentence of 18 months and a $15 million dollar fine for his conviction on 10 felony counts related to options backdating. Prosecutors had asked for 37 months and a $137 million dollar fine. Mr. Reyes was emotional at his sentencing:

When Reyes got his opportunity to address Breyer, he stood at the lectern silently for a few seconds, and then broke down sobbing. [His attorney] read his statement for him.

“I am a shell of the man I once was,” he read.

Breyer said he was quite moved by the 400 letters sent in on Reyes’ behalf, as well as the financial and emotional support he extends toward others. Yet a message must be sent to executives that deceiving the public markets is a serious crime, Breyer said.

The judge cited one more reason for a prison term.

“White-collar defendants, unlike most defendants I see in court every day, have choices,” Breyer said, adding that he had just sentenced a man to more time than Reyes because he illegally re-entered the United States to see his 5-year-old son.

In two weeks, Breyer will sentence another man whose drug addiction began when his father shot him up with heroin when he was 11.

“What choices did that young boy have?” Breyer said.

[From Law.com]

The best CEO of any high tech company?
I met Mr. Reyes a couple of times when both of us wanted Sun to buy FC switches to make Sun’s early FC array more maintainable. I was at Sun at the time. He was an excellent salesman, but some idiot had decreed no FC switches for the storage group.

Storage Newsletter had an odd bit of history as well:

In 2002, we asked Steve Duplessie, well known consultant, to told [sic] us who was the best CEO in the storage industry. His answer: “[The best CEO] would be Greg Reyes of Brocade.”"

The 2 critical success factors for salesman are: a capacity for self-delusion – so you can sincerely and honestly tell your prospects how good it is; and a resolutely short term focus, because making this quarter’s numbers is what counts. Don’t hire a salesman to design your products or your strategy.

The StorageMojo take
Given Brocade’s current problem – they’ve been for sale for over 9 months and there are no takers – and his own, Mr. Reyes was no strategist. But Brocade’s IPO timing made fortunes for Mr. Reyes and co-founders Paul Bonderson and Kumar Malavalli. Isn’t that what really counts?

But Mr. Reyes can be forgiven if he feels unfairly singled out. Here we are 2 years after after the big Wall Street meltdown, where the big ibanks were packaging and selling crap and calling it gold, when mortgage companies and rating agencies had gone wild, and who’s gone to jail for that?

At the same time, Maher Arar, a Canadian who was arrested in 2002 by U.S. officials while changing planes in New York on a trip to Montreal and then rendered by US officials to a Syrian jail was denied a hearing by the US Supreme Umpires. According to the findings of fact, Mr. Arar

. . . was in Syria for a year, the first ten months in an underground cell six feet by three, and seven feet high. He was interrogated for twelve days on his arrival in Syria, and in that period was beaten on his palms, hips, and lower back with a two-inch-thick electric cable and with bare hands.

So buck up, Mr. Reyes, things could be worse. In 18 months you will have paid your debt to stockholders and you will still be among the richest 30,000 or so people in the world.

Courteous comments welcome, of course. America is a nation of laws, not of men, unless the men are fighting terrorism.

Related posts:

1. Brocade’s ex-CEO convicted on 9 felony charges In a 2nd trial, Brocade’s former CEO Gregory Reyes was...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Brocade’s ex-CEO convicted on 9 felony charges In a 2nd trial, Brocade’s former CEO Gregory Reyes was...

Storage Newsletter reports that Tape Drive and Media Revenues Decreased by 25% in 2009. The data comes from a report by the Santa Clara Consulting Group.

The numbers show us how old tape formats die: slowly. While the overall market for drives and media was $1.58B it was split among LTO, DLT, DAT, 8mm and even, gasp, QIC.

The good news: drive sales were $629M, suggesting that media sales will continue for years to come. LTO had over 83% of drive sales – $534M – with DAT (!) drives making most of the rest – $69M – and DLT much of the remainder.

The media numbers are revealing. Overall, media sales were only about 50% greater than drive sales or $955M. But in the case of DAT, media sales of $45M were less than drive sales. Buyers aren’t making much use of their new drives.

8 mm and QIC bring up the rear. Somebody bought over a million units of AIT media and over $16M of QIC media.

The StorageMojo take
The long tail of tape is longer than I’d thought. There must be ancient systems out in retail or OEM equipment that use the media. Military, too.

But why that 25% drop in the overall tape market? I’d need more time series data to draw any firm conclusions, but here’s what I’d look at:

• The Great Recession. The overall slowing in business and capital expenditures is a piece of that. But the world economy did not decline 25%, thank goodness, so that can’t be the full cause.
• D2D. Data de-duplication is aimed at making disk competitive with tape. Looks like 2009 was the year it took a byte out of tape.
• Tape capacity growth. The LTO folks have been increasing LTO tape capacity at a rate near that of disk. More data, fewer tapes. Disks, of course, wear out, so the replacement market is huge.
• Drive cost. At $3-$4k for an LTO 5 drive and $125 per 3 TB tape, the use of tape is moving upmarket, which means smaller volumes.

Some people love what tape does. But others don’t: I haven’t seen a new tape or disk-based camcorder introduced in over a year. Everyone is going to flash.

$1.5B markets don’t die overnight – even dropping 25% a year. Tape will be around for a long time to come.

Courteous comments welcome, of course. I kicked off DLT for DEC back in 1991 and have always wondered why Quantum just rolled over for LTO instead of fighting. Oh well.

Related posts:

1. Will a 70 TB cartridge save LTO? IBM and Fujifilm have demonstrated a technology that, if productized,...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Will a 70 TB cartridge save LTO? IBM and Fujifilm have demonstrated a technology that, if productized,...

Kaminario has introduced the world’s fastest SAN storage, the K2. If time is money, this is for you.

DRAM
Kaminario’s K2 is fast because DRAM, not disk, is the primary storage. DRAM’s low latency, high bandwidth and durability breaks the tight link between capacity and performance that disks and flash impose. No need for excess capacity to ensure enough IOPS, bandwidth or service life.

The product
Kaminario is a software company. However, they configure customer systems and install the software to order. No home-baked integration here.

The basic hardware unit is a Dell blade server. The blade servers are either I/O directors or data nodes. The Dell server chassis is a passive box – no active components on the backplane – but some customers opt for dual chassis for redundancy out of caution.

I/O directors
The I/O directors use 8 gig Fibre Channel to servers and 10Gig/Ethernet to data nodes. The company says they can saturate both due to proprietary software optimizations.

Using FC switches, each I/O director can talk to multiple servers. Each I/O director can handle 150,000 random IOPS.

K2 architecture - courtesy Kaminario

The minimum config is 2 I/O directors and 4 data nodes with 500 GB of capacity. That’s 300,000 IOPS. They’ve been tested to 10 nodes and 1.5 million random read/write IOPS with support for 16 nodes – and double the IOPS – reportedly coming soon.

Under the covers
The I/O directors are clustered so when 1 fails the others pick up the load. The switched back end 10Gig Ethernet enables all I/O directors to access all data nodes.

The replication default is 2 copies of all data on different blades. Plus copies on disk.

All this runs on standard Dell blade servers. No specialized, low-volume RAID controllers or power-hungry disk shelves.

Software
The secret sauce is the software. Kaminario doesn’t say much about how they do what they do. In any high-performance cluster maintaining metadata coherence across nodes is one of the tough problems.

They did say they maintain hash tables that enable very short updates to all I/O directors after writes. I also suspect they also have implemented a low latency backend update protocol. Metadata serving is distributed across the cluster.

They must also have some creative ways to max out FC links. I’d like to know more.

Management
With storage this fast they say you need little tuning. Lay LUNs across the data nodes and fasten your seatbelt. The software includes optimizations, like pseudo-random block layout to minimize contention, automatic load balancing and demand-based block replication.

If your app calls for it you can tune chunk sizes and set replication policies. Kaminario says K2 is much easier to manage than typical high-performance storage – you don’t have to worry about disk-induced issues like stride.

Management is kept out of the data path on a dedicated GigE network.

Support
Kaminario says they have designed the product and their organization to provide mission-critical Enterprise support. The visible elements from configuration control and software installation to phone home and remote diagnostics back that up.

Who needs this?
If you are hammering a few TB of data for stock trading, real-time business intelligence or TLA government work, this could be the ticket.

Pricing
If you have to ask. . . .

Kaminario has a unique approach: pay for performance:

. . . we price the solution based on the customer IOPS and capacity needs, so basically the way we present such a platform price is by $/GB/IOPS.

I *think* small configs start around $200k. For the performance market price is something like #7 on the list. The first 3 are performance/availability – 2 sides of the same coin, really.

This removes SPEC shadow puppetry between application requirements and storage performance. Of course, you have to know what performance you want. But anyone who’s performance tuning high-end arrays will know that.

The StorageMojo take
Kaminario is opening a new niche at the performance end of the market.

The current Big Storage vendors claim that they too can do a million IOPS. And they can, for millions. A price that makes a few TB of DRAM look cheap.

Since high-end disk – ≈$1/GB retail – makes up 5-10% of the cost of a high-end array, replacing disk with DRAM might be expected to double the cost of an array. But K2 does away with all the low-volume kit – controllers, shared cache, disk packaging and more – and replaces it with high-volume blade hardware. That lowers costs a lot.

Kaminario has opened a new niche: hyper-performance data storage. While a few TB doesn’t sound like much, it is more text than all but the world’s largest libraries place on miles of shelves.

The data arms race has kicked up another few notches. It is more competition for the big iron arrays where they least expected it: at the high-end of the market.

Courteous comments welcome, of course.

Related posts:

1. Hot data, smart cache Okay, we’ve figured out how to produce protected storage for...
2. Flash bash: disks and DIMMs Flash memory is opening a second front in its war...
3. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Hot data, smart cache Okay, we’ve figured out how to produce protected storage for...
2. Flash bash: disks and DIMMs Flash memory is opening a second front in its war...
3. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...

I did something last week that I haven’t done for a year: read a book. I do plenty of reading, but somehow books fell off the list of preferred input devices.

Time to buy a Kindle or an iPad?
In retrospect I was taking a vacation from blogging. If I’d known that in advance I’d have let you know.

A bad book might have put me off from books for years, but I chose well: The Strangest Man by Graham Farmelo, a biography of the eminent theorist of quantum mechanics, Paul A. M. Dirac. A brilliant book about a difficult man and an even more difficult subject.

The author, Farmelo, earned a PhD in physics. And now he consults on technical communications. The book demonstrates that he knows how to take the counter-intuitive abstractions of quantum physics and describe them through artful examples.

But more than that, the book details the tangled history of the development of quantum mechanics, covering not just Dirac, but Heisenberg, Bohr, Fermi, Feynmann, Oppenheimer, Wigner, Bethe, Schrodinger and Kapitza. He evokes the flavor of the major pre-war research centers of European physics and makes the story a page-turner.

Dirac did not join the Manhattan Project, despite winning the Nobel Prize in his early 30s. He was a loner, possibly autistic, and made little effort to conform to social norms. The title comes from Niels Bohr’s comment that of all physicists who had visited his research institute, Dirac was the strangest.

The StorageMojo take
Quantum physics is at the root of the revolution in solid state devices that has made the computer industry incredibly successful. We don’t need to understand it to use computers, but there is something about the intellectual struggle to make sense of the universe – a process far from complete – that is endlessly fascinating.

If you are also interested in science, The Strangest Man is highly recommended.

Courteous comments welcome, of course. Dirac was appointed to the most prestigious professorship in Cambridge, the Lucasian Chair of Mathematics – once held by Isaac Newton – at the age of 29.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Today in the United States we celebrate a national holiday, Memorial Day, to honor those who have fought for our country and its Constitution.

I thought to re-post one of my Memorial Day posts from last year, Hospital ship Haven in Nagasaki, Japan, 1945 about my father and his World War II experiences. Highly recommended, in any case.

Instead a shorter and slightly newer story was presented to me last night at a party that a friend held for her father’s 85th birthday. He spoke for a few minutes, vigorous and alert, about his life.

He said that he was really only 61 years old. Why?

Because he came to America in 1949 – 61 years ago – after surviving Auschwitz. Here he built a life for himself and his family.

His life, and his love of his adopted country, is its own memorial. And a small reminder of what is best about America.

Related posts:

1. StorageMojo@NAB 2010 StorageMojo’s Global HQ will move to Las Vegas for a...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo@NAB 2010 StorageMojo’s Global HQ will move to Las Vegas for a...

A hybrid of SSD and hard drive that gives you the best of both worlds. That’s the theory anyway. But I won’t keep you in suspense: I think Seagate has hit a home run with their new hybrid XT architecture.

Specs
Take a standard issue 7200 rpm, 500 GB notebook SATA drive. Add 4 GB of fast and reliable single-level cell (SLC) flash. Make it look like a standard drive to the OS by keeping all the magic internal.

Give it smarts to learn about high-frequency small-block transfers. Put those blocks in the flash and voilà: super-fast small block access; leaving big sequential I/Os to the disk. The algorithm

The SSD is 2 SLC chips, each with about 75 MB/sec bandwidth. If the flash should fail – more likely than wearing out IMHO – you still have a perfectly good 7200 rpm disk. The algorithm looks to provide the most benefit on the most used apps.

Theory
Disk drives offer cheap capacity and good large read/write (R/W) bandwidth. The R/W bandwidth improves over time as drive capacities grow due to higher bit density.

Flash SSDs offer sub-millisecond access times at a high price: $2-$3/GB. SSD access times are about 150x faster than a notebook hard drive, while R/W bandwidth is only about 2x faster.

Maximum bang for the buck? SSD to store many small files – like DLLs in Windows – to minimize accesses, and let the disk handle the large R/W traffic. Why pay 40x for a 2x performance boost?

Flash capacity? 4 GB can store 4 million 1k files. That saves a lot of access time.

Test
An early 2009 17″ unibody MacBook Pro with 4 GB RAM and 2.66 GHz Core Duo 2. Swapped the current 7200 rpm, 500 GB Hitachi for a 7200 rpm, 500 GB Momentus XT with 4 GB SLC flash.

For the first test I formatted it with HFS+ using the Mac’s Disk Utility. Then I used the excellent Carbon Copy Cloner to clone the system disk to the XT.

Results
Boot performance sucked compared to the old drive’s ~45 second boot:

• First boot took 2 minutes 4 seconds
• Second took 1 minute 19 seconds
• Third took 1 minute 6 seconds

After consulting with Seagate I did a clean install of OS X. Reformatted the XT, installed Mac OS 10.6 Snow Leopard, migrated accounts and data. Took 3 hours, but the process worked perfectly.

New boot times went to ~45 seconds again. Not what I had hoped – the SSD on my MacBook Air booted in 15-20 seconds – but competitive. Other testers have had better results so Seagate is sending me another drive. I’ll update the results later this week.

Application results were much more impressive. Because the drive learns, the first time you bring up an app it happens at disk speed. But the 2nd time!

Mail startup went from 5 seconds to 1.5. Microsoft Word startup went from about 12 seconds to less than 3. See for yourself in this 15 second clip:

And the bigger the app, the bigger the potential speed up. Look at – or not – this 1 minute clip of the 1st and 2nd time opening Final Cut Pro, the Mac pro-level video editing app:

From 45 seconds to 10 17? Way faster than my Mac Pro’s 10k system disk.

Note that rarely opened apps won’t do as well as the algorithm favors apps that get used more often. Pure SSDs don’t have that variability.

Pricing
Announcement pricing:

• 500 GB/$156
• 320 GB/$122
• 250 GB/$113

These prices are about double the street prices for the non-hybrid drives. Once Seagate gets some competition they should drop.

The StorageMojo take
Looks like a home run for Seagate. The architecture is clean, the performance advantages are real, and pricing is not too bad – especially compared to SSDs.

SSD adoption has stalled because flash prices have firmed up. Few are willing to pay $200-$300 for a smallish SSD in a $700 notebook. But $100? OK.

Seagate could use cheaper MLC flash in 8-16 GB caches without wearing it out. There is every reason to put this in 3.5″ drives as well. After all, if the flash does fail all your data is on the hard drive – no loss there.

By blurring the performance difference between disk and SSD, these drives will ensure that hard drives dominate for at least another decade. And they’ll put pressure on SSD prices.

But don’t count on these showing up in RAID arrays soon. Arrays already have a lot of moving parts and hybrid drives add some subtle wrinkles. Like how do you know what a small block transfer is? Or ensure you don’t have stale data? Not sure cached array controllers need this.

Courteous comments welcome, of course. No money changed hands but I’m not excited about returning the review unit. Also I sent a note off to CCC’s developer to alert him to what I saw – maybe he can figure it out. Update:As a number of commenters pointed out – thanks! – I screwed up: the original video showed a reboot from system cache. When I got the time I reshot the FCP startup after a reboot. It was still 40% of a regular drive and still much faster than a 10k drive on my Mac Pro. Sorry about that! End update.

Related posts:

1. TPC-C: comparing SSD & disk Steve Jones of BT sent in the following, which I...
2. WD’s new 2TB drive delivers on green promise Triple your data center’s storage capacity – without increasing power...
3. Is STEC’s lead sustainable? Chris Mellor of The Register considers whether STEC’s lead in...

Related posts brought to you by Yet Another Related Posts Plugin.

1. TPC-C: comparing SSD & disk Steve Jones of BT sent in the following, which I...
2. WD’s new 2TB drive delivers on green promise Triple your data center’s storage capacity – without increasing power...
3. Is STEC’s lead sustainable? Chris Mellor of The Register considers whether STEC’s lead in...

Shouting at a disk drive will cause it to stop. But what about the constant nagging they get in busy data centers? That’s a bigger problem.

Bad, bad, bad vibrations
The use of consumer-grade SATA drives in the enterprise raises the concern. A 2005 study, Performance Impact of External Vibration on Consumer-grade and Enterprise-class Disk Drives, by Thomas Ruwart and Yingping Lu found that consumer

. . . disk drives are more sensitive to the vibration from physically coupled adjacent disk drives. . . . [G]reater care needs to be taken in enclosure design, particularly for the 3.5-inch form factor disk drives due to their higher-energy seek operations when compared to seek operations on a 2.5-inch form factor disk drive.

Yet another reason to end-of-life 3.5″ drives.

Mechanical engineers know that drive vibration is a serious problem. How serious is the question. We do know that steel does a good job of transmitting vibrations as this graph shows:
Courtesy Rockwell Automation
An internal study of a vibration-damping rack by a major storage and server vendor obtained by StorageMojo – and not available on the web – found a

20% decrease in IO throughput performance (read) and 25% increase in total time to perform a task of typical 2U servers resulting in 25% increase in energy consumption . . . when vibration level went up from 0.1 GRMS to 1 GRMS. This is a conservative result, as newer HDDs are more sensitive to vibration, “write operation” is 20% more sensitive than “read operation” to vibration and effect of vibration on rotating components , HDDs & fans, are not accounted for.

[GRMS - properly G_rms - is the root mean square of acceleration measured in g's. Check out The Calculation of GRMS (pdf) for a technical discussion.]

Drives use accelerometers to compensate for shock and linear and rotational vibration but the info isn’t available externally so it’s difficult to directly quantify vibration impact. But data centers are noisy places quivering with the vibration of fans, air conditioners, disk drives and 60 cycle hum, all supported by steel racks.

A limited study
In a USENIX paper presented at the SustainIT ’10 conference Julian Turner reported on limited tests of a prototype anti-vibration rack. The AVR-1000 is an engineered carbon fiber composite rack designed to dissipate vibration across a wide frequency range.

His observations include:

Performance improvements for random reads ranged from 56% to 246% while improvements for random writes ranged from 34% to 88% for a defined set of industry benchmarks. Streaming sequential reads and writes had a much smaller performance improvement. . . .

The internal study found similar results:

IO throughput performance (write) increased by 285% and total time and energy to perform a task of a 4U storage server with 48 HDDs decreased by 64% when vibration level went down from 1 GRMS to 0.1 GRMS.

Evidently the combination of random head movements and vibration has a substantial effect on disk read performance.

For the enterprise and Internet-scale data centers the implications are substantial:

• Energy savings. Anti-vibration racks can save power by improving disk performance and reducing run times.
• SSD value. Flash SSDs have fast random read access. But disks can improve their performance by 50% through vibration damping, changing the SSD value proposition.
• Array sizing. Enterprise arrays are over-configured to improve performance. If disks were suddenly 50% faster that could be reduced or, alternatively, the utilization could be increased.

The StorageMojo take
The research is limited, but everything we know about disks and vibration today suggests this is real. Assuming further research finds similar results we could see an explosion of products using engineered materials to improve disk performance.

This is already the norm in chip fabs, where nanometer precision requires extensive vibration damping. Disk feature sizes are even smaller, so why not?

Drive vendors should make accelerometer data available to researchers. Vibration-damping racks could make disks much more competitive with SSDs – and mean billions to drive vendors – but only if we have the data.

For home users of multi-drive towers it may be that damping carbon fiber towers or disk mounts could improve performance. It may not be economic, but it sure would look awesome.

The impact for Internet-scale infrastructures is more nuanced. Google, for one, doesn’t use high-density storage – maybe 120 drives per rack – so the performance gains may not be economic for them. But users of many drive servers with, say, 300+ drives in a rack would find it easier to justify.

Courteous comments welcome, of course.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Harmonic is acquiring video production infrastructure and storage provider Omneon for $274 million. They’d raised about $100M since their founding.

Omneon Video Networks is a specialized storage company that provides broadcast quality storage for digital media, along with the gear needed to convert video streams to bits. They do clustering, in their MediaGrid product, a sophisticated architecture that can handle a 7×24 beating.

Founded in 1998, venture-backed Omneon started offering storage in response to customer demand. They chose a commodity-based cluster and built their own storage software, MediaGrid, whose architecture hews to the post-array Google-style storage model:

• No RAID – slices are replicated one or more times based on policy or demand
• Single global namespace
• Out-of-band meta-data servers manage content servers

Omneon’s content servers do more than serve content. They put their unused CPU power to work doing jobs like transcoding – translating content from one format like HD to iPhone-suitable QuickTime.

The StorageMojo take
Omneon is more than a storage company, but their storage made them a competitor to Isilon in the broadcast market. Harmonic is big in the rest of the video workflow, especially distribution in multiple formats. It looks like the 2 firms complement each other nicely.

Omneon was not a pure play storage company. But the fact that they were able to build a competitive storage product as an adjunct to their main business points up how low the barriers to entry are in scale-out storage.

Courteous comments welcome, of course. I’m still at EMC World. YottaYotta’s technology is front and center in the VPLEX product. More on that later.

Related posts:

1. Cisco’s UCS limited scale Should a Unified Computing System be able to scale to...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Cisco’s UCS limited scale Should a Unified Computing System be able to scale to...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Leaving Sunday for beautiful downtown Boston. Really: much nicer since they finished the Big Dig.

Hoping to learn more about Atmos. Did you know it is available as an online service?

And I want to talk to some of the RSA researchers working on storage, but I’m not sure they are part of EMC world. Maybe I can squeeze in a visit to their Cambridge lab.

Any thoughts on stuff I should see? If you’ll be there and would like to talk please comment or email.

The StorageMojo take
Heard that of the 15,000 or so folks expected, some 5,000 will be carrying an EMC badge. Nice break from cubicle heaven.

Shades of DECWorld. But back in the 80s the minicomputer business was getting killed by PCs. The threat to EMC’s business is commodity-based scale-out storage, something they are wisely embracing with Atmos.

DEC founder and president Ken Olsen grew DEC for 30+ years, a record I don’t believe has been equaled by any other computer industry entrepreneur, though Bill Gates comes close. But even Ken didn’t know how to navigate the shift from vertical integration to assembling horizontal commodities. Nor did he see how commoditizing DEC’s proprietary technologies, such as DECnet, VMS and VAXclusters, could have ensured the company’s long-term survival.

That’s why Atmos is important to EMC. As our data continues to cool, commodity-based scale-out – or specialized hardware with commodity prices – storage is critical. Tucci seems to get that.

Courteous comments welcome, of course. Looking forward to seeing springtime Boston – if the weather cooperates.

Related posts:

1. Atmos architect moves on Patrick Eaton, the Berkeley PhD. who architected EMC’s Atmos cloud...
2. StorageMojo traveling Off to Boston for a group blogger tour today and...
3. StorageMojo off to HP’s Storage Tech Day A re-education camp for bloggers. If I start gushing on...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Atmos architect moves on Patrick Eaton, the Berkeley PhD. who architected EMC’s Atmos cloud...
2. StorageMojo traveling Off to Boston for a group blogger tour today and...
3. StorageMojo off to HP’s Storage Tech Day A re-education camp for bloggers. If I start gushing on...