Spring Cleaning = New location

Life Zero — Tue, 03 Mar 2009 12:10:52 PST

After many, many years of using Typepad...

I've decided to move my blog.

The new URL is here --> LifeZero at SquareSpace

OR, if you wait a day from this entry, www.LifeZero.org should redirect you there without problems.

For those who are subscribed via RSS, the new RSS feed URL is here --> Life Zero RSS

Super Fast 4TB RAID for Under $1,500 (was $2k)

Life Zero — Wed, 20 May 2009 13:53:46 PDT

HEY YOU - YEAH, YOU! MY BLOG HAS MOVED.
GO TO WWW.LIFEZERO.ORG for the new blog!

Updated: October 7, 2008: 6:50PM PDT (added new benchmarks & CalDigit Response)

Updated: December 4, 2008: 2:30PM PDT (added new links, updated title)

Updated: 2009 (Added HD Speed Test benchmark ... take that, @jasondiamond )

Background

If you are a film editor or dealing with large files, especially video, there's always a time when you run out of space and when you need more speed for editing or other work. This has been the problem for a long time and, frankly, decent RAID systems are prohibitively expensive for mortals, without the fat coffers of large company money.

Updated: I will soon have a video podcast which will walk through how to build this system and what you can expect, plus configuration options and a walk-through of setting it up. Stay tuned.

If you, like me, have decided to create a home studio, where you edit HD video and you're looking for a fast, powerful, stable and cheap RAID solution -- you've likely found everything aplenty, save for the cheap part of the solution.

Let's set a few things straight:

RAID is not cheap. Not like a lot of things. Good RAID requires hardware and hardware costs money.
Pre-packaged RAID solutions are expensive as hell. $6,000 for a fast, 4TB RAID system is pretty much the norm.
Do It Yourself RAID systems suck. And, not just kind of. They are unreliable. They are flimsy and half-baked and, if you're really cheap, they use an inferior RAID card or a software RAID.
RAID systems are notoriously slow for the money you're paying.
Vendors stats are bogus. Their stats are "best case" and are measured when the RAID is empty or 10-20% full, which is the sweet spot for RAID systems.

Okay, let's look at the other side:

You are cheap. You don't want to spend more for RAID than you did for your computer.
You want fast, cheap and reliable.
You don't have a Ph.D. in RAID technology and putting together a RAID from scratch is hard, to say the least.
You are editing HD footage and need something that keeps up with 1080p, even on a bad day.

I've searched everywhere, looking at possibilities, and the solution is a kind of hybrid of a standard, tested RAID chassis, a RAID card, a few cables and some cheap drives. I've been able to put a 4TB RAID system together that achieves 400 MB/s speeds (even when 80% full) for under $2,000.

System

I'm running a Mac Pro PCI Express system with 8GB RAM (basically, the newest Mac Pro). Theoretically, and according to everything I've seen and read, this configuration should work on ANY PCI Express-based computer, from OS X 10.4 through Windows XP and up to Windows Vista.

Considerations

Existing RAID solutions, even the ones with the best reviews and the best prices, are still pitifully underpowered (slow) and woefully expensive (by a lot).

In addition, because I'm doing audio and video editing, it is important for the RAID (or any hard drive system) to be quiet. I've tried a number of enclosures, drives and RAID configurations and the one I finally settled on is - by far - the most quiet and has the least vibration and noise.

I should also point out, for the record, I don't care about redundancy. If I want something backed up, I'll use external drives, or better yet, a Drobo. However, this RAID supports RAID 5 and is still very fast (just not fast enough for me).

Most people consider the CalDigit HDPro RAID to be the gold standard. But, even at 4TB, it comes in around $6,500 and it's too slow (yes, I said "too slow").

May people talk about the CalDigit HDElement because it's modular and you can stack them together. Firstly, it's not that modular (or, more directly, it's no more modular than other RAID systems) and secondly, it's expensive for what you get.

Most people, when they think about cheap, gravitate toward the imported, shoddy, badly made RAID solutions, like those from Directron. Resist the temptation.

Cheap also means, in many cases, Firewire, USB or eSATA. Resist this temptation as well. eSATA sounds fast, at 3Gb/s. But, consider that 3Gb/s is shared across 4-8 drives and each drive is theoretically capable of 3Gb/s. So, you're dividing the high-speed 3Gb/s by 8 drives. In a RAID, you want all drives cranking away at their best speed, without bandwidth limitations.

Other, cheap RAID solutions involve software RAID. This is bad, slow and devours your operating system at higher bandwidth use. Ignore software RAID if you want the highest quality, speed and reliability.

And, don't even think about a LaCie or other "closed system" where you can't add, remove or replace your own drives. If something breaks, you have a brick on your hands... until the RMA happens, assuming you're still under warranty.

What you want in a RAID:

Faster than eSATA (fiber, seriously)
8+ Removable, reliable, FAST drives
Dedicated RAID hardware or firmware
A decent RAID chassis

The Actual RAID

Your setup could be more or less organized, it's really up to you.

PROAvio makes a great RAID chassis. It has removable drives, high-speed interfaces, just about everything you'd want. The problem with it is cost. When you purchase the system as a complete unit, it can run upwards of $4,000 -- even without the cables or the right card.

And, if you want something both Windows and Mac compatible, it's tough to find the right bundle. Worse, all the bundles are labeled MS, ML and several other options.

I'm not sure why there's such a high premium on drives and RAID cards, but the reality is, if you can install four screws onto a simple tray and you can plugin a PCIe card and connect cables, then browse to a web interface; you have the skill to setup your own RAID. There are even detailed instructions.

So, you'll need:

8x fast hard drives (I'm a fan of the quiet, fast Western Digital WD5000AAKS)
The PROAvio EditBox 8ML
2x Infiniband cables
1x great PCI-Express RAID card (the HighPoint RocketRAID 2322 is a winner)

The RocketRaid 2322 is both fast (two InfiniBand to MiniSAS cables and PCI-Express) and easy to administer and setup. It uses a standard web browser with no additional application hardware to install. It does, unfortunately, require a driver on both Mac OS X (I use 10.5.4) as well as all versions of Windows.

Still, the web interface is simple and functional and allows e-mail warnings for any errors. The RAID can be RAID 0,1,5 or variations like RAID 10. I use RAID 0, because I want speed, but the RAID 5 configuration is still very, very fast.

Overall Cost

Finding the right place to order parts, or at least a reliable and inexpensive place, is tough. I've done the homework and have found the following to be the best in price plus reliability for all the parts needed:

The Edit Box 8ML is the hardest part to find, without drives attached. DV Warehouse has it for around $550 US, but I ordered through TechnoAvenue, even though they no longer stock the item. The DV Warehouse version is black, for some reason, which might be cool to some people.

InfiniBand is sometimes called MultiLane number 8470 and MiniSAS is number 8088.

You'll be buying the following items, from the following links:

DV Warehouse Edit Box 8ML
B&H Photo Video has the 8MS enclosure for about $650 (which should work, but I haven't tested it).
( if you have ad-blocking enabled, you may not see the next three links )

The breakdown of cost looks like this:

$550 - Edit Box 8ML

$ 75 - 2x InfiniBand to MiniSAS cables

$275 - RocketRaid 2322

$520 - 8x Wester Digital 500GB Drives

------

$1,420  (without tax or shipping)

This leaves plenty of room for upgrading hard drives to 750GB (about $130) or even 1TB (about $200). However, the smaller 500GB drives, because of their size, allow more spindles and the RAID will be faster than a 750GB or 1TB drive.

There's always a size / speed trade-off and I'll leave that to you to decide.

Setup

Follow the directions. Install the drivers for your system, install the RocketRaid 2322 Card. The biggest pain is that hard drives come in wrap and you have to pull each one from a box, unwrap it, then put the drive on rails and insert the drive into the Edit Box.

The RocketRaid 2322 PCI-Express Card

Once everything's installed, reboot and open a browser.

Installation is easy, until you get to the non-obvious web interface.

The web interface requires a login and password, which took me forever to find.

Browse to: https://localhost:7402

PASSWORD: hpt (case matters)

Once you're logged in, the interface is pretty simple. You have to create an array, then format that array with the operating system.

Once done, you'll see something like this:

There's even temperature monitoring:

Speed

I've seen various speed benchmarks for many different RAID systems. The CalDigit HDPro boasts the fastest RAID at 325MB/s write and 450MB/s read:

Empty, our RAID really screams:

Even at 60% full, our "Under 2K RAID" is faster (Oct 7; this test has been updated with only a few applications running, rather than a loaded system in the previous test. The spikes are smaller, by the way);

Our RAID at 50% full;

And, for the purpose of comparison, I've included a benchmark of a SOFTWARE RAID 0 setup on the Mac Pro, which includes 4 of the exact same drives as the RAID configuration above (as opposed to 8 drives). Each drive is attached to its own eSATA II connection and is mounted inside the computer (also known as the Mac Pro Poor-Man's RAID).

Notice the significant speed difference with four drives and software RAID.

And, just so you don't think I'm AJA biased, here's a benchmark using the BlackMagic Intensity card and the BlackMagic Speed Test for the latest version (2.1.1):

CalDigit Responds

Jared Picune, from CalDigit, responded with a nice e-mail about this article. He maintains that the RocketRaid 2322 is a software RAID solution. He suggests that's the reason I'm seeing "spikes" in the AJA tests. In the spirit of being fair, I've included his e-mail and tests below.

From his e-mail:

Just read your blog post Super Fast 4TB RAID for Under $2k. Great read, and very informative.

http://lifezero.typepad.com/blog/2008/10/super-fast-4tb-raid-for-under-2k.html

Since I deal with this technology on a daily basis I wanted to point a few things out that you might find interesting.

The High Point 2322 card is a software RAID card. It does not have an independent CPU or cache. This mean that it requires valuable system resources to operate. Also it does not include an environmental monitor. Our CalDigit RAID Card is 100% hardware RAID, and we are very proud of that.

Also I just wanted to note that our speed tests are not doctored. We use proprietary technology called ASTT (active sustained transfer technology) that was developed by our engineers to make sure throughput stays consistent. You can see in this test below the actual difference in read and write speeds. This test was done by one of our distributors in Denmark.

He sent us these results.

You'll note the speed is impressive, given the RAID 5 option. I'd love to see a comparison of the CalDigit and the RocketRaid 2322 I'm using. Hopefully, I'll be able to test this myself rather than rely on independent claims. If I'm wrong, I'm willing to admit it.

Conclusion

The RAID is fast, cheap and is a pleasure to build and use. I've color corrected feature films (at 2k), worked on That Media Show (1080p), edited RED footage and done a lot more. The RAID just keeps going and going and doesn't seem to slow down or have any real issues.

In RAID 5 configuration, expect 300 MB/s writes and 400 MB/s reads on average, ignoring the manufacturers claims (I've tested the system in RAID 5).

If you're still reading, I encourage comments, suggestions or complaints. Just leave a comment or e-mail me.

If I've left anything out of this already long article, please let me know.