Constant Thinking

I am a Mobile Sensor Network, Collecting Big Data

constant — Mon, 23 Jan 2012 15:40:48 +0000

Don’t worry, this is not a desperate attempt at SEO for my blog (although I do appreciate your likes, Tweets, RSS subscriptions and other ways you help me reach a wider audience), nor is this my entry into the latest contest of IT BS Bingo.

It just occurred to me yesterday that Big Data is everywhere. Even during your weekend jogging run.

Collecting Fitness Data, Step by Step, Heartbeat by Heartbeat, on Your Phone

For Christmas, I bought myself a Wahoo Fitness Key* and its matching ANT+ heart rate monitor (HRM)*. The key plugs into your iPhone and provides connectivity to the ANT+ wireless sensor protocol. The HRM is another dongle that straps around your chest and electrically registers every heart beat, then transmits the data to the Wahoo key. If you have an iPhone 4S, you can do without the key and just buy a Bluetooth HRM like the Wahoo BlueHR, because iPhone 4 supports Bluetooth 4.0 which includes a low power version of the protocol that supports sensor collection devices such as HRMs that run off of a coin cell.

So iPhone + Wahoo + HRM = Wireless Sensor Network. And if your idea of a network involves more than two participants, Wahoo also sells an ANT+ pedometer* to measure your stepping frequency along with heart beat data as well.

(Android users: I'm sure you'll find a similar solution for yourselves as well. I just happen to prefer quality over popularity.)

Running 2.0

Thanks to modern gadgetry, apps like iSmoothRun on my phone can now tell me how I'm doing while I’m running, including time, distance (thanks to GPS, which is another sensor), pace, cadence (using the phone’s accelerometer or a wireless pedometer*) and heart rate. I can also set up a target running profile (like “No more than 70% of max. heart rate so I can stay in the aerobic zone, please.”) and my phone will duck the music and tell me to slow down whenever I go beyond target heart rate.

Pretty cool.

Social Network Running

But we live in the age of web 2.0 so there's obviously more to do if you want to maintain your running geek-cred: The iPhone also collects all data (position, heart-beats, and steps) over time and at the end of the run, it will not only present me with my running statistics, possibly spiced up with current weather data etc., it will also offer to upload the data to one of the emerging fitness social networks, such as RunKeeper.com.

Sites like Runkeeper take the data and create web maps with my running path, complete with nice graphs that I can dive into for analyzing my own running behavior including altitude, pace, heart rate, cadence etc. They also collect other data such as weight and body fat percentage (yes, using a Withings Scale* for example, you can track weight/bodyfat data too, even data from a sleep tracking system* can be collected!) and show you your running (or fat loss) progress over time.

And thanks to social network goodness, you can run with friends over the network and compare statistics even if you’re not physically running at the same time. Or the same place.

And this is where Big Data comes into play, but what is it and how does it work?

The Advent of Big Data

The first time I heard about big data was during an internal workshop about the Sun Cloud in 2009 (you know, the old Sun habit of being way before our time). While we contemplated the implications of cloud computing for enterprises, someone mentioned that this would be nothing compared to the implications of Big Data. Back then, Big Data was reserved to web giants like Google and Yahoo! and the occasional large research institute such as CERN.

Big Data is the art of handling (surprise!) large amounts of data. “Large” can be anywhere starting at a dozen of Terabytes or a couple of Petabytes or any large number that no-one in their right mind would place into a single database on a single server.

Big Data has been made popular by innovations from web companies like Google, Yahoo, Facebook or Twitter, who pioneered new ways of handling huge amounts of data.

Today, Big Data is about to cross the chasm* from the domain of a few innovators and early adopters to the early majority, as businesses start to realize its value.

The Four V’s of Big Data

Big Data is typically associated with four V’s:

Volume, meaning lots and lots of data from sensors, devices, social networks, the web, retail offices, mobile fleets, etc.,
Variety, meaning there’s no predefined structure in the data that one can rely on: Unstructured data. This is the main differentiator against classic data ware-housing, which is strictly structured.
Value, meaning that somewhere within that data, there is some valuable information to extract, though most of the pieces of data individually may seem valueless.
Velocity, meaning quick turnaround cycles, quick, almost real-time processing and also short innovation cycles. Fail fast, fail often is the mantra, until you hit data gold.

RunKeeper and Big Data

Let’s come back to our running example: RunKeeper is a Big Data company because it collects GPS, heart rate, cadence and other data from its millions of users. Assuming that only half of their 6 million users actually use the service for real, and that they run once a week and assuming a data size of 50 KByte per run (including GPS positions), we get 7.8 TByte of data per year. This is not a lot by Big Data standards, and it is quite structured, but when you combine this data with Tweets, Facebook status updates, other exercise data and nutrition/sleep data (RunKeeper does all of the above), then data volume easily increases to more than 10 TB per year, which is quite a lot to wade through.

And if you start counting records, the complexity is overwhelming: Each GPS sample is about 100 Bytes, which means that RunKeeper’s 10TB per year translates into roughly 100 Billion records to correlate, analyze and create meaning from.

What meaning?

The Meaning of Big Data

And that is the goal of Big Data: To create meaning out of billions of records that seem so innocent, if looked at individually. In the RunKeeper example, they create graphs of your running history and help you analyze and optimize your fitness either for free or as a paid, “pro” service. And thanks to their Health Graph API, an eco-system of other applications and companies emerges who slice and dice RunKeeper’s data in other creative ways, trying to create valuable (and monetizable) meaning out of it. Example: World-Rank.in collects data from RunKeeper and Twitter, then ranks runners into its own top 30 lists.

Other companies use Big Data to identify patterns in their customer’s behavior, find threats or opportunities to act upon, or simply alert hospitals that a new flu epidemic is about to hit them.

How Big Data Works

Most Big Data use cases work around the same pattern:

Aquire: Data is collected. Speed and scalability is critical here, not necessarily high availability. It’s ok to be offline for a few minutes, even hours, or to lose the occasional data record, but it’s important to catch as much as possible. The Hadoop framework and file system and/or NoSQL databases are key tools at this step.
Organize: To make large amounts of data manageable, a divide and conquer approach is taken: Data is mapped to some interesting nomenclature/metric/attribute (For example: is this a positive of a negative tweet? What company is this tweet about? Was that run a new record or a below-average result?), then the data is reduced into a more condense form (“Number of negative tweets that mention our company”, “Fastest 10k runs per country”, etc.). These two steps are the key in the MapReduce framework and can be used repeatedly and creatively to compose a new, more valuable data source, like: “Top 10 keywords associated with negative feelings towards your company.”, or “Top runners per age category and country”, or “Best training improvement over the last year”, or even crazy stuff like “Fastest music to run with”.
Analyze: With that kind of data and processing at your fingertips, new kinds of insight are possible that can be analyzed and acted upon. Which flights are delayed and carry unhappy social media celebrities so you better take good care of them? How about sponsoring your top runner in a certain category and create a celebrity out of her/him? Of course, valuable analysis can also be monetized. Perhaps some research institute or some sports company is interested in getting access to all that heart beat, speed and nutrition data (anonymized of course)?

Oracle and Big Data

Don’t worry, this commercial break will be brief, but interesting:

Oracle’s big strengths of course are in handling commercial data warehouses and analyzing business information data, as well as building Engineered Systems that remove the pain of setting up an IT shop while optimizing the usefulness you get from your systems.

Big Data’s strength lies in its innovation to handle and organize unstructured, large data sets, through the Hadoop filesystem, the MapReduce framework, the R statistical language and other emerging technologies. But analyzing data after these steps is still in it infancy.

By combining the worlds of Big Data, Data Warehousing and Business Intelligence, running on Engineered Systems, Oracle can offer unique value to businesses who want to leverage Big Data for their benefit, without going through the trial/error/research of running their own Big Data development operations.

Learn more from Oracle’s Big Data White Paper, it’s really good, and check out Oracle's Big Data home page.

Building your own Sensor Driven Big Data Collection Network

As you can see, Big Data is fun and healthy. Here are some gadgets* to get your own Sensor Network based Big Data collection infrastructure set up that feeds into RunKeeper and other Big Data collecting social networks for your analytical pleasure:

Big Data and You

What are your favorite Big Data examples? Do you see Big Data being used in your company? Have you played with collecting, organizing and analyzing Big Data yourself? Leave a comment and share!

Finally, here's a video that shows the beauty of collecting, organizing, analyzing and visualizing of Big Data:

And if you want to see my own small chunks of running data, feel free to join my Street Team on RunKeeper.

Disclaimer: Neither me nor Oracle are affiliated with RunKeeper (Not that I know of). I just think it’s a cool service.

*Disclosure: Some product links in this article are affiliate links. If you buy through them, I’ll get a small kickback to help with hosting costs for this blog at no extra charge to you.

Engineered Systems and Enterprise Architecture (or: How to Sell Dog Food Online)

constant — Mon, 16 Jan 2012 16:21:45 +0000

One of the first things that customers and sales teams realize when dealing with Engineered Systems is: They fundamentally change the IT architecture of a business.

Change is good, it means progress. But change is sometimes seen as a bad thing: Change comes with fear.

The truth is that Engineered Systems really empower IT architects to add value to their business, application and data architectures, without worrying about the technology architecture.

To understand this, we need to dig a bit deeper into Enterprise Architecture, specifically the TOGAF flavor of it.

One of my first tasks as a member of my new team was to get TOGAF certified. TOGAF is “The Open Group Architecture Framework” which is a fancy name for a giant document full of definitions, models, and methods for creating IT architecture out of a given business capability need.

Let’s look at a simple example: Suppose you’re the owner of a big dog food business empire. Your core competency is to know all about creating the best dog food, knowing your customer’s dog feeding preferences and habits, and having the best suppliers of dog food ingredients lined up from partners to help you excel at selling dog food.

And now you want to take your business to the next level and start selling dog food online. Here’s the typical progression of steps, assuming your company has successfully introduced TOGAF as your method of doing Enterprise Architecture, according to the TOGAF Architecture Development Method (ADM):

TOGAF ADM for Dummies

Architecture Vision: Your boss told you to come up with an architecture for adding an online shop to your dog food business. The first step then is to create an architecture vision: How does it look like to sell dog food online? Who are the stakeholders for such a capability and what are their concerns?
Business Architecture: Your enterprise architects sit down through numerous meetings with your business people, identifying a business architecture for selling dog food online. This is a bunch of diagrams, catalogs and matrices describing exactly how your online shop capability ties into the rest of your dog food business: What’s your online dog food business model? What are the use cases you want to support through your new capability? What existing processes does the new capability tie into? What new processes need to be defined to add the online shop capability?
The output of this stage uses business building blocks: Business processes and capabilities, modeled at a very high level, and technology-agnostic, specific to your enterprise.
Information Systems Architecture: Your enterprise architects translate the business architecture created in step 2 into an information system representation, which is composed of two architectures, a data architecture and an application architecture:
1. The Data Architecture describes what kind of data you need to take care of for your online shop capability. Customer data, product data, payment data, shipping data, dog pictures, food pictures, dog food ingredient lists, all of it. It also describes where the data comes from, where it flows to, who needs what data when and so on. We see two types of data here: Structured data (what you'd place into a database) and unstructured data (i.e.: flat files).
2. The Application Architecture describes what applications you need to handle your data: Web portals, CRM applications, supply chain management applications, shipping and tracking applications, order management applications, payment processing applications, dog food personalization and mixing order processing applications etc.
  More diagrams, more catalogs and more matrices are created to describe this stage of building your dog food online shopping empire. Notice that there are no particular technologies and no products considered at this stage either.
The building blocks at this stage: Applications, database schemas, flat file specs. All particular to your enterprise.
Technology Architecture: The next step is to translate the information systems architecture into a technology architecture. Now we’re getting somewhere: This is where our enterprise architecture heroes get to describe what systems are needed where, how to map applications and data to those systems, what networks to use, which servers are connected how at what locations, etc.
And here comes the cut: The building blocks are now generic! A database server is a database server, no matter what it stores, be it dog food recipes or car configurations. An app server is an app server, no matter what app runs on top of it, same goes for a file server, a firewall or a router. All components of Technology Architecture are generic, there's hardly anything specific about selling dog food online here.
Opportunities and Solutions: This is the earliest stage at which particular technologies (such as the Oracle database) come into play and where specific instances of a technology ("Our main CRM system running on Oracle on SPARC SuperCluster.") are mentioned.
Migration Planning: Here's where the hand-over from the enterprise architects to the project managers occurs: Implementation and migration plans are forged, projects are created and prioritized, resources are allocated, hands become dirty.
Implementation Governance: This stage ensures synchronization between Enterprise Architecture and its implementation, as carried out by you project managers, through running reviews and monitoring progress.
Architecture Change Management: Finally the project is done, the dog food online shop goes live and hopefully the first orders come in. Parties are thrown and people get drunk. The next day, lessons are learned and captured, and fed into the next ADM cycle, starting again with an architecture vision. Perhaps the beta users identified a few bugs or RFEs, or the product managers have new ideas for what else the online dog food store could sell, but that would require a new capability to be added to the architecture: The cycle starts again.

The Limits of Creating Business Specific Value

What does this have to do with Oracle's Engineered Systems? The answer is that it's all about who creates unique value at what level of the architecture and who doesn't:

Steps A-C are highly specific to the enterprise: This is where enterprises truly innovate: The first online shop exclusively for dogs! Online dog food customization! Geo-aware local dog food sourcing! The value at these stages is unique to the particular enterprise, or at least the vertical business it operates in.
But step D (Technology Architecture) is different: All building blocks here are generic, and there's hardly any value for a typical enterprise to create here that differentiates themselves from other enterprises. How is a database cluster developed for a dog food database different in design than a database cluster developed for a car configuration database? Or why would a file server for hosting dog pictures be significantly different from a file server for hosting book covers?
All other steps starting from 4 follow the same pattern: Building blocks, best practices and implementation details are the same, there's nothing specific to the particular business that they support. They all need to be secure, performant, reliable and more or less highly available. All business use the same trucks to deliver their goods, only very specialized business may occasionally need a special truck, but they're quite a minority.

And yet, businesses spend enormous amounts of resources for coming up with their own custom made database cluster design, their own custom made app server farm design, their own custom made file server infrastructure design, their own custom made web infrastructure design and so on.

This is where the rise of Engineered Systems takes place.

The Engineered System: TOGAF Stage D in a Box

Oracle's Engineered Systems (you know, Exadata, Exalogic, SPARC SuperCluster, ZFS Storage Appliance and so on) are nothing more than the TOGAF stage D of the ADM in a few, easy to plan, buy, install, manage boxes, without the usual headaches that used to occur when planning Technology Architecture.

Want a new database for your dog food recipes? Just ask the DBA to allocate one for you out of an Exadata. Want to host the entire chain of app servers and portal that forms the application layer of your dog food shopping empire? Push a button on your PaaS portal to instantiate a virtual assembly on one of your Exalogic boxes. Want something more powerful? Perhaps with more transactional crypto-oomph? Ask you friendly PaaS portal for a slice off of your shiny new SPARC SuperCluster. Wanna analyze heaps of personalized dog food recipes and correlate them with mailman assault reports, dog flu epidemic data and weather forecasts? That's what the big data appliance and Exalytics are for.

The point here is: No business should spend time, money and resources for creating yet another personalized flavor of a RAC cluster, an app server tier or a web portal infrastructure. This is what IT companies like Oracle can do better than anyone else: Design Technology Architecture.

Instead, use your energy and resources to create real business value. What's your online dog food architecture?

P.S.: Please forgive my dog food analogy. Here's the Guy who planted it into my mind:

TOGAF ADM picture taken from Wikipedia, used under NASA public domain policy.
Dog picture by digital_image_fan on Flickr, used under Creative Commons license.

Iron Sky Update: A Behind the Scenes Video Interview

constant — Tue, 27 Dec 2011 19:58:13 +0000

If you've been following my blog or the Systemhelden.com podcast for some time, then you probably know that I'm a big fan of and small investor in the movie Iron Sky, a crowd-funded science-fiction comedy about the Nazis who went to the dark side of the moon in 1945 and come back to conquer Earth in 2018.

A few weeks ago, I had the privilege of attending a pre-screening of the movie's beta version in Helsinki, Finland. What can I say? It exceeded my high expectations!

The script is ingeniously written, just the right mix of humor, action, character and political satire. All actors are brilliant: Götz Otto, Christopher Kirby, Tilo Prückner and Udo Kier gave the movie a unique blend of coolness, style, creepiness and comedy while Julia Dietze and Peta Sergeant stood out in excellently played character roles. Special kudos to Stephanie Paul for being the most hilarious US President in movie history!

Director Timo Vuorensola found just the right balance between comedy, action and the occasional emotional moment. The whole production has a very high quality feeling, despite the limited (by hollywood standards) budget of about 7.5 Million Euros.

Video Interview

After the pre-screening, I had the opportunity to interview Tero Kaukomaa, producer of Iron Sky and Pekka Ollula, Social Media Manager. We talked about the audience's first impressions, a day in the life of a movie producer, and some cool new side projects such as the Iron Sky Comic Book or the brand new Iron Sky board game.

Here’s the full interview (sorry for the slightly shaky camera):

Wanna see more videos? Check out an interview with the fathers of Iron Sky from last year. Or go watch some cool Iron Sky teasers.

Participate in Iron Sky!

Iron Sky is scheduled for world-wide release in April 4, 2012. If you want to participate, you still can:

There’s some room for a few more fan investors: Check out the Iron Sky Finance page for prerequisites to becoming an investor in Iron Sky and become part of the biggest crowd-funding effort in movie history!
For only EUR 1 (or more), you can take a sneak peek at the full production process of the first 5 minutes of the movie, from script to storyboard to shooting to special FX and final result.
If you want to see your company logo on Times Square within the movie, there may still be an Iron Sky Times Square sponsoring opportunity left!
Every T-Shirt, board game or other fan item bought in the Iron Sky Store helps promote the movie while contributing to the production budget.
Spread the word! Visit the “Demand Iron Sky” page and help sell the movie to distributors, like the Iron Sky page on Facebook, follow Iron Sky on Twitter or just tell all your friends. With more than 80,000 fans on Facebook, the movie is already a social media phenomenon, let's reach 100k fans on Facebook by the time it reaches the theaters!

Special Constant Thinking Iron Sky T-Shirt Contest!

Win an official Iron Sky T-Shirt by promoting this interview! Just tweet (or retweet) a link to this article along with the #ironsky hashtag and enter a draw to win a t-shirt!

On January 15th, I’ll randomly draw the lucky winner out of all participants and send her/him an official Iron Sky t-shirt.

Disclaimers, Disclosures and Stuff

Standard disclaimers apply for the T-Shirt contest, this is no official promo, no legal hassles please, this is just for fun among friends.

Disclosure: I'm a proud investor in Iron Sky which means that I'm biased, have a commercial interest in this project and that I want you to see the movie and tell all your friends about it.

Update (2012-01-01): A stupid bug killed the intro part of the article, restored from a draft. Also corrected the praise for Julia and Peta since my poor English didn't do them enough justice before :).

Update (2012-01-06): Corrected the year to 2018. In 2038, earth will face a different problem. Thanks to Max who pointed that out in the comments!

Update (2012-01-20): The results are in!

Today I determined the lucky winner of the Twitter #ironsky retweet contest. 15 different people retweeted this article before or on January 15th, here's the list:

00 @TAKIM0T0
01 @sascha242
02 @MThorr
03 @zarkon
04 @Okona
05 @bayoda
06 @sunian314
07 @ironskyfilm
08 @optionstrict
09 @BrodyK91
10 @shift_plusone
11 @jenksho
12 @digisubculture
13 @apfelgriebs72

Special mention to @deltabps who retweeted this article, but was a bit late. So 14 people altogether qualified for the t-shirt draw.

Here's the first winner, determined by a roll of Unix dice:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
4

Congratulations @Okona!

Now, I personally know Okona who is a regular guest at the HELDENfunk podcast, so I didn't feel good about keeping the t-shirt in the family so to speak, so I decided to give away a second t-shirt! Here's the second roll of dice:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
12

Congratulations @digisubculture!

Damn, again someone I know personally from my university days in Clausthal :).

Ok. I'm rolling again until I get someone I don't know. Here's lucky winner #3:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
13

Congratulations @apfelgriebs72!

That's it, congratulations to you all! I'll send you some DMs now and ask for your shipping addresses so I can send you your T-Shirts.

But wait! Since Twitter and I aren't infallible, there's a slight chance that someone who re(tweeted) a link to this article with the #ironsky hashtag before or on January 15th didn't come to my attention and so wasn't included in the above list by pure bad luck. If that is you, then send me a link to your Tweet using the contact form of this site to prove it and I'll send you another T-Shirt!

(Just to avoid getting broke due to very unlikely events, this offer is only valid for the first three people to qualify for this, and only for the remainder of January 2012.)

The Rise of Engineered Systems

constant — Wed, 07 Dec 2011 14:11:38 +0000

I changed into a new role at Oracle: I now work for the EMEA Engineered Systems Architecture Team (ESAT). We support Oracle’s EMEA Engineered Systems business by engaging with customers, enabling our field organization with trainings and through evangelization.

You can call me biased towards Engineered Systems now, but that would be like accusing a Mac fanboy of suffering from the Stockholm Syndrome, when it’s actually the other way round.

The other side of the “biased” medal really is that I have a choice of where I want to work, and one of the reasons I changed from my cozy SPARC/Solaris Technology camp to the Engineered Systems crowd is: I believe the world of IT is changing.

Let me explain.

Back to the Future

Once upon a time, when computers were new, they came as a whole: You bought yourself a “computer” and it came with everything you needed to get started: Hardware to compute with, storage to store data with and software to work with. Those times culminated in the rise of mainframes that offered everything a business wanted to do with computers as one whole, integrated, single package.

Even through the 70s, when mainframes got replaced by minicomputers, they still came as an integrated whole: Storage, Computing hardware, OS and development tools with libraries, what we would today call “Middleware”. (For a fascinating travel in time back to the minicomputer era, read “The Soul of a New Machine”*).

But then the integrated approach was seen as too proprietary and too limiting, which gave rise to a new wave of systems in the 80s: Open Systems. Now, vendors offered the pieces (You know: Storage, Servers, OS, Tools and Middleware, Applications, Networking and so on), while businesses (or system integrators, for a fee) put the pieces together into a customized whole.

30 years later, we’re still living in the 80s, when it comes to running an IT project.

A Typical IT Project

So how does a typical IT project work?

Someone has an idea for a new business capability.
“We need an app store for our iToaster company!”
Bright people create an IT architecture to support that new capability.
“Here's our blueprint for a private cloud based, full-featured, interactive, augmented reality, HTLM5, AJAX iToaster App Store system, complete with CRM integration, web portal, BI engine and ToastHabits™ loyalty card integration!”
A round of RFPs goes out to procure the components for the new system: Storage, Servers, Networking, Virtualization, OS, Database, Middleware and some Applications.
Some of these are standardized, but we still end up with more than a handful of them, multiplied by our company’s list of n>1 favorite vendors to play against each other.
RFP responses are evaluated.
Technologies are studied, claims verified, proof-of-concepts are run, pilots are conducted and so on. Lots of traveling, playing, arguing.
Finally, the components of the IT system are decided on.
The result tends to look like a camel (which allegedly is an animal that was created by a committee): The number of vendors involved is only slightly smaller than the number of components across the storage-to-application stack, thanks to elaborate purchasing rules, standardized buying practices, old relationships or just habits of the people involved in the process.
Then the pieces come in and the IT department gets to play.
Storage is set up and plugged to servers through some network, servers are set up and provisioned with virtualization and operating systems, databases are set up and connected to middleware which is being rolled out to support the actual applications that matter and so on.
In the process of putting the components together, problems emerge.
The storage doesn't like to play well with the server, which turns out to be a firmware issue after weeks of finger-pointing between vendors and countless hours on their hotlines. The OS needs a specific patch and some settings in /etc/system to support the right version of the database so it can optimally talk to the middleware, but there’s still a network latency or routing issue which drags down performance because the NAS device tends to overload specific segments more than the middleware communication to the database so different networking paths need to be set up, and so on.
Finally, after months of delay the new IT system is running.
Kinda. A bright guy came up with a creative solution so the project doesn’t have to be delayed again. The “solution” also compromises security a little bit, but that’s a small price to pay for cutting weeks or months off time-to-market and reach the huge potential of hundreds of thousands of new customers (and their credit cards) and turn them into revenue through the magic of the new system. And we’ll fix the security bugs later. When we have the time.
Only, it is not running very well, yet.
Performance problems arise, once more than 10,000 customers try to place orders at the same time, which happens on the very first day because the new service was announced with a huge marketing campaign. On the management floors, biological end products hit rotational dynamics which only worsens the problem because now even more people try to use the new system just to see what the problem is. The press isn’t helping either. The test department guy says they shouldn’t have brought the system online without running any further tests while the business people point at charts with "opportunity costs" due to the already mind-blowing project delays.
I’ll spare you the details of the first successful hacking attempt.

True, this is exaggerated for reasons of entertainment, but things like this happen all the time in current, “modern” IT shops. I've seen them all in two decades of working in IT and I’m sure you’ll recognize a few of them, too.

The point is:
Building IT systems is complicated, time-consuming, error-prone, unpredictable, resource-intensive, expensive and risky.

Or, more shortly:
The way we build IT today is broken.

The Return of the Engineered System

Wouldn’t it be nice if you could just buy a full system that comes with everything you need to get started on your new business idea? A system where storage, networking, servers, virtualization, operating system, database and middleware, applications are engineered together, tested together, certified together, delivered together, managed and patched together?

Unpack, set up and get started in days, not months. Roll out your application without the risk of it blowing up due to unforeseen incompatibilities. Profit from better performance because all components are pre-integrated, so they can use a faster networking technology than the generic standard. Less suffering from bugs because most of them have been fixed already at the factory level rather than at the customer site. Less support suffering because the guy at the other end of the phone has access to exactly the same system and can therefore understand your issue much better. Better security because everything has been designed together with security in mind and in a pre-packaged way with controlled interfaces in and out of the system as a whole. Shorter time-to-market because steps 3-10 of the above process are reduced into one step that only takes a few days: Unpack and configure your Engineered System.

That's what Oracle’s Engineered Systems are about.

It sounds similar to the early times of computing, where the 10-step process from above was just two:

Buy computer.
Get to work.

And that’s the goal of Engineered Systems: Get rid of the maddening complexity of building IT systems and let the people sort it out that do this actually for a living, 100% of the time and with much better expertise.

And the goal for you is to get to work on what you are best qualified for. Building App Stores. Selling stuff through e-commerce. Collecting big data and finding sense therein. Running social networks. Figuring out where the economy is going. Making sure that parcels go from A to B on time during the christmas season. And so on.

But isn't this going back to monolithic, proprietary and closed systems?

Of course it is not. The thing is that we're getting the best of both worlds:
Simple IT deployment, and open standards with open interfaces across all components of the stack.

There is no lock-in: All software that comes with Engineered Systems is identical to the software that you can buy independently as a component of your 10-step home-brew IT deployment process. If you don't like the Engineered System, just build your own IT and move your stuff onto it, while keeping the same OS, middleware or database software.

Yes, there’s the occasional tool that is specific to the Engineered System it ships with, but these are extra components for convenience, like the customized dashboard on your Volkswagen that doesn't make sense if you duct-tape it onto your Toyota. Which brings us to:

Analogies Galore!

Engineered Systems are often compared to cars. And the analogy is striking: When was the last time you ran RFPs for tires, engines, transmission, seats, dashboards, chassis and bodywork?

Indeed, when cars were invented, each car was individually built to the driver’s specs. Until Henry Ford came along, and the rest is history:

Any customer can have an Engineered System painted any color that he wants as long as it is gray.

A friend of mine came up with a much better analogy: Beer!
Ever tried brewing your own beer? Sure, a liter of home-brew is cheaper than a liter of your favorite beer brand, but good luck going through the elaborate process of setting up, cleaning, roasting, fermenting, brewing and bottling your own beer. One word: Hygiene.

The result can taste great, but often enough, something (usually bacteria related) goes wrong and you end up with a beverage that tastes, well, home-brewn. And then you're sitting on 20 liters of something that your friends politely called “interesting” and that you’re too proud of to dump into the sewer for a few bottles of good, proper German beer.

Which is not far from “interesting” IT solutions we see a lot, their creators being too proud to replace them with something that actually works.

Engineered Systems are the Bavarian beers of IT.

You can find a great analogy much closer to traditional IT: Just walk into any coffee shop, better yet, visit any conference and look at the number of MacBooks vs. laptop PCs, iPhones, vs. Android phones, and of course iPads. Particularly their users and the amount of “work” vs. “fix” they do on their devices.

The iPad is the ultimate consumer Engineered System: Hardware and software and applications merged into a single, optimized system that lets you “just work”.

Change is Coming

Expect professional IT to change the way the car industry changed, or the way Apple changed the consumer industry: Companies are starting to build IT out of large, pre-engineered building blocks, vs. re-inventing wheels at the component level.

When was the last time you pondered whether to use Hitachi, Seagate or Conner drives for your storage project? Do you really care about the chip sitting inside your networking card or storage controller? Are you still busy testing, sourcing and integrating hardware and software components while your colleagues are treating the system as a component?

The focus of professional IT has now been elevated: From building systems out of components to building clouds out of pre-engineered systems. That’s where IT is going in 2012.

So What Do They Look Like?

Oh, I almost forgot. This is not a product pitch, so I’ll be brief:

Exadata is the Engineered System for running Databases.
Exalogic is the Engineered System for running Middleware and Applications, including SAP. (And one of my first duties as part of my group was to write a white paper about “Running SAP NetWeaver on the Oracle Exalogic Elastic Cloud”)
The Oracle Database Appliance is the easiest to use, all-in-one database appliance for small and medium enterprises or individual departments.
The Big Data Appliance makes it easy to pre-process Big Data for your database, NoSQL included.
Exalytics is the Business Intelligence System that leverages 1 TB of RAM to slice and dice huge amounts of data at lightning speed.
SPARC SuperCluster is the most flexible Engineered System: It comes with both Exadata and Exalogic technologies to boost database and middleware performance within the same box, and it can run any other application, too. A full Enterprise IT stack in a single box. I can’t wait to see SAP certified on this baby!
The ZFS Storage Appliances are part of Oracle’s Engineered Systems, too. The one-stop shop for a wide range of storage requirements. It's also used within Exalogic and SPARC SuperCluster as a common repository for compute nodes.

Your Take on Engineered Systems

What do you think? Do you see how Engineered Systems change IT? Or do you still see value in building stuff from scratch, over and over again?

What are your experiences with Engineered Systems? What aspects of Engineered Systems would you like to see covered in this blog?

Drop me a few lines of comment below, or send me mail. Let’s discuss!

*: Amazon affiliate link. Buy cool books at no extra charge, and I’ll get a small kickback. We both win!

Car picture by Flickr user flamesworddragon, used under Creative Commons License.

Solaris 11 Launch Blog Carnival Roundup

constant — Sat, 12 Nov 2011 16:31:27 +0000

Solaris 11 is here!

And together with the official launch activities, a lot of Oracle and non-Oracle bloggers contributed helpful and informative blog articles to help your datacenter go to eleven.

Here are some notable blog postings, sorted by category for your Solaris 11 blog-reading pleasure:

Getting Started/Overview

A lot of people speculated that the official launch of Solaris 11 would be on 11/11 (whatever way you want to turn it), but it actually happened two days earlier. Larry Wake himself offers 11 Reasons Why Oracle Solaris 11 11/11 Isn't Being Released on 11/11/11.

Then, Larry goes on with a summary: Oracle Solaris 11: The First Cloud OS gives you a short and sweet rundown of what the major new features of Solaris 11 are.

Jeff Victor has his own list of What's New in Oracle Solaris 11.

A popular Solaris 11 meme is to write a blog post about 11 favourite features: Jim Laurent's 11 Reasons to Love Solaris 11, Darren Moffat's 11 Favourite Solaris 11 Features, Mike Gerdt's 11 of My Favourite Things! are just three examples of "11 Favourite Things..." type blog posts, I'm sure many more will follow...

More official overview content for Solaris 11 is available from the Oracle Tech Network Solaris 11 Portal. Also, check out Rick Ramsey's blog post Solaris 11 Resources for System Administrators on the OTN Blog and his secret 5 Commands That Make Solaris Administration Easier post from the OTN Garage.

(Automatic) Installation and the Image Packaging System (IPS)

The brand new Image Packaging System (IPS) and the Automatic Installer (IPS), together with numerous other install/packaging/boot/patching features are among the most significant improvements in Solaris 11.

But before installing, you may wonder whether Solaris 11 will support your particular set of hardware devices. Again, the OTN Garage comes to the rescue with Rick Ramsey's post How to Find Out Which Devices Are Supported By Solaris 11. Included is a useful guide to all the first steps to get your Solaris 11 system up and running.

Tim Foster had a whole handful of blog posts lined up for the launch, teaching you everything you need to know about IPS but didn't dare to ask: The IPS System Repository, IPS Self-assembly - Part 1: Overlays and Part 2: Multiple Packages Delivering Configuration. Watch out for more IPS posts from Tim!

If installing packages or upgrading your system from the net makes you uneasy, then you're not alone: Jim Laurent will tech you how Building a Solaris 11 Repository Without Network Connection will make your life easier.

Many of you have already peeked into the future by installing Solaris 11 Express. If you're now wondering whether you can upgrade or whether a fresh install is necessary, then check out Alan Hargreaves's post Upgrading Solaris 11 Express b151a with support to Solaris 11. The trick is in upgrading your pkg(1M) first.

Networking

One of the first things to do after installing Solaris 11 (or any operating system for that matter), is to set it up for networking. Solaris 11 comes with the brand new "Network Auto-Magic" feature which can figure out everything by itself. For those cases where you want to exercise a little more control, Solaris 11 left a few people scratching their heads. Fortunately, Tschokko wrote up this cool blog post: Solaris 11 manual IPv4 & IPv6 configuration right after the launch ceremony. Thanks, Tschokko!

And Milek points out a long awaited networking feature in Solaris 11 called Solaris 11 - hostmodel, which I know for a fact that many customers have looked forward to: How to "bind" a Solaris 11 system to a specific gateway for specific IP address it is using.

Steffen Weiberle teaches us how to tune the Solaris 11 networking stack the proper way: ipadm(1M). No more fiddling with ndd(1M)! Check out his tutorial on Solaris 11 Network Tunables.

And if you want to get even deeper into the networking stack, there's nothing better than DTrace. Alan Maguire teaches you in: DTracing TCP Congestion Control how to probe deeply into the Solaris 11 TCP/IP stack, the TCP congestion control part in particular. Don't miss his other DTrace and TCP related blog posts!

DTrace

And there we are: DTrace, the king of all observability tools. Long time DTrace veteran and co-author of The DTrace book*, Brendan Gregg blogged about Solaris 11 DTrace syscall provider changes. BTW, after you install Solaris 11, check out the DTrace toolkit which is installed by default in /usr/dtrace/DTT. It is chock full of handy DTrace scripts, many of which contributed by Brendan himself!

Security

Another big theme in Solaris 11, and one that is crucial for the success of any operating system in the Cloud is Security. Here are some notable posts in this category:

Darren Moffat starts by showing us how to completely get rid of root: Completely Disabling Root Logins on Solaris 11. With no root user, there's one major entry point less to worry about.

But that's only the start. In Immutable Zones on Encrypted ZFS, Darren shows us how to double the security of your services: First by locking them into the new Immutable Zones feature, then by encrypting their data using the new ZFS encryption feature.

And if you're still missing sudo from your Linux days, Darren again has a solution: Password (PAM) caching for Solaris su - "a la sudo".

If you're wondering how much compute power all this encryption will cost you, you're in luck: The Solaris X86 AESNI OpenSSL Engine will make sure you'll use your Intel's embedded crypto support to its fullest. And if you own a brand new SPARC T4 machine you're even luckier: It comes with its own SPARC T4 OpenSSL Engine. Dan Anderson's posts show how there really is now excuse not to encrypt any more...

Developers

Solaris 11 has a lot to offer to developers as well. Ali Bahrami has a series of blog posts that cover diverse developer topics: elffile: ELF Specific File Identification Utility, Using Stub Objects and The Stub Proto: Not Just For Stub Objects Anymore to name a few.

BTW, if you're a developer and want to shape the future of Solaris 11, then Vijay Tatkar has a hint for you: Oracle (Sun Systems Group) is hiring!

Desktop and Graphics

Yes, Solaris 11 is a 100% server OS, but it can also offer a decent desktop environment, especially if you are a developer. Alan Coopersmith starts by discussing S11 X11: ye olde window system in today's new operating system, then Calum Benson shows us around What's new on the Solaris 11 Desktop. Even accessibility is a first-class citizen in the Solaris 11 user interface. Peter Korn celebrates: Accessible Oracle Solaris 11 - released!

Performance

Gone are the days of "Slowaris", when Solaris was among the few OSes that "did the right thing" while others cut corners just to win benchmarks. Today, Solaris continues doing the right thing, and it delivers the right performance at the same time. Need proof? Check out Brian's BestPerf blog with continuous updates from the benchmarking lab, including Recent Benchmarks Using Oracle Solaris 11!

Send Me More Solaris 11 Launch Articles!

These are just a few of the more interesting blog articles that came out around the Solaris 11 launch, I'm sure there are many more!

Feel free to post a comment below if you find a particularly interesting blog post that hasn't been listed so far and share your enthusiasm for Solaris 11!

*Affiliate link: Buy cool stuff and support this blog at no extra cost. We both win!

Join the Solaris 11 Launch Party!

constant — Wed, 02 Nov 2011 19:36:06 +0000

In about a week, on November 9th, 2011, the long-awaited final version of Solaris 11 will be launched. If you happen to be near New York that day (and assuming there'll be no power outages), you're invited to join the official Solaris 11 launch party!

Solaris 11 has been in the making since 2005, when Solaris 10 was launched. In fact, every major Solaris release is just a fork of the ongoing Solaris development train, so the very first uber-pre-release of Solaris 11 was actually generated only weeks after Solaris 10 hit the shelves.

Since then, Solaris 11 (or: Project Nevada as it was called) has seen a lot of OS history: An open source adolescence called OpenSolaris, growing adoption and community work, a broad range of ground-braking new features, long overdue re-writes, brand new concepts, controversial discussions, a major acquisition, rules changed and rules kept, siblings and offsprings, lots of investments, entire companies built on top of its source code, generations of processors and hardware, lots of systems in production, the Cloud and what not.

And all that before it was even born. Quite an achievment, eh?

So when you finally hold that DVD in your hand... Nah, DVDs are last century. Let me try again.

So when you finally finish that download, here are a few things about Solaris 11 to try out and dig into:

Brand New Package Management, a Brand New Root File System and a Brand New Installer

This is perhaps the biggest change from Solaris 10 to Solaris 11: The way Solaris is installed in the first place. Over time, the old System V package manager and patching system has become quite a burden, so it just had to be re-written:

A modern, network-based, finely-granular package manager called IPS.
No more patching, just updates of existing packages.
No more entanglement between installation and configuration.
A new, powerful concept for root file systems: Boot environments.
Everything is neatly integrated with ZFS.
A new, network-centric installation mechanism that works hand in hand with IPS and ZFS: Automated Installer (AI).
Backward-compatibility with existing System-V packages.

I know some people had their issues with the new ways things are handled with IPS, boot environments and AI, but I'm sure they'll eventually get over it: Every change is difficult at first, but when the power of the new concepts start shining through, we'll all be glad we did it. And over time, and with more experience from the installed base, I'm sure that the Solaris team will iron out any missing bits.

Learn more about this by visiting the Installation and Packaging Community on OpenSolaris.org.

A Brand New Way to Support Existing Installations

Whenever a new release of Solaris came out in the past, it always took a long time until it would finally be adopted in production. The reason was simple: Migration. Granted, there's a 100% binary compatibility guarantee with Solaris that (technically, not legally) dates back to the ancient times of Solaris 2.6 and that only one competitor can remotely compare itself with (can you name it?), but still there was quite some work involved in moving a Solaris release to its successor, plus formal certification, of course.

Now, with Solaris 11, we'll get Solaris 10 Containers: Automatically pack your existing Solaris 10 installation into a Solaris 10 Container, then import into Solaris 11 as a branded zone, done. "Brand" new, indeed. Sorry for the pun.

No need for re-scripting stuff. No need to re-certify. No need to re-install or re-configure. Take the existing stuff and run unchanged, with a new OS under the covers. Migration heaven.

Yes, there may be the occasional detail to observe, but for the utter majority of existing installations, I expect the migration process to be smooth and slick: Containerfy, then import, done. A huge step in terms of ease-of-migration, and one that I expect will greatly help adoption of Solaris 11.

More about Virtualization in Solaris 11 is available in this OTN feature.

Brand New Networking

Solaris 10 changed the way Applications and Services are run and introduced virtualization at just the right level and with no friction: Zones.

Now, Solaris 11 will do the same to the network: Thanks to Project Crossbow, Solaris 11 users can enjoy limitless network virtualization including virtual NICs, virtual switches and routers, a built-in load-balancer and including full resource management.

Check out this introductory article with examples by Sunay Tripathi to get a better feel about Crossbow.

And if you're looking for 10x faster latency and 4x more bandwidth, then check out the new Infiniband stack that is included with Solaris 11. This will give your Exadata and Exalogic (and any other Infiniband based hardware) a nice, OS-level upgrade. Catch up with Ted Kim's Weblog if you want to learn more about InfiniBand in Solaris 11.

A whole new way of looking at networks.

Brand new Storage

Solaris 11 comes with a brand new version of ZFS, including two new major, much anticipated features: De-Duplication and Encryption, plus lots of new and improved stuff under the hood.

But there's a lot more to discover: The new COMSTAR framework will turn any Solaris 11 system into a first-class block-based storage engine citizen, be it on a SAN, an Infiniband or an Ethernet network: Create arbitrarily large volumes in seconds with ZVOLs, export them via FC, iSER or iSCSI, then enjoy the beauty of ZFS data integrity, snapshots, deduplication and storage replication with easy administration. Better than the big storage boys, but without the cost.

If you happen to be at the DOAG 2011 Conference in Nürnberg, Nov. 15th-17th, feel free to show up at my talk on "Neue ZFS-Funktionen in Solaris 11" on Tuesday, Nov. 15th, 15:00.

More DTrace

Solaris 10 brought us the ultimate X-Ray tool for the kernel and the rest of the OS: DTrace. more than 30,000 probes for your observation and analytical pleasure.

With Solaris 11, DTrace extends to the whole stack: With a total of 77982 probes (at least on my plain vanilla system, give or take a few dozen), you'll be able to observe your application throughout the whole stack: From kernel to OS to the web server, middleware (including Java and PHP) up to the browser.

Full stack observability, at run-time, in production, without any code change. What more can developers and administrators ask for?

If you want to refresh your DTrace know-how in time before Solaris 11 comes out, check out this postfrom the Observatory with a real-life use-case of DTrace, including an example on how to analyze a web application with MySQL DTrace probes.

Of course, the real authority in terms of DTrace literature is Brendan Gregg's and Jim Mauro's book DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD (Oracle Solaris Series)* which is highly recommended. If you want to get a preview, here's a series of cool videos about the DTrace book and its content. Their favorite DTrace scripts are included in every Solaris 11 copy in the /opt/DTT directory, and this is where you'll get much more!

More Security

Similarly to DTrace and ZFS, the new security concepts from Solaris 10 have been enhanced in Solaris 11. Most notably, root is no longer the all-powerful user (though you can still make it so if you want), but just a role. This introduces accountability to the way sysadmins handle root passwords: No more anonymity by "just becoming root for a while".

We already saw encryption being part of the ZFS overhaul and in addition, Solaris 11 includes more and better implementations of common and new encryption algorithms, plus TPM support and of course support for hardware encryption in new processors, both from Intel and from Oracle.

Finally, Security is now deeper ingrained in other sub-systems such as SMF, IPS, Kerberos, Networking and Trusted Extensions.

Here'a a short overview of some new security features in Solaris 11.

Tune In and Follow the Launch

I'm sure the Solaris people have a few more good news to share during the November 9 event, so make sure to tune in and register for the event, be it in-person or through the live webcast.

Your Take on Solaris 11

What are your expectations about Solaris 11? Which new feature do you expect to deliver the biggest benefit to you and your organization? What are your plans for Solaris 11? Drop me a line in the comments and get yourself heard!

Update: Added a reference to the DTrace book. How could I have forgotten? Sorry, was a late night post...

* Affiliate link: Buy cool stuff here and I'll get a small kickback at no extra cost for you. We both win!

Solaris 11 Available for Early Adopters

constant — Mon, 12 Sep 2011 07:42:00 +0000

Maybe I should write more frequently, though that would mean shorter, less elaborate articles. This is the first one of that kind. Let me know what you think!

Recently, the Oracle Solaris 11 Early Adopter Release became available on the Oracle Technology Network (BTW, can I have a date with that Java Developer, please?). Here's the gist:

Latest Update: This is an update to the Oracle Solaris 11 Express release from almost a year ago, and it's the first publicly available update (though you need to join the OTN program and accept the early adopter license).
Fully functional: This release is feature complete: All the features expected for Solaris 11 are available in this release. Now, Solaris 11 developers and users can start qualifying their software, hardware and environments against it.
New license: This release is covered by a special OTN Early Adopter License. Read it before installing.
Release Notes: They cover known issues, numerous ways of installation, including Auto Installer and USB stick, and a list of obsolete packages and new packages since Solaris 11 Express 2010.11.

Go download it now, try it out and let me know what you think!

Hat tip to Stefan for the nudging :).

ZFS: To Dedupe or not to Dedupe...

constant — Wed, 27 Jul 2011 14:26:11 +0000

...that is the question.

Ever since the introduction of deduplication into ZFS, users have been divided into two camps: One side enthusiastically adopted deduplication as a way to save storage space, while the other remained skeptical, pointing out that dedupe has a cost, and that it may not be always the best option.

Let's look a little deeper into the benefits of ZFS deduplication as well as the cost, because ultimately it boils down to running a cost/benefit analysis of ZFS deduplication. It's that simple.

ZFS Deduplication: What Value Do You Get?

ZFS dedupe will discard any data block that is identical to an already written block, while keeping a reference so that it can always reproduce the same block when read. You can read more about ZFS deduplication and how it works here.

Before you decide to use deduplication, it's good to know what value you'll get out of it. Here are a few options to figure out how much space you'll save as a result of using ZFS deduplication:

Test it with some real data. This is the most accurate and straightforward option: Set up a test pool, enable ZFS deduplication on it, then copy a representative amount of the data you are considering onto it. Then use zpool list and look at the DEDUP column for the deduplication ratio. The important thing here is to use a representative amount of data, so you get an accurate estimate of how much savings to expect.
Simulate it by applying the zdb -S command to an existing pool with the data you want to deduplicate. This option is less accurate than using real data with a real deduped zpool, but it can provide you with a ballpark estimate based on your existing data.
Guess it, based on the knowledge you have of your data. Not the best option, but sometimes, setting up a test pool is not feasible and simulating dedupe on existing data doesn't work because you simple don't have any data to analyze with. For example, if you plan to run a storage server for virtual machines: How many machines do you support? How often are they patched? How likely will people apply the same software/patches/data to your machines? How many GB of dedupe-able data is this likely to generate? Can you come up with a representative test case after all to make the guess less guessy?

In any case, you'll end up having an expected deduplication ratio for the data: For every GB of data you actually store, how many GBs of retrievable data will you get? This number can have any value: Some people see a value of 1.00 (no duplicates whatsoever), others see some moderate savings like 1.5 (store 2 GB, get one free), and some very lucky people can see as much as 20x, for example a virtualization storage server with a very repetitive usage profile.

Now take your total amount of storage and divide it by the dedup ratio, then subtract the result from your total amount of storage. That is your expected storage savings as a result of deduplication:

Total Storage - ( Total Storage / Expected Dedupe ratio ) = Expected Storage Savings

As a fictional example, let's assume that we're looking at a 10 TB storage pool to be used for storing virtual machine images in a virtual desktop scenario. In a quick test, we set up a 1 TB pool and copy some existing VM data to it, which yielded a dedup ratio of 2. This means that we only need about 5 TB of capacity to provide the 10 TB of data thanks to deduplication, hence we would save 5 TB of disk storage.

Let's assume that the average cost of 1 TB of disk (including controller, enterprise class drives, etc.) is at $1000 for the sake of simplicity: Then dedup would save us $5000 in this particular example.

So what do we need to spend in order to realize these cost savings?

ZFS Dedupe: The Cost

Saving space through deduplication doesn't come for free. There is a cost. In the case of ZFS, it's memory: ZFS keeps a dedup table in which it stores all the checksums of all the blocks that were written after deduplication was enabled. When writing new blocks, it uses this table to determine whether a block has been written yet, or not.

Over time, the table becomes larger and larger, and since every write operation has to use it, it should be kept in main memory to avoid unnecessary extra reads from disk. To be clear: ZFS can work perfectly well even if the table is not in memory. But the bigger the deduplication table grows, the slower write performance will become, as more and more writes trigger more and more extra reads for dedup table lookups.

How much memory does one need to keep the ZFS dedup table in memory, and hence your system happy?

According to the ZFS dedup FAQ, each entry in the dedup table costs about 320 Bytes of memory per block. To estimate the size of the dedup table, we need to know how many blocks ZFS will need to store our data. This question can be tricky: ZFS uses a variable block size between 512 bytes and 128K, depending on the size of the files it stores. So we can't really know in advance how many blocks ZFS will use for storing our data.

If we mainly store large files (think videos, photos, etc.), then the average block size will be closer to 128K, if we store small files (source code, emails, other data), we'll probably be closer to a few K. Here are some ways to find out for sure:

Option 1: Count Your Blocks With ZDB

The most accurate way to determine the number of blocks is to use the zdb -b <poolname> command, which will print out detailed block statistics. But beware: This command may take a long time to complete as it will scan all of the metadata blocks in your pool:

constant@walkuere:~$ zdb -b oracle                                              
 
Traversing all blocks to verify nothing leaked ...
 
	No leaks (block sum matches space maps exactly)
 
	bp count:          306575
	bp logical:    19590419968      avg:  63900
	bp physical:   17818332160      avg:  58120     compression:   1.10
	bp allocated:  17891765760      avg:  58360     compression:   1.09
	bp deduped:             0    ref>1:      0   deduplication:   1.00
	SPA allocated: 17891765760     used: 52.48%

The number to look for here is bp count which is the number of block pointers, hence the number of blocks in the pool.

In this example, the dedup table would take up 306575 blocks * 320 bytes, which yields around 100 MB. This file system has a size of 32GB, so we can assume that the dedup table can only grow to about 200 MB, since it's about 50% full now.

Option 2: Estimate Your Average Block Size, Then Divide

If you don't have your data in a ZFS pool already, you'll have to guess what your average block size is, then divide your expected storage capacity by the average block size to arrive at an estimated number of ZFS blocks. For most cases of mixed data like user home directories, etc., we can assume an average block size of 64K, which is in the middle between the minimum of 512 bytes and 128K. This works reasonably well for my own example:

constant@walkuere:~$ zpool list oracle                                          
NAME     SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
oracle  31.8G  16.7G  15.1G  52%  1.00x  ONLINE  -

we see that the average block size of my work pool is 16.7G divided by 306575 which yields about 54K. Your mileage will vary, especially if you use different kinds of data such as VM images, databases, pictures, etc.

Example: If a zpool stores 5 TB of data at an average block size of 64K, then 5TB divided by 64K yields 78125000 blocks. Multiplied by 320 Bytes, we get a dedup table size of 25 GB!

The Total RAM Cost of Deduplication

But knowing the size of your deduplication table is not enough: ZFS needs to store more than just the dedup table in memory, such as other metadata and of course cached block data. There's a limit to how much of the ZFS ARC cache can be allocated for metadata (and the dedup table falls under this category), and it is capped at 1/4 the size of the ARC.

In other words: Whatever your estimated dedup table size is, you'll need at least four times that many in RAM, if you want to keep all of your dedup table in RAM. Plus any extra RAM you want to devote to other metadata, such as block pointers and other data structures so ZFS doesn't have to figure out the path through the on-pool data structure for every block it wants to access.

RAM Rules of Thumb

If this is all too complicated for you, then let's try to find a few rules of thumb:

For every TB of pool data, you should expect 5 GB of dedup table data, assuming an average block size of 64K.
This means you should plan for at least 20GB of system RAM per TB of pool data, if you want to keep the dedup table in RAM, plus any extra memory for other metadata, plus an extra GB for the OS.

The Alternative: L2ARC

So far, we have assumed that you want to keep all of your dedup table in RAM at all times, for maximum ZFS performance. Given the potentially large amount of RAM that this can mean, it is worth exploring some alternatives.

Fortunately, ZFS allows the use of SSDs as a second level cache for its RAM-based ARC cache. Such SSDs are then called "L2ARC". If the RAM capacity of the system is not big enough to hold all of the data that ZFS would like to keep cached (including metadata and hence the dedup table), then it will spill over some of this data to the L2ARC device. This is a good alternative: When writing new data, it's still much faster to consult the SSD based L2ARC for determining if a block is a duplicate, than having to go to slow, rotating disks.

So, for deduplicated installations that are not performance-critical, using an SSD as an L2ARC instead of pumping up the RAM can be a good choice. And you can mix both approaches, too.

Adding up the Dedup Cost

Back to our example: Our 10 TB pool is expected to just utilize 5 TB which, at an average block size of 64K would need a dedup table that is approximately 25GB in size. Last time I checked, an enterprise-class SSD with 32GB was in the range of $400, while 25GB of Memory was in the range of $1000. So this is the range of what using deduplication will actually cost in terms of extra SSD and/or RAM needed.

Putting Together the Business Case

There you have it: For our fictitious example of a 10 TB storage pool for VDI with an expected dedup savings of 5 TB which translates into $5000 in disk space saved, we'd need to invest in $400 worth of SSD or better $4000 of RAM. That still leaves us with at least $1000 in net savings which means that in this case dedup is a (close) winner!

But if we assume the same amount of raw data (10 TB) but only a dedup savings factor of 1.1, then our equation would be different: We'd still save close to 1 GB of disk storage (ca. $1000) but we'd need to build up a dedup table that can manage 9 TB of data, which would be in the range of 45GB. That means about $600 in SSD capacity for storing the dedup table. For RAM, we'd need 4 times that amount (180GB), since only 1/4 of ZFS ARC RAM is available for metadata. That doesn't look very attractive to me.

So it really boils down to what you get (= amount of space saved) and what you need to spend in order to get that (= extra SSD and/or RAM for the DDT), depending on whether you want to sacrifice some performance (by going SSD only) or not (by adding enough RAM).

Now, you can do your own calculations on a case by case basis.

Finding the Break-Even Point

Given some standard cost for disk space, SSDs and RAM, you can calculate a break even point. That way, you can just ask: What's the expected dedup savings factor? Then you can instantly decide whether it's worth deduplicating or not.

At our fictitious values of $1000 per TB of disk space, $400 for a 32 GB SSD and $1000 for 24GB of memory, and assuming an average block size of 64K, we can derive two break-even dedup ratios depending on the performance requirements:

For applications where performance isn't critical, you can get away with no extra RAM for the DDT, but with some extra space for storing the DDT at least on an SSD in L2ARC. Each TB of pool capacity will cost 5GB of dedup table, no matter how much dedup will save. 5GB of dedup table will cost $62.5 when stored in a standard SSD ($400 for 32GB). Hence, for each TB of pool capacity, at least 62.5 GB need to be saved through dedup for the SSD to pay for itself (1000 GB cost $1000, 62.5 GB saved will save 62.5$, the price of having that dedup table stored in SSD). That translates into a minimum dedup factor of 1.0625 to warrant the extra SSD capacity needed.
For applications that are more performance-sensitive, you'll need the same amount of memory for the DDT per TB (5GB), but this time you want to store it fully in RAM. ZFS limits metadata use in RAM to 1/4 of total ARC size, so we need to make sure our system has at least 20GB of extra RAM per TB of stored data. That means each TB of deduped pool data will cost us approx. $834 in x86 memory for storing its dedup table, so the minimum dedup savings factor needs to be 1.834 here.

Rules of Thumb

These are all fictitious numbers, YMMV, but I think some good rules of thumb are:

If performance isn't critical, and if you expect to save more than 20% of storage capacity through deduplication, then go for it but add at least 5GB of L2ARC SSD capacity per TB of pool data to store the dedup table in.
If performance is important, wait until you expect at least a 2:1 reduction in storage space through deduplication, and add 30GB of RAM to your system for every TB of disk capacity to make sure that the dedup table is always in memory so optimal write performance is ensured..

Why the extra 10GB of RAM in the latter rule of thumb? You don't want to fill up ZFS' metadata RAM cache entirely with the dedup table. The other metadata you want to have quick access to are ZFS' pointer data strcutures so it knows where each data block is stored on disk etc. That can be estimated at 1% of total pool capacity which is 10GB per TB of pool data.

Conclusion

The decision to use ZFS deduplication or not is almost always a simple cost/benefit analysis. When using deduplication, one needs to plan for at least some extra L2ARC SSD requirements, or better some extra RAM for storing the dedup table in a manner that doesn't negatively impact write performance.

Especially RAM can become a decisive factor in deciding for or against deduplication, so usually a dedup savings factor of 2 is a necessary threshold for deduplication to become a real cost saver.

Your Take

So here are some fictitious numbers. Did you do your own dedup analysis? What were your results? What do you base your decision to use deduplication on? Place a comment below and share your own dedup cost/benefit analysis cases.

The Solaris Eco-System is Expanding

constant — Mon, 13 Jun 2011 19:58:21 +0000

More than a while ago, I wrote about the birth of Illumos, a project that aims at substituting the last non-open-source bits from the OpenSolaris kernel with replacements, in order to create a 100% open source Solaris kernel.

On May 20th, I had the opportunity to attend the Nexenta European User Conference 2011 in Amsterdam, where Solaris and storage enthusiasts from all over the world met to discuss their favorite technology: ZFS. Of course there was also a lot of talk about Illumos and related projects.

Now I've given a lot of Solaris presentations to customers, always highlighting the big, growing and powerful community behind the Solaris OS. But this conference added a new dimension to the Solaris Eco-System for me!

Prior to the conference, I had a peek at the Nexenta Partner Training, and I was quite impressed to see about 80 people listening to the gory details of ZFS technology, taught by Craig Morgan (@craigmorgan) and his colleagues, effectively becoming a new army of ZFS supporters.

Nexenta European User Conference

The conference itself was quite interesting (see the agenda). It started with a "Storage is growing like crazy" keynote by Nexenta's Evan Powell, complimented by Chris Evans' irresistible review of 55 years of storage evolution, ending in the firm belief that the future of storage is open, while asking for better storage virtualization (wasn't NFS invented to virtualize storage in the first place?).

The rest of the conference was a good mix of user experiences, cloud and virtualization discussions. Notable highlight: Phil Harman (@pgdh) and his hilarious "High Availability matters" talk (see a version from OpenStorageSummit here on YouTube).

As with many conferences, the stuff that is between the sessions is often the most interesting. And this meeting was no exception. While talking to attendees, it became clear to me that the Solaris eco-system has become a vibrant, self-sustaining and rapidly growing movement, beyond what had started as "OpenSolaris".

The ZFS Working Group

A lot of discussions revolved around the "ZFS Working Group", a group of developers from all ZFS variants that aim at defining and coordinating common interfaces and standards for the various development efforts around ZFS. Read the silent announcement here.

One of the first projects that the ZFS Working Group is working on is to define a more granular approach to zpool and zfs on-disk formats that aims at facilitating interoperability between different versions of ZFS, even if they come from different vendors.

Illumos

Another emerging player in the Solaris eco-system is the Illumos Project, a project that aims at creating a Solaris kernel and basic operating system core that is 100% open source. "Wasn't OpenSolaris open source already?" you ask. Yes, mostly. There were still a few parts that nobody got around to port/review/get clearance for, like Internationalization, some drivers or the NFS locking daemon. Illumos is about replacing these last bits with 100% open source equivalents in order to complete the OpenSolaris promise.

Garrett D'Amore (@gedamore), founder of Illumos was so kind to give me an interview about the history of and his plans for Illumos. It's published in the latest episode of the HELDENfunk podcast from SYSTEMHELDEN.com. The podcast is in German, but the interview is in English, just skip to 03:29 if you don't understand German.

OpenIndiana

Illumos is just a kernel and the associated utilities, not a full operating system or distribution. Other players like Nexenta, Joyent and Delphix pick up Illumos and incorporate it into their own products and distributions.

But OpenIndiana is different: It is a complete, generic, fully open source Solaris distribution that picks up where OpenSolaris has left, providing users with a complete OS for servers and desktops.

In a second interview, Alasdair Lumsden (@alasdairific), founder of OpenIndiana and UK hosting company Everycity, and Andrzej Szeszo (@aszeszo) talk about OpenIndiana, how it started and their current plans and challenges for the future. Again, it's part of the current German HELDENfunk episode but you can skip right to 17:16 and listen to the interview part which is in English.

Conclusion

Solaris is growing. Whether it's driven by Oracle's Solaris 11 or the extended Solaris community, there's no question that the Solaris eco-system continues to evolve, innovate and deliver cool stuff on a regular basis and for a wide range of companies, users and developers.

What's your opinion on the Solaris eco-system? Have you checked out Solaris 11, Illumos and OpenIndiana lately? Share your experience in the comments!

Disclosure and Disclaimer Many thanks go to Nexenta for inviting me to visit the Nexenta European User Conference and paying for my travel costs. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle, my employer.

How to Render the Mandelbrot Set in the Joyent Cloud with Node.js

constant — Sun, 15 May 2011 20:36:48 +0000

First of all: Apologies for not posting for a long time. The reason? I was having too much fun with node.js and the Joyent Cloud :).

What started as a small experimental hack turned quickly into an exciting new pet project involving the good old Mandelbrot Set, as a web service, running in the Joyent cloud, programmed in node.js.

But first things first: Let's take a look at node.js as a language and programming model, at the Joyent Cloud and how it relates to Solaris and finally some details on how the picture you see was rendered inside the Joyent Cloud, including an interactive Mandelbrot Set explorer you can play with now, written as a web app.

I think the first time I heard about node.js was from Bryan Cantrill in this post where he explained the story about Joyent and a node.js programming contest and how they added DTrace probes to Node to enable some analytics for their contestants' machines.

What Is Node.js?

node.js (or short: Node) is both a programming language and a runtime system. The programming language is V8 JavaScript with a few extensions, the more important part is the runtime system and the programming model: Node is server-side, event-driven and non-blocking. Which makes it ideal for writing highly concurrent, scalable web applications, without caring about processes, locks, threads, etc.

Check out the nodejs.org homepage for details and please do spend an entertaining hour watching Ryan's introduction to node.js. Don't worry, I'll wait.

Node is lower level than most web application frameworks: HTTP and sockets are first-class citizens in Node, and "Hello world" in Node is actually a web server. Other frameworks like Java, PHP etc. typically sit on top of an existing web server (though it is possible, but not common, to implement web servers in these languages as well). Node's intimacy with networking protocols make it interesting from a minimalistic point of view: Develop the web server with the web app in sync, and only implement the pieces that are really necessary, leaving out the rest.

You can also implement pure socket based protocols with node.js, which makes Node interesting to game developers, for chat servers, messaging in general, etc.

There's a rapidly growing library of Node extensions called Modules, including a package manager called npm that makes installation of modules very easy. Most low level web server programming needs can be addressed easily with the connect framework, higher level web apps will enjoy working with the Express framework.

Getting started with node.js is easy: Download the software, build it using the easy to follow instructions and start playing with it. The node.js wiki is a great resource to start digging in, and Mozilla.org's JavaScript Reference will help you polish up your JavaScript skills in a breeze.

Why Node is Cool

Here's why I think node.js is so interesting:

It's event driven. This means that a node.js program is not just a sequence of stuff to do. It gives you the ability to attach pieces of code to events: Someone connects to your server? Fire up this code. A task has finished? Go run this other code. Got a new packet coming in? Go do these three things with it. Timer's up? Here's what to do. And so on. Just like JavaScript in a browser is based on browser events (you know: onload, onclick etc.), node.js is based on server events.
This allows for symmetric web app development: If you have some JavaScript skills that you developed by writing AJAX apps on the client, you'll be familiar with node.js on the server very easily. If you want to develop a new web app, you can now do both the server and the browser sides at once, with the same set of skills, libraries, models, etc. Millions of JavaScript hackers are becoming server side hackers overnight, thanks to Node.
node.js removes the complexity of dealing with concurrency, locking etc. by ignoring these concepts: If everything is driven by little pieces of code that run independently, triggered by events, there's a complete disconnect between them. This is a good thing: No need to watch out for the other guys, no need to do any locking or other complicated stuff. Just write small pieces of code that do their job little by little and let the database, the OS or the node.js runtime deal with the messy concurrency stuff. Eventually, there will be a way to fire off parallel threads, but it will be batch-oriented and message based, through the web workers API.
It's easy. I managed to put together a Mandelbrot Set rendering web service in about an hour. From scratch. And I'm not a real programmer.
It's new. There's a lot of innovation around Node. It's a completely new world, which makes it fun and exciting, too.

Joyent and node.js

As you may know, Joyent is a startup company that delivers cloud software and cloud hosting services. Joyent is probably the most innovative company in terms of cloud computing right now, and they're big supporters of Solaris. In fact, their own "SmartOS" operating system for the cloud is based on OpenSolaris. Cool people including sysadmin god Ben Rockwood, DTrace gods Bryan Cantrill and Brendan Gregg (Buy his DTrace book now*, it's really good!), and of course node.js inventor Ryan Dahl work there.

IMHO, Joyent is where the future of Clouds is made.

Anyway, the Joyent Software Stack uses node.js at the top to deliver a powerful, asynchronous, open source PaaS platform.

Getting Started with Your Own Joyent SmartMachine for Node.js for Free

Of course, you can play with node.js on your laptop, but it's more fun when you use it inside a real cloud.
Thankfully, this is really easy (and free) with Joyent's SmartMachine for node.js service:

Go to http://no.de/ (yay, a German domain!) and sign up for a Joyent account there.
Read the Getting Started With a Node SmartMachine guide, including how to apply for a provisioning code so you can provision your first Joyent SmartMachine.
Upload your SSH key, set up your Git repository and off you go!

The beauty of a Joyent SmartMachine is that it's actually a Solaris Container: You get all the flexibility and power of being king on a Solaris system, with all of the security, reliability and efficiency of a Solaris system, but without the hassle or without annoying your other buddies running on the same physical machine.

The node.js service running inside a SmartMachine is nicely integrated with Solaris: It's configured as an SMF service and the pre-supplied commands node-service-start, node-service-stop, node-service-log etc. make great use of it. Even better: As soon as you push your Git repository into your Joyent SmartMachine, the node.js service is automatically restarted through SMF to immediately update your service. And if anything fails, it will automatically revert back to the previous version. How cool is that?

But the real killer here is the integration of node.js, DTrace and the Joyent Cloud Analytics feature. As your node.js app gathers more and more users, you can use Cloud Analytics to diagnose, monitor and better understand the dynamics of your service. Watch the video with Bryan Cantrill and Brendan Gregg on Cloud Analytics to get an overview of how it works.

Joyent gives you a simple package manager with a large amount of pre-configured packages called pkgin. npm, the Node Package Manager is pre-installed, but some extra packages are recommended for a smoother install of some npm modules.

Introducing the Constant Thinking Mandelbrot Rendering Web Service

So I started playing with Node in a Joyent SmartMachine. And it was fun! After a few weeks of trying stuff out, optimizing and tinkering I can now present you with my first real web app, the Constant Thinking Mandelbrot Set Web Service!

Most of you already know that the Mandelbrot set is a beautiful mathematical object that can be generated using a simple formula over and over again over the coordinates of a complex number plane. I think I wrote my first Mandelbrot set program at the age of 12 or so, when these things became fashionable in the home computer crowd.

Now here's how Mandelbrot 2.0 works:

On this page, click anywhere within the image to set a new center for the image computation.
Click on "Zoom" to zoom into the set, and on "Zoom out" to back out. The math says there's infinite detail hidden in the set. The limits of floating-point calculations will put an end to this but as soon as I start seeing precision errors, I'll work on a more precise implementation.
The amount of detail is limited by the maximum number of iterations. If after a few rounds of zooming into the image the black areas start to look boring, hit the "more details" button and you should see more cool stuff. This simply doubles the maximum number of iterations for the computation.
There's some anti-aliasing, too: If you select one of the anti-aliasing options, the image will be generated at a larger resolution (3x3 or 5x5), then shrinked into the target resolution using one of two filtering methods (simple Gaussian or the more sophisticated Mitchell-Netravali filter). In theory, this should make the image smoother. In practice, the infinite detail of the set counters that a bit. Still worth a try.
Finally, there's some timing displayed and you have a choice of different optimization algorithms. The fastest algorithm is chosen by default, but if stuff looks funky, send me the parameters that were displayed and how you got there, and go with one of the lower optimization levels to get a decent picture.

Feel free to play with the Mandelbrot service and place a comment below. Or send me a message if you find a bug or have any other suggestions.

Experiences, Gotchas and Further Plans

My first impression of node.js: Boy is this fast! Rendering a 512x512 pixel Mandelbrot set picture takes less than a second, including color mapping and generating the PNG image from the raw pixel data, even without any optimization. This is possible because under the covers, node.js uses Google's V8 JavaScript engine which has some pretty advanced optimization techniques.

Like any new platform, it takes some tinkering until you figure out things and find your ways around it, but thanks to the Node Wiki, the Mozilla JavaScript Reference and the Joyent Node Wiki, most answers are just a few clicks away.

Since Node is so new, there are a couple of minor annoyances: There's no easy way to do parallelism (it's not encouraged yet, until the web workers API is built in), the amount of available modules is rapidly changing or incomplete, and sometimes the best documentation is to read some example source code. All of these are just symptoms of a good thing: A new, innovative and rapidly evolving world.

Bonus: Source Code and Some Mandelbrot Set Geekiness

Most of the time I spent writing this web service was actually spent on optimizing the algorithm for computing the Mandelbrot set. For the standard image you see in the beginning of the Mandelbrot web service page, the speedup is more than 2x, so it was worth the effort. Here's a crash course in Mandelbrot set optimization:

The two main big bulbs that the Mandelbrot set is made of are actually well understood (see the Wikipedia article on the Mandelbrot set). This means one can use a simple formula to detemine whether a point is inside one of the two bulbs, then decide whether to skip or continue the standard, lengthy iteration formula.
Exploiting connectedness: The Mandelbrot set is connected. This means that there are no "islands". This also means that if all points of the circumference of any area are inside the set (= black), the whole area is. To exploit this, an algorithm can subdivide the set into rectangular blocks, walk the blocks' edges and try to find blocks with circumferences inside the set through binary (or any other convenient) subdivision. For such blocks, further iterations can be skipped and the whole block can be marked as inside the set.
Major areas of the Mandelbrot set are well-known: This means that we don't need to test for bulbs if we're rendering the piece that's above them, so we can skip the check. We can also skip the check for bulb 1 if we're left or right of it, etc. This optimization therefore computes the intersection of different areas with predefined optimization settings and the areas that are actually to be rendered, then uses the right optimizations for the right pieces of the image.
Exploiting symmetry: The Mandelbrot set is symmetrical along the real axis. So if the area to be rendered crosses that axis, up to half of the image can be recycled through mirroring.

The highest optimization setting (which is also the default) makes use of all of the above and should be pretty good for most cases. If you have any more cool and easy to implement optimization ideas, let me know!

That said, the only thing left for me to do in this article is to give you the source :). Like any good Node citizen, I've set up an account on Github and uploaded the current version of the source:

Check out the Mandelbrot set generator in node.js source here.

Of course the usual disclaimers apply: Use at your own risk, don't sue me, forgive any code uglinesses, let me know if you find bugs, send me your suggestions and generally have fun with it.

Conclusion and Further Plans

Many thanks to the fine people at Joyent for bringing node.js to the world and giving it such a cool place to live! Thanks to their great support people who helped me find the magic incantation for getting a provisioning code. I'm looking forward to seeing Node grow and continue playing with it :).

Actually, the fun has just started: Right now, the Mandelbrot set rendering web service ist pretty basic, and it doesn't use many of the cool Node features. Over time, I'd like to add some more:

Caching should be easy to do in Node, and I already have an idea on how to exploit it.
Parallelism is the next big thing: Put together a stack of batch jobs that make up the total rendering process, then fire them at some worker nodes, then collect the pieces together. Should be fun to implement while learning something new.
Integration with C: While Node is pretty fast already, it would be fun to implement the basic number-crunching in C so I can learn how the integration of C works with Node.
While we're at it, I'd like to increase rendering precision through implementing the computations in a purely integer manner. You know, floating-point is evil.
The user interface on the web browser side could use some brushups, perhaps with some jQuery magic.
Colormaps should be configurable.
And so on. You know, some projects are never finished...

Your Turn

Did you try out Node already? What are your experiences? Any cool libraries, hacks, tips to share? Do you like the Mandelbrot set web service, or do you think this is too 80's? Let me know in the comments!

* Affiliate link: You buy great stuff that's good for you, I get a small commission to help finance this blog, we both win!

Checking Out the Amplidata Storage Cloud Technology

constant — Mon, 28 Mar 2011 20:14:23 +0000

Last week during WorldHostingDays, I had the opportunity to visit Tom (@tomme), a former colleague of mine who came with Q-Layer to Sun, then to Oracle.
Today, he works for a new Belgian startup called Amplidata, a company that specializes in building storage clouds. He introduced me to Wim, their CEO and we discussed their optimized object storage technology, some parallels to ZFS and the newest trends in cloud computing storage.
Amplidata is a spin-off of Incubaid, a technology incubator which is responsible for the success of two good old Sun friends: Innotek (VirtualBox) and Q-Layer (The company that powered the Sun Cloud).

The Amplidata idea is simple: Take the data and spread it across many storage nodes in a clever form, so that it is available in a scaleable, reliable and power-efficient way. Cloud storage in a box, err in multiple boxes, if you will.

The Technology

The way this works is slightly more complex, and Amplidata's secret sauce is composed of three technologies:

BitSpread: This is a clever new algorithm that splits up a given block of data into multiple blocks that can be spread across multiple nodes.
The trick here is that it can reconstruct the original data out of a sufficient number of blocks, which is smaller than the total number of blocks.
Sounds weird? Let's try an example: Take a block of data and pass it through the BitSpread codec and it will yield, say, 10 blocks. The codec is set so that 7 blocks are sufficient to reconstruct the data. Now, you can take any subset of 7 blocks out of the 10 and the codec will be able to reconstruct the original data. Kinda like RAID, but with no distinction between data and parity blocks, and with an arbitrary number of total blocks and sufficient blocks, which is configurable.
The more obvious advantage is that now any 3 blocks (assuming we stay with 10 total blocks and 7 sufficient ones) can be missing and we can still reconstruct the data out of the remaining 7. Since blocks are spread across nodes, it's a bit like a distributed RAID algorithm.
The other advantage is that when you retrieve data, you can ask all ten nodes to send you their blocks, and you can deliver the original block as soon as the first 7 have arrived. This can help performance if some of your nodes are more loaded than others.
Or, you can choose to bring down up to 30% of your storage nodes to save power and still be able to deliver your data. Neat.
There's also a security aspect to this: When data is spread across many nodes, an attacker can't learn much if he only steals one node's worth of data (or even a rack, if the installation is big enough).
The BitSpread parameters are configurable: For each file, the user can specify the number of resulting blocks to spread across and how many of them are to be sufficient for reconstruction.
Finally, and contrary to RAID-5 and similar algorithms, the BitSpread codec is designed to have a low overhead, so it doesn't get in the way of performance.
BitLog: Many of the concepts behind BitLog reminded me of ZFS although they're implemented quite differently. It's a combination of smart caches (similar to ZFS' ZIL and L2ARC) to store written and read data locally in order to help performance, combined with a special data structure that enables snapshots, clones, I/O optimization and other more advanced data services. Again, BitLog is spread across nodes and helps the whole system to provide transactional robustness and higher level features.
BitDynamics: This is the management software that manages the Amplidata storage nodes. It manages all the housekeeping between the nodes such as data integrity, scrubbing, node management, self-healing and garbage collection.

The three technologies form the foundation of the Amplidata storage system, which adds block device, iSCSI, Fiber Channel and Amazon S3 interfaces on top. This enables customers to build storage clouds out of Amplidata's nodes and deliver services similar to both traditional SANs and cloud computing vendors.

Two Business Models

One of the main challenges for a startup is figuring out how to bring their technology to market. Amplidata has two models:

The Amplidata company offers their technology as an OEM solution to partners who want to build storage clouds.

The AmpliStor product is an optimized object storage solution for petabyte-scale, unstructured data applications.

AmpliStor comes with its own hardware, using high capacity storage enclosures, low-power drives, pre-installed software and some extra temperature and power management intelligence to make the solution more energy-efficient. For example, AmpliStor can switch off individual drives or full nodes when not used, to save energy.

Uses and Applications

Amplidata and AmpliStor are different from your good old NFS server, therefore it's important to know where they fit and where not. The distributed nature of the system is a powerful concept, but it comes at the price of higher latency compared to direct attach or traditional NAS storage (though I haven't seen benchmarks yet). Here are some typical applications:

Web 2.0 Style Media: Petabytes of photos, movies, audio etc. are created every day by social networks and their users. This is a natural fit for the Amplidata storage model, because the data is largely unstructured (here's my data and a hash, when I show you the hash, gimme the data back) and the distributed nature makes it easy to provide reliable storage at web-style performance for affordable cost.
Media Archives: There's a growing space in archiving that requires less power-use than disk storage, but is more demanding than tape storage. For example, digital media archives (think YouTube) want to archive large amounts of data, but they're not always happy with the latency of tape (think YouTube, but with minutes of waiting before the video starts). Still, they want to save power and cooling, so this would be an attractive solution here.
Cloud Storage: Of course, if you're a service provider or a public cloud, you'll likely want to provide something similar to Amazon's S3 service. This solution comes with a pre-built S3-compatible API, too. Cloud storage in boxes.

The Verdict

Amplidata has some very innovative technology. It reminds me of a distributed kind of ZFS, though some parts are quite different. Today, they're using standard Linux nodes as the base of their storage servers. It would be cool to try their solution with ZFS and Solaris on their nodes for some extra robustness, performance, and data integrity (through ZFS), reliability (through Solaris FMA and SMF) and observability (DTrace). Maybe something for the future?

Anyway, if you're looking for an optimized object storage solution in the cloud, check them out.

More Information

The Amplidata web site is evolving quickly, so check often. There's a small page on technology but you can request a technology paper if you want more details.

Full disclosure and disclaimer: Tom gave me a free ticket to visit WorldHostingDays in Rust, where Amplidata had a booth, though with no strings attached. Thanks, Tom! Also remember that I'm an Oracle employee, but this is solely my own, personal and independent opinion and not my employer's.

How to Set Up a ZFS Root Pool Mirror in Oracle Solaris 11 Express

constant — Mon, 14 Mar 2011 16:45:27 +0000

One of the first things to do when setting up a new system is to mirror your boot disk. This protects you against system disk failures: If one of the two mirrored boot disks fails, the system can continue running from the other disk without downtime. You can even boot from the surviving mirror half and continue using the system normally, until you have replaced the failed half.

At the currently low prices for boot drive sized disks, this is a no-brainer for increasing your system's availability, even for a home server system.

Unfortunately, the steps to complete until you're running off a mirrored ZFS root pool are not yet a no-brainer. While there is a piece of documentation entitled How to Configure a Mirrored Root Pool, it only covers how to add a second disk to your root pool, it does not cover how to prepare and layout a fresh disk so Solaris will accept it as a bootable second half of an rpool mirror.

Which, for historic reasons, is slightly more complicated than just saying zpool attach.

Over the weekend, I sat down and played a bit with the current Oracle Solaris 11 Express release in VirtualBox and tested, re-tested and investigated all currently necessary steps to get your root pool mirrored, including some common issues and variations.

Here's a complete, step-by-step guide with background information on how to mirror your ZFS root pool:

The Basic Plan

After a standard install of Oracle Solaris 11 Express, we'll have our system disk configured as a ZFS root pool called rpool. The rpool disk is set up as an fdisk partition with some SMI partitions (= "slices") on top. The fdisk part is for compatibility with other OSes, the SMI slicing is done in order to reserve some room on the physical disk for the boot blocks and GRUB.

This is different from a regular ZFS data disk which would normally use EFI (not fdisk) labels and no further partitioning.

So here's the basic plan on how to turn a fresh disk into an rpool mirror:

First, we'll figure out what disks we have on the system and what their device names are.
For x86 systems, we need to create an fdisk(1M) primary partition on the second disk (the one to be mirrored), so it uses the same partitioning technology as the original rpool disk and so it can support booting.
We'll then set up an SMI label on the second disk with the same layout (slices) than on the first disk. This includes the slice that reserves space for boot blocks and GRUB and the slice that will contain the rpool's second mirror half.
Now that we have sliced and diced the second disk the same way as the original rpool disk, we can use zpool attach to let ZFS mirror its data on top of it.
For x86 systems, and after ZFS has finished resilvering the rpool mirror, we'll install GRUB onto the second disk so it becomes equivalent to the original rpool disk in terms of bootability.
As a final step, you need to configure your BIOS or OpenBoot PROM to try booting from the second disk if the first one is not available.

You see, the official documentation only covers step 4 above, and let's you guess about the other steps. Here's the full sequence of stuff to do to create a proper mirror in more detail:

1. Figure Out Your System's Disks

Hard drives in Solaris show up in the /dev/rdsk directory as raw devices and the same drives with the same names show up again in /dev/dsk. The former are used to perform raw partitioning and low-level options, while the latter is the standard way to access disks from a day-to-day point of view such as setting up ZFS pools.

Here's a typical device name: c0t0d0s0. The naming convention is simple: Controller 0, SCSI target 0, disk 0 and Solaris slice 0.

Of course, the digits may vary and even become multi-digits in larger systems such as c12t18d5s8, but the convention is always the same.

PATA systems omit the t0 part, because PATA doesn't support "targets" like SCSI or SATA does. This will give you devices like: c0d0s0.

Sometimes, when dealing with DOS partitions, you'll see a p0 part instead of the (Solaris specific) s0 piece. This simply refers to DOS partition 0 (or any other DOS primary partition).

So before we do anything, we need to figure out what disks we are dealing with, what device names they have and if they're used somewhere else already. Two commands will help us here:

zpool status will print information about running zpools. This should tell you what the device name for your existing root pool ("rpool") is. On my system, I get this:

admin@s11test:~$ zpool status
  pool: rpool
 state: ONLINE
 scan: resilvered 2.61G in 0h16m with 0 errors on Sun Mar 13 21:01:06 2011
config:
 
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c7t0d0s0  ONLINE       0     0     0
 
errors: No known data errors

This means my rpool sits on controller 7, target 0, disk 0 and slice 0.

The easiest, interactive way of figuring out all of your disks in the system would be the format command, but we don't want to spend time going through menus and needless interactivity. Here's a less common, but effective option: cfgadm. This command will tell you what disks we have in the system:
```
admin@s11test:~$ cfgadm -s "select=type(disk)"
Ap_Id                          Type         Receptacle   Occupant     Condition
sata0/0::dsk/c7t0d0            disk         connected    configured   ok
sata0/1::dsk/c7t1d0            disk         connected    configured   ok
```
Not surprisingly, the second disk in our system therefore sits on target 1 of the same controller. Since cfgadm only knows about hardware, not (software) slices, it omits any "s" part.

Now we know what disks we have, which of them is used for rpool already, and which ones are available as a second mirror half for our rpool.

2. x86 only: Set Up a Single Fdisk Partition on Your Second Disk

Solaris disk partitioning works differently in the SPARC and in the x86 world:

SPARC: Disks are labeled using special, Solaris-specific "SMI labels". No need for special boot magic or GRUB, etc. here, as the SPARC systems' OpenBoot PROM is intelligent enough to handle the boot process by itself.
x86: For reasons of compatibility with the rest of the x86 world, Solaris uses a primary fdisk partition labeled Solaris2, so it can coexist with other OSes. Solaris then treats its fdisk partition as if it were the whole disk and proceeds by using an SMI label on top of that to further slice the disk into smaller partitions. These are then called "slices".
The boot process uses GRUB, again for compatibility reasons, with a special module that is capable of booting off a ZFS root pool.

So for x86, the first thing to do now is to make sure that the disk has an fdisk partition of type "Solaris2" that spans the whole disk. For SPARC, we can skip this step.

fdisk doesn't know about Solaris slices, it only cares about DOS-style partitions. Therefore, device names are different when dealing with fdisk: We'll refer to the first partition now and call it "p0". This will work even if there are no partitions defined on the disk, it's just a way to address the disk in DOS partition mode.

Again, we could use fdisk in interactive mode and wiggle ourselves through the menus, but I prefer the command line way. Here's how to check if your disk already has some kind of DOS partitioning:

admin@s11test:~# fdisk -W - c7t1d0p0
 
* /dev/rdsk/c7t1d0p0 default fdisk table
* Dimensions:
*    512 bytes/sector
*     63 sectors/track
*    255 tracks/cylinder
*   2088 cylinders
*
* systid:
*    1: DOSOS12
*    2: PCIXOS
*    4: DOSOS16

(lots of id specifications omitted...)

*  191: SUNIXOS2
*  238: EFI_PMBR
*  239: EFI_FS
*
 
* Id    Act  Bhead  Bsect  Bcyl    Ehead  Esect  Ecyl    Rsect      Numsect
  0     0    0      0      0       0      0      0       0          0         
  0     0    0      0      0       0      0      0       0          0         
  0     0    0      0      0       0      0      0       0          0         
  0     0    0      0      0       0      0      0       0          0

The second - tells the W option to write to standard out instead of to a file.SUNIXOS2 (191) really means SOLARIS2. This is the partition type that we'll create soon.

Here's how to apply a default Solaris fdisk partition to a disk in one simple step:

admin@s11test:~# fdisk -B c7t1d0p0

That's it. Be careful and double-check that you got the device name right! If you're unsure, you can still use the interactive version (fdisk c7t1d0p0) and work through the menus by hand.

Now let's verify that we got what we wanted:

admin@s11test:~# fdisk -W - c7t1d0p0
 
* /dev/rdsk/c7t1d0p0 default fdisk table
* Dimensions:
*    512 bytes/sector
*     63 sectors/track
*    255 tracks/cylinder
*   2088 cylinders
*
* systid:
*    1: DOSOS12
*    2: PCIXOS
*    4: DOSOS16

(stuff omitted...)

*  191: SUNIXOS2
*  238: EFI_PMBR
*  239: EFI_FS
*
 
* Id    Act  Bhead  Bsect  Bcyl    Ehead  Esect  Ecyl    Rsect      Numsect
  191   128  0      1      1       254    63     1023    16065      33527655  
  0     0    0      0      0       0      0      0       0          0         
  0     0    0      0      0       0      0      0       0          0         
  0     0    0      0      0       0      0      0       0          0

Here's the fdisk partition we wanted. Its type is 191 which equals to SOLARIS2 (you can double-check using the interactive version of fdisk), and it spans the whole disk.

3. Set Up an SMI Label With the Same Partitioning on the Second Disk

Before ZFS can do its magic, we need to tell it where on the disk the rpool's mirror is supposed to be, and what blocks are off-limits because they're supposed to host the GRUB bootloader. This is done by using a Solaris SMI label that breaks down our Solaris2 fdisk partition into Solaris "slices".

Again, there's an interactive possibility using the format command, which involves many interactive steps (print out the original disk's layout, set it up step by step on the second disk, write the label), but we want to be cool here, so we'll do it in a single step, again:

admin@s11test:~# prtvtoc /dev/rdsk/c7t0d0s0 | fmthard -s - /dev/rdsk/c7t1d0s0
fmthard:  New volume table of contents now in place.

That's it. You can check how the new Solaris-style partitioning looks like on the second disk and compare to the first one. Here's my first disk:

admin@s11test:~# format
Searching for disks...done
 
 
AVAILABLE DISK SELECTIONS:
       0. c7t0d0 <ATA    -VBOX HARDDISK  -1.0  cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,2829@d/disk@0,0
       1. c7t1d0 <ATA    -VBOX HARDDISK  -1.0  cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,2829@d/disk@1,0
Specify disk (enter its number): 0
selecting c7t0d0
[disk formatted]
/dev/dsk/c7t0d0s0 is part of active ZFS pool rpool. Please see zpool(1M).
 
 
FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> p
 
 
PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 2085 + 2 (reserved cylinders)
 
Part      Tag    Flag     Cylinders        Size            Blocks
  0       root    wm       1 - 2084       15.96GB    (2084/0/0) 33479460
  1 unassigned    wm       0               0         (0/0/0)           0
  2     backup    wu       0 - 2084       15.97GB    (2085/0/0) 33495525
  3 unassigned    wm       0               0         (0/0/0)           0
  4 unassigned    wm       0               0         (0/0/0)           0
  5 unassigned    wm       0               0         (0/0/0)           0
  6 unassigned    wm       0               0         (0/0/0)           0
  7 unassigned    wm       0               0         (0/0/0)           0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
  9 unassigned    wm       0               0         (0/0/0)           0
 
partition> q

And here's my second disk:

admin@s11test:~# format
Searching for disks...done
 
 
AVAILABLE DISK SELECTIONS:
       0. c7t0d0 <ATA    -VBOX HARDDISK  -1.0  cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,2829@d/disk@0,0
       1. c7t1d0 <ATA    -VBOX HARDDISK  -1.0  cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,2829@d/disk@1,0
Specify disk (enter its number): 1
selecting c7t1d0
[disk formatted]
 
 
FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> p
 
 
PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 2085 + 2 (reserved cylinders)
 
Part      Tag    Flag     Cylinders        Size            Blocks
  0       root    wm       1 - 2084       15.96GB    (2084/0/0) 33479460
  1 unassigned    wu       0               0         (0/0/0)           0
  2     backup    wu       0 - 2084       15.97GB    (2085/0/0) 33495525
  3 unassigned    wu       0               0         (0/0/0)           0
  4 unassigned    wu       0               0         (0/0/0)           0
  5 unassigned    wu       0               0         (0/0/0)           0
  6 unassigned    wu       0               0         (0/0/0)           0
  7 unassigned    wu       0               0         (0/0/0)           0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
  9 unassigned    wu       0               0         (0/0/0)           0
 
partition> q

Note: This is a typical x86 layout. It's likely different on SPARC systems as they don't use a special slice for boot block hosting. But the basic idea on how to replicate the partition table is the same.

Great! We're almost there.

4. Set Up the ZFS Rpool Mirror

Now that our second disk is prepared, the rest is quite easy. From now on, we can just follow the standard Solaris documentation for mirroring the root pool.

The right command to use here is zpool attach. Notice that this is different from zpool add: By attaching a disk to an existing disk, we mean attaching it to its mirror (you can attach more than one disk to a mirror). By adding a disk to a pool, we mean expanding the pool size in the sense of striping in another disk (or sets of mirrored/RAID-Z disks). For mirroring, zpool attach is the way to go. Remember? Slice 0 is the one we reserved for the rpool's mirrored data:

admin@s11test:~# zpool attach rpool c7t0d0s0 c7t1d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c7t1d0s0 overlaps with /dev/dsk/c7t1d0s2

Wait, what happened? ZFS is complaining that two slices are overlapping. If ZFS uses slice 0, and something else uses slice 2, it may overwrite some of ZFS' data!

In this particular case, ZFS' worries are unfounded: Slice 2 by convention spans the whole disk and is named "backup" (see the output of format above), so traditional disk backup solutions have a way of easily performing raw backups of whole disks. Today it's hardly used, but the convention remains for historical reasons.

Therefore, we can safely override this little nit and get our mirror done:

admin@s11test:~# zpool attach -f rpool c7t0d0s0 c7t1d0s0
Make sure to wait until resilver is done before rebooting.
admin@s11test:~# zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Tue Mar 15 18:17:32 2011
    13.9M scanned out of 2.72G at 594K/s, 1h19m to go
    13.3M resilvered, 0.50% done
config:
 
        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c7t0d0s0  ONLINE       0     0     0
            c7t1d0s0  ONLINE       0     0     0  (resilvering)
 
errors: No known data errors

Great! Everything's working fine now. Before we make the second disk bootable, we should really wait until it has finished resilvering. We don't want to boot into a half-baked root pool, do we?

Here's the end state, freshly resilvered:

admin@s11test:~# zpool status
  pool: rpool
 state: ONLINE
 scan: resilvered 2.72G in 0h15m with 0 errors on Tue Mar 15 18:33:23 2011
config:
 
        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c7t0d0s0  ONLINE       0     0     0
            c7t1d0s0  ONLINE       0     0     0
 
errors: No known data errors

5. x86 only: Make the Second Mirror Half Bootable

Since x86 systems depend on a bootloader that is installed on disk, we need to perform a final step so that the system can boot off the second disk, too, in case the first one fails completely.

This is a simple install of GRUB onto the second disk. GRUB, ZFS and Solaris will then figure it out automatically in case you have to boot from the second disk instead of the original one.

admin@s11test:~# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c7t1d0s0
stage2 written to partition 0, 277 sectors starting at 50 (abs 16115)
stage1 written to partition 0 sector 0 (abs 16065)

Since we're dealing with a low-level operation (boot blocks etc.), we want to address the devices using the raw device paths. The s0 part is still needed so GRUB knows what slice to boot from.

Almost done!

6. Add the Second Disk to the BIOS' or OpenBoot PROM's List of Bootable Devices

This is one of the little things that often gets overlooked but then becomes critical in case of a real failure: The system crashes because the first disk is completely borked, or you force a reboot and the first disk fails to come up again. How does the system know it's supposed to boot from the second half of the mirror?

Managing the Solaris boot behavior and its mechanism is described thoroughly in the documentation:
SPARC: Here you usually set up aliases for your bootable mirror halfs in the Open Boot PROM, then assign them to the boot-device variable as a list of possible devices to boot from (e.g.: "disk1 disk2 net"). Check out the SPARC Enterprise Servers section of the Oracle System Documentation area, find the administration guide for your particular system, then consult the sections on booting.
x86: Most BIOSes have a section where you can configure what disks to boot from, in what order and what to do if a disk is not bootable. Here's a list of current Oracle Sun x86 system documentations. Again, look for the boot section of your system's admin manual.

Play With It! And Check Out Some Man Pages!

How do you know if this really works? How do you develop confidence for something critical like booting from a second mirror half, surviving a disk disaster, etc.?

Here's the easiest option: Use VirtualBox to set up a test system like I did. It comes with ready-to use suggestions for a standard Solaris machine. Then, configure a second virtual disk and play with the commands above. Set up a mirrored rpool, bring down the machine, unconfigure the original disk, then see if it can boot from the second mirror half and so on.

BTW: I did not find a way to tell VirtualBox what disk to boot from (it only allows to specify what type of device to boot from, not what individual disk), so I reverted to just pull out (figuratively speaking) the original boot disk, then test if if boots from the mirrored one.

In short: Play, experiment, break it, etc., until you know what's going on and are confident to make it happen on your real system.

Finally, here's a list of useful man pages to check out, including links:

cgfadm(1M): Asks about and modify's your system's hardware.
fdisk(1M): Manipulate DOS-style partitions.
prtvtoc(1M): Print out a disk's partitioning information in a machine-readable format.
fmthard(1M): Write a partitioning table to a disk.
format(1M): Interactive formatting utility.
zpool(1M): Manipulate ZFS pools.
installgrub(1M): Install the GRUB boot loader.

I hope this article has made rpool mirroring a little easier for you from now on!

Your Take

There are endless variations to the above, and sometimes I've been more verbose, or more simplified for the sake of ease-of-use. I'm sure there are many different ways to achieve the same result, so here's your chance to share your favorite mirrored rpool tricks!

What's your routine for mirroring rpools? Did you find other good tutorials to share? ('cause I didn't, at least nothing obvious in Google...) What are your preferred rpool mirroring tricks?

Feel free to write a comment!

Update: Wow, this article got a lot of comments, thank you! Make sure you check them out as they contain a lot of useful additional information.

Frequently Asked Questions About Flash Memory (SSDs) and ZFS

constant — Sat, 19 Feb 2011 15:51:43 +0000

A few weeks ago, a reader asked me a couple of questions about SSDs and ZFS, hinting that this might be a good topic to write a blog post about.

Sure enough, just last week, a couple of similar questions came up, this time from a customer and a colleague at work.

Well, if that's not a sign from heaven, I don't know what is, so here's a collection of frequently asked questions about flash memory (also known as solid state disks, or SSDs) and ZFS, with answers and some useful links, and an index, too.

Index

What Are SSDs?
Are Flash Memory and SSDs the Same Thing?
Are SSDs "Better" Than Traditional Hard Disks?
Are All Flash Memory SSDs Equal or are there Different Variants?
How Can ZFS Benefit From SSDs?
How Do I Know if an SSD Will Really Help My Write Performance?
How Do I Know if an SSD Will Really Help My Read Performance?
Are different types of SSDs better suited as read or write accelerators?
Should I mirror flash memory drives?
Can I use the same SSD both as a ZIL and as an L2ARC?
How much space do I need for a ZIL?
How much space do I need for an L2ARC?
How do I configure ZFS to use SSDs?
That is too complicated, is there an easier way?
Can I benefit from Flash Memory/SSDs even if I don't use ZFS?
More Questions and Answers

What Are SSDs?

"SSD" stands for "Solid State Disk". In short: A disk that doesn't have any moving parts. Today, most SSDs are made using flash memory, but some SSDs use a combination of RAM and some form of battery back up to make them permanent. The key is to leave out the moving parts so the SSD has much faster performance characteristics, especially in terms of IOPS. Read more about SSDs on Wikipedia.

Are Flash Memory and SSDs the Same Thing?

Technically, no: SSDs are a drive category, and flash memory are a memory technology. But in practice, people use both terms synonymously: A drive that looks like a harddrive but uses flash memory to store data instead of rotating rust is an SSD, which is the same as a "Flash Memory Drive". But much if not all that is being said about Flash Memory SSDs can also be applied to any other storage medium that has very fast IO performance characteristics, especially in terms of IOPS.

Are SSDs "Better" Than Traditional Hard Disks?

It depends on what you're looking at: Flash Memory SSDs are about two orders of magnitude faster in terms of IOPS (the number of IO operations per second) than traditional harddisks. But in terms of throughput performance (MB/s) they're in a similar league. Since there are no moving parts, SSDs consume a lot less energy than traditional hard disks. For the same reason, they're also more reliable. On the other hand, the cost per GB is much higher with SSDs.
This creates an interesting phenomenon: While it would be cool to throw out all traditional disks from your datacenter, PCs and laptops, it would also be expensive. Therefore, the key to taking advantage of SSDs is to use them at the right place, and where their benefits most matter.
Think of SSDs as being a new layer of memory between RAM and traditional harddisks: Faster than drives, slower than RAM, cheaper than RAM on a per GB basis, but more expensive than disks.

Are All Flash Memory SSDs Equal or are there Different Variants?

No. There are different categories of flash memory and occasionally, new technologies are invented that expand the scope of SSD variants.
Today, the two most important types of flash memory are SLC (Single Level Cell) and MLC (Multi Level Cell).
SLC flash memory uses a single level per cell to store a bit of data. This results in faster performance, better reliability and lower power consumption than MLC.
MLC flash on the other hand uses multiple levels per cell. This allows each cell to store more values than just 0 and 1. Therefore, an MLC cell can store two (most common) or more bits of data. This makes MLC flash less expensive on a per GB basis and more data can be stored in the same form factor.
When selecting SSDs, you need to weigh these factors according to the way you want to use your SSD: Do you need better reliability and performance? Or are you after a lower cost, more dense option?

How Can ZFS Benefit From SSDs?

There are two ways in which SSDs can be used with ZFS (See the ZFS source tour diagram to better understand the following):

ZFS uses a logging mechanism, the ZFS intent log (ZIL) to store synchronous writes, until they're safely written to the main data structure on the pool. The speed at which data can be written to the ZIL determines the speed at which synchronous write requests can be serviced: The faster the ZIL, the faster most database, NFS and other important write operations become. Normally, the ZIL is part of the regular pool on disk. But ZFS offers the possibility to use a dedicated device for the ZIL. This is then called a "log device".
By using a fast SSD as a ZFS log device, you accelerate the ZIL and synchronous write performance improves. See also Solaris ZFS, Synchronous Writes and the ZIL Explained.
ZFS also has a sophisticated cache called the "Adaptive Replacement Cache" (ARC) where it stores both most frequently used blocks of data and most recently used ones. The ARC is stored in RAM, so each block of data that is found in the RAM can be delivered quickly to the application, instead of having to fetch it again from disk. When RAM is full, data needs to be thrown out of the cache and is not available any more to accelerate reads.
SSDs can be used as a second level cache: Blocks that can't be stored in the RAM-based ARC can then be stored on SSDs and in case they're needed, they can still be delivered quicker to the application than by fetching them again from disk. An SSD that is used as a second level ARC is therefore called an L2ARC, or a "cache device".

In summary, SSDs can be used in two important parts of the ZFS architecture: A log device will accelerate synchronous writes, and a cache device will accelerate reads.

How Do I Know if an SSD Will Really Help My Write Performance?

Good question. Sometimes, the situation is so borked (or the bottleneck so non-obvious) that an SSD for the ZIL won't help you much. Before you invest money in an SSD, it's better to check.
Richard Elling has written a cool utility called zilstat that answers the question: How stressed is my ZIL?. Run this script, put your server under a typical load and watch for yourself. If you see a lot of ZIL activity, you'll likely benefit from a ZIL that is offloaded onto an SSD.

How Do I Know if an SSD Will Really Help My Read Performance?

Again a very good question. Sometimes, the most popular data fits into your RAM, so an L2ARC won't give you much extra benefit. Sometimes, though, the ARC is so busy that it really could use some extra help from an SSD. Again, there's a cool script that will help you decide, this time from Ben Rockwood: arc_summary will collect a number of useful ARC statistics and his Explore Your ZFS Adaptive Replacement Cache (ARC) article will teach you what numbers to look for.
The short version: If you see a lot of ghosts (= blocks that were eliminated from the ARC but that were discovered later to be useful), then you should buy an SSD for an L2ARC to keep them from becoming ghosts.

Are different types of SSDs better suited as read or write accelerators?

Yes:

SLC flash is faster for writes and more reliable, therefore it's the best choice for a ZIL. You can still use it as an L2ARC, though. Reliability and speed never hurts.
MLC flash will give you more capacity for your money, but is less reliable and less fast than SLC. Therefore, MLC makes a good read accelerator when you're budget-constrained. But don't use MLC flash as a ZIL: If a write to a single cell breaks, other bits may be affected and the failed write may affect blocks that were previously written.
Data on a ZIL is an important backup mechanism, you don't want to compromise that.

Should I mirror flash memory drives?

Interesting question. On one hand, hardware breaks all the time, and SSDs are no exception. On the other hand, SSDs don't have any moving parts and so they're statistically much less susceptible to failures.
It really depends on your risk tolerance:

The ZIL is the last resort to go to if the system crashes before data that was promised to the application to be "safe" is actually written to disk. Then, upon reboot, the system reads back the ZIL and performs the missing updates on the actual ZFS storage pool. Since the ZIL is so important for ensuring data integrity, it should therefore be mirrored and ZFS supports that quite nicely.
The L2ARC is a read cache: It stores data for convenience and speed only, but every bit of data in the L2ARC is also available elsewhere. So mirroring an L2ARC SSD is not really necessary (though Marcelo has a very good point in that a dramatic loss of performance may actually justify an L2ARC mirror). Instead, ZFS will use any extra SSDs you give it for L2ARC in order to expand the amount of space for caching data.

Can I use the same SSD both as a ZIL and as an L2ARC?

Yes, you can. (I always wanted to say that phrase, btw...)
In theory you can split up an SSD into two slices through the format(1M) command.
In practice, this means that you'll have two streams of data (ZIL writes and L2ARC writes and reads) instead of one, competing for the limited resources of the SSD's connection and controller. That may compromise your ZIL performance as two mechanisms step on each other's feet.
Better try it out: split up the SSD, configure the ZIL part of it, see how much it improves your write performance, then hook up the L2ARC part, while observing if the ZIL performance is still good. Use zilstat for monitoring ZIL performance.

How much space do I need for a ZIL?

The role of the ZIL is to store a transaction group until it has safely been written to disk. After that, this can be safely deleted and the space used for the next transaction group.
So the question becomes: How much transaction group data is "in flight" (i.e. not yet written to disk) at any time?
ZFS issues a new transaction group (and consequently a new pool update) every 5 seconds at the latest (more if the load is higher). While one transaction group is written to the ZIL, the previous one may still be in the process of being written to disk, so we need enough space to store two transaction groups, which means 10 seconds of maximum write throughput worth of data.
What's the maximum amount of data that your server writes in 10 seconds? Well, an upper boundary would be the maximum write speed of your SSD. At the time of this writing this was about 170 MB/s for an Intel X25-E, times 10 that would be just short of 2 GB for a typical ZIL.
So for ZILs, a little can go a long way.

How much space do I need for an L2ARC?

This is more difficult, or more easy, depending on how you put it. More is always better, but too much would be a waste if it's not used. Check your L2ARC usage with arc_summary and if you still see a significant amount of ghosts after adding an L2ARC, you'll likely benefit from even more L2ARC space.
Another way to estimate L2ARC need is by looking at your working set: The amount of data that is used most frequently. Depending on your application, this could be your top 10 research projects, your top 20% of recurring customers, your most popular 100 products etc.

How do I configure ZFS to use SSDs?

Read the zpool(1M) man page, in particular the sections on Intent Log, Cache Devices, the zpool add and zpool attach subcommands and the EXAMPLES section.

That is too complicated, is there an easier way?

It's actually not complicated, ZFS can be administered with just two commands!
But if you're looking for a GUI version, check out the Oracle ZFS Storage Appliance products for an easy-to-use NAS version of ZFS. Some third party vendors offer other appliances or appliance-ready OS distributions that are based on OpenSolaris or FreeBSD. They don't support all the latest features that the Oracle ZFS Storage Appliances (or Oracle Solaris 11 Express) offer, but they may be good enough for some. Examples include Nexenta, EON and FreeNAS.

Can I benefit from Flash Memory/SSDs even if I don't use ZFS?

Certainly. Here are a few common scenarios:

Any filesystem that you place on an SSD is going to be faster due to the physical nature of SSDs versus rotating rust.
Databases often allow you to place their database log onto a separate file system. SSDs are a great choice for database log files. Actually, a database log and the ZFS ZIL are very similar mechanisms.
The Oracle database in particular has a Smart Flash Cache Feature that works similarly to the L2ARC described above: It's basically an extension of the SGA so that frequently needed data remains accessible from fast flash memory even if the SGA (which is in memory) is not big enough.
I'm sure you can come up with other great uses, but that almost always involves some sort of intelligent distinction between data that is accelerated with flash and data that is stored on regular disks. The beauty of ZFS is that it does this job automatically for you.

Video: An Interview With the Fathers of "Iron Sky"

constant — Tue, 25 Jan 2011 21:30:36 +0000

A deserted street in the docks. The rain is pouring.

A small, old, flower-power style painted Volkswagen van drives by at high speed, then stops below a lifting crane, tires screeching.

The door opens and a woman stumbles out of the vehicle, as if trying to escape. A tall man in uniform catches her violently. Intense exchange of words, probably four-lettered. She hesitates, then follows him back into the van.

"Cut!"

My brother and I are watching from the side of the road. It's a scene from the upcoming movie Iron Sky, a science-fiction comedy about the Nazis who escaped with UFOs to the dark side of the moon and who come back in 2018 to conquer the earth.

What?

It's a long, fun story, but you better see it for yourself. Early 2012, that is, when the movie is done.

If you've ever met me at a party, you'll know that I'm a movie fan.

As a student, I've worked for the local movie theater: Distributing flyers, setting up the online movie schedule, as a projectionist, etc. In exchange, I got to see all the movies for free, often a few days in advance. For example, I've seen Speed and The Mask during theater owner previews, and my friends and I got to watch Toy Story the day before it was released.

I still carry some pieces of real film in my wallet as a lucky charm and to illustrate how different movie sound formats like DTS, Dolby Digital, Dolby ProLogic work, but I digress.

Sudden Snow Storm Freezes Frankfurt!

It is Wednesday, December 8th. My brother and I took the day off to drive to Frankfurt, where the Iron Sky team is shooting. The week before, they got to explode one of Frankfurt's main streets, which was great fun, I hear. Today, we're less lucky: The rain is turning into snow, Frankfurt airport is about to be closed and chaos is spreading across Frankfurt's streets and autobahns as the blizzard sets out to conquer the city, making our shoot impossible.

But none of that matters, this is the coolest day of the year for us: We are on a real movie set!

You see, Iron Sky is not your normal Hollywood-style movie. It is one of the first major cinema projects that is financed with the help of its own fan community: If you want to support the film, you can become an investor and help make the movie happen. If the movie becomes profitable, you can get your money back. If the movie becomes a success, you may even earn a little extra.

But that's not the point. Iron Sky is all about community involvement: Fans all over the world are involved in the filmmaking process: You can witness the production process through blogs and video diaries, you can "demand to see Iron Sky" as a way of showing distributors and theaters how popular the movie is, you may be invited to spend a day with the cast and crew while they shoot the movie, or you can have a sneak peek at the whole film-making process of the first 5 minutes: From the script to the final movie.

And you can ask the director and the inventor of the movie to sit down and do an interview for your blog, which is what I did a few hours before the van scene was shot.

Watch the Interview

In this interview, we talk about how the original idea of the movie came into being, the role of technology in Iron Sky and what "to steampunk a gun" means. We also talk about the role of social media and communities in the making of the movie. Enjoy!

(Sorry for the shaky camera and funky sound, we're still practicing :).)

Due to the snow, we had to interrupt the shooting and look for alternatives. We returned to the "studio" which was an abandoned area with three skyscrapers in the middle of downtown Frankfurt. I was told that the skyscrapers used to belong to Lehman Brothers who went broke in 2008. I told them "Cool, so you have more investors than you think".

Waiting, Chatting, and a Long Night

The utter majority of time on a movie set is spent waiting: Waiting for the set to be prepared, waiting for makeup, waiting until stuff is ready, and waiting for people who have been waiting for whatever reason.

This gave us lots of time to talk and get to know the team: We saw cool concept drawings, heavy gun mock-ups, space pilot uniforms and other cool "steampunked" stuff.

The actors are cool and fun people to hang out with: Götz Otto is hilarious when he puts on his "Of course the Nazis are on the moon, I was there, too!" expression in front of the journalists who attended the press conference. Chris Kirby told us how he took his son to see the Matrix movies, where he played "Mauser", resulting in naming their cat "Neo". And Julia Dietze charmed the press with her ambition to study every tiny move of Marlene Dietrich, so her role as "Renate Richter", who grew up on the moon, culturally frozen in the era of the 40ies, becomes more convincing.

After dinner, the scene needs to be modified to a different surrounding. Peta Sergeant and director Timo Vuorensola discuss the motivations, background and conflicts of their characters, so the new scene can be make to "work". A discussion among actors, the director and fans emerges, while they brainstorm different possibilities for the course of the scene. It's amazing how much attention to detail and dedication the team puts into the movie: From the tiniest mechanical bits inside computer-generated machinery to gorgeous space suits made of sauna tubes to subtle humor in the naming of characters or space ships. Everything about this movie is made with love and hard work.

Around midnight, the team found a new location: A closed fairground parking garage where the crew managed to set up the new scene in record time, complete with lights, camera, catering, heating, A/V equipment and so on. We shoot until 3 AM and after a short night, my brother and I try to beat the snow storm on our way back to Munich.

We will return.

How You Can Help

If you're like more than 45,000 fans who "like" Iron Sky on Facebook, and if you too want to make this crazy Finnish-Australian-German science-fiction comedy Co-Production happen, here' s what you can do to help:

Support Iron Sky for free: See how the movie is being made by following the Iron Sky Blog. Or Demand to see Iron Sky so the producers can convince more and more distributors and movie theaters to license the movie for screening.
Support Iron Sky for $1 or more: For only a dollar (or any amount you choose), you can witness how the first 5 minutes of the movie are made: From the original script to storyboards, shooting, CGI, post-production to the end result, you'll get one new piece of the puzzle each month until the movie is released.
Support Iron Sky for EUR 7.50 or more: By shopping in the Iron Sky fan store, you can buy cool stuff like T-Shirts, mouse pads, concept art, posters, "War bonds", even pre-order the special collector's edition of the movie. Of course, any profit flows directly into financing the movie.
Support Iron Sky by becoming an investor: If you are from Europe and are ready to invest EUR 1000 or more, you can apply to become an investor in Iron Sky. Certain rules apply, but there are also perks: You'll be more involved in the making of the movie, get a chance to visit the team and who knows, the movie might even turn a profit that you can benefit from.

Meanwhile, enjoy the trailers, and stay tuned for more tales from the dark side of the moon at the Iron Sky blog.

German-speaking readers: More background on Iron Sky is available in episode #51 of the HELDENfunk podcast and in this "Blick über den Tellerrand" interview about crowd-funding.

Or Do You Prefer a TV Show?

Before we close, I'd like to quickly introduce you to another cool project:

Pioneer One is a TV series about a mysterious incident at the border of Canada, where a radioactive capsule with a strange survivor has been found. The FBI and the Canadian authorities are trying to solve the mystery, under their government's pressure who think this may be a terrorist attack. Slowly they realize how strange their survivor really is...

This series follows a similar crowd-funding concept: Fans can donate money and as soon as the goal for an episode is reached, it gets produced, then distributed through BitTorrent for free. You can download the first two episodes now for free and trust me: This is a very ambitious and well-scripted story. Great stuff!

Are You a Movie Buff?

How about you? What's your favorite movie? Are you a movie investor? Do you believe that crowd-funding is changing the world of film or is this just an exotic indie gig? Share your thoughts in the comments!

Disclosure

As you have guessed by now, I'm an investor in Iron Sky, so I have an interest in promoting the movie, its funding and its commercial success.

How to Save the World with ZFS and 12 USB sticks: 4th Anniversary Video Re-Release Edition

constant — Mon, 24 Jan 2011 21:11:45 +0000

About 4 years ago, a few colleagues and myself got together and we created a short video about the coolness of two of the most innovative products from Sun of the last decade: ZFS and the X4500 Server.

Today, nearly 4 years later, the video has been downloaded more than 100,000 times (across the original German and the English dubbed version, plus the full resolution downloadable files) and shown to a lot more people during tradeshows, customer demos, etc.

Now YouTube and Google Video (remember?) don't allow for highest video quality and the old Sun Mediacast server, where we hosted the original MP4 file, no longer exists. Instead, Vimeo has emerged as my video hoster of choice for a variety of projects (check out my video collection on Vimeo) and so it was time to give this video a new home.

For your viewing pleasure.

Watch the English dubbed version:

Or if you understand German (or have a taste for movies in original language), watch the original:

The original CSI:Munich blog post explains some of the technical background of the video, as does its German counterpart post "Solaris ZFS auf 12 USB-Sticks: Ein "Thumper" für Arme!".

It's amazing to see how far ZFS has made it today. Still, there's no other file system I know of that can be configured in seconds, provides data integrity at mathematically deep levels, implements database-like transaction safety, comes with a high-end featureset and gives you so much as ZFS.

And the possibilities of ZFS continue to expand. Stay tuned.