My first real job was manually sorting printed superannuation statements. I had to go through about a dozen boxes, each with about 2000 pieces of paper, and separate them into piles: single-page statements, and multi-page statements.

Why? Because the program that printed the statements just printed them all at once, so you ended up with the single-page statements mixed in with the multi-page statements. The mailout company needed to know whether a packing run was to have single-pages folded into envelopes, or 2 pages, or 3. Otherwise, someone might get 2 pages; 1 page of their financial info, and 1 page of someone else’s. Bad news for a financial company.

So they gave the job to a 16 year old wanting 2 weeks of holiday work. Go figure.

Mind numbingly tedious? Hell yeah. Also: paper cuts.

Industrial Robots

Fast forward a few years, and I wanted to do Computer Systems Engineering, because that meant I could build robots and program them to do stuff.

Stuff like solving a Rubik’s cube in under 12 seconds (with Lego), or assembling cars, or stacking pancakes. What boldly geeky 18 year old wouldn’t want to do that?

I learned how to write the software that runs the ticket barriers in train stations (no, I didn’t work on Myki). I also learned a lot about magnetic hysteresis and calculus in three dimensions, which I’ve mostly forgotten.

Then I got a job in IT, and still haven’t built a robot. Yet.

Why is IT So Awful?

I’ve written before about my thoughts on how and why IT sucks for business. Watching those videos has crystallised why it annoys me so much.

We’ve been automating factories for decades. All that work going on here used to be done entirely by humans.

We’ve been systematising production for centuries. Adam Smith wrote about division of labour back in 1776!

And yet every large company I’ve worked for in the past 10 years has been universally awful in the way they use IT.

Car companies re-tool their automated factories to produce a new car model every year. Some companies still have critical business applications running on Windows NT 4.

That pancake manufacturer has a robot that stacks pancakes automatically for them, but some companies are still manually typing in commands to their routers over telnet.

The humans in IT are making mistakes, breaking production equipment, and producing an endless supply of hand-crafted, unsupportable, poor-quality products. IT projects have a failure rate of well over 50%, and people just shrug their shoulders and say “Ah well, what can you do?”

The only reason it hasn’t been done yet? You’re not trying hard enough.

My Ultimate Goal

Now I’ve finally got a clear, concise way of explaining what I want to do for IT. The next time someone isn’t grokking what I mean, I can just say:

This. This is what I want to do.

No related articles.

I finally got around to uploading a gallery of photos I took from the Australian Synchrotron Open Day last year.

What is a Synchrotron?

A synchrotron is basically a particle accelerator, but instead of trying to find the Higgs-Boson (or create black holes and cause the end of the universe, if you’re a nutcase who believes that sort of nonsense), it uses a radiation by-product to do high energy science.

Synchrotrons accelerate electrons, not protons or other matter. When you have electrons moving about, you get electro-magnetic radiation; i.e., light. This light is high-energy light, so think X-rays. And just like your doctor can use X-rays to look inside your leg to see how broken it is, researchers can use high energy light to look inside materials, do fancy crystallography, all sorts of things. This is why the tagline for the Australian Synchrotron is “the brightest light in the southern hemisphere”.

This is really useful for all sorts of practical, every-day industrial purposes, like analysing soil samples. Synchrotrons are another awesome example of brilliantly useful things coming out of pure science research, even though it wasn’t what we were looking for in the first place. This is one of the great things about science; just trying to figure out how stuff works so other people can come along later and go “Ooh! I can totally use that to do this really useful thing over here!”

Open Day

Mmmmagnets!

The Australian Synchrotron has held an open day for the past 2 years that I’ve lived in the area. It looks like they do free tours if you book, but I couldn’t see anything on their website about the Open Day being a regular event.

There was an amazing level of public access, as you can see from the photos. I booked ahead for a guided tour of the control systems, which ended up being a tour of pretty much the whole facility. Not only did we get to see the control room, complete with inflatable kangaroo mascot, we also got to see inside the concrete bunker of the beam path itself.

At one point we had to clamber over the linear accelerator. We were close enough to touch the beam path! How cool is that!?

It was a really popular event, and I hope they keep holding it.

Check out the full gallery of high energy science/industrial machinery porn by clicking here.

Like this one? You might like these too:

• Slow Light and Quantum Interference

The chairs are awful. They’re bulk-bought, partially adjustable, cheap, and nasty. Sometimes a company has a policy for how you can get a better one, usually involving medical certificates, being 7 foot 2, or weighing 148kg.

I’m 5 foot 6 and 62kg. No snazzy chair for me.

If I’m lucky, the desks are adjustable. Usually though, the furniture police have the special key thing you need to adjust the height, or (as in one place I worked at), they’ve actually bolted other, non-adjustable pieces onto the adjustable bits, so none of it can be moved. Again, I’m 5 foot 6, so I perch on a chair that’s too high so I can reach the keyboard, and need phonebooks so my feet reach the floor.

This sucks, but I have a solution.

My New Hiring Test

Should I ever interview for a permanent role, here’s a question I’m going to ask:

“I would like a small, discretionary budget I can use for things like a chair, LCD screens, whiteboard markers, that sort of thing. Five to ten grand per year. Would that be possible?”

It’s a question, not a demand, because I’m interested in how the question is answered, more than the specific answer.

I’m expecting that most companies would just say no. A slightly more progressive company may at least entertain the idea. A better company will ask me about why and be willing to be persuaded. The best companies will already have something like this in place.

Here’s why.

Death by a Thousand Cuts

A crappy chair annoys you every day. A too-high desk is irritating all the time. A too-small screen is frustrating every time you open a large spreadsheet.

I want the power to make these small annoyances go away, because small annoyances grow into large ones over time. If I have a comfy chair to sit it, doing boring work tasks sucks that little bit less. I can deal better with all those minor annoyances you get with any job.

A small budget of my own is great for several other reasons:

I get to solve my own problems. Since people are different, there will be loads of different problems. Jill over there might be fine with the standard chair, but wants a footrest. Adam might want a different mouse. They get to solve the problem that affects them specifically, all by themselves.

I’m accountable. I only have $4k to spend (say). If I buy a $3k chair, the money’s gone. I can’t decide half a year later that I should have bought a $2k chair because now I don’t have enough money to buy extra pens. It teaches people that company money isn’t just an endless pile that they get to spend on whatever they like, because hey, it’s not my money.

It demonstrates trust. The company doesn’t assume that I’m stupid and need a manager to figure out what sort of chair I want. People are pretty good at figuring out what their own personal preferences are, and in my experience, managers are almost universally woeful at it.

Penny wise, Pound foolish

A really good chair, say an Arthur Miller ‘Aeron’, costs about $1300 retail. LCD screens are, what, $300-$500 each? An Aeron will last more than 10 years, and an LCD screen will last 3-5 easy.

Try getting your work to buy you one. Listen to the excuses. No doubt there will be company policies, or the need for managerial (or worse, finance) approvals. If you’re super-lucky, you may get to fill in a business case!

Then think about this: If you’re on $100k a year, you cost the company $400 a day. If you goof off for just one day, that’s an LCD screen worth of wasted company time. You can waste an equivalent amount of money with no oversight required at all. But you can’t buy an LCD screen for yourself.

The higher your salary, the worse this is. At my level, salaries are $175k and up, so it’d take about 2 days of watching YouTube videos to buy an Aeron. Or, I could just go to 16 hours of pointless status meetings over the year. Same thing.

Implementing a change that saves a team of 5 people an average of 10 minutes a day (by, say, improving the timesheeting system) is a cost saving of $15k. Enough for everyone to have an Aeron and 2 LCD screens each.

I have total discretion to waste significant amounts of company money with no oversight at all. But I can’t buy my own pens. How empowered am I really?

What would you buy if you had a personal budget for the year?

Like this one? You might like these too:

• Why Chargeback Doesn’t Work

1. Speak up

Nothing is more frustrating than straining to hear someone mumbling quietly to the room. If you’re talking, presumably you want other people to listen to you, right?

You’re probably familiar with those lumpy phone things that get used in meeting rooms for conference calls. They have a couple of microphones for picking up the voices in the room, but usually don’t have per-person mikes. This means you get background echo mixed in with your voice, and the mike is a fair way away from you.

The person on the other side of the table is close, and doesn’t have phone system attenuation to deal with. Plus, they can get clues about what you’re saying by lip reading, watching body language, etc. The people on the phone get none of that.

You need to speak up, projecting your voice so that it can be heard clearly in all parts of the room. That way, the mikes have the best chance to pick up what you’re saying, and the people on the phone can hear you.

Also, practice speaking with clear diction. The voice algorithms used by phone companies tend to blur phonemes, so if you slur from one word to the next, the phone will make it worse.

2. Mute your phone

If you’re not talking, mute your phone.

No one needs to hear your children crying, or the dog barking, or your other phone ringing.

Take care to know when your phone is muted and when it isn’t. I’ve been on a few calls where someone made a snide aside to someone in the same room on their end that wasn’t intended to be heard by everyone.

Only they weren’t muted at the time. Oops.

3. Announce your presence

There are some sneaky people who dial in to a conference call, and then sit silently (with their phone muted, if they’re super-sneaky), listening to what others have to say.

It’s true that you can find out some interesting things with this technique. Many a time I’ve heard people badmouth a colleague on a conference call, only to have them pipe up and inform everyone that they heard every word.

It’s important that everyone in a meeting knows who else is there. The sneaky technique above is underhanded and disrespectful to your colleagues. Don’t do it.

4. Be on time

A conference call is just a meeting. Being late for a meeting is poor form. So it is for conference calls.

Some conferencing systems announce your presence to the whole call (see tip 3 above), so being late interrupts the person currently speaking. That’s like walking into a room 5 minutes late, slamming your notebook down and saying “Hi everyone! I’m here!” in the middle of the VP’s opening statements.

5. Take turns politely

If you’re on the phone, and there’s a room full of people on the other end with three talking at once, you can’t pick them out spacially because you only have one sound source: your phone. This makes it even harder to figure out what’s being said.

Make sure there’s only one conversation going on at a time. And don’t talk over one another. It’s rude normally, but it’s worse on a conference call.

Conferencing systems tend to favor a single voice over all the others, too, so if you’re talking, you can’t hear anyone else, and they can’t talk over you.

I’ve been on a few calls where someone on the phone was droning on and on about something irrelevant, and the meeting chair couldn’t get a word in edgewise to interrupt and keep things moving. That person’s reputation suffered as a result.

Don’t hog the spotlight. Make sure you yield frequently so other people have a chance to speak.

Like this one? You might like these too:

• Pick up the phone
• Working From Home

I’ve heard various people complain about it, and it seemed to be much the same stuff as we get here: the security theatre, queuing, delayed flights, hassles and pain.

Well yes, all those are true. And yet somehow the US managed to be worse.

I’ve done a lot of domestic air travel in Australia this year, so I’ve well versed in how it all works. However, I only had two domestic flights while in the US, so this isn’t a wide ranging survey.

It’s probable that other airlines (I flew US Airways) and airports (I only saw LAX and PHX) are better. At least I hope so. If not, you have my pity, you poor US airline travelers.

Airport Layout

One of the first things I noticed was that Australian airports tend to be more streamlined in their layout. The US experience felt messy and confusing.

I’m a seasoned air traveller, so I shudder to think what novice travellers would experience.

Checkin

Checkin is done using electronic kiosks, for the most part, and then you just drop off your bag. The first time you use it is a little unusual, but the service people are there to help you if you need them. Or you can check in online.

Finding the right bag drop queue is simple; there’s clear signage, and different queues depending on if you’re a frequent flyer, club member, etc.

If you’re unsure where to go, there’s a person or two hanging around near the front of the queueing area to help with queries, and they know what they’re talking about. I’ve never been sent to the wrong place.

For US Air, checkin and bag drop happens at the same time (in PHX). There’s a bunch of kiosk things, and you need a credit card to pay for each bag. There are only one or two service people to help, and there’s no real queue. It’s confusing and unclear what you’re supposed to do, and no real signage to help.

Common to both Australia and the US are arbitrary bag weight limits. This is important for takeoff weight calculations, I know, but I say bring on weighing of passengers like they did back in the early days of air travel. I only weigh 60kgs, so I should be able to put more in my bag if I want.

As it was, I had to transfer 5 pounds of books from my checked back to carryon to get under the 50 pound limit.. which went on the same plane. My bags were checked all the way through to international (where I have a higher limit that I couldn’t use), so my experience was limited by US Air’s annoying policy.

Is Australia, all they do is tag your bag with ‘Heavy: nn kgs’ and get on with it. Maybe the baggage handlers in the US are more concerned about lower back injuries?

Security

Lots of theatre. No surprises there.

In Australia, I don’t have to show ID to get on a plane. In the US, they check that your boarding pass matches some form of ID. They don’t check it again when you board, though. That’s trivial to bypass, as pointed out by Bruce Schneier. So it’s annoying and ineffective. Well done, TSA.

Security screening itself takes significantly longer. I don’t have to take off my shoes in Australia. However, we also have the entirely non-random explosives screening process.

There was just an extra layer of paranoia across the whole thing in the US. Land of the free? Way to win the ‘war’, folks.

The weirdest thing was the layout of the exit from the security area. You get funnelled through the X-ray machines, and then there were no obvious pointers to the exit. People just kinda milled about before stumbling across the exit. This created a blockage, and slowed the whole thing down.

In Australia, there’s a clear exit path, so it’s easy to get out after you’ve gone through all the probing and prodding. Perhaps our significant experience with sheep dipping has prepared us for designing a better system?

One final question: what are the success criteria for winning the War on Moisture?

Boarding

The boarding algorithm for planes is suboptimal. For a moment, I thought US Air were trying to use a better one by boarding in ‘zones’. I figured, ok, yeah, board from the back of the plane to the front, sure. Makes sense.

Nope. I couldn’t detect any relationship between the calls to board by zone and any kind of optimisation of the boarding algorithm.

I timed it. It took 25-30 minutes to get a 737 (25 rows of 6, 3 each side) completely boarded, with no late passenger delays.

In Australia, boarding takes half that, and it’s still not optimal. What on earth is going on?

Also, the draconian checked back limits means people have metric assloads of carryon, so the overhead space fills up in seconds. Maybe that’s the cause of huge boarding times: people searching for somewhere to store their bag?

I just bung my laptop bag under the seat in front. Saves a bunch of hassle. But I’m short and don’t need the extra legroom. One advantage, at least.

Finding Your Way Around

Airports in the US are much the same as in Australia, with one exception: signage.

I found it trickier to locate things like bathrooms and baggage collection. Perhaps I don’t have the same social cues that US folks are used to, but I had to really concentrate on locating the appropriate signage to point me to where I needed to go. Often there’d be one sign saying “it’s up this way”, but that would be it. I couldn’t find the Qantas club lounge in terminal 3 at LAX at all.

Asking official airport people gave mixed results. Several times I was advised to go a certain way, and I did, only to find 25 metres later that, no, I should have gone the other way. Or, yes, I could have also gone the way I asked about, but I got sent the long way instead for some unknown reason.

Most odd was the sheer number of people supposedly there to help you. I’d guess at 2-2.5 times the number of people doing the same job as a single person in Australia. I don’t know if that’s because Australia is understaffed, or if the low minimum wage in the US makes hiring more people cheap, though I strongly suspect the latter.

However, there’s one thing I absolutely loathe about Melbourne International: the duty-free store is directly in the way after you clear customs on the way out. It’s like walking directly into the perfume department of a big department store, at the same time as 200 other people.

You have to wend your way between all the merchandise before, finally, after 50 metres, you’re in relatively clear space. Ugh.

At least on the way back in there’s a corridor on the side that bypasses all the displays.

Conclusion

I can totally understand why US airlines don’t make money. The experience sucks.

The Australian experience isn’t as fun as it used to be, but it’s still a notch or two above ’sucks’ most of the time. The US experience basically defaults to ’sucks’, and dips below that more often than it rises up to ‘tolerable’.

Which is a huge shame, because the US is mostly a nice place that I’d like to see more of. The prospect of having to fly around it kinda puts me off, though.

So bravo to the Australian domestic airlines, and a supporting bravo to the airports themselves. You’re doing much better than your US compatriots.

No related articles.

It’s an All Innovation, All the Time mega-festival.

Or not.

No, It’s Not

Most of what’s happening isn’t that innovative. It’s incremental improvements that are largely obvious, but still good and useful. Nothing wrong with that.

And sure, doing anything even slightly new can be lumped under the catchall term ‘innovation’ if you want to. It’s a very shiny bandwagon, so why not jump on? You’re totally a unique and innovative slowflake. Just like everyone else.

More insidious is the pointless re-implementation of existing ideas, but in New Fashion Colours! and calling that innovation.

So you’ve discovered Karl Marx or Ayn Rand? Wow. No, I’m sure no one has ever though of applying those ideas to how people live their lives today. Yeah, you should totally start a blog about that.

Spare me your derivative Manifesto, Mission Statement, Vision or Information Product (How To Twitter Your Way to Total Happiness, available for a strictly limited time at the low, low price of just $97!)

Innovation? Bah.

The Infection Spreads

This culture of the Perpetual New has infiltrated even the most conservative of public institutions. What were once staid and traditional places (politics, banking and finance) are now seemingly overrun with innovation pundits.

Derivatives caused us all kinds of problems. We need better, faster derivatives! We have the technology, we can rebuild the world economy!

Doing things well using the stuff we have is old hat. We need to be New! Vibrant! Exciting!

What new piece of software can we buy to solve all our problems? What about Silver Bullet 2.0?

If I wear this new dress, maybe my customers will like me again?

The Focus is Wrong

Now you might be thinking that I’m becoming a neo-Luddite (my contribution to innovation. You heard it here first!). Maybe, maybe not.

I’m a big fan of purposeful change. I think getting rid of old, broken things in favour of new, working things is great. I want things to be improved upon. I want things to suck less.

That’s not what’s happening.

My issue is the neglect of things that were working quite decently until they started to get ignored. Things require maintenance, and when you neglect them because you’re spending all your time on building new things, the old things start to fall apart.

The fixation with newness is causing resources that used to be spent on maintenance to be spent instead on New Shiny things for no good reason other than that they’re new and shiny.

And then people act surprised when the old stuff stops working properly. But hey, that gives me an idea! An innovative idea!

Why not just throw out the old stuff that’s breaking and replace it with this shiny new stuff!?

What could possibly go wrong?

Shoulders of Giants

Here’s what the smart people are doing. Quietly, because that’s their style. You won’t hear about this in the papers, because it’s not New and Exciting:

They’re using the stuff they already have!

The smart folks look at the problem they need to solve, and they first try to use the stuff they already have to solve that problem. If they can, great, problem solved. Total cost: $0.

Maybe they need to change the way they use the stuff they already have a little. A bit of tweaking, problem solved. Total cost: $tiny

Maybe they have most of the solution, and an upgrade here, an extra part there, and bam! Problem solved. Total cost: $small

They don’t look at a problem and immediately decide that they need to completely rebuild the whole factory for total cost: $bignum without first seeing if there’s a way to reuse what they’ve already paid for.

Sure, it’s not as exciting. It may not look as dramatic on your CV. But it works, it’s faster, and it’s cheaper. And then you have a bunch of free time and cash to spend on being really, truly innovative. And you’re still making money from the job you were supposed to be doing all along.

If you’re competing with the smart folks, and you’re spending $bignum while they spend $small to solve the same problems, who’s being the most innovative, do you think?

Here’s a hint: It isn’t you.

Like this one? You might like these too:

• A Solution Is Not A Product
• Why You Should Stop Using Multiple Redo Logs with Oracle on NetApp
• I Believe You Have My Stapler
• What’s in a Name?
• Infrastructure Isn’t Agile

It reminded me that snapshots, that most basic of NetApp features, really are very cool. When something awesome (like lightbulbs) are part of your every day, it’s easy to forget just how awesome they really are.

So here’s a primer on NetApp snapshots for those who haven’t used them, and a reminder of their awesomeness for those of us who take them for granted.

Basic Snapshots

In the beginning, snapshots were pretty simple: a backup, only faster. Read everything on your primary disk, and copy it to another disk.

Simple. Effective. Expensive.

Think of these kinds of snapshots as being like a photocopier. You take a piece of paper, and write on it. When you want a snapshot, you stop writing on the paper, put it into the photocopier, and make a copy. Now you have 2 pieces of paper.

A big database might take up 50 pieces of paper. Taking a snapshot takes a while, because you have to copy each page. And the cost adds up. Imagine each piece of paper cost $5k, or $10k.

Still, it’s faster than hand-copying your address book into another book every week.

It’s not a perfect analogy, but it’s pretty close.

Copy-on-Write Snapshots

Having to copy all the data every time is a drag, because it takes up a lot of space, takes ages, and costs more. Both taking the snapshot, and restoring it, take a long time because you have to copy all the data.

But what if you didn’t have to? What if you could copy only the bits that changed?

Enter copy-on-write snapshots. The first snapshot records the baseline, before anything changes. Since nothing has changed yet, you don’t need to move data around.

But as soon as you want to change something, you need to take note of it somewhere. Copy-on-write does this by first copying the original data to a (special, hidden) snapshot area, and then overwriting the original data with the new data. Pretty simple, and effective.

And now it doesn’t take up as much space, because you’re just recording the changes, or deltas.

But there are some downsides.

Each time you change a block of data, the system has to read the old block, write it to the snapshot area, and then write the new block. So, for each write, the disk actually does two writes and one read. This slows things down.

It’s a tradeoff. You lose a bit in write performance, but you don’t need as much disk to get snapshots. With some clever cacheing and other techniques, you can reduce the performance impact, and overall you save money but get some good benefits, so it was often worth it.

But what if you didn’t have to copy the original data?

NetApp Snapshots

NetApp snapshots (and ZFS snapshots, incidentally) do things differently. Instead of copying the old data out of the way before it gets overwritten, the NetApp just writes the new information to a special bit of disk reserved for storing these changes, called the SnapReserve. Then, the pointers that tell the system where to find the data get updated to point to the new data in the SnapReserve.

That’s why the SnapReserve fills up when you change data on a NetApp. And remember that deleting is a change, so deleting a bunch of data fills up the SnapReserve, too.

This method has a bunch of advantages. You’re only recording the deltas, so you get the disk savings of copy-on-write snapshots. But you’re not copying the original block out of the way, so you don’t have the performance slowdown. There’s a small performance impact, but updating pointers is much faster, which is why NetApp performance is just fine with snapshots turned on, so they’re on by default.

It gets better.

Because the snapshot is just pointers, when you want to restore data (using SnapRestore) all you have to do is update the pointers to point to the original data again. This is faster than copying all the data back from the snapshot area over the original data, as in copy-on-write snapshots.

So taking a snapshot completes in seconds, even for really large volumes (like, terabytes) and so do restores. Seconds to snap back to a point in time. How cool is that?

But wait, there’s more.

Snapshots Are Views

It’s much better to think of snapshots as a View of your data as it was at the time the snapshot was taken. It’s a time machine, letting you look into the past.

Because it’s all just pointers, you can actually look at the snapshot as if it was the active filesystem. It’s read-only, because you can’t change the past, but you can actually look at it and read the data.

This is incredibly cool.

Seriously. It’s amazing. You get snapshots with almost no performance overhead, and you can browse through the data to see what it looked like yesterday, or last week, or last month. Online.

So if you accidentally delete a file, you don’t have to restore the entire G:, or suck the data off a copy on tape somewhere. You can just wander through the .snapshot (or ~snapshot) directory and find the file, and read it. You can even copy it back out into the active file system if you want.

All without ringing the helpdesk.

Celebrate the Humble Snapshot

I’m really glad I had to write this up for people new to NetApp, because it’s given me a renewed appreciation for this most basic feature.

Have you been taking your snapshots for granted?

Like this one? You might like these too:

• Why You Should Stop Using Multiple Redo Logs with Oracle on NetApp
• Background on NetApp Dynamic Data Center
• Administer Your NetApp Filers With Python
• Who Runs The Show?
• ModiPy update: Automated NetApp simulator installation

In a small company, the IT department consists of one or two admins who do everything. They look after the database, the network, the PCs, the phones. Everything. The one guy is always wearing the hat.

In a large company, there are dedicated teams of people who do nothing but respond to automated alarms. There are lots of hats, being worn by different people.

Dealing with both of these markets simultaneously has been a significant challenge for vendors thus far. Their tools are continually confused by who needs to be wearing which hat at what time.

Here’s why.

Empower the Users

Empowerment sounds great. I mean, you’re giving people power, right? Giving end users the power to look after themselves has got to be good, right?

Well, maybe. But not always. Because self-service means I have to know how to do things. Sure, I could change my own oil, but I’m a techo who would rather go to work and earn money than spend three hours of my weekend doing what I consider to be work that I don’t get paid for and don’t enjoy.

Yet this is how many vendors design the administrative interface for their tools. As the user, you can look after yourself. Even if you don’t want to.

I want to be able to give my hat to someone else sometimes.

You Break It, You Bought It

So what if I screw up and break something? If it’s my car, it’s my problem. I know it’s my fault, and I pay the penalty.

But what if I didn’t get a choice about whether I had to look after my own stuff? What if I never got the right training in how to use the tools?

How might I feel when the storage guys call me to ask why I deleted the base snapshot? Or filled up the snapreserve because I did a data load last night?

I didn’t ask to be given this hat. And it doesn’t fit properly. I hate this hat.

The User is the Admin

In a small company, the database guy is the storage guy. If I need to create a new LUN for the database using SnapDrive, I can. Awesome.

If I need to check that the overnight snapshot backup worked ok, I can log into NetBackup and see for myself. Easy.

If I want to fail the system over to DR to check the process works, I can log in and kick it off myself.

Coordination is easy when I’m coordinating with myself. I always know who’s wearing the hat, because it’s me.

The User is Not the Admin

If we scale up to a big company, now the user isn’t the admin. Remember our dedicated teams? The database is looked after by a completely different team to the storage team.

In fact, the database team might report to a completely different level three manager. They might even work for a completely different company, because we outsourced.

So now how do I deal with the empowerment? If I’m using SnapManager^[1], the DBAs now manage their own backups, but I still have to grow the volumes, or create new vFilers.

Or if the application guys do a data load and use up all the snapreserve space, so it grows into active filesystem and runs out of space, and then the database falls over, and I get to fix it.

Because the DBAs and app guys don’t look after the storage. That’s my job. I have the hat.

Who’s Job Is It?

I don’t want to give the database guys full administrative control of the storage, because they’re database guys, not storage guys. They don’t have the training, experience, or more importantly, the time to admin the storage the way the storage team wants it done. They’re busy running AWR reports, creating indexes and doing statistics updates.

I’m busy configuring SnapVault relationships and figuring out where to put new volumes. I don’t want to have to create new database indexes, or do reorgs. That’s the DBA’s job.

They run the database show.

I run the storage show.

Division of labour is one of the biggest contributors to productivity of all time. So why do vendors keep trying to undo all that by assuming that the same person will be doing all the admin tasks?

Why does there have to be just one admin hat?

A Better Way

You don’t have to choose between one way or another. You can have both.

How? By thinking in terms of hats.

When I’m growing a volume, I’m wearing my storage admin hat. When I’m taking a database backup, I’m wearing my DBA hat and my backup hat.

So when you’re designing an administrative tool, keep in mind which hat you need to be wearing when you use it. Make it easy for me to change hats.

Provide Role Based Access Control that allows different people, or the same person wearing different hats, to do different things.

Check your assumptions constantly. Can the database system do this via the vFiler, or do they have to access vFiler0? Can I delete a qtree as easily as I can create one?

It’s far, far easier for me to put on three hats than to have three people wearing one hat.

Because whoever’s wearing the hat, runs the show.

^[1] I’m picking on NetApp in this post, but I know the other vendors have the same problems, so don’t go crowing about this on twitter. You know who you are. All hardware sucks, all software sucks. D,NA.

Like this one? You might like these too:

• What is a Project, Exactly?
• How to Install NetApp DFM on Ubuntu
• All Things NetApp
• Background on NetApp Dynamic Data Center
• Why You Should Stop Using Multiple Redo Logs with Oracle on NetApp

If you grabbed the document from my last post about the NetApp Dynamic Data Center, you’ll have noticed the concept of a Project.

Figuring out what exactly a Project is can be one of the hardest parts of using this architecture.

Small, Medium or Extra-Large?

The problem stems from each organisation running their business differently from others. It’s hard to do one-size fits all when there’s such a wide range of sizes.

Some businesses have applications that are basically silos. They don’t share any information with each other. If they do, it’s through some sort of specialised brokered channel.

Other businesses have applications that are more tightly coupled, sharing data with each other all the time, often in a fairly informal manner.

You can use this model with either end of the spectrum, but it’s important to get your idea of a Project straight in your head first.

Let me explain.

It’s All About the vFilers

The vFiler is the core piece of this system. You want everything that’s connected to a given vFiler to be able to share data.

If you have two hosts connected to the same vFiler, sharing data between them is relatively easy. If they’re connected to different vFilers, you have to do some more complicated network things to make it happen. More complex is bad, because it makes it harder to look after.

The whole point of using this stuff is to get economies of scale by doing things in standard ways. If you go breaking the model, well, why did you buy it in the first place?

Each vFiler belongs to one storage VLAN at a time. This is what gives you your secure isolation from other projects. It’s a feature. The trick is deciding on the level of isolation you want.

Which brings you back to the vFiler, and what needs to share.

The simple rule is: if you want to share data, ever, you’re in the same vFiler. If you put things in different vFilers, you’re explicitly preventing things from sharing storage.

Everything flows on from that.

Types of Sharing

Here are some examples to make this easier to understand:

Vertical Isolation

• The company website; a 5-way n+1 tier of webservers, 3 app servers and a database server, are all on one vFiler. This is one Project.
• The payroll system; 3 application servers and a clustered database, are on another vFiler. This is another Project.
• The Exchange servers for company mail, and the Symantec Enterprise Vault servers, have another vFiler. This is your email Project.

Here we have isolation vertically, by application.

We can share data between the database servers and the application servers via the storage. Some companies write their apps like this so they can write static HTML out from the application servers directly to the filesystem used by the webservers to serve traffic.

Some of the downsides of this approach are issues with multiple-physical-site hosting, and a proliferation of ‘projects’ requiring a lot of vFilers, VLANs, and networks to be set up.

But you get high security, and problems within one vFiler/VLAN/network only affect one application.

Horizontal Isolation

• All the webservers are in one vFiler. A webserver Project.
• Internet facing application servers are in another vFiler. Internet app Project.
• All internal Solaris servers share another vFiler. Solaris Project.
• All the Windows SQL Server boxes are on another vFiler. SQLServer Project.

Here we have isolation horizontally by server function.

This can work really well, because you can isolate some servers that are in low-security networks (Internet facing, DMZ), and you can share common data between servers with the same function. Boot-from-SAN for all your Unix servers, great de-dupe of common files, a common binaries area (a couple of copies of Oracle instead of one per server, for example), there are a bunch of neat applications of this approach.

The downside is lower security isolation, though there are some techniques to help with this. You also have more eggs in your vFiler baskets, so you need to take better care of them.

Plan, Then Execute

A good place to start is to look at who the administrators for the servers are. They have root access, so they can munge anything on the servers anyway. Group all the servers with a common administration team together. Keep your internet and DMZ servers separate from everything else.

The important thing to remember is that you need to make a decision of which approach to use before you start. These two methods are quite different, and you can’t use both at the same time.

If you start with one and then decide to change 8 months later, you’ll have a bunch of production servers that need to have their storage designs changed. Bummer.

It’s rare that you’ll be able to do that without incurring an outage. And it’s an outage that has almost no value to the businesses who are using their applications quite happily, thank-you-very-much. Way to win friends and influence people, dude.

Instead you have a bunch of legacy stuff using one method, and a bunch of newer stuff using a different method. And your Ops Manager hates you.

Seek Help

This is enterprise cloud computing we’re talking about, so you’re not expected to know exactly what to do. Seek the advice of people who’ve done it before.

Get a hold of your NetApp rep and get them to bring in someone from their DDC/cloud team. I know quite a few of them, and they’re smart, helpful people.

If you can, talk to existing customers about their experiences.

Or, if you get desperate, you could email me.

But if you do nothing else, at least take a few minutes to think about how your organisation shares data.

Like this one? You might like these too:

• Background on NetApp Dynamic Data Center
• Who Runs The Show?
• All Things NetApp
• Why You Should Stop Using Multiple Redo Logs with Oracle on NetApp
• Fix high CPU usage in Ubuntu 9.04 Jaunty

The Beginning

You’ll want to start here.

More specifically, you’ll want to read this paper on the solution at Telstra. That’s where it all began.

Go get it. I’ll wait. It’s only 4 pages, and page 3 is the one I want you to look at.

Let me draw your attention to the picture on page 3. That’s the guts of the thing, right there.

Horizontally scalable. Vertically scalable. Secure partitioning.

1.7 Petabytes of storage when it was published in early 2008. It’s much bigger by now.

That was a fun project.

I was reminded by a colleage recently that one of the great things about it was how we used (relatively) old technology to build something new and cool.

Quick Rundown

The switch gear is all Cisco, like it says. Gigabit or better, and dedicated for storage. You pick the models that are appropriate for your environment, and the fan-out ratio that suits.

A lot like your more traditional FibreChannel SAN, right? Hosts connect to it, only you don’t need special FC HBAs at $1k a pop. Standard Ethernet cards, times thousands of servers. Cost savings, right there.

NetApp Filers at the backend. Whichever ones you want. FAS 270 enough for you? Cool. Need the big 6080s? No problemo. They all run the same DataONTAP, and they all talk Ethernet, so you can mix and match depending on what you need. That’s really neat.

And ONTAP supports CIFS, NFS and iSCSI over Ethernet, so you get 2 types of file and 1 of block storage over the same wires, with no special translator droids to slow everything down. With FCoE, you can add FibreChannel. I don’t see the need, but hey, some people like it.

Need more storage? Add more Filers. Need more hosts? Add more switches. It really is very easy. I’ve done it, as have lots of other people.

Oh, and because it’s all NetApp, if you decide the 270 just can’t hack the load you’ve dumped on it, you can do this and move it all to the 6080.

The Grey Bit Is Important

Enough breathless hype.

You know how GUIs often grey out the parts you can’t use? Not the case here. The grey ellipse bit is the most important thing in that picture.

Let’s take a closer look at the grey bit, though.

The caption is VLAN + vFiler + volumes. Let’s see what that means:

VLANs

So it’s a layer 2 network. There is sooo much to write about this one area. For now, think about how VLANs work. If you don’t know, I encourage you to read up.

VLANs are really old technology, as far as networking goes. In the same way that hypervisors now give us multiple virtual machines on a single physical server, VLANs give us multiple virtual LANs over the same physical wires, instead of having to cable up each computer multiple times in order to isolate different kinds of traffic.

And that’s what they’re used for here. They logically isolate different project’s storage traffic from one another, so a web application in a project over on the left can’t access payroll data in the database project on the right.

That’s not the only bit of virtualisation going on here.

vFilers

NetApp’s MultiStore feature gives you virtual Filers. In the same way as VLANs gave us multiple LANs on the same piece of wire, vFilers give us multiple Filers on the one physical chassis.

As far as each project is concerned, the vFiler looks just like their very own Filer. They get storage over NFS, or CIFS, or iSCSI, over their storage network (in a VLAN, remember!) so it looks just like they have their own dedicated hardware.

Only they don’t. Which is heaps cheaper and easier to manage. You can buy a bunch of disk in a single, say, FAS3050A, and then slice it up into virtual pieces, and give a slice to each project.

Volumes

Back in the 6.5.x days, TradVols were a tad unwieldy, but still better than mucking about with host based volume managers. When FlexVols arrived, wow. Soo much better.

You just buy a whole mess of disks, and put them into dirty big aggregates. Lots of spindles == better performance. And then you slice up the aggregate into volumes. All the volumes get to benefit from the high spindle count, so they all get to go faster.

And you can have different volumes on the same aggregate belonging to different vFilers. So you might only have 2 aggregates, but you can have 20 projects all sharing disk in the back end, but all getting the benefit of 50-odd spindles (model dependent, of course). And they still all look like they have their own storage.

RAID-DP protects the data, and a few spare disks gives you hot swap. Fault tolerance with great performance. What’s not to love?

How Is This Cloudy?

When you build a system like this, the storage and network become commodity pieces. Cloud computing is all about economies of scale.

The end customer doesn’t need to care what model of Filer their storage lives on. They don’t even need to know which physical Filer it lives on. You can move stuff around, add more, take some away, with far less disruption to the end customer than if they had dedicated gear.

Half a dozen customers (or more) can all be sharing hardware and never even know. Because they don’t have to.

Now they can get on with writing new applications instead of managing storage.

But Wait, There’s More!

More fun stuff coming up in future posts. It’s not all smooth sailing, so I’ll give you some pointers on how to avoid the traps for new players.

TrendsMap don’t use NetApp storage, as far as I know. A guy I know worked on the site, apparently, and I just thought it was cool and wanted to share it with you.

Image by TopTechWriter at Flickr

Like this one? You might like these too:

• ModiPy update: Automated NetApp simulator installation
• What is a Project, Exactly?
• Administer Your NetApp Filers With Python
• All Things NetApp
• Why You Should Stop Using Multiple Redo Logs with Oracle on NetApp