SysAdmin1138 Expounds

Bringing the Head Committee to order

2012-05-16T15:40:42Z

A client has just asked to move a maintenance window to some other time due to a sudden emergency. The in-house Project Managers, being the customer-service oriented people that they are, have just asked if it is possible to move the window to something less likely to get in the way. You are staring at the reply screen to the email.

Scene

A few people clustered around a table. The Herald of Duty has just departed, having delivered the demand that a decision needs to be made. A gavel sounds.

Bleeding Heart: Really? Well then. We'll move it.

Grumpasarus: For ****** sake, we had contractual a maint-en-ance win-dow for a ****** reason! ******, we took a quarter of what we COULD have! Tell 'em to ***** off and lump it. They were warned!

Schemer: Yeah, really. We told them four weeks ago and now it's a problem?

Slacker: Dude, I planned my day around that. Moving the work totally ruins my schedule

Voice of Reason: Well, it's a problem they have now and didn't know about then.

BH: Think about it! They don't know their emergencies four weeks in advance. It's not fair to them to have a problem only to have us unavailable. We owe it to them, it's the humane thing to do.

Grumpasarus: So ****** what? Not OUR problem.

Schemer: You know.... if we play our cards right we just might get a special gift from the client. Hm.

Grumpasarus: Who ****** cares?? It'll just go to the PM's not us.

VoR: I point out that the PMs have been known to be effusive in their praise when we do favors.

Slacker: But moving the window means I'll lose sleep. It no workie for me.

Schemer: You know, if we push back a little now and then give in, the chances of something special coming our way get higher.

Grumpasarus: Really? Like what?

Schemer: Clients have been known to send us things like large boxes of chocolate, cases of beer, or comps to swank restaurants.

Grumpasarus: Looks thoughtful.

Slacker: DUUUUUDE! Loss. Of. Sleep. Hello!

VoR: You can take a nap later.

Slacker: But I was going to play Skyrim all day, if I nap I won't get as much!

Schemer: Did I mention the Chocolate?

Slacker: But that's like, work and stuff. No, dude.

VoR: You'll nap less than the sleep you lost, so you'll actually get more Skyrim in.

Slacker: ... good point.

Schemer: Looks to the awaiting Page, We have come to a decision! We will push...

BH: Wait! This is the wrong course of action. You have to work with these people, and pushing back now will make you seem like you're hard to work with.

Schemer: And that means they have to buy me off. I don't see the problem with this.

BH: Then they won't tell you things.

VoR: And when they don't tell you things, you get cut out of decision loops.

Grumpasarus: For ****** sake just tell them to ****** lump it already. Or deliver the ****** chocolate. It's not like this'll get done without us.

Slacker: But that's, like, conflict and stuff. I don't like that.

Grumpasarus: ****** ****** you too? *******! Grow a ****** backbone!

Slacker: Not my job, man.

BH: Going along with this now will make us look reasonable, and willing to work around our customer's needs. Be a team player.

Grumpasarus: ****** team player. We ****** told them ****** four weeks ago that we'd ****** take the ****** system down. They can ****** well bend over and take it. We notified them! This window is in the ****** contract they ****** signed and ****** well didn't read.

VoR: I remind you again. When the notice for this outage went out, they didn't know about this problem. And secondly, the contracted window is much larger than the one we gave.

Schemer: I still say we should push back a little. This is last minute, there should be some costs.

Grumpasarus: ****** right there should be!

BH: Pushing back makes us look hard to work with.

Schemer: I really have to council against blithely rolling over on this one. We don't want to get to a spot where others can dictate our schedule. It'll mean dark-of-night for everything.

Slacker: I hate that!

Schemer: I know. And as we're East Coast and have West Coast clients, it'll mean morning outages not late-night ones.

Slacker: Duuuuude. No.

Schemer: I know. They do need to be made aware that this is not a trivial sacrifice on our part.

Slacker: Sleeeeeep! Skyrim!

BH: They already are aware. Look at the email. See? They know.

Grumpasarus: That's just ****** 'customer service' weasel-words.

Schemer: Hm, I'm not so sure. That sentence could go either way. I'm willing to be persuaded this once and see how it goes.

VoR: Fred has usually played straight with us in the past.

Grumpasarus: You're ***** serious??

VoR: Yes. Always.

Slacker: No. Uh uh. We gotta push a little. Make 'em think twice before asking to totally blow our schedule.

Grumpasarus: Better! Tell them to ****** off!

Schemer: No, we can still make some hay with this. Push a little, not a lot. Just enough.

BH: I really wish we didn't have to push back.

Grumpasarus: We don't ****** have to. We just tell them ****** no. No pushing, just ****** standing our ****** ground.

Slacker: I, uh, don't want to say no outright. It'll get us in trouble later. So kinda no?

Grumpasarus: For ****** sake, grow a ****** backbone you ****** lazy ****** ****** ******. ***** this. Stomps off in a huff.

BH: Points at the retreating Grumpasarus. That's an abstention!

VoR: I agree.

Slacker: Turns to waiting page, We have a decision....

To: Fred Gerkin
Subject: RE Pushing the maintenance window

Hi Fred,

We warned 'em, but I do understand emergencies. I had plans, but I'm pretty sure I can get them moved. The new window will be 4 to 7am. I'll let you know if I can't flex my plans.

Coming up for air: moving an office

2012-04-30T11:32:19Z

Yes, I've been light on the blogging this month. I've been helping to move an office, and it takes quite a lot of effort. Chief among the work contributers being that we have two (2) support-type people in the DC office, and I'm one of 'em. Where in this case 'support' means 'supports the office's primary function', which in larger offices would include:

The Facilities team who keep everyone in heat, cooling, and power, as well as put a damper on structural decay
The network engineers who keep the phones and data going
The office-admins who make sure all the various supplies are kept up to stock, usually invisibly.
The people-facing IT team who fix broken computers

Our DC office has at most 12 people in it during a normal day, only two of whom fit into the above. Me and the person who fills the third bullet point up there. She was doing 9/10ths of the job of getting the new office built, ordering movers, coordinating new furniture, talking to the contractors whenever they hit a snag, and all that stuff. Me? I'm the Speaker to Networks. This is a new role for me, but I've got enough depth in the field I can fake it reasonably.

I'm also better-than-average at the whole HVAC thing, what with HVAC being a primary concern for server health. However, that only came up during the 5 times I got asked "you want HOW much cooling in that closed room?" (4KVA if you must know). They kept not believing me, even after I pointed to the pair of L5-30 outlets they were going to install. Where's that power going, huh? I'd rather over-cool than under-cool, you know.

But... we're in. Even though this move was a true tragi-comedy of errors, the old hands assure me that the previous one was worse. I'll have to get THAT story once the raw wounds from this one heal over a bit more.

I'll probably post pictures once I've had more sleep. 19 hours of work since close-of-business Friday.

Poorly phrased discovery requests are expensive for everyone

2012-04-13T15:19:52Z

Kind of an obvious statement, but an object lesson has been provided. In the linked case, the discovery request included the phrase "unallocated space" and included keywords with general meanings.

The result? An unmanageable wad of data only scant slivers of which were 'responsive', and would cost well over a million dollars to find it.

"Unallocated space" is what gives me the shivers. That would require sorting through the empty spots of partitions looking for complete or partial files and producing those files and fragments. I know WWU wasn't equipped for that kind of discovery request, and we'd be knock-kneed about how to handle "unallocated blocks" on the SAN arrays themselves. It would suck a lot.

And in this case it cost quite a bit to produce in the first place.

But this also shows the other side of the discovery request, the expense of sorting through the discovered data. My current employer does just that; pare down discovered data into the responsive parts (or make the responsive parts much easier to find during manual review). And yes, it costs a lot.

Pricing is somewhat complicated, but the dominant model is based on price-per-GB with modifiers for what exactly you want done with the data. OCR costs extra. Transformation into various industry-common formats costs extra. That kind of thing. The price has been dropping a lot lately, but it's still quite common to find prices over $200/GB, and very recently prices were hovering around $1,000/GB.

Many sysadmins I know pride themselves in their ability to phrase search queries into Google to get what they're looking for. It doesn't take long to locate exactly what we're looking for, or some hint on where to look next.

Lawyers have to get the search query right on the first try. Laziness (being overly broad) costs everyone.

Parallel vs Distributed Computing

2012-04-04T01:01:38Z

Inspired from a closed ServerFault post:

I was just wondering why there is a need to go through all the trouble of creating distributed systems for massive parallel processing when, we could just create individual machines that support hundreds or thousands of cores/CPUs (or even GPGPUs) per machine?

So basically, why should you do parallel processing over a network of machines when it can rather be done at much lower cost and much more reliably on 1 machine that supports numerous cores?

Well.

The most common answer to this is really, Because it's generally far cheaper to assemble 25 4-core boxes into a single distributed system than a single 100-core box. Also, some problems don't parallelize well on a single system no matter how large it gets.

A Distributed Computing system is a loosely coupled parallel system. It generally does not present as a single operating system (a "single system image"), though that is an optional type. Latencies between processing nodes are very high, measurable in milliseconds (10^-5 to 10^-2) or more. Inter-node communication is typically done at the Application layer.

A Parallel Computing system is a tightly coupled parallel system. It presents as a single operating system. Processing nodes are very close together; on the same processing die or a very short distance away. Latencies between processing nodes are slight, measurable in nanoseconds (10^-9 to 10^-7). Inter-node communication is typically done at the Operating System layer, though Applications do have the ability to manage this if designed to.

Looking at those two definitions, you'd think the parallel computing system would win every time. But...

There are some workloads that don't behave nicely in a single system image, no matter how many cores are present. They tie up some key resource that blocks everything else. Each process consumes huge amounts of RAM. Multiple instances running in parallel don't behave nicely. Any number of things can prevent a massively parallel system from doing it's thing. In such cases it makes very clear sense to isolate these processes on smaller systems, and feed them work through a distributed work-management framework.

What kinds of things can "block key resources"?

The application requires a GUI element to be presented. This is not an application designed for highly parallel systems, but people do weird things all the time. Such as an Email-to-PDF gateway that utilizes MS Office for the rendering.
The application is attached to specialized hardware that is single-access only (an ancient ISA card, or a serial-connection to scientific gear).
The application uses hardware that generally only handles one or two devices per install, at least without extremely expensive engineering (GPU computing).
Each running instance of the application requires a hardware dongle due to hard-coded licensing requirements.

Also, distributed systems have a few advantages over their closely coupled brethren. First and foremost, it's generally a lot easier to scale them out after the initial build. This is very important if the workload is expected to grow over time.

Secondly, they tend to be more failure-resistant. If a single processing node fails, it does not always cause everything else to come to a halt. In a massive parallel computing system, a failure of a CPU or RAM can cause the whole thing to stop and that can be a lot of lost work.

Third, getting 1000 cores of processing power is generally a lot cheaper if that's done in 125 discrete 8-core servers than if it is done in a single box. If the problem being solved can tolerate a loosely coupled system, there are a lot of economics that make the distributed route very attractive. As a rule, the less custom engineering you need for a system, the cheaper it'll be.

And that is why we build large distributed systems instead of supercomputers.

Moving /tmp to a tmpfs

2012-04-01T00:29:36Z

There is a move afoot in Linux-land to make /tmp a tmpfs file-system. For those of you who don't know what that is, tmpfs is in essence a ramdisk. OpenSUSE is considering the ramifications of this possible move. There are some good points to this move:

A lot of what goes on in /tmp are ephemeral files for any number of system and user processes, and by backing that with RAM instead of disk you make that go faster.
Since a lot of what goes on in /tmp are ephemeral files, by backing it with RAM you save writes on that swanky new SSD you have.
Since nothing in /tmp should be preserved across reboots, a ramdisk makes sense.

All of the above make a lot of sense for something like a desktop oriented distribution or use-case, much like the one I'm utilizing right this moment.

However, there are a couple of pretty large downsides to such a move:

Many programs use /tmp as a staging area for potentially-larger-than-ram files.
Some programs don't clean up after themselves, which leaves /tmp growing over time.

Working as I do in the eDiscovery industry where transforming files from one type to another is common event, the libraries we use to do that transformation can and do drop very big files in /tmp while they're working. All it takes is one bozo dropping a 20MB file with .txt at the end (containing a single SMTP email message with MIME'ed attachment) to generate an eight thousand page PDF file. Such a process behaves poorly when told "Out Of Space" for either RAM or '/tmp.

And when such processes crash out for some reason, they can leave behind 8GB files in /tmp.

That would not be happy on a 2GB RAM system with a tmpfs style /tmp.

In our case, we need to continue to keep /tmp on disk. Now we have to start doing this on purpose, not just trusting that it comes that way out of the tin.

Do I think this is a good idea? It has merit on high RAM workstations, but is a very poor choice for RAM constrained environments such as VPSs, and virtual machines.

Are the benefits good enough to merit coming that way out of the box? Perhaps, though I would really rather that the "server" option during installation default to disk-backed /tmp.

Tape!

2012-03-29T16:47:37Z

Tape isn't going away, much like mainframes never actually went away. However, their utility is evolving somewhat.

The emergence of the Linear Tape File System is quite interesting. It's an open tape format (nothing new there) that looks to have broad acceptance (the new part). Especially since the LTO governing body has adopted it, and LTO is the de-facto standard for tape in the industry right now.

Open standards make the long-term archive problem easier to tackle, since the implementation details are widely understood and are more likely to either still be in use or have industry expertise available to make it work should an old archive need to be read. They also allow interoperability; a "tier-5" storage tier consisting of tape could allow duplicates of the tape media to be housed in an archive built by another vendor.

In my current line of work, a data-tape using LTFS would be a much cheaper carrier for a couriered TB of data than a hard-drive would. We haven't seen this yet, but it remains a decided possibility.

Understandably, the video content industry is a big fan of this kind of thing since their files are really big and they need to keep them around for years. The same technology could be used to build a computer-image jukebox for a homebrew University computer-lab imaging system.

Looking around, it seems people are already using this with CloneZilla. Heh.

It's hard to get excited about tape, but I'll settle for 'interested'.

A complaint

2012-03-25T23:38:34Z

This came across on a private twitter feed, otherwise I'd attribute the bejebers out of this one.

I know it's not my field, and probably not yours if you're a regular around here, but even I have noticed a steady uptick in ninja rockstars on job postings for people who do software engineering all day.

So, so true.

That said, it does bring up an opportunity for when you need a brush an employer off for being too skeevy.

"I'm sorry, it turns out I'm not a good fit for $Company. I'm more of a rockstar, not a ninja."

Time off

2012-03-10T13:50:05Z

I'm not going anywhere, but yesterday the following article was thrown under my nose:

Unlimited Vacation Doesn't Create Slackers -- It Improves Productivity

The base premise is very roughly:

If you let employees take the time they need, when they need it, they'll perform better than if you tracked (and metered) the time they are allowed to take.

True, dat. My last three jobs:

Current Job: Unlimited time off
Last Job: 5 weeks of vacation a year, 3 weeks sicktime.
Last +1 Job: Seniority based vacation times, starting at 2 weeks vacation + 3 sick, topping out at 5 weeks vacation + 3 sick for 20 year veterans.

Since my previous two jobs tracked time, both of them had the phenomena of 'use it or lose it' for when you hit the max time carry-over. Last +1 was December, Last job was at your hire anniversary. The fact that people were hitting their carry-over max shows that they were not using all of the vacation allotted to them (or alternately, were consciously building up a severance-pay pad due to vacation cash-out).

And I must say, when you work some place that's "hey whatever" about hours you get less stressed about work. And you also are tempted to 'work' outside your normal bounds. The article mentioned studies that show people working 'from vacations' or suchlike. They're out of the office (yay!) but still being productive. A win for the company, and a win for employee agency.

However, having lived with 5 weeks of vacation a year (or as I've called it in the past, "five freaking weeks of vacation.") the effect is almost the same as unlimited. You just take time off when you need it and don't sweat it. When your carry-over hits max, that's when the differences show up; we had people take two week 'call me if you need me, I'll be around,' vacations just to burn it off.

As a highly paid IT professional, I've shown I have sound judgment. It's nice that my employer recognizes that and treats me like I can be trusted with company-time management.

Also, the "Fool's Errand" idea is genius. I'll have to see if we can do something like that at work.

The cost of insecurity

2012-03-05T22:22:58Z

This rant is a familiar one to a lot of desktop-facing IT professionals.

Today the person who handles our bank stuff came to me with a problem. The check-scanner wasn't working. I poked around but couldn't make it work, and advised her to talk to the bank since they provided the scanner, the scanner driver, the IE-plugins it worked with, and Windows didn't recognize it as a scanner.

So she did.

Their advice?

You need to turn off Windows Update. Every time it runs it changes things in IE and you have to go through and do a bunch of things to make it work

I gave her the look. Then I remembered myself and redirected the look to a harmless corner until I could speak again. Even getting a piece of that baleful stare caused her to cringe. Oops.

Like all right-thinking Windows admins, I'm a believer in leaving Windows Updates turned on if I'm not doing something else to manage that particular risk. Our fleet of Windows machines is not big enough for me to bother with WSUS (we have a LOT of Mac users), and we most definitely do not do anything like blocking browsing at the border. So, I want those updates thanks.

So I pulled a laptop out of the dead pile. That laptop will now be a dedicated machine for talking to the bank and nothing else. All because our freaking bank can't run secureable software. Makes you question your trust in them, it does.

Going off half-cocked about graphical shells

2012-03-03T14:20:00Z

In the last month-ish I've had a chance to read about and use two new graphical shells that impact my life:

Windows 8 Metro, which I already have on my work-phone
Gnome 3

Before I go further I must point out a few things. First of all, as a technologist change is something I must embrace. Nay, cheer on. Attitudes like, "If it ain't broke, don't fix it," are not attitudes I share, since 'broke' is an ever moving target.

Secondly, I've lived through just enough change so far to be leery of going through more of it. This does give me some caution for change-for-change's-sake.

As you can probably guess from the lead-up, I'm not a fan of those two interfaces.

They both go in a direction that the industry as a whole is going, and I'm not fond of that area. This is entirely because I spend 8-12 hours a day earning a living using a graphical shell. "Commoditization" is a battle-cry for change right now, and that means building things for consumers.

The tablet in my backpack and the phones in my pocket are all small, touch-screen devices. Doing large-scale text-entry in any of those, such as writing blog posts, is a chore. Rapid task-switching is doable through a few screen-presses, though I don't get to nearly the window-count I do on my traditional-UI devices. They're great for swipe-and-tap games.

When it comes to how I interact with the desktop, the Windows 7 and Gnome 2 shells are not very different other than the chrome and are entirely keyboard+mouse driven. In fact, those shells are optimized for the keyboard+mouse interaction. Arm and wrist movements can be minimized, which extends the lifetime of various things in my body.

Windows 8 MetroUI brings the touch-screen metaphor to a screen that doesn't (yet) have a touch-screen. Swipe-and-tap will have to be done entirely with the mouse, which isn't a terribly natural movement (I've never been a user of mouse-gesture support in browsers). When I do get a touch-screen, I'll be forced to elevate my arm(s) from the keyboarding level to tap the screen a lot, which adds stress to muscles in my back that are already unhappy with me.

And screen-smudges. I'll either learn to ignore the grime, or I'll be wiping the screen down ever hour or two.

And then there is the "One application has the top layer" metaphor, which is a change from the "One window has the top layer" metaphor we've been living with on non-Apple platforms for a very long time now. And I hate on large screens. Apple has done this for years, which is a large part of why Gnome 3 has done it, and is likely why Microsoft is doing it for Metro.

As part of my daily job I'll have umpty Terminal windows open to various things and several Browser windows as well. I'll be reading stuff off of the browser window as reference for what I'm typing in the terminal windows. Or the RDP/VNC windows. Or the browser windows and the java-console windows. When an entire application's windows elevate to the top it can bury windows I want to read at the same time, which means that my window-placement decisions will have to take even more care than I already apply.

I do not appreciate having to do more work because the UI changed.

I may be missing something, but it appears that the Windows Metro UI has done away with the search-box for finding Start-button programs and control-panel items. If so, I object. If I'm on a desktop, I have a hardware keyboard so designing the UI to minimize keystrokes in favor of swiping is a false economy.

Gnome 3 at least has kept the search box.

In summation, it is my opinion that bringing touch-UI elements into a desktop UI is creating some bad compromises. I understand the desire to have a common UI metaphor across the full range from 4" to 30" screens, but input and interaction methodologies are different enough that some accommodation-to-form-factor needs to be taken.

On appropriate metrics, a tale of servers and fashion

2012-02-24T20:10:00Z

One of the questions that SysAdmins frequently get asked is:

I have a web application based on $Platform. It needs to support $NumUsers concurrent connections. How much server do I need?

$Platform can be anything from 'php' to 'tomcat' to the incredibly unhelpful 'linux'. $NumUsers can be anything from a reasonable number to completely unreasonable numbers representing the anticipated worst-case (or maybe that's 'best case') scenario (50,000 users! Two meeeeelion Users!).

The answer they're looking for is:

Two AWS large instances.

The answer they'll get is:

It depends on the application code.

They're laboring under the misconception that $Platform and $NumUsers are the only variables in the Grand Equation of Scaling. HAHahahahaahaha. There actually is a GEoS, but I'm getting ahead of myself.

The part of the garment industry that focuses on pants has a problem. People shopping for off-the-shelf pants want to only have to look at one, maybe two, numbers to determine fit. Also, the fewer number-combinations possible reduces the number of different sizes they have to make in order to cover everyone.

And yet, there are there are four measurements that determine how well a pair of pants fit:

Waist
Hip
Rise (crotch to waist)
Inseam

This? This is a problem. There is no way to sell off-the-shelf pants with four measurements on the tag. Well, you could but retailers would hate it since they'd have to shelve umpty different permutations, and customers would hate it since finding the right one would take far too long. Clearly a non-starter. What to do?

Statistics!

The question here is, "How related are these four measurements, and is there a single one that would provide the best predictive power for the rest?"

What they did was measure a lot of people. They've done this several times over the last century, but the most recent dataset is far more multi-cultural than the ones used back in the 1950's. It actually has non-white people in it!

What they found was:

For men, Waist is a strong predictor of Hip
For women, Waist is a poor overall predictor of Hip
For women, there are clusters around certain ethnicities where Waist is actually a pretty good predictor of Hip

Add this into several decades of marketing-habits they've learned over the years:

Men can handle two numbers on the label, so can handle a separate Inseam measurement
Women expect only one number on the label
Rise is dictated by overall fashion trends instead of individual bodies (compare 1980's pants vs today's)
Inseam for women is dictated equally by body measurements and fashion (you wear different pants when wearing heels)

Men's pants are easy: Waist, Inseam, done.

Women's pants.... trickier.

Pants?? What about Servers!

Consider what an enterprising new clothing manufacturer faces when designing pants. How do they get the best fit for their picky customers? Like the question at the top of this article, all they know is that women's pants have a single size number on them, and come in petite/regular/tall. But what does that mean? How should they size their pants?

I'm a new clothing designer for women's ready-to-wear. What's the hip measurement for a 28" waist?

The answer they're looking for is "40 inches".

They answer they'll get is, "Who are you selling to?"

Does this sound familiar at all? It should.

The AppDev looking for server-sizing is expecting their problem to be a men's-pants kind of problem; one sizing-style fits everything. When in fact, it's far more complex.

So, what is this Grand Equation of Scaling I talked about above?

Take a look at how the fashion industry solved their sizing problem. They took a lot of measurements, ran quite a bit of analysis over it, watched things over the course of years, and adjusted to fit.

They measured things relevant to the problem (body measurements)
They researched customer preferences (shopping data and market research)
They analyzed trends in the data

With these three things done, they were able to construct a sizing regime that works pretty well for them, their retailers, and their customers.

The same bullet-points apply for figuring how much hardware (physical or virtual) is needed for a web or mobile application. The easiest to quantify metrics are $NumUsers and $NumServers, but they're just parts of the overall dataset needed to be analyzed to appropriately answer the question.

Measuring things relevant to the problem

When figuring out how many resources are required to support a given application, any of the following variables will need to be tracked (and I'm sure I'm missing some for edge-cases):

Number of Users
Number of Web Servers
Size of WebServer
Number of Databases
Size of DatabaseServers
Caching tier efficiency
Number of concurrent accesses
Number of concurrent sessions

And much, much more.

These variables can be discovered in pre-deployment testing, and by monitoring production performance. Since things like user concurrency is not a static value, a range needs to be assessed.

Research customer preferences

This will tell you how your customers expect your application to perform, when they start getting peeved at slow-downs, how they work through the application, and a bunch of other things. The item easiest to measure is perceived performance thresholds. People using it on mobile networks will be more forgiving of stuttering than those on traditional networks. It is through this research that you find the performance envelope you have to stay within.

Analyzing trends in the data

A pile of data means nothing unless you have someone who can make sense of it. It is through this process that the Grand Equation of Scaling is derived. Now that you have a pile of data about how your application works, and you know what performance goals you have to hit, you can start constraining your equation. This is where you get to use the higher maths you got in college, and is one of the reasons why Google likes to hire Ph.D's in Mathematics.

Because this equation? For complex environments it can involve Calculus and Discrete Mathematics. This is why they sometimes call us Systems Engineers, not Administrators.

With these three steps you can actually answer the "how do I build an infrastructure that can support 2 meeeeelion users" question.

The same steps can be used to figure out whether or not your new web-app can be run on shared-hosting or needs more expensive dedicated-hosting. It won't even involve calculus for something this small!

Either way, you do have to know what to measure. For pants, see above. For IT infrastructure, find a Systems Administrator/Engineer. We'll even wear pants if you ask.

A change in mind-share, or just a mistake?

2012-02-22T17:36:47Z

Today while running through my feeds, I ran across an article describing how nifty Windows 8 Server will be for storage. And tripped when I got to the following line:

The new OS will support Hyper-V and SQL Server running over the Samba file sharing protocol Server Message Block (SMB) v2.2, including remote direct memory access (RDMA) over Ethernet and InfiniBand.

Waitwhat..

The Samba file-sharing protocol? Since when?

Perhaps this is what happens when a linux-person writes an article on a Windows technology. Who knows.

When upgrades fail, why the cutting edge cuts

2012-02-20T21:11:20Z

Since I had some time this weekend and actually had my work laptop home, I decided to take the plunge and get OpenSUSE 12.1 onto it. I"ve been having some instability issues that trace to the kernel, so getting the newer kernel seemed like a good idea.

Oops.

The upgrade failed to take for two key reasons:

VMWare Workstation doesn't yet work with the 3.1 kernel.
Gnome 3 and my video card don't get along in a multi-monitor environment.

The first is solvable with a community patch that patches the VMware kernel modules to work with 3.1. The down side is that every time I launch a VM I get the "This is running a newer kernel than is known to work" message, and that gets annoying. Also, rather fragile since I'd have to re-patch every time a kernel update comes down the pipe (not to mention breaking any time VMWare releases a new version).

The second is where I spent most of my trouble-shooting time. I've been hacking on X since the days when you had to hand-roll your own config files, so I do know what I'm looking at in there. What I found:

When X was correctly configured to handle multi-monitors the way I want, the Gnome shell won't load.
When the Gnome shell was told to work right, GDM won't work.

Given those two, and how badly I need more screen real-estate at work, I'm now putting 11.4 back on the laptop.

Looking to the future person

2012-02-07T15:30:58Z

It is becoming increasingly clear that we'll be hiring a second full-time IT-type person some time this year. What's more, they're likely to be directly reporting to me. That would turn me into a straight up Manager, something I haven't been before. This is somewhat scary! The closest I've been was two jobs ago when I was providing work direction to two other people, but the actual time-card signoffs and vacation approvals was handled by someone else.

Which means I'm giving thought on what we'll need in terms of skillset for this nebulous person. As I see it, there are five primary knowledge domains that we're interested in:

The "buying things" one is new to this chart, in the past I've always had either a purchasing department to handle things like chasing down late orders, or have had single-source contracts that take a lot of the choice out of what we can buy. When you get to the senior levels and get 'recommend' powers (if not straight up purchasing authority) this kind of thing is actually pretty key. In fact, right now I have a specific order that I need in my hands by next Thursday OR ELSE, and I'm having to apply mallet to a supplier to get it. I never had to do this kind of thing before.

Anyway.

The tricky part is, what weight do we assign to each knowledge domain? Based on my workload right now, I'd have to say "automation coding" is primary. However, come August the pain we're feeling may be somewhere else entirely. Come August, our product should have been released for several months and we'll then have a lot of operational experience, and what we may need most of all is someone to share the midnight callout duties more than anything else.

And then there is the whole, "Now that I'm a manager-type person, what work is most suited to that?" question.

For me, of those five domains the Automation Coding portion is the domain with the highest interrupt costs; it takes me a good while to get back on track after someone asks me something random. Considering it's my job to be interrupted (even though that's a bad thing for systems administrators), this suggests having Someone Else do the automation coding part is a good idea. On the other hand, for large web-scale and highly homogeneous infrastructures automation coding is the majority of the systems engineering work needed to make it all function.

Right this moment, it's looking like my #2 will have to be someone with some years under their belt, which runs against our company's proven track-record of hiring promising people right out of college. Automation coding is that curious mix of a lot of Software Engineering crossed with the domain specific knowledges of OS lore, management frameworks, and application-specific functioning. Of these, it's the OS lore that's the hardest to train for.

However...

If I've done my job right, by the time we release the automation framework should be largely completed and should be good enough to handle at least half a year's worth of scale. If we release, and the industry loves us like a loving thing, we'll be scaling out madly. At that point, having another set of hands well versed in the Network Configuration and Server Hardware parts will be more important than a systems programmer. THAT is someone who could be a fresh-from-college person, much like our summer intern IT last year.

Won't know until we get there.

Judicial rubber-hoses

2012-01-24T15:50:50Z

The other day a Colorado court ordered a defendant to produce the unencrypted contents of their own laptop. This is what I called "rubber hose cryptography", and previously we've heard of efforts in the UK to compel decryption. It has now happened here, and not at the US border. Unlike the UK, this decryption demand in Colorado is not based on a law that specifically says that courts can demand this.

Wired article

The counter-argument is quite clearly the 5th amendment right guaranteeing the ability to not self-incriminate. If that decryption key only exists in your head, and disclosing it would incriminate you, then you don't have to yield the key.

This judge disagreed. I'm not a lawyer, so I can't tell what legal hairs were split to come to this decision. But the fact remains that this judgment stands. The only concession he appears to have made for the defendant is to preclude the prosecution from using the act of disclosure as a 'confession', but the data yielded by the disclosure is still admissible.