In Free advice is worth what you pay for it, I wrote about the genesis of my blogging – it all started with my LiveJournal, and my friends complaining that they didn’t “get” what I was writing…so I moved to blogspot, then to my own VPS.

The blog has really come a long way from this look!

Every once in a while, someone will ask me how to blog. My answer is almost always the same, and it sounds disingenuous, but it isn’t. The answer is to write. Write. Write. Write.

My theory behind blogging is that if you start a blog, you want to eventually be successful and have lots of people read it. If you want lots of people to read it, you want to have stuff for them to read, so you need to write a lot, especially in the beginning. Even if your writing is bad (especially if it’s bad, because then you’re getting practice, too!). Even if you don’t have anything amazing to say.

Just keep writing. Build a body of work, because some day, you’re GOING to write something amazing, and people are going to wonder, “gee, I wonder what else that person has written”, but if you don’t have a body of work, they’re not going to have anything to read, and they’ll leave. So you need to write.

If you look at the early posts on Standalone SysAdmin, there are a few gems among a lot of cruft. But I kept writing anyway. Eventually I wrote better stuff, and more people read it. But it took time to build that body of work, and it took time to build a group of readers.

When I celebrated my first blogiversary, I had a bit over 500 subscribers, and I was highly impressed. Given how my writing was at that time, I’m still impressed ;-)

As it stands right now, I have over 2,700 subscribers (thank you everyone who subscribes!), and in the past six months, I’ve had around 2 million page views (!) according to CloudFlare. That is simply amazing.

I want to thank all of my readers, whether they subscribe or just visit regularly (or even just find me on a Google search). Thanks to everyone who has emailed me over the years offering support, advice, and asking for help. I’m thankful for all of the opportunities you all have given me, and I look forward to blogging for years to come.

This past year at LISA’11, I won a drawing for a free pass to the Technical Sessions at LISA’12 this year in San Diego. Now, that’s kind of silly, because as a member of the LISA Blog Team, I get free admission anyway. So what should I do with it? I guess I could put it on Ebay…

Or I could give it away! To one of you!

So here’s how it’s going to go. You have to work for your entry, but only just. Fill out the following survey for a chance to win the pass to the LISA’12 Tech Sessions pass. This doesn’t get you into training, but it does include 3 days of tech sessions, plus the hallway track and BoFs. I don’t know the value because the prices aren’t up yet, but it’s worth a lot of money.

LISA’12 is in San Diego, California from Sunday, December 9th through Friday, December 14th. The Technical Sessions (what this pass gets you into) is from Wednesady, December 12th through Friday, December 14th. It covers the conference pass only. No airfare, no hotel, no food (except that which may be provided by the conference itself). It’s literally a pass to register for the tech sessions.

So there you go. Fill it out to throw your hat in the ring. Also, I included a thing for “don’t include me in the contest”, since some of you may not want (or need) to win a free pass.

If you have any questions, please throw them in the comments or email me directly at standalone.sysadmin@gmail.com. Thanks for 4 great years!

After some discussion, my friend Brandon Burton and I started a podcast called SysAdministrivia. We released 2 episodes of the 4 that we recorded, but life intervened. Brandon changed jobs, I quit mine, and I moved to Ohio. And 18 months passed.

Now, though, we’re back. Episode 003 was released today. It’s only 25 minutes, so it should be easy to listen to. There are show notes with links on the website.

Please check it out, and if you like the format and idea, subscribe to the RSS. We’ll work on getting iTunes integration, but if there’s anything we should be doing that we aren’t, drop me a line or a comment below and tell me!

Thanks for listening!

It’s not without its warts, but it is software that I’ve used for several years despite them, and I feel like progress is being made.

Monitoring is important to me, and it should be important to you, if it isn’t already. It’s one of those things that you can always get better at, and with a tool lik Nagios, there are dozens of ways to get better at it.

I am intending to go to the Nagios World Conference this year in St. Paul, MN. At least a little bit of it is that I get in free, thanks to being named a Nagios MVP, but most of it is that I really, really want to be better at monitoring. (Also, I have some crazy ideas that I want so share with the group, so I’m going to be submitting a talk (and if this kind of thing interests you, then you should too!)).

Anyway, I wanted to let you know about this conference that’s coming up in September in plenty of time so that you can start planning now. As an aside, it’s nice to see the middle of the country getting some conference love. Aside from Ohio Linux Fest (incidentally, which overlaps the first day of Nagios! Really?!?), there’s not much in the American MidWest. Kudos.

I’ve had this blog post sitting in my queue for a week. I really thought this whole thing would blow over, but people keep talking about it, and as an interested conference-goer, it’s not outside of the realm of my interests…it’s just that I think that this whole “drinking at conferences” thing is pretty overblown.

Work is the curse of the drinking class
–Oscar Wilde

A lot of people go to conferences and drink while they’re there. A subset of people go to conferences SO that they can drink. Which, speaking economically, makes no sense to me. But whatever, there you have it.

Now, the post that I linked to above, Ryan Funduk’s Our Culture of Exclusion begins with two paragraphs about a culture of exclusion (and hey, I think that there really is a small group of individuals who actively try to exclude other people. I wrote Xenophobia and Elitism in the Community two years ago), but then he dives headlong into an accusatory diatribe over the “Alcohol Clique”.

Let me confront this label head on.

There isn’t an alcohol “clique” at tech conferences. There is a culture of technical people, and one aspect of that culture is alcohol. It isn’t the reason that (most) people come together, but it is something that is shared by many members of the community. That isn’t a clique, that’s mainstream.

Enough of the label, let’s actually get to the substance of the problem.

Ryan views drinking as exclusionary because some people don’t drink, don’t want to frequent the places where alcohol is drunk, and don’t want to be around people that have been drinking.

I sympathize with people who don’t drink, don’t want to frequent the places where alcohol is drunk, and don’t want to hang out with people who have been drinking. It has to be frustrating when 95% (99%? I don’t know the numbers) of a conference disappears in order to go partake in something you cannot or choose not to partake in.

At the same time, though, the activities of this vast majority are not exclusionary. You are not being excluded, you are excluding yourself if you choose not to be around people who are drinking. This isn’t a case of a disability barring you from entering a building – it’s a choice that you’re making not to be around people.

That being said…I agree with a lot of the observations that Ryan makes. There certainly are people (too many of them) that drink until they’re witless. Who are loud and obnoxious…and who frequent places that are loud and obnoxious. I don’t like that particular “scene”, so I don’t go there or hang out with those people.

Some of the things that go on around other conferences (like VMworld) are kind of…off the chart, as far as bad decision-making goes, so I’m going to exclude them and deal with the ones that I have more experience with, such as PICC and LISA…and it’s not all sunshine and roses, but it’s not all bad, either.

As you probably know, LISA has Birds of a Feather sessions, or BoFs, after normal conference hours. These are “unconference” sessions where people can create a meeting about a particular topic, and then (the theory goes) anyone with interest in that topic shows up and you can have an engaging and informative discussion. The BoFs are scheduled each night, Monday-Thursday until 10pm or so.

That’s exactly what happens, too. There are a lot of interesting BoFs on good topics that are all well attended. Until Wednesday, that is. Wednesday is when the actual LISA conference begins, with the previous two days having been training only. On Wednesday, the floodgate of attendees opens and the BoFs change.

Instead of having 6 rooms where each has 4 hours of BoFs full of people, you have 3-4 rooms of poorly-attended BoFs because enterprises grab up big chunk of BoF time…which I’m not complaining about. Vendors have the same right to BoF time as individuals (note that I don’t know if a BoF time block is negotiated as part of a sponsorship package or not, but it may be, since a couple of the vendors get 2 hour BoF-space). What I do complain about is that these things are called BoFs in the first place.

Look at the LISA’11 BoF schedule. Take note of everything that says “vendor BoF”. My impression, from the majority that I’ve been to, is that they are far-less BoF and far more marketing and headhunting opportunities, in some cases with a party atmosphere thrown in. There’s often very little actual discussion of the company’s technologies other than maybe a slideshow or a technical “state of the union”-type address.

If there’s so very little signal in all of the noise, why do people go? In a word…beer. At the very least, there are buckets full of beer at the front door of the room when you walk in. The higher-end ones have wine, too, and although I’ve never seen liquor in an official BoF, there very well could be. I don’t know.

The thing with the vendor BoFs is that the vendors see buying beer as either a community service of sorts, or a marketing expense, or maybe as a human-resources endeavor, but whatever the reason, it virtually guarantees that any BoF that runs up against, say Google’s, is going to have next to no one attending it.

But what are you going to do? As someone who throws a BoF or two every year, I have a choice to make. I can schedule my BoF before the big vendor BoFs start, or I can schedule it late enough that all of the vendor BoFs are over, or I can schedule it against what I judge to be the weakest vendor BoF in my time frame. I’ve done all of those. But let me be very clear – I am not being excluded. I am making a choice to not attend a vendor BoF and partake of whatever it is that they’re offering, be it alcohol, company, or ambiance. It is my choice.

Yes, the situation sucks from the perspective of someone who feels like they have something to offer people who are busy drinking somewhere else, but that is the situation at hand. Would eliminating vendor BoFs solve the problem? Hell no. LISA is held in the heart of metropolitan areas for a reason – good access to local restaurants, pubs, and other attractions. Enterprises would LOVE it if the LISA staff got rid of vendor BoFs, because hotels charge an astounding amount of money for providing beer, and it would be cheaper to rent an entire pub than it is to supply hotel beer and wine to the probably 400+ people who showed up to Google’s BoF last year.

If you eliminate vendor BoFs then you not only continue to keep vendors involved, but you actively draw people away from the conference location. This isn’t even “not winning”, it’s losing worse.

There is no way to win the war on an “anti-alcohol” ticket. Ask the 21st amendment of the United States Constitution.

So instead, winning the war means picking your battles and deciding what you want to fight against.

Here is what I am against, that Ryan referenced in his post. I am against parties that are so dark you can’t see the person you’re trying to talk to. I am against parties that are so loud that you can’t hear the person you’re trying to talk to. And I’m against parties that are so crowded that you can’t find the person you’re trying to talk to. If you’ve been to conferences, you know the parties that I’m talking about, and at LISA, I can think of at least one that was a “vendor BoF”.

So, if I were to take the stance that I was against a party of that sort, my first step would be to identify why people wanted to be at that party.

Is it the ambiance? For a few people, sure, but there’s far more ranting against that than praise of it. Since the people who like that “club” feel are going to seek it out, and my goal isn’t to be in a place like that, our differences on the subject are irreconcilable, so I’ve got to move on.

Is it the giveaways? Sure, that has a lot to do with it. If someone is giving away an iPad or an Android, or a pass to LISA next year, but you’ve got to be present to win, you’re going to have a much higher attendance than you would otherwise.

But I think that the big reason is the alcohol, because free beer is a big draw for people who like to drink, and large vendor BoFs have it in spades. It’s the biggest draw that I hear about whenever I talk to someone who’s going to those BoFs.

So, in a thought-experiment, how could you fix the fact that people are going to vendor BoFs instead of content BoFs? You could eliminate the alcohol and giveaway aspect, but we already established how that turned out. Or you could level the playing field.

If all of the BoFs had the same access to alcohol, snacks, sodas, etc that the vendor BoFs did, then the vendor BoFs would be on equal footing with the guy who wants to hang out with people and talk about shell scripting, or configuration management, or anything else that no vendor would ever sponsor on their own. If there were no incentive to attend a useless BoF, then you wouldn’t. And vendors COULD move away from the conference and have unofficial BoFs at local pubs, but they wouldn’t, for the same reason they don’t now – competition from other more convenient (i.e. on the conference site) BoFs that offer the same or similar levels of desirable features (i.e. alcohol).

Anyway, enough about conference BoFs, and back to the culture of drinking stuff.

Again, yes, our culture does include drinking as one of its hallmarks, but I have never, ever, not even once, been (or seen anyone) put down for deciding not to drink. And I know a lot of people that can’t or won’t drink. In fact…

I didn’t write about this here, because, well, I was “busy”, but a few weeks ago, during a Tech Field Day event, I had to go to the hospital because I was experiencing a condition called atrial fibrillation, which is where the top part and bottom part of your heart disagrees about when and how often to pump. It’s very uncomfortable, and can be dangerous if left untreated, so I spent a few days in the hospital.

One of the outcomes of that was, because they feel that my having alcohol was a contributing factor, they advised me not to drink to excess. I decided that three days in the hospital was enough, and so I decided that the easiest way not to have a second beer is to not have the first. I haven’t had a drink of alcohol since the night I went in to the hospital.

That’s a big change for me, because I’m not a heavy or regular drinker, but I really enjoy good beer and good scotch, but I’m making a decision not to have them any more (although I reserve the right to taste new and interesting alcohols, I’m not going to be sitting down and having one).

By the arguments of Ryan and several other people who have weighed in on the topic, I could now consider myself excluded from the events where drinking takes place. Hogwash.

Were I not to be around people who are drinking, it would be my decision. As it is my decision to drink or not, is it not my decision to share company with people or not? If I don’t like the place they’re hanging out, I can hardly say that I’m being excluded for my own preferences.

No, I’m not going to take this exclusionary view. Instead, I am going to continue to hang out at the BoFs (both with and without alcohol), I’m going to continue to go to the hotel bar after hours. And I’m going to continue to not drink alcohol, and I have every confidence that no one is going to give me a second thought over it. And if the people I’m with drink too much to the point that I, in my sober state, can’t stand to be in their company, I’ll leave and find someone else, or go sleep. But I’m not being excluded.

So there, that is my very long, very complaint-filled view on this culture of exclusion thing. Let me know how wrong I am in the comments.

…an online store where customers can find, buy, and quickly deploy software that runs on AWS.

You can select software from well-known vendors including CA, Canonical, Couchbase, Check Point, IBM, Microsoft, SUSE, RedHat, SAP, and Zend as well as many widely used open source offerings, including WordPress, Drupal, and MediaWiki.

AWS Marketplace includes pay-as-you-go products, free software (AWS infrastructure fees still apply), and hosted software with varied pricing models.

When you find the software you’d like to purchase, you can use AWS Marketplace’s 1-Click deployment to quickly launch pre-configured server images, or deploy with familiar tools like the AWS Console. You’ll be charged for what you use, by the hour or month, and software charges will appear on the same bill as your other AWS services

Great. Awesome idea, where you can not only spin up AMI instances from the community, but from enterprises as well, and pay more money per hour for the privilege.

But surely it’s worth it, right? Lets take a look.

Here are the current prices for running EC2 instances:

Now, lets take a look at the prices for running a well known OS, Redhat Enterprise.

The costs for a license of RHEL when buying it from Redhat are very clear, with scaled amounts per support tier as well as price increases for enhanced feature sets.

The costs for paying for a license of RHEL hourly are here:

So, doing a bit of math, if we look at the cost of a “large” AWS instance for a year we get:

8,766 hours in a year * $0.32 per hour = a server that costs $2,805.12 a year


Now, with the Redhat Value Add:

8,766 hours in a year * $0.38 per hour = a server that costs $3,331.08


That’s a difference of $526, when an RHEL subscription costs $349 per year. So what are they charging me the $177 extra for? Support? I’m afraid not:

I can’t figure out why they think anyone would do this. ESPECIALLY considering that there is already an RHEL AMI available to use!

Here’s a pair of terminals I’ve got running:

The only difference is that the terminal on the right costs me $526 more per year!

Now, I’m not saying that all of the Marketplace offerings are bad…but you need to watch out.

Things to look for are whether or not the stated price includes the cost of the EC2 instance, because some of them don’t. All of the 0.00/hour instances still cost money because they are running on EC2, and if you spin up a large instance of Tomcat from the Marketplace, you’re still going to get charged 32 cents every hour.

And for the love of all things holy, if you do decide to spin up an instance of something commercial, do price checks to make sure that you aren’t getting gouged. This layer of abstraction is the perfect place to hide invisible price increases, and none of us are in a good enough shape that we can throw away a lot of money on a little convenience.

Stephen Foskett is hosting the all-day event which features a keynote address by Robin Harris of StorageMojo fame.

Chris M Evans, Howard Marks, and Nigel Poulton will moderate a panel of innovators in the enterprise solid state market.

The companies involved are:

Admission is free, so register at EventBrite and join us at the San Jose DoubleTree on Wednesday, April 25th at 9am.

In the past week, I’ve gotten no less than three messages, all from different people who are going through certain decision-making opportunities, and asking how it was that I got into system administration, and wanting to know how I gained the skills that I did.

If there’s anyone else reading who wonders about this, or is looking for an example of how someone learns system administration, then maybe this will suffice.

Note: this was a response to someone about to graduate in three weeks with their BS in Information Systems. They currently work for a webhosting company and handle a decent amount of Linux. They’ve been in both Windows and Linux shops, “done a bit of everything, run ESXi servers at my house with 2008 R2 AD, WSUS, FreeNAS, etc”. They’re getting ready to choose between two jobs, an easier one and one they’re afraid might be more than they can handle.

I learned almost entirely from books and from working on the products, just like you.

I went to college for a year (well, sort of – I went to DeVry before I figured out that it was a scam) but it taught me nothing relative to what I do right now. I learned Linux on my own by running it on my desktops over the course of a very long time (16 years so far).

My first IT job was in my first year of college doing remotely outsourced technical support for the BellSouth.net dialup ISP. Then I moved to a different project, doing dialup technical support for MCI Worldcom (again, outsourced to the same company that BellSouth used).

When I moved back to West Virginia where I grew up, I got a job at a local ISP doing tech support, and I just happened to know Linux better than either of the two administrators who were implementing it in the server infrastructure.

They slowly “adopted” me into being a junior administrator-type. They taught me a little bit NT4 domains, and by the time I was an “official” junior admin, there was only one senior admin, and he and I taught ourselves Windows 2000 while I taught him more Linux. I eventually took over after he got a better job.

Back in those days, I tried to learn, but I took a kind of lackadaisical view of everything. I didn’t push like I do now. I just kind of went with things, implementing stuff as I needed, and I wasn’t anywhere near a “senior” sysadmin, I was probably a SAGE Level 2 when I left to move back to Columbus to take a programming job.

It didn’t take long before I found another sysadmin position, and my “stats” at that point were high-skill-level Linux admin, low-to-medium-skill-level Windows admin, and I was not incredibly well versed in the good overall tao of system administration.

You can look at my path of becoming a better system administrator in two ways: my skillset and my attitude. The second is, in my opinion, more important than the first.

What really turned the corner for my outlook was that I had started a thread on a forum that I frequented devoted to the “interesting command-line of the day”. Every day, I would post an interesting hack that I would develop using bash. I did things like write a command-line RSS reader in a bash shell, or report the most-recent earthquakes in the world over a 5.0. Just weird little things that I thought were interesting.

I took to writing longer entries (or diatribes about technology) on my LiveJournal, but one day, a friend of mine basically said, “look, it’s great that you’re writing this here, but no one knows what you’re talking about”. So I thought, “maybe I’ll make a blog”. So I made a blog to share all of my technical writing.

This introduced me to the sysadmin blogger community, where I linked to (and read all of the posts by) people like Bob Plankers, Jeff Hengesbach, Michael Janke, Saint Aardvark the Carpeted, and Ben Cotton. And I really do mean that I read everything they wrote.

I became aware of people like Tom Limoncelli, Ben Rockwood, and John Allspaw.

I kept writing, and got linked to by a lot of people. A weird group called LOPSA-NJ added my blog to their RSS page. I checked it out, and in February of 2009, I became a member. It seemed natural to go to the meetings when I moved to NJ, but they were a long way away. New York City was closer, so I started my own chapter there. Then when I moved back to Columbus, I started a chapter here, too.

In terms of my skillset, when I started at my job in Ohio, I was fairly well versed in all of the technologies they were using. They had a more involved backup plan than I was used to, and more physical sites, and way more data to move around. I had to begin thinking about data *en masse*, as opposed to in small, discrete files.

My boss there, the CTO, functioned (and still does) as the defacto DBA, and I learned the operation and administration of Oracle and PostgreSQL from him, from scripts, and from breaking it and fixing it.

Across the board, I worked to implement the ideas that I had been reading online in my own infrastructure – automation, separation of duties, structured management, repeatability. I learned more about backups from reading people like Curtis Preston and Preston DeGuise, and I worked to learn more about the underlying technologies.

Rather than working really hard to implement things the way my boss wanted, I began asking myself why we were doing things certain ways. I began to think about what my pain points were, and what caused them, and I began considering other ways.

We were an all-Linux shop, and we ran Slackware everywhere except on the desktops, which were split evenly between Mac and Windows. One of the things I hated was user account administration. Changing a user’s account on all machines, plus the pertinent Samba servers, plus ftp, etc etc really sucked. It got to the point that Moves, Adds, and Changes (MACs) were taking up most of my time. Then I found Likewise Open, and it changed my life. It allowed a Linux machine, almost ANY linux machine, to become part of an Active Directory infrastructure. We were a Linux shop, but I got a demo copy of 2003 Server, built an AD infrastructure, got some CentOS test machines up and running, and authenticated them against the directory. Then I showed my boss.

One of the things that Likewise-Open couldn’t do was run on Slackware, mostly because of the lack of PAM and package management. That meant migrating our entire infrastructure over to something that could work, which in this case was CentOS, since I had a very small amount of experience with it.

That was one example of implementing a new technology with a small amount of previous experience. SANs were a different story.

We had DAS on our file server, DAS on our backup file server (manually rsync’d between them), DAS on our database servers, web servers, etc etc etc. We didn’t use virtualization at all.

Because of the growth of data, we were going to need a new storage solution, and because of the age of our servers, an infrastructure refresh was called for as well. We were primarily a Dell shop, so naturally, we reached out to Dell to see if there was a deal they could cut us.

As it turned out, the 1955 Blade Servers were being EOL’d, and Dell was willing to cut a ridiculous deal to unload some on us. Add into that the two Dell-branded EMC AX4-5 SAN storage arrays that we needed, and we got an entirely new infrastructure for quarters on the dollar. And as a bonus, I got to build them in parallel with our existing infrastructure, so I could take as much time as I needed, within reason.

I’d never dealt with blades before. Or SAN storage. So I read as much as I could about them in documents from the manufacturer. You know those CDs full of docs that come in the boxes, but everyone always throws away? I actually put them in my CD-Rom and read them. And what I could find online. I searched for other people running them and read those blogs. I started reading some of the dozens of storage blogs that are out there talking about SAN storage. I took in as much information as I could find, and since I had my own lab full of this stuff, I built and played and broke and fixed and learned as much as possible. Then I implemented.

Your story won’t look like mine – no two stories ever look alike. That’s why the attitude and outlook is so important. A good healthy approach to this job will allow you to reach your maximum potential, and will help toward you not hating the job.

Always take the long view, and think about the ramifications of the decisions you’re making. Don’t be afraid to over-reach from time to time, but try to balance that out by knowing what your limitations are, and knowing that you can always exceed them by learning and by trying.

Deciding what job you’re going to take is always difficult. I’ve never once regretted going with the harder or more interesting choice, whenever I’m presented with two. I tend to be aggressive, because I have faith in my ability to grow into whatever I find myself in. But, at the same time, I always represent myself as I am, not as I want to be…that helps people underestimate me.

I don’t know if you’re biting off more than you can chew, but if you don’t try, then you don’t know either.

LOPSA Columbus (website forthcoming) is having our April meeting tomorrow at 6:30pm at the Datacenter.bz offices and datacenter.

This is the first of a series of Datacenter tours that LOPSA Columbus is taking. We’re going to be doing one per quarter, and each tour will be accompanied by a presentation and discussion about datacenter and server room infrastructures.

This month’s presentation will be given by LOPSA Columbus co-organizer Warner Moore, who will be discussing a vendor neutral review of what to look for when selecting a data center, after which we will be touring the actual datacenter at Datacenter.bz.

If you’re in the Central Ohio area, you should come, too! Datacenter.bz is located at:

If you’re in the area, make sure to sign up at the MeetUp page for TechLife Columbus. You do not need to be a LOPSA member to come to the meetings (but you should be a LOPSA member anyway, just for the record).

Come, have fun, and tour datacenters with us.

TELNET had some good things going for it — a local-echo mode and a well-defined network virtual terminal. Then SSH came along and added minor enhancements like confidentiality and authentication, at the cost of losing the local-echo mode and the well-defined terminal semantics.

Terminal emulators are a hard sell. I mean, yeah, we want that functionality, but at the cost of selling our souls? OK, if not our souls, how about our sanity?

Ever had a misbehave program freak out and corrupt your display? Or maybe you’d cat the wrong binary, then suddenly f?Kfjd!ܾ!j?sRc?CH?U^, it looks like your mom picked up the phone during your BBS session, and you have to type ‘reset’ to fix it. It’s not world-ending, but it is irritating.

As amusing as the telnet quote is, it’s right. Local echo is nothing if not immediate. We get feedback. Granted, in the case of telnet, that feedback doesn’t mean a whole lot, but still, it’s nice to be able to type and not see the letters show up several seconds later just because you’re typing into a terminal from 35,000ft using an internet connection that cost you the price of a month of dial-up back in the days when mom could interrupt your session of Legend of the Red Dragon.

Enter Mosh, a new take on terminal emulators.

Mosh is a client/server model. You run it almost exactly like you would ssh:

mosh msimmons@servername

It actually connects over ssh, then runs mosh-server on the other end. That’s when things get a little strange.

Mosh actually performs its transmissions via encrypted UDP datagrams. In the spirit of the heathen connectionless-UDP-protocols, it also doesn’t try to deliver each and every individual packet that the server spits out – so when you screw up and run find /, you don’t have to wait 10 minutes for your futile ctrl-c keystrokes to get to the server. Plus, and maybe most damning, it allows you to close your laptop, get off the plane, go into the terminal, pay ANOTHER $20, and connect back in immediately, all without running a screen session. Clearly, this is the work of Satan.

In all seriousness, though, this is pretty handy. What they actually do is, instead of treating the terminal like a stream (which is what every command line terminal emulator ever has done), they treat it like a state machine, and the network transmissions are all about bringing the client state into sync with the server state. That’s a cool idea.

It uses AES-128 in OCB mode. Of course, the Layer-4 headers are still unencrypted, but IP Over Unicorn doesn’t exist yet. *ahem*

Check it out. It’s free, available for a lot of systems, and the formal paper will be presented at this year’s USENIX ATC’12 conference during Federated Conferences Week. Cool, right?

Edit
So, after I wrote this, I spun up an Ubuntu server in the HP Cloud, and started playing with it. Very interesting.

The first thing I noticed was the confidence degree that the server received what I typed. I noticed this because the mosh shows how confident it is by underlining the words as I type them. For instance, when I type

echo “Four score and seven years ago our fathers brought forth on this continent”

as I type that, the underline “chases” my cursor. When the underline disappears, the client is 100% confident that the server and the client are in the same state.

Interestingly, though, when I paste in that same string, the underline is nonexistent. At least, I don’t see it at all. I’m on a Mac, and I’m using Terminal.app (yeah yeah, I know). I’m not sure if there’s an intrinsic difference in how Terminal.app handles pastes vs keyboard inputs, but the server becomes immediately aware when I paste, but not when I type. Odd.

Also interesting is the output of ps:

10512 ? S 0:00 mosh-server new -s -c 256
10513 pts/1 Ss+ 0:00 \_ -/bin/bash
10614 pts/1 S+ 0:00 \_ screen -T screen-256color ...
10633 ? Ss 0:00 \_ SCREEN -T screen-256color ...
10708 pts/2 Ss 0:00 \_ /bin/bash
10870 pts/2 R+ 0:00 \_ ps axf


All of the screen stuff is normal Ubuntu 10.10 cruft, but you can see that even though it does use the original ssh control channel to start up the connection, it drops it immediately.

I’ll run packet captures if anyone is curious, but if you’re that curious, you should just play with it yourself! Just remember to open UDP ports 60000-61000 to the machine on your firewall (which I almost forgot, natch). Let me know what you think of it in the comments below!

One of the things that always kind of annoyed me was that not everyone uses IRC. Some people don’t care for it, or can’t use it where they are, or just don’t bother to log on. That makes it hard to see what’s going on with the meeting until a while later when the transcript is posted online.

I thought about this for a little while, and decided that it couldn’t be that hard to make a web-enabled viewing screen for IRC conversations. The only trick was doing it.

I started by making a github repository, because hey, don’t all good things start with github? Anyway, I knew I’d need it.

I had been looking for a project to use as an excuse to learn RabbitMQ, and after considering the things I wanted to do, I thought this fit the bill nicely.

As RabbitMQ installed, I got down to brass tacks and learned to build an IRC connection using Net_SmartIRC, a PHP pear plugin that handles those kinds of things reasonably well.

If you’re asking why PHP, well, it’s the language that I’m most familiar with. I would have loved to have done it in Ruby, but I was crunched for time. I started this project on Tuesday, and LOPSA-Live was today. I didn’t have time to learn a new language in addition to a new daemon and two new APIs. I’m only human ;-)

So I gradually wrote irc.php, which began by connecting to an IRC server and printing channel outputs, and eventually grew to implement the various AMQP features necessary for the full project.

(Also, if you’re familiar with AMQP, I’m currently using a fanout exchange, but I’d like to switch to a topic exchange, so that web viewers can opt-out of channel join messages, etc without me needing to parse the messages myself.)

After things were going well with irc.php, I wrote filewriter.php to receive those messages, convert them to JSON, and save them to a flat file. It’s definitely not the most sexy method of storing data, but it is relatively quick to access. The file reads are linear, and can be cached by the OS simply. It’s not going to stay like this, but it was acceptable for a short hack.

Honestly, the hardest part was probably target.html, because I’m developmentally-disabled when it comes to javascript. I don’t know why, but that language has never clicked with me. I struggle with even simple things.

So I got all of these things done before today at noon, which is exciting, because I was able to get them ready in time for the LOPSA-Live session. I made a temporary page on my blog and embedded the target.html file in an iFrame, and it worked great.

I presented it to the LOPSA leadership and they really liked the idea, so as a test, Philip posted a trial run of the live feed to see how it went over.

I couldn’t stay at home today and take part in LOPSA-Live, but I got to eat my own dogfood by watching via my web form on the way to West Virginia to see my brother. Although I’m going to have to work out some bugs specific to the mobile browsers, I was able to read and keep up to date on the meeting with very few problems.

Overall, I’d say my efforts were a success. I just wanted to share with you all since I think you’re awesome, and like telling you what I’m up to. Hopefully you were interested. As I get more into it, I’m sure I’ll be writing more about RabbitMQ and AMQP in general. Thanks!

For a long, long time, I would hear people mention multicast, and I would be like, “oh, multicast, yeah, talking to lots of hosts at one time, yeah, that’s cool…” then do my best to not join in that conversation, because I had less than nothing to add. In fact, all I could do was show my ignorance, and THAT’S no fun.

So my shame of not knowing it drove me deeper into my cave. “It’s not like anyone even uses it. I’ve been doing networking for years and its never come up”.

Then I started learning more about IPv6, and multicast would creep into the conversation. And I’d ignore it, and sure enough, the next paragraph wouldn’t mention it anymore, and I could go back to pretending it didn’t exist. But it kept coming up, and one day I slipped.

I thought to myself, “so, I know that multicast is all about sending data to lots of hosts at once, but how could it really do that?” And so I slipped into the inescapable void of my curiosity.

As it turns out, it’s really not all that crazy, and it’s really not that difficult, at least on the surface. There are two sides of the proverbial coin. There’s the theory of how it works, and there’s the practical application. Lets look at the practical side.

A prototypical example use-case would be video streaming. Every multicast application (and by application, here, I mean every solution, such as a NASA video stream and the like) picks or is assigned a particular multicast IP address, then everyone on the internet that wants to watch the stream joins that multicast IP address…and when I say it joins that multicast IP address, I mean that the machine uses a protocol called IGMP to tell its local router that it wants in on the action.

The local router is hopefully configured to know about IGMP so it listens. If it’s configured properly, then it uses a protocol called PIM (or one much like it) to let the next closest router to the video source know that it has someone who wants to watch. That router then sends PIM updates to the router it thinks is closer to the source than it, and so on.

Eventually, you wind up with a tree starting with the root being the video server, and forming the branches are all of the routers who have clients that used IGMP to let them know that they wanted to watch. The video source sends out one packet, to its closest router, and the router remembers which connections joined that group and sends the packets to those routers, which send the packets to the next routers, and the packets eventually wind up at the clients who are watching the stream.

No one machine sends a ton of packets, and no routers have to remember much besides which interfaces belong to which groups. It works out nicely, even though, and I say this with the greatest apology now that you’ve read the preceding paragraphs, almost no one uses it like this.

Sure, there are some internal networks that use it for things like teleconferencing, but I bet yours doesn’t. Except you Googlers and Yahoos out there. Your university probably does, I guess, especially if it’s on that whole Internet2 thing, but the vast majority of you will never have to deal with multicast video streams.

That doesn’t mean that multicast is going away. Actually, with IPv6, it’s getting more airtime.

The biggest difference between IPv4 and IPv6, in my opinion, is the idea that your interface isn’t going to belong to only one layer 3 (IP) network.

You know how, right now, you are probably on a private block, like 192.168.whatever? Well, with IPv6, you’re still going to have a private block, only it’s called link-local. Your machine, and every other machine, will have a link-local network address on their interface. It doesn’t route to the internet at all. When you have one machine that wants to talk to another local machine, it’s going to happen over the link-local interface.

Of course, your computer will also want to talk to the internet, probably, so you’re going to have an IP with the “real” network prefix (which is probably going to be assigned automatically by the IPv6 autonegotiation process, which I’ll write more about later). This “real” network IP will allow you to communicate with the outside world (and it with you, unless you set up your firewall rules correctly). That’s two IP addresses so far. How many more?

Well, there is also going to be the “All Nodes” multicast group, FF01::1 for “node-local” (AKA “all interfaces on the same machine”) and FF02::1 for the link-local (AKA “the local broadcast domain” because routers don’t forward link-local packets) group. Your machine will belong to these multicast groups, because things like ARP are going away with IPv6.

Instead of ARP (and ping and passive IP collision detection), IPv6 includes a protocol called NDP, which provides for all kinds of things…and most of those features come from using multicast addresses.

So what happens is really just a smaller version of above, where the host tells the router that it wants to join a multicast group. Except that in IPv6, we don’t use IGMP anymore, we use MLD, but it works mostly the same way. The router sends us any packets that are sent to that IP address, and things go swimmingly.

If the machine that you’re working on IS the router, well, then it’s going to join any number of IPv6 multicasts…the “All Routers” multicast (ff02::2), maybe the All DHCP Servers group (ff02::1:2), and seriously, a ton of other multicast groups.

The big reason for all of these groups is service discovery. It’s kind of like, in IPv4, if you had a specific private IP block for, say, DNS servers. If you just knew that somewhere in 192.168.32.0/24, there would be a DNS server, you could join that network, ping the broadcast, and the DNS server would respond. You could use that to add it to your configuration. It’s kind of like that, but instead of a broadcast, it’s multicast.

If you’re the rare networking wonk who didn’t know multicast, you’ve probably already asked, “yeah, but how does that work at the switch?”, in which case I invite you to read the wiki entry on IP Multicast: Layer 2 Delivery.

So despite my earlier fears, multicast doesn’t operate by magic, just by protocols that I’ve never configured because they didn’t matter to me. But even with the scant knowledge that I just imparted, you can feel a little bit better about needing to use multicast all over the place with IPv6.

Apparently, USENIX hosts the Computer Failure Data Repository, which is a series of datasets from various installations that includes failure data for various components.

There are around a dozen different data sets, most from HPC clusters, but they’re actively soliciting for additional data:

If you already have your data public on your reference web page so that any one can download it, then all you need to do is to send us a pointer to your reference web page and a brief description of the data.

But otherwise – if you want to make the first release of your data through the CFDR – then the data contribution procedure is as follows:

We need to have a necessary paperwork on file to show that we actually have permission to host this data. You need to sign or find someone at your organization to sign our contributor’s agreement.
If the data contains some sensitive information like user or vendor names, you need to sanitize (anonymize) it. If you don’t have proper sanitization tools, we will try to help you.
Please provide any available documentation or description of the data you are contributing. If no documentation is readily available, it would be helpful to create one in the form of a FAQ with answers to frequently asked questions on the data. You can take a look at the FAQ accompanying the LANL data sets to get an idea of the kind of questions people commonly ask about failure data.
Make your data accessible for us, then we will host it on the CFDR server.

I’m looking forward to going through some of these data sets, just to see what kinds of things I can glean. The idea kind of reminds me of the Public Datasets that Amazon hosts, but more specific to my interests.

It’s not hard to see how crunching these kinds of numbers to get interesting statistics could be an interesting passtime (or maybe I’m just a giant geek).

Do you know of any other resources for long-term statistics on failure data related to IT? What else have I been missing?

On occasion, though, I do get some downtime to discuss things with my readers, and the other day, I got a good question that I don’t think I’ve written about, so I thought I’d share with everyone else.

Here’s the email:

Matt,

This might sound like a nooby question, but I will sally forth…

What is the difference between using a Private cloud solution and a Virtual Machine environment?

I am planning to update my test environment, and if possible use a personal Could based solution for testing various software platforms. Based on my limited knowledge of Cloud based platforms, The VM and Cloud based environments are pretty much the same, except when you shutdown an instance of cloud based machine, it loses it configuration.

I have a VM environment right now, but would like to be able to provision new machines at will (depending on hardware restrictions), but would like to keep some of these machines as configured.

Thanks in advance

Here was my answer:

Hi!

I don’t think it’s a “nooby” question. It’s one that a lot of people struggle with, because there isn’t a clear definition, and the marketing in the industry tries to confuse the matter at every turn.

I’ve thought about it pretty extensively over the past couple of years, and I actually have a definition in my mind of “Cloud” that I think is logical and based on the origination of the term. Remember that the word cloud came from network diagram icons that indicated, for the purposes of that particular diagram, it didn’t matter what specific machines or software filled that particular space, only that there was a resource available that provided a service. At its core, it is a term of abstraction.

In my mind, cloud is still a relative term. Amazon’s AWS cloud is absolutely not a ‘cloud’ to the people who work on the discrete components of it. It *IS* a cloud to you and I, though, because when we interface with it, we see no distinction between the particular constituant machines. We only see the interface to the API that controls the creation and management of the resources that are hosted within the cloud. We don’t know which actual machine or machines inside the Amazon cloud are handling our resources, and we don’t care.

It should also be noted that there are different *types* of Cloud, too, and it all depends on what you want to abstract. If you want to create entire machines and network devices, there is Infrastructure as a Service (IaaS). Sometimes, you just want to write software and not worry about administering the web server that it runs on…then you want a Platform as a Service (PaaS). It’s also possible to just want to rent a service, like hosted Email, in which case you want Software as a Service (SaaS). But since you mentioned building virtual labs, IaaS is what we’re talking about here.

So, this brings us to your question about a private, internal cloud. It IS possible to build an internal cloud such that you can interface with it in an abstracted way, never knowing or caring (other than for debugging purposes, since you’re not only running things on the cloud, you’re running the cloud itself) where the resources are hosted. But for the immediate future, its almost always overkill in a smaller shop.

The Eucalyptus Project is devoted to creating private cloud platforms that can be administered using an abstracted API – in fact, it’s the basis for the new HP Cloud that’s currently in beta.

That being said, is it worth setting up for a lab environment? I guess the answer is, “It depends”. It would be an interesting exercise to build a cloud, just for the sake of doing it, but only you know if you have the number of machines available with the (admittedly low) required resources.

It may be possible to configure your current virtual environment to deploy new images quickly and without much manual intervention, but without knowing more details, I can’t be sure of the implementation. I would, at the very least, research the API for your virtualization platform and check out something like Cobbler.

I hope this answered any questions you have. Please feel free to follow up if you have any other questions. I’m happy to help.

Thanks!

–Matt

Confusion and misconceptions about what a “cloud” is run rampant because, for a very long time, it was essentially an undefined marketing term. As the idea gels, we get a better idea of what to expect from the service providers.

Also, if you’ll note that I said that running a private internal cloud is overkill for the immediate future. That’s because I suspect more and more software will be developed with the inherent idea of “cloudiness” built-in. I’d be surprised if software like Tomcat didn’t have server abstraction built in during the next couple of releases, where you install Tomcat on a few machines, and then you deploy your application to a central controller, which then deals with the “cloud” instances. There are already commercial services for this, but there’s no reason it shouldn’t be added into the software itself.

Likewise with an abstraction layer above KVM, so you just launch new VMs onto an array of hardware, and you don’t know or care which machine it runs on. This is what Eucalyptus does, of course, but right now, Eucalyptus is exceptional for doing so, where as in a few years, it’s going to be par for the course and built in. More and more administration will be done through APIs, I’m telling you…

Of course, I could be wrong! Let me know what you think in the comments below.

There are tons of tutorials on all kinds of sysadmin-related topics, and if you couldn’t get into the tutorial you wanted, there are a whole lot of talks, too. Just check out the schedule. It’s impressive.

If you read my blog on its website, Standalone-SysAdmin.com, you’ll also notice in the upper right of the page, there’s also the PICC’12 ad. If you’re on the other side of the country, then you should register for PICC (early-bird is over at the end of the month, so do it soon!).

I should also note that both of these conferences are operated by their LOPSA Local groups. If you’re a member of LOPSA, but there’s no chapter near you, maybe you should consider starting one. The process is simple, and I’d be glad to give you a hand. LOPSA is also working on a handbook for new chapters to help kickstart the process, so if you’re interested, drop a comment on this blog entry and I’ll get you in touch with the right people.

Why are there no peer-to-peer VDI solutions? (or are there, and I haven’t found them)?

There are some big issues with large-scale Virtual Desktop Infrastructures. The biggest of which I know is the so-called “boot storm”, where everyone is requesting their images at once. This causes stress to the link from the desktops to the server and on the server’s IO channels.

There are a few workarounds that people use, such as staggering boot times or throwing more spindles/SSDs at the problem, but I’ve never heard of a solution where hosts in the same classroom could disseminate the image to their peers.

It wouldn’t be hard to verify security – it could be verified by a centrally-distributed hash. I/O at the central server wouldn’t suffer NEARLY as much, since it would only need to fullfill the request once per logical grouping of clients, which could then seed themselves similar to torrents.

Am I missing something big, or is there just nothing like this? If it’s a good idea and no one has implemented it, feel free to give me full credit ;-)