A Place For My Thoughts :: gWaldo

On the Artifacts of an Incident

2021-06-21T17:12:00.004-04:00

“Never let a crisis go to waste.”

-- Apocryphal

The dust has settled, service has been restored, the monitors are green, the status update announcing the issue is resolved has been posted, and the Incident Commander thanked everyone for their effort.

The issue may be resolved, but the incident is not yet done.

The Artifacts of an Incident

When addressing the incident, actions will be taken in the course of the diagnosis, troubleshooting, and experimentation to solve the issue. How things are done is almost as important as what is done to resolve the issue. If, for instance, isolated changes are made manually that aren’t reflected in codebases, it is likely that they will be overridden in the next deployment, and lost.

It’s less than ideal to have to make changes that way, but it is something that happens in real life. It’s possible that a config change needs to be made on the fly that isn’t well supported by your existing deployment systems. This is why it’s vitally important to make a record of changes being done. This should be made in the running chat log of your incident. They should be tagged for review (during the postmortem?) to determine if they should be reverted or captured and made a permanent part of the environment.

Further, if that change is needed in Production, should it be rolled out to other environments as well?

Equally important is determining how the change impacted the incident, and whether it should be treated as an experiment to revert the change.

For these reasons, in the course of your incident, it should be noted in the timeline actions that are taken and how (#review, changed config manually), properly merge code that was deploy out-of-band (#todo, implement the bar limits to foo), and other housekeeping and hygiene that is noticed along the way (#todo, database is running high on it’s connection count) (#todo, add a bounding limit to the workers queue).

While these items should become tickets in whatever work-tracking system that you use, an active incident isn’t the time to do that. This is part of what a postmortem is for.

If you personally have a lull in the action, it’s ok to create these kinds of tasks then, and include them in the chat. That will make it easier to track them during the postmortem.

The Postmortem

One of the hallmarks of a healthy postmortem culture is the expectation that after the incident is resolved, a report of the incident is produced and published.

A Brief Note on Terminology

In most cases, software outages are not responsible for a loss of life. It is for this reason that in many cases the term “Postmortem” is recommended against. While I agree that terms like “Incident Review” or “After Action” are more appropriate, I currently find the term “postmortem” more convenient. I may review my stance at some time in the future, but for now the use of the term will stand.

The Postmortem

I won’t be covering the culture of “Blameless Postmortems”, as that topic and its benefits are well-covered elsewhere. Let us assume that the incident was not the result of a bad actor, and that the people who were involved were authorized to act in the ways that they did, and were working with good intentions. The short version is that a postmortem should identify contributing factors to an incident and identify remediations; it should rarely result in someone losing their job.

In fact, the topic of conducting a postmortem is a large topic by itself. In the interest of not performing a disservice, I will recommend that you look elsewhere for recommended practices for performing a postmortem.

In extremely short detail, all of the participants should come to the postmortem with any supplemental materials that they saw, used, or created during the remediation of the incident. These will be used as supporting documentation in order to build the timeline of events and draw conclusions of the findings during troubleshooting.

Postmortem Supplements

There are a variety of supplements that should be used during a postmortem meeting, and may be included in the report. Here are many of the most common ones.

Notifying Signals: These can be anything that draws attention to the fact that a problem exists. Typical examples could include a pager notification or other alert, or an observation from a customer support team that there is an uptick in customer complaints.
Metrics, Traces, Logs, etc: These may be noticed during the course of other activities, but any metrics, traces, or logs that were helpful for identifying contributing factors should be included.
Chat Logs: While each individual chat log won’t be included in the report itself, it is helpful for constructing the overall incident’s timeline.
Working Notebooks: In organizations that use a shared notebook or other co-edited document, these can be helpful for drafting the postmortem in real-time, as the incident is being worked on. These can also include avenues of approach that don’t ultimately pan out, but they can help tell a more complete story.
Other Timeline Events: These can include anything else that happened during the course of the outline, but outside of the above components. These may include things such as pulling in additional people, or publishing updates to customers.

Actions Taken

In many of the incidents that I’ve participated in, it was very common to attempt small fixes, often somewhat manually, to see if a local experiment shows improvement. These are among the items that should have been tagged by individuals as they communicate during the incident. These should be collected and noted to either clean up experiments that didn’t yield results (undoing changes or re-deploying fresh assets), or be committed and made part of the relevant repositories so that the next deployment doesn’t erase the changes that stabilized the incident.

Long-term Remediations

No team is immune to a “quick solution that we’ll replace later” that lives much longer than intended. The postmortem is also a time to surface work items that may have been identified in the past but have never been prioritized. It is a good time to scan the backlog for old issues that your team had intended to circle back to, “and do it right this time”...

Monitors and other Tests

A healthy question that should be asked in any postmortem is “what could have helped us discover this problem earlier?” In fact, this question should come up for each of the contributors to the incident.

This is something to keep in the back of your head as you work on an incident and you discover contributing factors. As these come up during the incident, note #todo comments of monitors in chat or the working notebook that would have helped alert of the problem earlier.

Additionally, if a software fix was needed, it’s possible that a test for that behavior should be added to the application’s test suite.

Where we got lucky

Sometimes there was foresight or an accidental decision made in the past that kept this incident from being worse than it could have been. It’s a healthy practice to call out these things. Keeping these in mind as you build new products can help bring good practices to the fore, and contribute to building more defensive and resilient applications from the beginning.

The Report

Once all is said and done, you have a detailed, involved document that details the timeline, actions taken, and tasks to be completed. This is a document that you can take pride in. It is a document that most likely very few people will read.

And that is alright.

The document should be widely available, and attached or linked to a broadly-sent announcement of the postmortem, but the announcement will contain what most people will consume: The summary.

The summary doesn’t need to be complicated. A couple paragraphs describing the core dates and times, a broad overview of what was broken and how it was fixed, some highlights of the longer-term fixes that need to be made, and the link to the full postmortem report.

Fixes!

The final artifact that should come out of an incident are closed tasks! It should go without saying that the remediation items that are identified during the incident and postmortem are prioritized, but it is worth calling out.

Those contributing factors have shown themselves capable of contributing to an incident! The mitigations that can be put in place, should be. I would go so far as to say that they should take priority over new-feature development, in fact.

If you should be challenged on that, I would offer that availability is the most important feature of any service!

The Purpose

This may seem like a lot of effort to “rehash yesterday’s news”. However, postmortems can be a key to your future. If your customers see repeated mistakes, that will quickly erode their confidence in you, regardless of whether they are internal or external customers. Postmortems are an opportunity to pause the often frantic pace of development and reflect on past decisions, and should be used to improve the resilience of your systems.

Almost every mistake that I have made in my career has been forgiven. There have been instances where I’ve been rewarded for creating “new and novel” mistakes! No matter how much good will that you have, it will surely erode by repeating the same mistakes.

On the Incident Commander

2021-05-28T15:30:00.001-04:00

As described in my last post, it’s important that stakeholders can get information on an ongoing incident, and trust that the information that they have is relevant, recent, and reputable.

Without a clear understanding of your teams’ incident management protocols, or a lack of trust in the information that they are presented with, it’s understandable that management and other stakeholders will attempt to get status info in the best way that they know how.

As I described last time, this will mean interrupting engineers who are actively working on a problem. This adds stress to the situation, and clogs communication channels, both of which can easily exacerbate the problem.

Enter the Incident Commander

The Incident Commander (IC) is a role that exists to coordinate and delegate effort, and to provide communications. They are the single source of truth for the status of an incident. They are the hub for all an incident’s communications.

One facet of their purpose is to serve as a buffer between stakeholders and the various subject matter experts (SME) who are working to resolve the incident. They are responsible for updating the outward-facing communications channels that the stakeholders rely on for status, whether they be a status website, a chat channel, or a call bridge, or something else entirely. Even when the status of an incident hasn’t meaningfully changed in awhile, such as in a long-running fix, the IC will still update the status. Seeing an updated timestamp on the status, even if that status is still “we’re working on it” helps to reassure the stakeholders that they are being provided with current information.

When engineers are troubleshooting an incident, it’s important that they stay as focused as possible, and delegate as many communication tasks as possible. When SMEs determine that people (teams, skills, etc) are needed, they will call upon the IC to pull in the needed assistance. The IC will be responsible for “paging” out to different teams and individuals, bringing them up-to-speed on the incident, and having them join in the channels where the incident is being coordinated.

When an incident is declared, someone capable of taking on the role of IC should ask the group (whether in a call or chat) “Is there an Incident Commander here?” If nobody identifies as the IC, that person will announce “I assume the role of Incident Commander”, make note of that in the incident timeline, begin gathering status, and communicate outward to stakeholders.

This begs the question “what makes someone capable of taking on the role of Incident Commander?” At this time there are no specific qualifications for someone to take on that role. Some organizations have more rigorous internal programs, but PagerDuty has provided some guidelines at https://response.pagerduty.com/training/incident_commander. Characteristics to look for in an IC is someone who can communicate effectively, delegate, and are familiar with the organization.

In the case of larger or more complex incidents, an IC may find themselves in need of assistance. A common tactic is to delegate some or all of the communications duties. The IC is still the “hub” of all communications, but the communications are executed by a Communications Manager.

It’s important to note that the IC does not actively work to resolve incidents; they aren’t hands-on-keyboard working to resolve the issue. Should it become necessary that the IC becomes hands-on to resolve the issue, it is incumbent upon them to pass on the IC duties to someone else.

In the end, as PagerDuty says, the single sentence description of the Incident Commander’s purpose is to “Keep the incident moving towards resolution.”

On Incident Management

2021-04-16T13:53:00.005-04:00

If you’re making software, you’re breaking software.

You are probably familiar with the yarn from the DevOps community that the source of friction between the historical groups is that Developers want to make changes, and Operators want stability.

The customers want both.

When your company is small enough to fit around a conference room table, and most people can hold the state of the entire environment in their head, incidents don’t require a capital-P “Process”. It’s a very different story when that’s not the case anymore.

When you grow beyond that - where changes are being made without your knowledge, and you can’t predict the impact that one services’ changes might influence another service - that is when you need to consider a more formal process for managing incidents.

Story Time

Once upon a time, I worked for a large company. (Well, this has happened several times…) This company had a lot of customers, and a lot of products, and ran a lot of services with various levels of interdependence. The team that I worked on was a relatively large systems/operations engineering team, and we supported a vast array of services. We were responsible for everything from the OS on up, databases, queues, caching, app routing, monitoring, config management, CI tooling, deployment workflows and tooling, and the availability of the in-house applications themselves.

We also supported a vast number of development teams, all of whom were constantly creating new services, and updating existing ones.

While we had an “Incident Management Process”, it was relatively chaotic. Not for any technical reason. Well, for lots of technical reasons, but what made things worse was the non-technical reasons. As you might expect, this company had many, many layers of leadership hierarchy.

What typically would happen is that when the incident was called, a phone bridge and chat channel would be opened for the engineers to collaborate on the issue. Almost invariably, someone from leadership would join the call asking for a status update. This is not a problem; they should be able to get the current status.

The problem was that they felt the need to “go to the source”. The reason that it was a problem was that demanding to be updated in the channels meant that they were clogging the channels where people were trying to diagnose and solve the issue.

Even that by itself isn’t a huge problem. But as the scale of the company increases, so too do the numbers of stakeholders. And they all want to “go to the source”.

More leaders would join the call, ask for the status, and each would ask for updates.

Eventually, the engineers would be flooded out. They’d start ignoring the requests for updates, and take up backchannels to actually get work done.

What can we take from this?

Those experiences that I described above aren’t unique, but the layers of management strongly exacerbated the problem.

Lack of communications discipline, lack of trust in the update mechanisms, and not trusting that postmortems would be produced all contributed to the frustrations experienced by all involved.

In my next posts, I will describe what an Incident Management process looks like, the artifacts that should come out of an incident, and I will describe a special role that can mitigate these issues.

On Brownness

2020-06-04T01:30:00.001-04:00

If you wonder about the protests:

This is rage at a system that has been abusing it’s unbalanced power for time immoral.

This is a symbol of unjust killings and lesser mistreatment by those who should be protecting all of us!

And I say this as a straight, white, CIS, disabled war veteran of the USMC.

If you are not with the protesters, you aren’t paying attention.

If you don’t live in fear of extrajudicial murder, you’re not paying attention.

If you’re not sure that your death would be adequately investigated, you’re not paying attention.

I have close family members who are police.  I love them.

I have partaken of lethal-force training, and riot-control training, from which I still have physical scars. My (minor) injuries are far more than today’s “law enforcement” have sustained before attacking peaceful protesters.  (To be clear: I suffered a severe abrasion on top of my right wrist when detaining/wrestling a resisting “protester”.  An extremely minor injury.)

I’m ashamed for our current military.  I ache for the soul of our law enforcement.

As the palest among us, I proudly say that “We are all Brown.”

What happens to any of us can happen to all of us.

We are all brown.

On Windermere Sound, Part 2

2019-04-24T01:17:00.002-04:00

If you have the opportunity to move into Windermere Sound in the Orlando area, don’t.

Also, specifically Cindy Riggs of Wemert Realty, you should know that she is a toxic enabling person who is not to be trusted.

You can do better. And not far away, either.

On Windermere Sound

2018-05-23T22:45:00.001-04:00

A little over a month ago, I moved to a new house.

When I mentioned that I was moving to people in the community (such as doctors, dentists, and other professionals in my area), the topic of "Why" would come up. The answer was "Problems with the HOA".

This person would inevitably sigh knowingly and heavily, and launch into a story about the HOA notice that they received saying that their tree bark was exactly the wrong shade of grey-brown, but that tree had been planted at least 7 years ago.

That is not the kind of "HOA Troubles" that I was having.

I can honestly say that it's refreshing where "Crazy HOA Stories" involve notices that the Internet Cable is hung in an unsightly way, rather than "The HOA Board are having parties that exceed 80 decibles as measured inside my closed house, or setting off illegal fireworks", or "The HOA President is setting off a siren while a hurricane is approaching, or he's physically threatening residents". It's refreshing. I haven't had to call the Orange County Sheriff's in over a month.

If you are looking to move to the Orlando area, I would caution you to be aware of the HOA that you are moving into. The HOA's behavior and demeanor can make life either wonderful or awful. When looking at a home, ask who the board members are.

If the board includes a man who's first name sounds like he could be Dr. Frankenstein's assistant and who's last name sounds a bit like "Thor's Son", move on.

If the board includes a woman who's first name sounds like "Sindy", and who's last name might include either a Japanese surname and/or sound like Murtaugh's partner, move on.

Windermere Sound is a nice neighborhood with the exception of a few people, but those people happen to be on the never-voted-upon HOA Board. Life is too short, and these people and those houses are not worth it. Some of their friends are quite morally compromised, too. They're obviously not complete fuck-ups, because they have great kids with better morals than their parents, but at this point being involved with the Windermere Sound HOA in Windermere, FL is a condemnation.

I hope that this changes, and the board should become filled with responsible adults. I'd love to be able to take this post down with a massive "EDIT" at the beginning. But that will require that the existing HOA to follow it's own rules, provide proper notice, and run a legitimate Board vote. Unfortunately, it is not in their own best interests to do so.

Central Florida is nice. I like it here. I used to love it, and would tell everyone I met how great my neighborhood was. This was until the people who became the HOA turned toxic.

I'm happy with my new house. The neighbors are quiet, respectful adults. Which is to say "they act like Adults". Their children are well-behaved. We don't have any fear about letting our kids outside to play, for fear of an unbalanced person harming them. I haven't called the Sheriff's department in over a month.

Things are good in Windermere, FL. Just mind where in Windermere you are.

On Perfume

2017-11-12T21:20:00.002-05:00

Perfume. Originally created to cover the fact that you may have never bathed, and that you may have one or more infections that are rotting your flesh away.

Now, it serves mainly to warn the herd that someone named "Chad" has wandered into your proximity, and may try to corner you into listening about Crossfit.

Men and women, if you MUST, here's how to apply it:

1) If there is a pump that sprays the perfume out, point the nozzle away from you, pump once, then walk into the cloud. Now put the perfume away. Do not pump more than once. Do not repeat. You have enough.

2) If there is no pump, with the cap on the bottle, shake the bottle. Remove the cap, and prepare to stick one finger into the cap. NOTE: Not the bottle! Now, barely touch the part of the inside of the cap that is wet. Put the cap back on the bottle, and put the perfume away. Lightly touch the perfumed part of your finger to your collarbone, or behind the ear. Do not do this more than once. Do not repeat. You have enough.

Here are a couple Don'ts:

* Do not wear perfume to the office. You're not there to pick up anyone unless you're a walking HR Violation. It's only distracting, and not in the good way.

* Do not wear perfume out to dinner. You're fucking with everyone else's sense of smell and taste.

=====

This part is for the men.

Guys, look, I get it. I used to bathe myself in perfume because I thought women were into guys that smelled like the inside of bottles that were shown in the foreground of cowboys. I didn't want to have "too little" perfume on, and miss out on them being unable to keep their hands off of me. I get it.

I even had one girlfriend who Loooooved a particular perfume, so bought me a bottle of "Curve". Yes, she did like that perfume.

Here's the thing: If you want to attract the kinds of women that are attracted to a perfume, you're going to be easily replaced. Shallow can be fun for awhile. But if some bottled chemicals are what separate you from the next guy, I feel sad for you, bro.

So, that girlfriend? She tried to play a bunch of headgames to get me to... fuck, I don't know what she was doing. Maybe get me to try to prove to her that I was worthy, or mental judo to convince myself that putting in that much effort meant that she must be worth it. Needless to say, it didn't last, and the way she ended things won her no friends. She bought the new boyfriend some Curve, too.

Perfumes might be a cheat against pheromones, but that's all they'll be. If you stink, you'll be better off getting clean.

If you want a woman to miss your smell when you're not there, you want that smell to be you, not something from a cosmetic. If "your smell" that she misses is a perfume, you're just a vibrator and a heated blanket away from being completely replaceable.

In the days before I went to Iraq for most of 2005, I set aside a t-shirt, and wore it at night while I slept. As I was doing my final packing up, I put it into a gallon ziplock bag, and set it aside in a drawer. After I got in-country and could make a call, I told my wife what I'd done and where it was. This was so that when she missed me the most, she could open that bag, and have a little bit of my smell. I don't know how long she kept it, but it was appreciated, especially as time passed and my smell faded from our house.

As a side note, if you take exception to it being called "perfume" and insist on calling it "cologne" or "body spray", I challenge you to tell me the fucking difference, especially if yours wasn't created and packaged in a particular region of France.

=====

Those ads with hordes of women swarming a guy because he's spraying himself are appealing. What if you look at it this way: What if they're not drawn to him? What if they're rushing at him to beat him to death? He has marked himself, and they figure that one murder shared among them is worth not having to hear any more goddamn Crossfit stories.

WWDC, Again, 2017 edition

2017-06-04T21:28:00.000-04:00

Hey, it's time for another WWDC! Granted, I haven't been doing iOS development in awhile, and find that I'm not eyeing the latest hardware as much as I used to, but it's still at least a little fun.

For a lot of people, there is a lot of fun and money to be had from the Apple Speculation Game.

For me, I don't care nearly so much. Historically, I have bought iPhones every 3 years, and iPads nearly as often. My most recent iPhone is a 6S, so, I probably won't buy a new phone in this coming year.

But here's what I hope for:
-An iPhone 6S,
-Retaining the headphone jack,
-With more storage available,
-Faster would be fine, but isn't required,
-And rather than be thinner,
-Make the front and back flush with the thickest part (the camera),
-Taking up any extra space with battery.

There. That's it for iPhone.

I love my 6S. It's height and width are fine. I like having a Lightning connector as well as a headphone jack. I use my headphone jack REGULARLY. I have several different pairs of headphones. I often swap what pair of headphones is on my head into a different device while sitting at work; I use different headphones for different things. Likewise, I may swap what headphones are in my device.

Bluetooth does not give me that flexibility. The cords don't bother me that much. I will probably buy a nice pair of Bluetooth headphones in the next year or so, but I don't want it to be my only option.

As far as the body being flush on front and back, I hearken back to my iPhone 5 (as well as the 4/4S and 5S phones). My 6S is the first phone that I felt that I needed a case on it. I had my 3GS and 5 on my person day-in and day-out for each of their 3-year lives. While the backs were a little worn, that didn't bother me. I loathed the idea that I would have to make the device thicker just to protect something that I had no intention of selling, so I didn't need to protect it. They each suffered a couple falls, but nothing damaging.

My 6S, on the other hand, drove me nuts, because the camera sticks out. The phone does not lie flat on a table. So, in order to keep from damaging the camera, or causing stress to the phone when laying it on a table repeatedly, I bought a case. I got a leather case which I'm quite pleased with, but I hate that I needed it.

I would much rather have a thicker phone with better battery life.

Apple's product lineup from this last year disappointed me. I didn't care about the iPhone 7, but I wasn't going to get a new iPhone unless my current one broke. But I wanted to buy a new Mac Laptop, or a cheap laptop, and an iMac. Typically I would buy the mid-level 15" MBP, but this year I'd intended to buy a pretty much maxed-out 13". But then Apple "Innovated". "Bravely."

I don't know anyone who cares about the touch-bar. I know a lot of people who care about a goddamn Escape Key. I care about the Escape Key.

I know people who want an Apple Desktop. I was ready to buy an Apple Desktop. Guess who didn't buy an Apple Desktop.

In fact, I was so disgusted by Apple's lineup, I went all the way around, and found what is almost my ideal setup: A Razer Stealth Blade with the Razer Core. This is an Ultrabook, paired with a Thunderbolt 3 Dock that allows me to run a desktop GPU. I love this setup.

My two complaints are 1) the trackpad isn't as nice as Apple's, and 2) it's running Windows. Now, this isn't a knock on Windows, per se; I had administered Windows for a long time. I can make it work. But I'd rather be running MacOS, by a long shot. Aside from gaming, my main application categories are a web browser, terminal, and text editor. Editors and browsers are fine on Mac, but I want ZSH with oh-my-zsh, my .zshrc, and a Unix toolchain. Yes, I can install a *nix distro on it, but I'd rather have the MacOS without having to jump through a shitload of hoops.

I'm sorry, but Synaptics is not nearly as good as Apple's touchpad. I wish it weren't the case.

The one upside is that the number of usable games in my Steam library doubled immediately, and I've been able to play a fair number of games that used to be completely inaccessible to me before.

Too bad Apple couldn't provide something that I could use. I needed a laptop, but all Apple provided were severely compromised options.

Maybe this year will have better options.

Dear Vendors

2017-03-16T12:52:00.001-04:00

Dear Vendor,

Since I didn’t see an unsubscribe link, I replied with the subject changed to “Unsubscribe”.

That didn’t mean reply some more.

When I replied to that with the subject and body changed to “Unsubscribe”, that was a less-subtle hint.

When you reply to that (maybe including a quote or some byline of mine) not along the lines of “sorry to trouble you”, I read that as “please block and report me as spam."

I know that you're a person with a job to do, but when I express a lack of interest, please take the hint. Let's not waste both of our time.

That Product Team Really Brought The Room Together

2016-12-23T00:01:00.000-05:00

As with last year, I participated in SysAdvent again, a tradition of Systems Administrators to submit to write a blog article of their choosing. Similar to a Conference, a Call for Proposals is published, and if interested, you propose a Topic (a title and brief description are all that are required, if I recall correctly. My proposal was accepted, an editor was assigned, and due dates were set.

I'd had a lot of notes of things that I wanted to say, but I ended up struggling with constructing a coherent narrative, but lots of smallish topics that would be highly overlapping in a Venn diagram, but didn't present linearly in my mind. Again, being something of a followup of last year's article/talk, there's a lot of related material, but also a lot omitted for time and space reasons. (I needed to end it _somewhere_...)

Again, I offer my thanks to my editor Cody and the rest of the SysAdvent team for maintaining this tradition, and I sincerely appreciate the work that goes into keeping it going. While I'd had this bunch of ideas floating around in my head (largely as a conference presentation), it was their work that made me sit down and write it out as prose, rather than a slide deck.

Now, with permission from the author, I present:

Day 23 - That Product Team Really Brought The Room Together

Written by: H. “Waldo” Grunenwald (@gwaldo)
Edited by: Cody Wilbourn (cody@codywilbourn.com)
There are plenty of articles talking about DevOps and Teamwork and Aligning Authority with Responsibility, but what does that look like in practice?
Having been on many different kinds of teams, and having run a Product Team, I will talk about why I think that Product Teams are the best way to create and run products sustainably.

HEY, DIDN’T YOU START WITH “DEVOPS DOESN’T WORK” LAST TIME?

Yes, (yes I did). And I believe every word of it. I consider Product Teams to be a definitive implementation of “Scaling DevOps” which so many people seem to struggle with when the number of people involved scales beyond a conference room.
To my mind, Product Teams are the best way to ensure that responsibility is aligned with authority, ensuring that the applications that you need are operated sustainably, and minimizes the likelihood that a given application becomes “Legacy”.

What do you mean “Legacy”?

There is a term that we use in this industry, but I don’t think that I’ve ever seen it be well-defined. In my mind, a Legacy Product is:

Uncared For: Not under active development. Any releases are rare, using old patterns, and are often the result of a security update breaking functionality, causing a fire-drill of fixing dependencies.
In an Orphanage: The people who are responsible for it don’t feel that they own it, but are stuck with it.

If there is a team that actively manages a legacy product, they might not be really equipped to make significant changes. Most of the time they are tasked only with keeping this product barely running, and may have a portfolio of other products in similar state. This “Legacy Team” might have some connotation associated with it of being “second-string” engineers, and it might be a dumping ground for many apps that aren’t currently in active development.

What are we coming from?

The assumed situation is there is a product or service that is defined by “business needs”.
A decision is come to that these goals are worthwhile, and a Project is defined.
This may be a new product or service, or it may be features to an existing product or service. At some point this Project goes into “Production”, where it is hopefully consumed by users, and hopefully it provides value.
Here’s where things get tricky.
In most companies, the team that writes the product is not the same team that runs the product. This is because many companies organize themselves into departments. Those departments often have technical distinctions like “Development” or “Engineering”, and “Quality Assurance”, and an “Operations” and/or “Systems” groups. In these companies, people are aligned along job function, but each group is responsible for a phase of a product’s lifecycle.
And this is exactly where the heart of the problem is:
The first people who respond to a failure of the application aren’t the application’s developers, creating a business inefficiency:
Your feedback loop is broken.
As a special bonus, some companies organize their development into a so-called “Studio Model”, where a “studio” of developers work on one project. When they are done with that project, it gets handed off to a separate team for operation, and another team will handle “maintenance” development work. That original Studio team may never touch that original codebase again! If you have ever had to maintain or operate someone else’s software, you might well imagine the incentives that this drives, like assumptions that everything is available, and latency is always low!
See, the Studio Model is patterned after Movie and Video Game Studios. This can work well if you are releasing a product that doesn’t have an operational component. Studios make a lot of sense if you’re releasing a film. Some applications like single-player Games, and Mobile Apps that don’t rely on Services are great examples of this.
If your product does have an operational component, this is great for the people on the original Studio team, for whom work is an evergreen pasture. Unfortunately it makes things more painful for everyone who has to deal with the aftermath, including the customers. In reality it’s a really efficient way of turning out Legacy code.
Let’s face it, your CEO doesn’t care that you wrote code real good. They care that the features and products work well, and are available so that they bring in money. They want an investment that pays off.
Having Projects isn’t a problem. But funding teams based on Projects is problematic. You should organize around Products.

OK, I’LL BITE. WHAT’S A PRODUCT TEAM?

Simply put, a Product Team is a team that is organized around a business problem. The Product Team is comprised of people such that it is largely Self-Contained, and collectively the team Owns it’s own Products. It is “long-lived”, as the intention behind it is that the team is left intact as long as the product is in service.
Individuals on the team will have “Specialties”, but “that’s not my job” doesn’t exist. The QA Engineer specializes in determining ways of assuring that software does what’s expected to. They are not responsible for the writing of useful test cases, but they are not limited to the writing of tests. Notably, they’re not solely responsible for the writing of tests. Likewise for Operations Engineers, who have specialties in operating software, infrastructure automation, and monitoring, but they aren’t limited to or solely responsible for those components. Likewise for Software Engineers…
But the Product Team doesn’t only include so-called “members of technical staff”. The Product Team may also need other expertise! Design might be an easy assumption, but perhaps you should have a team member from Marketing, or Payments Receivable, or anyone who has domain expertise in the product!
It’s not a matter of that lofty goal of “Everyone can do everything.” Even on Silo teams, this never works. This is “Everyone knows enough to figure anything out“, and ”Everyone feels enough ownership to be able to make changes."
The people on this team are on THIS team. Having or being an engineer on multiple teams is painful and will cause problems.

You mentioned “Aligning Authority with Responsibility” before…

By having the team be closely-knit, and long-lived, certain understandings need to be had. What I mean is that if you want to have a successful product, and a sustainable lifecycle, there are some understandings that need to take place with regards to the staffing:

Engineers have a one-to-one relationship to a Product Team.
Products have a one-to-one relationship with a Product Team.
A Product Team may have a one-to-many relationship with it’s Products.
A Product Team will have a one-to-one relationship with a Pager Rotation.
An Engineer will have a one-to-one membership with it’s Pager Rotation.

Simply put, having people split among many different teams sounds great in theory, but it never works out well for the individuals. The teams never seem to get the attention required from the Individual Contributors, and an Individual Contributor is in a position of effectively doubling their number of bosses having to appease them all.
Some developers might balk at being made to participate in the operation of the product that they’re building. This is a natural reaction.
They’ve never had to do that before. Yes, exactly.
That doesn’t mean that they shouldn’t have to. That is the “we’ve always done it this way” argument.
This topic has already been well-covered in another article in this year’s SysAdvent, in Alice Goldfuss’ “No More On-Call Martyrs”, itself well-followed up by @DBSmasher’s “On Being On-Call”.
In this regard, I say is that if one’s sleep is on the line - if you are on the hook for the pager - you will take much more care in your assumptions when building a product, than if that is someone else’s problem.
The last thing that amazes me is that this is a pattern that is well-documented in many of the so-called “Unicorn Companies”, who’s practices many companies seek to emulate, but somehow “Developers-on-Call” always is argued to be “A Bridge Too Far”.
I would argue that this is one of their keystones.

WHO’S IN CHARGE

Before I talk about anything else, I have to make one thing perfectly clear. If you have a role in Functional Leadership (Engineering Manager, Operations Director, etc), your role will probably change.
In Product Teams, the Product Owner decides work to be done and priorities.
Within the team you have the skills that you need to create and run it, delegating functions that you don’t possess to other Product Teams. (DBA’s being somewhat rare, and “DB-as-a-Service” is somewhat common.)
Many Engineering and Operations managers were promoted because they were good at Engineering or Ops. Unfortunately it’s then that it sets in that, in Lindsay Holmwood’s words, “It’s not a promotion, it’s a career change”, and also addressed in this year’s SysAdvent article “Trained Engineers - Overnight Managers (or ‘The Art of Not Destroying Your Company’)” by Nir Cohen.
How many of you miss Engineering, but spend all of your time doing… stuff?
Under an org that leverages Product Teams, Functional Leaders have a fundamentally different role than they did before.

Leadership Roles

Under Product Team paradigm, Product Managers are responsible for the work, while Functional Managers are responsible for passing of knowledge, and overseeing the career growth of Individual Contributors.

Product Managers	Functional Managers
Owns Product	IC’s Professional Development
Product Direction	Coordinate Knowledge
Assign Work & Priority	Keeper of Culture
Hire & Fire from Team	Involved in Community
Decide Team Standards	Bullshit Detector / Voice of Reason

Product Managers

The Product Manager “Owns the Product”. They are ultimately responsible for the product successfully meeting business needs. Everything else is in support of that. I must stress that it isn’t necessary that a Product Manager be technical, though it does seem to help.
The product owner is the person who understands the business goals that knowledge and those stakes, they assign work and priorities such that it’s aligned with those business goals.
Knowing the specific problems that they’re solving and the makeup of their team, they are responsible for hiring and firing from the team.
Because the Product Team is responsible for their own success, and availability (by which I mean, of course, the Pager), they get to make decisions locally. They get to decide themselves what technologies they want to use and suffer.
Finally, the Product Manager evangalizes their product for other teams to leverage, and helps to on-board them as customers.

Functional Managers

At this point, I expect that the Functional managers are wondering “well what do I do?” Functional Managers aren’t dictating what work is done anymore, but there is still a lot of value that they bring. Their job becomes The People.
I don’t know a single functional manager who has been able to attend to their people’s professional development like they feel that they should.
Since technology decisions are made within the Product Team, the Functional Management has a key role in coordinating knowledge between the members of their Community, keeping track of who’s-using-what, and the relevant successes and pitfalls. When one team is considering a new tool that another is using, or a team is struggling with a tech, the functional manager is well-equipped for connecting people.
Functional Managers are the Keepers of Culture, and are encouraged to be involved in Community. That community-building is both within the company and in their physical region.
Functional managers are crucial for Hiring into the company, and helping Product Managers with hiring skills that they aren’t strong with. For instance, I would run a developer candidate by a development manager for a sanity-check, but for a DBA, I’d be very reliant on a DBA Manager’s expertise and opinion!
Relatedly, the Functional Manager serves as a combination Bullshit Detector and Voice-of-Reason when there are misunderstandings between the Product Owners and their Engineers.

The Reason for Broad Standards

Broad standards are often argued for one of two main reasons: either for “hiring purposes”, where engineers may be swapped relatively interchangably, or because there is a single Ops team responsible for many products, who doesn’t have ability to cope with multiple ways of doing things. (Since any one Engineer might be called upon to solve many apps in the dark of the night.)
Unfortunately, app development can often be hampered by those Standards that don’t fit their case and needs.
Hahahaha I’m kidding! What really happens is that Dev teams clam up about what they’re doing. They subvert the “standards” and don’t tell anyone, either pleading ignorance or claiming that they can’t go back and rewrite because of a deadline. Best case is that they run a request for an “exemption” up the flagpole, where Ops gets Over-riden. And Operations is still left with a “standard” and pile of “one-offs”.

Duplicate Effort

Another claimed reason for broad “Standards” is to “reduce the amount of duplicated effort”. While this is a great goal, again, it tends to cause more friction than is necessary.
The problem is the fallacy that comes from assuming that the way that a problem was solved for one team will be helpful to another. That solution may be helpful, but to assume that it will, and making it mandatory is going to cause unnecessary effort.
At one company, my team ran ELK as a product for other teams to consume. A new team was spun up, and asked about our offerings, but asked my opinion of them using a different service (an externally-hosted ELK-as-a-Service). I was thrilled, in fact! I want to see if we were solving the problem in the best way, or even a good way, and to be able to come back later for some lessons-learned!

Scaling Teams

At some point, your product is going to get bigger than everyone can keep in their head. It may be time to split up responsibilities into a new team. But where to draw boundaries? Interrogate them!
A trick that I learned a long time ago for testing your design in Object-Oriented Programming is to ask the object a question: “What are you?” or “What do you do?” If the answer includes an “And”, you have two things. This works well for evaluating both Class and Method design. (I think that this tidbit was from Sandi Metz’s “Practical Object-Oriented Design in Ruby” (aka “POODR”), which I was exposed to by Mark Menard of Enable Labs.)

What Doesn’t Work

Because this can be a change to how teams work, it’s important to be clear about the rules. If there is a misunderstanding about where work comes from, or who the individual contributors work for, or who decides the people who belong to what team, this begins to fall apart.
Having people work for multiple sets of managers is untenable.
Having people quit is an unavoidable problem in any company. Having a functional manager decide by themselves that they’re going to reassign one of your people away from you is worse, because they’re not playing by the rules.

WARNING: Matrix Organizations Considered Harmful

If someone proposes a Matrix Org, you need to be extremely careful. It’s important that you keep a separation of Church and State. Matrix Organizations instantly create a conflict between the different axes of managers, with the tension being centered on the individual contributor who just wants to do good work. A Matrix Org actively adds politics.
All Work comes from Product Management. Functional Management is for Individual Careers and Sharing Knowledge.
This shouldn’t be hard to remember, as the Functional Leaders shouldn’t have work to assign. But it will be hard, because they’ll probably have a lot of muscle-memory around prioritizing and assigning work.
Now, I’m sure a lot of you are skeptical about how a product team actually works. You might just not believe me.
If you properly staff a team, give them direction, authority, and responsibility, they will amaze you.

GETTING STARTED

As with anything, the hardest thing to do is begin.

Identifying Products

An easy candidate is a new intiative for development that may be coming down the pipeline, but if you aren’t aware of any new products, you probably have many “orphaned” products already running within your environment.
As I discussed last year, there are plenty of ways of finding products that are critical, but not actually maintained by anyone. Common places to look are tools arounddevelopment, like CI, SCM, and Wikis. Also commonly neglected are what I like to call “Insight Tools” like Logging, Metrics, and Monitoring/Alerting. These all tend to be installed and treated as appliances, not receiving any maintenance or attention unless something breaks. Sadly, it means that there’s a lot of value left on the table with these products!

Speaking with Leadership

If you say “I want to start doing Product Team”, they’re going to think of something along the lines of BizDev. A subtle but important difference is to say that you want to organize a cross-functional team, that is dedicated to the creation and long-term operation of the Product.
I don’t know why, but it seems that executive go gooey when they hear the phrase “cross-functional team”. So, go buzz-word away. While you’re at it, try to initiate some Thought Leadership and coin a term with them like “Product-Oriented Development”! (No, of course it doesn’t mean anything…)
What you’re looking for is a commitment to fund the product long-term. The idea is that your team will solve problems centered around a set of problems. The team is of “Your People”, that becomes a “we”. Oddly enough, when you have a team focused and aligned together, you have really built a capital-T “Team”.

SUSTAINED

The Product Team should be intact and in-development as long as the product is found to be necessary. When the product is retired, they product team may be disbanded, but nobody should be left with the check. Over time, the features should stabilize, and the bugs will disappear, and the operation of the application should stabilize to a low level of effort, even including external updates.
That doesn’t mean that your engineers need to be farmed out to other teams; you should take on new work, and begin development of new products that aid in your space!

CONCLUSION

I believe that organizing work in Product Teams is one of the best ways to run a responsible engineering organization. By orienting your work around the Product, you are aligning your people to business needs, and the individuals will have a better understanding of the value of their work. By keeping the team size small, they know how the parts work and fit. By everyone operating the product, they feel a sense of ownership, and by being responsible for the product’s availability, they’re much more likely to build resilient and fault-tolerant applications!
It is for these reasons and more, that I consider Product Teams to be the definitive DevOps implementation.

GRATITUDE

I’d like to thank my friends for listening to me rant, and my editor Cody Wilbourn for their help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

CONTACT ME

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts and Steam, and seldom refuse hugs (or offers of beverage and company) at conferences. Death Threats and unpleasantness beyond the realm of constructive Criticism may be sent to:

Waldo  
c/o FBI Headquarters  
935 Pennsylvania Avenue, NW  
Washington, D.C.  
20535-0001

On The DevOps Drinking Game

2016-11-01T10:58:00.003-04:00

Just for laughs, as part of my "Fear and Loathing in Systems Administration" conference talk, I saw a real gap in our community's resources. My research turned up nothing, so I took it upon myself to "be the change that I want to see in the world.

Thus was born a new GitHub project: The DevOps Drinking Game.

Enjoy, and Be Safe!

-Waldo

On the Loss of a Star

2016-08-29T18:54:00.002-04:00

Someone that I don't know died today. Apparently many of my friends have close personal relationships with famous actors and musicians. Apparently they don't invite me along when they hang out with their celebrity-friends. I can't blame them.

Fortunately for me, the works that I knew these celebrities from has been recorded, so that I may continue to enjoy their works as I always have.

For those of you who had a relationship with the deceased, I am sorry for your loss.

Fear and Loathing in Systems Administration

2015-12-31T20:00:00.000-05:00

This year I participated in SysAdvent, a tradition of Systems Administrators to submit to write a blog article of their choosing. Similar to a Conference, a Call for Proposals is published, and if interested, you propose a Topic (a title and brief description are all that are required, if I recall correctly. My proposal was accepted, an editor was assigned, and due dates were set.

After fleshing out some further notes and constructing an outline, I set out to procrastinate until a couple of days before my due-to-editor deadline. My editor, the fantastic Shaun Mouton (@sdmouton), promptly reviewed my content for style, sense, and sanity, and when we came to consensus that it was good enough, he submitted it for publication.

I offer my thanks to Shaun and the rest of the SysAdvent team for maintaining this tradition, and I sincerely appreciate the work that goes into keeping it going. While I'd had this bunch of ideas floating around in my head (largely as a conference presentation), it was their work that made me sit down and write it out as prose, rather than a slide deck.

Now, with permission from the author (), I present:

Fear and Loathing in Systems Administration

Written by H. “Waldo” Grunenwald (@gwaldo)
Edited by Shaun Mouton (@sdmouton)

“DEVOPS DOESN’T WORK”

The number of times that I’ve heard this is amazing. The best thing about this phrase is that the people who say it are often completely right, even if for very wrong reasons.

Who Says This?

Well, let’s talk about the people who most commonly have this reaction: SysAdmins. I’m going to use the term “SysAdmins” as a shorthand for a broad group. The people in this group have widely varying titles, but it is most commonly “Systems”, “Network”, or “Operations” follwed by “Administrator”, “Engineer”, “Technician”, or “Analyst”.

In some companies, these folks have the best computers in the place. In others, they have to live with the worst. Their workspace probably isn’t very nice, and almost certainly has no natural light. If there is a pager rotation, they are almost certainly on it. If there isn’t a rotation, they’re basically on-call all of the time.

During the course of a normal day they might have to switch contexts between disaster planning, calculating power and HVAC needs for a new datacenter, scrambling to complete an outage-driven SAN migration, rushing to address urgent requests to help people with their email, to troubleshoot a printing problem, or suss out why someone can’t get to their electric company’s bill pay website. They may be the sole person with database expertise in the company, or they may work on a team of dozens.

The work is largely invisible except when something fails, in which case it’s highly visible and widely impacting.

These are typically cynical people, because there are only so many times that you can’t make the team/department/company party for ostensibly “celebrating our successes” because something’s broken, and you’re left to clean up after the “success”. There are only so many times that one sees a new project announced and begins to hire more people. When asked who’s going to support the new project, the response is a blank look and “you are”. The “…of course.” may not be vocalized, but it’s probably there. When asked how many people they get to hire to help with the workload, the response is a combination of “sorry, but there wasn’t anything left in the budget”, “it won’t be that much more work”, or a variation of the “team player / good soldier” speech. There are only so many times one can take getting your requests for training or conference budget rejected out of hand, and have your requests for training or conference budget laughed out of the room.

They probably have basic working knowledge of a half-dozen programming languages, but most likely they often think in Shell. They probably know at least three ways of testing if a port is open, and probably have a soft spot in their heart for a couple of shell commands.

They may have seen or participated in a DevOps initiative that consisted of a team or position rename, or helpfully suggested that they install some Config Management and Monitoring software so that “we can DevOps now…” or “so we can do Agile”. When they hear “DevOps” or “Agile”, what they are hearing is is Let’s take the same people who can’t handle a planned release schedule or make whatever effort that they need to squeak by the Change Board and Release Management requirements, and give them unfettered access to Production. Clearly, I’m not paged often enough.

So what is one to do? How is one to maintain their sanity in the face of increasing job scope, increasing demand for access and velocity, and little hope for an effective new-hire count? Not to mention continuing to juggle the existing volume of requests, and continuing to grease the existing gears to keep the machine running.

GET HELP

Please note that I’m not saying “just”. There’s nothing just about this situation; there is nothing simple about any of this, and Justice hasn’t been seen in a long time in an environment where this is the norm. Most of these changes are difficult. They will take work, and will require convincing other teams to join in your cause.

Admitting you have a Problem

The problem (probably) isn’t technical. It’s almost entirely social.

Because SysAdmins are typically responsible for the environment, the easiest way to assure that the state is stable is to lock everyone else from it. While this helped with the goal of “keeping out unexpected changes”, it had a number of side effects.

First, a kind of learned helplessness has set in. Your customers and teammates became so used to being “hands-off”, that they don’t have the wherewithal to meet reasonable expectations. Since they’re uncomfortable making any changes, all changes must be made by the SysAdmins. This leads to your time being taken by having to perform lots of low-value tasks.

Some teams settle on the pattern of “hands off Production, but you have access to Staging”, but this is fraught with peril. The most common problem that stems from this is “Configuration Drift.” Config Drift is when you have different settings in one environment (or server) than the others. When the cost to discover what Production looks like is high, it’s more likely that people will either use defaults, make assumptions, or use the same configs that they use in their IDEs. “Works on my machine”, indeed.

This is a problem well-solved by Configuration Management tools, but you still need to be willing to trust your peers and give them access. If you want to be part of the process of validating changes, you could put in place the structures that allow a pull-request and code-review workflow, something that your Software Engineering peers should be very accustomed to! Granting access to see the existing configs and the ability to propose changes also shares responsibility for your team’s environments and contributes to feelings of ownership. Denying colleagues the ability to effect necessary configuration changes contributes to the root problems of configuration drift and learned helplessness.

Stop Feeding the Machine

Your value is not in doing the work, but rather being able to make the decision to do the work.
I’ll be the first to say that “Automating ALL THE THINGS” is a flawed goal. At work, it’s usually said in the context of a Project, rather than part of a philosophy of Continual Improvement (Think Toyota Kata). You shouldn’t have to engage in an “Automation Project” to improve your environment. Build into your schedule time to solve one problem. Pick something that is rough, manual, and repeatable. Remove a small piece of friction. Move on to the next one. Hint: Logging into a server to make a configuration change should be a cue to implement configuration management!

While I agree that everything being automated, not everything should be automatic. Decision-making is complex, and attempting to codify all of the possible decision-making points is a fantastic way to make yourself insane. Not to mention that documenting your decision-making processes may be an unwanted look inside your brain. Caveat Implementor. (Or perhaps that’s just me…)

All of the units of work should be automated. But the decision to run the now-automated tasks can be left to a human. When you find that there is a correlation between steps, those pieces should be wrapped together. Automation isn’t a project into itself. It should be iterative. Pick something that’s painful. Make it a little smoother. Repeat. Ideally, you have time blocked out for Continuous Improvement. If not, create a meeting, or create a weekly project to do so. Review the issues that you’ve experienced lately, and pick something to make better. It might be worth making into a project, but it won’t be an ALL-THE-THINGS project. Create a scope of effort. Take the time to plan goals and requirements.

Whatever you don’t automate must be documented. Beyond the typical benefits of documentation, it also serves as “Functional Requirements” for someone else to pick up when they can help you with providing a solution. Try to recognize whether documenting or automating takes longer. Perhaps this piece of documentation will bet better served by “Executable Documentation” (i.e. code).

Clarify Your Role

You should attempt to pick apart the parts of your work, and attempt to describe them. One way to make this a fun exercise is to use other job titles to describe the work.

Are you an “Internet Plumber”? How much of your job could be described as “Spelunking” into the deep dark caverns of Legacy systems?

If you want, you could ascribe Superhero names to these parts of your work. The added bonus is that it not only describes a role, but also a demeanor associated with them. When ‘bad code’ makes it to Production, do you go “Wolverine” on that dev team?

Could you describe part of your role as “Production Customs Official”? Are you the gateway to Production? If so, are you actually equipped to do that? Here’s a quick test: When you say “no, that can’t go”, do you get overridden?

More importantly, is this what you want to do?

Prepwork

You will need to prepare for this. Most SysAdmin teams do not have a healthy relationship with the rest of the business. You will need to initiate the healing.

Take someone to lunch. Preferably someone who you don’t know well. Ask questions, and listen to the answers. It is not time to defend yourself or your team. It’s time to find out what the business needs from someone else’s perspective. Ask what they think that your team’s role is in toward achieving that success. Ask what they think your team does well, and where there are gaps between what you have now and excellence.

Speak their language

You probably recognize their words, but you need to go out of your way to speak them. To communicate your message, you will need meet them on their turf. This may seem terribly unfair - “Why can’t they meet me on my terms?!” - but I’m guessing that has not been working out well for you so far.

Not only do you need to use their language, but you need to communicate over their medium. And identifying who they are is step one in learning to speak it. It’s probably not IRC, and only writing it in email is a good way for it to be ignored.

If you’re speaking to management, be prepared to write a presentation. Executives especially like to see a slide-deck. It doesn’t have to be slick. It probably shouldn’t have sounds or much in the way of transitions, but a presentation can help to lay the groundwork for a conversation.

Discuss Scope, Staffing, and Priorities

Now that you have described your role, we also need to describe everything that you support.
What Products do you support? It’s entirely possible (likely, even) that the people and teams that you support don’t actually know what you’re responsible for. It could be argued that most of them shouldn’t need to know. But if you have been saying “no” to protect yourself, it’s a sign that you are significantly overextended. You need to have a real discussion with your leadership about your role, scope, and staffing.

In order to have this discussion, you need to prepare. You need to come up with a fairly comprehensive list of the products and teams that you support. This is a list of every team, and their products, the components and tasks that belong to you for each. Don’t forget all of the components that “nobody owns” but somehow people come to you to fix or implement (CI, SCM, Ticketing, Project, and Wiki tools seem to be common examples). Are you also responsible for Directory Services? Virtualization platform? Mail/Chat/Phones? Workstation Purchasing and provisioning? Printers? Do you manage the Storage, Networking, etc? Don’t be afraid of getting into details. It can help to provide clearly written potential impacts the company if some of these “hidden” services stop working? Your leadership might not know what LDAP or Directory Services are, but they’ll understand if nobody can log into their machines, they can’t pull information to build reports, and by-the-way nobody can deploy code because it relies on validating credentials…

What is most important to the company? What do you need to succeed? How much more staff do you need? What tooling or equipment would help you work more efficiently? Does code deploy even when it fails testing? How many outages have arisen due to this happening?

Demonstrate Cost and Value and Revisit Priorities

In order to have meaningful discussions with people in your company who aren’t necessarily technical, you need to be able to relate to a language that they speak. Regardless of team duties, the lingua franca of most teams is money. As Engineers, most of us prefer to think in terms of the tech itself, but in order to describe an impact, a unit of monetary value is a proxy for impact that most non-technical people can understand, even if they don’t grasp the details.

It is a helpful (if difficult and uncomfortable) habit to get into, but I encourage you to consider the components of cost that goes into every incident or task.

What is the cost of a main-site outage? How much revenue does this feature bring in? Why are you spending so much on infrastructure and effort to make that component Highly-Available? Why does it matter that you do that piece of maintenance? Show the negative value of doing things they way they are (Opportunity Cost), versus investing time to improve the automation around it. Describe how doing this maintenance work reduces your context switching, unplanned outages, and lost reputation of your company. Describe the benefit in increased visibility to the business, and Agency to be gained by your peers on other teams.

Why put in place these tools to let product teams self-serve? Describe that the features that the company’s teams spend so much time and effort (read: “money”) creating means nothing if those features aren’t available for customers to use. That having those features not available costs money in terms of feature billing, and reputation cost. If they claim that they’re doing Agile, but can’t do Continuous Delivery, they’re not really Agile, and the whole point of that framework is to improve delivery of value to the customer and the business!

Further, show how systems relate. It doesn’t have to be terribly detailed. Describe that the features that the customers use are reliant on x, y, and z components of infrastructure. Draw the lines from LDAP to storage to your CI tool to testing code to artifacts delivered to Production. Then show some of the other systems that have similar dependencies.

Once the picture emerges showing how everything is reliant on unexciting things like LDAP, your Storage cluster, and that janky collection of angry shell and perl scripts that keep everything working, realization will begin to dawn.

Congratulations, you’ve just effectively communicated Value.

Align Responsibility with Authority

Are you held responsible for apps written by other people? Who gets paged when “the app” goes down? How does that make sense?

Get Devs on-call for their apps. SysAdmins should be escalated to. Devs can triage and troubleshoot their own apps more readily than you can. They get to call in the cavalry when they get stuck. They don’t need to know everything about the systems, and they don’t need to resolve everything. When a fault occurs and they need help, they stay on the call, pairing with you as you diagnose, troubleshoot, and resolve. That way, they don’t need to escalate to you for that thing the next time it occurs, and can collaborate on automating a permanent fix.

When teams aren’t responsible for their products - When they aren’t paged when it fails - they are numb to the pain that they inflict. They’re not trying to cause pain; they just don’t feel it. It’s especially easy to argue this for teams that proclaim that they use Agile development methods: If they claim to want “continuous feedback”, there is nothing more visceral for providing feedback than the feeling of being awoken by a pager in the middle of the night. When the inevitable exclamation comes that “we can’t interrupt our developers”, ask if it makes sense to interrupt someone else.
Even being aware of the pain (say, hearing how many times you were paged last night) can elicit sympathy, but that’s a far cry from the experience of being paged yourself.

Further, this is what that list of responsibilities is for. Asking each team to take responsibly for their own products, you will still likely have a hefty list of services that you provide that you are on-call for. As these set in, point out the staffing numbers. This may be a matter of the places that I have worked, but I have never seen a Developer-to-SysAdmin ratio of less than 5-1. In most places it is much higher. By adding these teams to pager rotations, they drastically reduce the load on you. By not adding them to pager rotations, they are complicit in your burnout.

Stop saying “No”

SysAdmins have a reputation for saying “No”. The people who are asking are probably not trying to make your life worse; They’re probably just trying to get their work done. They might not know what their “simple request” involves, and that it probably isn’t necessary.

But by not having Responsibility aligned with Authority, you may have been stuck with the pain of other people’s wishes. You know that fulfilling their request will cause you pain, so understandably, you say “no”. What often happens next is that they escalate until they hit someone sufficiently important enough to override you.

This is the basis for why SysAdmins feel steamrolled by everyone else, and everyone else feels held hostage by SysAdmins.

But all hope is not lost.

Stop saying “No”.

“Yes, but …” is a very powerful thing.

“Yes, but …” can be used to get you help.

“Yes, I can set that up for you, but we don’t have capacity to run it for you.” What happened there? You agreed that the request is reasonable. You set expectations of the level of support that you can give. You left the requestor with several options to continue the conversation.

They might have hiring reqs that they can’t fill. You can negotiate for some of them to go to your team, as you’re clearly understaffed.
Some of their engineers may join your team as a lateral move. They’ll need mentorship and training, but this kind of cross-training is invaluable. It’s a force multiplier. It also sets precedent.
They might take the responsibility for the Thing. They run it. They get paged for it. Of course you will probably have to be an escalation point to assist when it fails, but it’s their product. This again sets precedent.

Delegate

Most SysAdmins are stuck doing tasks that provide very little value because they restrict access to their peers. To my mind, there is one perfect example: “Playing Telephone”.

When I say “Playing Telephone”, I’m talking about the situation where someone (let’s say a Developer) wants logs from the application, but they don’t have access to get them. They request the logs from you. You fetch the log requested and provide it to them. “No, not that log, this log…” You fetch. “Hmm, I’m not seeing what I’m looking for, could you check in here for something that says something like this …?” And so on, and so on…

I don’t know what you’re hoping to prevent by restricting access, but if this scenario ever happens, you should know that you’re providing Negative Value. Again, let’s try to remember that your peers are not out to get you, and can probably be trusted to be reasonable humans if you meet them mid-way.

With that framework in mind, it’s time that you demonstrate some trust, and Delegate to them. Give them access. Your value is not in the logon credentials that you have, otherwise you’re just a poorly-implemented “Terminal-as-a-Service”.

Even better than giving access, is giving Tooling. Logging into a server should be an antipattern for most work! You need some better tooling. So, with the example of logging, let’s talk tooling.

Logging

First, logging into boxes to get logs is just dumb. Sure, you could wrap a tail command in a Rundeck job, but let’s Centralize those logs while we’re at it.

SysLog is better than nothing, but not by much. Shipping logs is easy, but consuming them as something useful is not. Batteries not included.

If your company wants to spend the money on Splunk, then encourage that. Splunk is a fantastic suite of tools, but I might wave you away from it if you’re not going to use it for everything. It’s going to be expensive, and if you’re not going to spend enough to use it for everything, there will be confusion as to what’s in there, and what’s stored elsewhere.

ELK (ElasticSearch + Logstash + Kibana, sometimes mistakenly simplified to “Logstash”), or a “Cloudy Elk” / “ELK-as-a-Service” is a good middle-ground. ELK is Free (as-in-beer), and very featureful.

Take your Centralized logging of choice, and provide your customers with the url to the web interface. Send them links to the “How to use” docs, and get out of their way!

Terminal-as-a-Service

If someone asks you to “run this command for me”, you need to put a button on it.

You don’t need to RUN-ALL-THE-THINGS!

Rundeck is a fantastic tool to “Put a button on it”. Other people use their CI tools (like Jenkins or Bamboo) for this. My friend Jeremy Price gave an Ignite Talk at DevOpsDays NYC 2015 that describes this.

Personally I like Rundeck, because it’s pretty easy to make HA, tie it into LDAP for credentials, manage permissions, and by shipping it’s logs (see what I did there?), you get Auditing of who ran what and when!

If you have some data that Must be restricted, try to isolate those cases from the rest of your environment. You shouldn’t have to restrict Everything just because Something does need isolation.

Deploying Code. Yes, to Production

Why would you want to have to deploy other people’s code?! Do you really provide any value in that activity? If the deployment doesn’t go well, you’re launching another game of “Telephone”.
What if you make it easy for them to do it? Empower them with trust and tooling, making it easy to do the right thing! Give them tooling to see that the deploy succeeded! Logs are a start, but Metrics Dashboards that show changes in performance conditions and error rates will make it plain to see if a deployment was successful!

This Freedom doesn’t come free. Providing tooling doesn’t absolve the development teams of the need to communicate; in fact, it’s likely that they’ll have to communicate more. They will need to be watching those dashboards and logs to see for themselves the success of every deploy. They will also be more readily on-hand to help triage the inevitable instances when it doesn’t go swimmingly.

US

I say “They” in this article a lot. And that is because, by default, most organizations that I have been a part of or heard stories of have had a strong component of “Us-Versus-Them.” It’s only natural for there to be an “Us” and a “Them”, but thinking in those terms should be a very short-term use of the language. Strive for the goal of a “We” in your interactions at work, and reinforce that language wherever possible. While it may not be My job to do “foo”, it is Our job to ensure the team and company is successful.

While that may sounds like some happy-go-lucky, tree-hugging, pop-psychological nonsense (and it is…:), the goal here is to get you, the beleaguered SysAdmin the help that you need, in order to improve the capabilities of the business.

CODA

There is so much more to this topic, particularly the shift away from a Systems team supporting a bunch of Project teams to a series of largely self-sustaining Product teams, but that will have to wait for another day.

The psychological damage done to SysAdmins by their peers can make us bitter and cynical. I encourage my people to try to see that “They” aren’t trying to make life difficult for you, but it’s very likely that Authority and Responsibility are misaligned. I likewise encourage my people to take steps to make their lives better. A ship’s course is changed in small degrees over time.

When someone says “DevOps Doesn’t Work”, they’re absolutely correct. DevOps is a concept, a philosophy, a professional movement based in trust and collaboration among teams, to align them to business needs. A concept doesn’t do work, and a philosophy does not meet goals - people do. I encourage you to seek out ways of working better with your fellow people.

GRATITUDE

I’d like to thank my friends for listening to me rant, and my editor Shaun Mouton for their help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

CONTACT ME

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts, and seldom refuse hugs (or offers of beverage and company) at conferences. Death Threats and unpleasantness beyond the realm of constructive Criticism may be sent to:


Waldo
c/o FBI Headquarters 
935 Pennsylvania Avenue, NW
Washington, D.C.
20535-0001

On SysAdvent 2015

2015-12-15T17:00:00.000-05:00

Knowing that I'd regret it if I didn't, I took on yet another task, and wrote a thing for SysAdvent.

"Fear and Loathing in Systems Administration"

This is a title that I'd had kicking around in my head for awhile, and have started shopping it around as a conference talk. I'm hoping to get some feedback on the article content in order to tweak the presentation content.

On Raymond Ramos

2014-07-02T20:26:00.000-04:00

I want to talk about my friend Raymond Ramos. My friend is dead. He's been dead for nine years, and I still miss him.

One night while deployed to Iraq, I found myself "on duty". At this particular instance, my duty was to inflate the ego of the CO and SgtMaj, and relay useful news to anyone who happened by the desk. While armed. In theory, if Al Qaeda came by, it was my job to kill them, but in reality there was no way for them to not have the jump on me, and the best that I could do is die a loud death. Even in Iraq, Egos must be stroked. And they were.

While I sat bored, uncomfortable, and awaiting zealous but non-specific rage (or a passing ego), someone who knew both of us told me that Ray had died. My friend, who had been honorably discharged months earlier, had died in Alabama while traveling for a job interview. My assistant took over the watch, as I walked away in tears.

He had made it through his enlistment. He was supposed to have been safe. That's probably what had hit me the hardest; "unsafe" was a deployment to a hotspot, or a mission. "Unsafe" meant armed, briefed, and geared-up. He was supposed to have been safe. He made it through his enlistment intact. Game over, you win, enter initials.

Nope. See, "unsafe" is a sliding scale. While I was on the far side of the scale (but not all the way over), in my mind, he was swaddled safe in civilian life.

While our CO was a raving asshole (which contrary to public image, is absolutely not normal), he did say exactly one useful thing when we landed in-theatre. I don't recall exactly what he said, but what I absorbed is this: "Death can be a lightning strike on a clear day. It can come at any time, from any direction. When it's your time, there's nothing to do about it."

I'd absorbed this for myself, and my friends and colleagues in-theatre. But Ray had been safe.

I never got an official story, but this is what I recall being told. Ray had been driving for a job interview, and lost control of his car, went off the road, and was severely injured. I seem to remember that he'd left messages for his girlfriend, but she didn't get them until he had died. I wonder if he didn't have cell reception, and just left voice memos. As I'd heard it, the last one (possibly the third), he was resigned to death, and said goodbye.

When a Marine departs a unit (whether a change of duty or when leaving service), it is customary - at least for those in good standing - to receive a plaque commemorating their service. Naturally, the more rank that one has, the more extravagant the plaque. Those who won't be missed get something generic, but a good plaque is apparent. Ray was Corporal when he got out. He may have picked up Sergeant right before he got out, but it would have been on his way out.

Normally there is a collection is taken from among everyone in a given unit for a given Marine's plaque. You pay into each plaque as one of the best and most direct examples of "paying it forward" that there is in this world. Plaques usually come from a local awards company (where you'd go have bowling trophies made), and the standard no-effort plaque has the Marine's name and dates with the unit, and the unit logo. Perhaps a motivational phrase or a quote that the Marine was fond of saying.

Now, Ray wasn't what you could call a "Marine's Marine." He wasn't "hard-core" or "gung-ho", or any of that bullshit. You couldn't imagine him shouting "Let's take that fucking hill" outside of a joke. In fact, Ray's demeanor could be said to piss some people off; Sergeants Major either loved him or hated him. You could reliably predict how much a given Staff NCO would like him based on how comfortable they were with technology newer than, say, the typical ballpoint pen. If they were the kind to send a "runner" than a email, they hated him; if they theoretically knew the difference between "Reply" and "Reply All", they probably loved him. (NOTE: Senior Staff NCOs are very rarely more savvy than this.)

No, Ray was never going to win a "Marine / NCO of the Quarter" board. He didn't really know Marine Corps history. He knew the day-to-day customs & courtesy, but not much beyond that. He shuffled when he walked, and never re-soled his boots, so much so that he and a few of our friends referred to them as his "Combat Slippers"; it didn't help that they weren't polished, and were only barely what you could describe as "black". His uniform looked like it had once met an iron, but they weren't more than casual acquantences. This was very foreign to me coming from an occupational field that was very starch-and-polish, as well as dirty-in-the-field; I was used to being one or the other.

Please keep in mind that this was circa-2004. Google was around, but not in common use yet. Technet and MSDN were around, but search sucked, and information was categorized. (Which meant invariably that the taxonomy made sense, but not to you.) There was no Stack Overflow. While Windows XP/2003 were out, most of our domains were NT, groupware was Exchange 5.5, and workstations were either NT or 2000. The only Unix was in the specialized Intelligence boxes, and I was probably one of 10 people on the base who'd even installed Linux. This is what we had.

Ray and I were SysAdmins. SysAdmins in the Marine Corps don't code. Programmers code, and SysAdmins aren't Programmers. Being a Programmer in the Marine Corps means that you will be spending 80% of your time working on the CO's Pet Project, which is usually some stupid website that nobody will really use and won't survive your departure. The consequence of this is that the work that you should be doing falls to your compatriots, who are few in the first place, and already barely keeping things afloat.

But he had this insatiable curiosity, and this provided a great intellectual focus for him. Unfortunately the big boss found out, and the next thing you knew, a Pandora's Box of pet projects exploded in our office.

So, one day Ray's hacking on something like a CMS for shit nobody cares about, or an in-browser chat client, and he's pondering. Pondering hard. "How can I....?" I don'
t remember if he exclaimed "EUREEKA!", but it might as well have been. Next comes "I KNOW! I'LL DO IT WITH A COOKIE!" This was the first time I'd seen demonstrated the pure joy that comes when a solution presents itself.

So, when it came time for him to leave, knowing that the unit would get him something generic, I took care of it. I wanted him to have something special.

Ray loved him some Star Trek. Original, Next Generation, whatever. So, I went to a hobby shop, picked up a roughly 10" version of NCC-1701, assembed, painted, and decaled it. I took it into the awards shop with an idea and a quote, and they said "no problem."

While the quote that Ray was most fond of was to reply to statements with "...But does that make it right?", what I had inscribed on his plaque was "I'll do it with a Cookie!"

A going-away party in the Marine Corps can range from a don't-let-the-door-hit-you-on-the-ass-on-the-way-out just about all the way through a weekend-long Roman orgy of an alcohol-and-grilling bender. Most of the time, it's making sure that your immediate office and some unit friends have lunch before you leave. Often it's with said Marine's car packed, and they're about to drive off base for the last time.

Ray's going-away party was a lunch at El Cerro Grande, the local Mexican establishment. We had a pretty good turnout. There were stories told, and a few last-minute items he'd forgotten. After the food and nostalgia finished, and the awkward and anxious quiet settled, some of us gave mini-speeches. MSgts Brown and Ayo said something, and they turned it over to me. I said a few things, reflecting on our past few years. Then I presented his plaque.

He cried and hugged me. I cried.

This is how I remember my friend Ray.

Ray indirectly taught me much more important things than he directly taught me. The most useful to my life and career is that being autodidactic is very useful. Being able to state a problem, ask an actionable question, and research an answer is amazingly useful. He also taught me that Programming is not (necessarily) evil, and formal training is not required to learn it.

Let us end with a toast. "To Absent Friends..."

-Waldo

Cooking for Geeks - A Review

2014-02-18T09:42:00.005-05:00

It seems that most of the writing that I've been doing is reviews. Fortunately I have plenty of articles to write in my head, but the times that I feel like writing are pretty inopportune (such as my commute.)

So, dear friend, we have another review of an O'Reilly book, and providing full disclosure, I received a free copy of this book in exchange for a review... All integrity accountability in-place, let us begin.

Chapter 1 opens with the headline "We geeks are fascinated by how things work, and most of us eat, too." This book makes it very plain that it is a book for geeks (who want to learn more about the alchemy that is cooking), by geeks (who know some things about said alchemy).

While I already have a basic competency in the kitchen, there was a lot of value for me in this book, even beyond the underlying explanations of "how" and "why".

One of the things that I appreciate most are the quick hacks to improve your life. Probably my favorite is the Pizza Stone tip. The tip is to keep your pizza stone in your oven; it will take longer for your oven to come up to temperature, but the stone will help to keep the temperature more even.

Probably the greatest value from this book is in technique. Rather than just a list of steps, there is a lot of explanation of "how" to do something as well as "why". Even though I have some experience cooking, I do appreciate that the assumption is that the reader is a thorough beginner.

All in all, I found this book to be pretty fun and very informative. I've already recommended it to others, and do so without reservation.

The book's website is http://www.cookingforgeeks.com/, and it can be purchased from Amazon, or from the publisher here.

-Waldo

On Hoodies

2014-01-27T22:25:00.002-05:00

I have yet to find the perfect hoodie. Unlike the perfect knife, I am surprised at this, because it doesn't seem like it should be that hard.

This may come as a surprise, but I've only recently come to appreciate Hoodies. I can only think of one that I'd had during my youth that I didn't particularly care for, and didn't have any during my time in the Corps.

Perhaps they've only now come into my life because my office is in an unheated basement, or perhaps it's that my last few jobs have allowed for more casual attire. In any case, as is natural for me, now that I've experienced a thing, I have opinions on the matter.

The most obvious is the question of Zippered or Pull-Over. This one is actually pretty easy for me. While I appreciate the decorability and unified front pocket of a pull-over, the flexibility of partial-openness (as well as lower static generation) make me a Zippered man.

The one Pull-Over hoodie that I have has one flaw that is unique to it's shape that causes me annoyance, and that is that the neck hole is too small. I may have a large head and neck, but I'm surprised that the hole is as small as it is. It's small enough that it's hard to pull my head through the hole, and the neck sits uncomfortably close on my neck. I may need to go so far as to cut the neck to extend it, but then I'll have to do some sewing to keep it from ripping further.

So as far as Zippered Hoodies go, there is a feature that is unique to them that deserves attention when looking for the ideal hoodie. The zipper border - the piping of the garment material outside of the zipper itself - should be large enough that it covers the zipper teeth when closed. One of my hoodies, which I like despite it's two significant flaws, has an almost completely exposed zipper. It doesn't bother me in the normal wearing of the thing, but I am incredibly aware of it when resting something on my chest or stomach, such as a tablet, laptop, or handheld game. At times like this, I can feel the device scraping against the zipper, and it's all the more annoying because it seems like a completely unnecessary oversight.

Next are the features that are of crucial importance to me, but aren't specific to either style of hoodie.

The pockets should be large enough to fit my hands in to the wrist, and even better, fit a largeish handheld game system (Nintendo DS XL or 3DS XL) completely, parallel to the ground. If the pockets aren't big enough to do that, they're too small. Additionally, and have raised sides so that it's rare that anything (such as a DS XL) could fall out accidentally. Surprisingly, I don't have any that fail at this.

The sleeves are something that I wouldn't have expected to be an item of consideration, but alas, here I am writing... The material of the sleeve should have ample room for at least the average man. My arms are not by any definition "large", but one of my hoodies has sleeves that only barely fit. This is only compounded by the next feature of the sleeves: the cuffs.

The cuffs should be - I would have thought this would go without saying, but then again, here we are - *elastic*. And not just a little elastic; I should be able to push them up to my elbow, and have them stay there. I don't know when or why I started doing this, but when I'm doing work, if I'm wearing long sleeves, they're probably at least half-way up my forearm. When I'm passively consuming information or media, they may be down to the wrists, but when I'm producing, my sleeves are probably up. This is my own bias and habit, but it is still a thing for me: Why someone would make comfort-wear without comfortable sleeves is completely beyond me.

Finally, the hood. It should go without saying that the main job of a hoodie is to be a sweatshirt, but with-a-hood. So, you could reasonably be expected that a sweatshirt pattern and material with a decently-sized hood would be pretty easy to come up with. Alas, here I am... A hood should be much larger than the head wearing it. In my opinion, I should be able to pull my hood up, with my head all the way to the back of it, and it should come down to just over my eyes, and be loose around the sides of my head. What we're talking about is the Sith-look. Unfortunately, most of my hoodies' hoods barely come up to frame my face. That makes me sad. They had one job to do that made them different from a sweatshirt, and they failed.

Most of my hoodies actually fail at the hood. In fact, the features that I've complained about in each of the last several sections all refer to the same hoodie. Amazingly, it isn't my worst hoodie. Due to an absolutely awesome hood, I love that one, but it's a strange love; it's a love of the best of mediocrity. If I could find a hoodie that was good at all of these, I'd buy 10 of them and probably give most of the rest away.

On Cultural Norms and Perception

2014-01-14T18:05:00.000-05:00

At both my last gig and the current one, coffee was available via a Keurig.

At the previous gig, it was declared that you should remove your used cup when your brew was done, and if the reservoir was empty, you were to fill it. There were some people who seldom (if ever) did this, and there was much snark and gnashing of teeth.

At the current gig, we do the opposite: When you open the cup holder, you will have to remove the previous person's cup, and if the water is empty, you will have to fill the reservoir. You do these things because you are incentivized to do so, as either condition is a barrier to you getting your drink. As a result, there is no grousing or emotion aside from perhaps being briefly delayed.

In the former, a lazy person has caused you more work. In the latter, the worst-case is that you lose a trivial lottery.

I think the second way is better.

The Macintosh Terminal Pocket Guide - A Review

2013-12-01T20:12:00.000-05:00

The Macintosh Terminal Pocket Guide - A Review

Disclosure: I received a free copy of this ebook in exchange for a review.

The "Macintosh Terminal Pocket Guide" by Daniel J. Barrett (published by O'Reilly Media) is a handy quick-reference for leveraging the power of the terminal to use your Mac more efficiently. This book is especially helpful for guiding you through the use of commands that are built-in with your Mac, which may vary from other Unix-like distributions that you may be more familiar with (such as Linux.)

As is typical for an O'Reilly Pocket Guide, the book is pretty terse, and it is not all-inclusive. It does briefly describe each command that it covers, but most entries are between a couple of paragraphs and a couple of pages. It varies depending on the complexity of each command, but most commands include a small bit of descriptive text (typically 1-3 paragraphs), a handful of the most-common options, and some examples. Typical use-cases are covered, but due to space constraints, they really are limited to the most common options; A book this size could probably be taken up solely by covering the options for `ssh`, `grep`, `tar`, and `ls`. (In fact, there is a Definitive Guide for SSH, and a Pocket Reference for `grep`.)

This book is not set up to be a friendly introduction to those who are completely new to the Terminal. However, if you are comfortable with the terminal and could use a handy quick-reference, this is a fine tool to work more efficiently with your Mac.

The book can be purchased here.

-Waldo

On the "Skills Shortage"

2013-01-10T09:29:00.000-05:00

Companies: You cannot complain of a "Skills Shortage" if you expect the moon and sky for "skills requirements" (esp. if you refuse to accept anything but a full match), are hiring for one person to fill the workload of three people, and pay below-market value.

There wouldn't be a skills shortage if you were willing to A) pay a decent wage for what you're asking for (if you must have a perfect fit), B) plan to hire someone "close enough" (especially with cross-compatible skills) , and/or C) provide training.

(Hint: if any of the people related to the hiring process mention recession or unemployment as reasons that people should be willing to accept lower pay, right there is where you're hurting yourself, and are part of the problem. If you do hire someone, they will leave you as soon as a reasonable offer comes.)

"On Crap Detection" By Howard Rheingold (O'Reilly Media)

2012-08-10T14:16:00.002-04:00

I found a video series titled "Crap Detection 101: How to Distinguish Good and Bad Information Online" by Howard Rheingold (available from Oreilly Media), and was instantly intrigued. When the opportunity came up to see it free as part of the Blogger Review Program, I felt that it would be a good choice.

The short version:

If you are a cynic and/or have been on the net for any significant amount of time, this series probably isn't for you. If you have children, it's a good idea to watch this so you can frame how you want to talk to them. If you know older (non-web-savvy) people getting on the net, perhaps for the first time, buy this for them.

The main points that I boiled this down to are:

Be aware of Bias
Look into the source
Wikipedia is a place to begin research, not end it. (See previous)

Accounting for my personal biases, and assuming the intended audience, I give it a 4/5.

The Long Version:

This video series by Howard Rheingold (UC Berkeley and Stanford University lecturer) addresses the basics of protecting yourself from misinformation and fraud while going about your daily online life. Broken up into 11 videos (including a short intro and conclusion video), this series is probably short enough for someone new to the material to take in all in one shot. Personally, being a cynic and long-time internet user, I found it to be slow and fairly basic. However, I am not his target audience.

Mr. Rheingold addresses his target audience as pretty much everyone who is encountering the internet for the first time. That is tricky, however, because even my four-year-old son has used the internet. A few years ago, my In-laws decided to get off of WebTV, and I was to be the instrument of their transition. (No, I didn't have a choice.) Had this video existed at the time, I would have purchased it for them without hesitation. I had given them a brief list of warnings ("No Nigerian prince is going to ask you for help with a financial transaction" and "Don't click on links in emails that claim to be from your bank, even if it looks real"), but I think that the presentation of this material was better.

Being a parent of youngin's, I think that they're going to grow up with a strong sense of cynicism. This video series may help inform you about topics to bring up, but anyone younger than middle-aged will not be able to sit through the video. The tone is conversational, and the pacing is fine, but Mr. Rheingold is just not a good fit for a younger audience.

The production and presentation of the video gives me mixed feelings. The video looks great. But it looks to be done in Mr. Rheingold's home office, which I find I have mixed feelings about. It's certainly better than the blank light-beige conference-room wall common to most informational videos, but the posters and monitors in the background are distracting. Rheingold is joined by Mark Brokering of Safari Books Online, who asks leading questions on each topic. Rheingold's sound level is fine, but Brokering's is often a little too low. My final criticism on the presentation is that Rheingold references his laptop, especially when demonstrating a site. When he does this, it seems to distract him, and he forgets what he was saying.

I think that this series covers very well the idea of "Being aware of Bias". In fact, I think that it is mentioned in every section. There is an excellent example early-on regarding Martin Luther King, and he covers ways to begin looking into the people presenting information to you.

Later on, he touches on some sources to leverage to investigate urban legends, hoaxes, etc, which I think is especially helpful to the target audience.

I especially appreciate that he addresses Moral Panics ("Protect the Children!"), and by pointing out that censorship is against the founding ideas behind the internet, and that censorship would destroy it.

I think that two topics that really deserved treatment were neglected (or under-served). The first is Privacy Settings. I think that making people aware that there are different levels of visibility to the internet (especially when using social media sites/apps), is of crucial difference. To this day, I see faux-pas on Facebook where an uncomfortable topic is posted on someone's Facebook Wall instead of in a private message or chat. (Personally, I don't like Facebook having that information at all, but sometimes it can't be avoided.)

The other topic, somewhat related, is Over-Sharing. My sister is 10 years younger than I, and when she first set up her Facebook account, her information was widely-viewable, and she'd posted her phone number, mailing address, and talked freely about her comings-and-goings. It took yelling at her a few times before she finally fixed some of the issues, but at least some of those were taken down.

In Conclusion:

All in all, this series is a great resource for older Net-Newbies, and a decent starting-place for parents. But if you've been on the net for some time, it's probably not for you. All in all, I give it a 4/5 stars (attempting to account for my biases.)

-Waldo

Indie Game: The Movie Soundtrack

2012-06-12T19:47:00.001-04:00

The digital edition of "Indie Game: The Movie" was finally released today. I was disappointed that the movie has been on-tour for so long before this happened, but I understand and acknowledge that they had their reasons. I look forward to seeing it.

But this blog post isn't about the film. As a special "thank you", Jim Guthrie's soundtrack for the film was released today as a bonus I believe to the Kickstarter backers. For this I am grateful.

Let me start by saying that I had purchased his previous album for the Superbrothers: Sword and Sworcery game "The Ballad of the Space Babies", and I have found that to be fairly amazing. There are a several tracks that I simply love, and I think I've listened to it every day for the past two weeks.

Unfortunately "Indie Game: The Movie (Soundtrack)" has not resonated with me thusly. In fact, it has left very little impression with me, (other than one song reminding me strongly of Sunny Day Real Estate.) My biggest disappointment is that I was hoping to experience a similar reaction to BSB, but instead it strikes me as a bunch of incidental music. It may complement the film spectacularly, however it is not an album that I think I will ever sit down and listen to and experience again.

I mean no disrespect to Mr. Guthrie; this is but one man's impression.

-Waldo

UPDATE: I've just finished watching the film. It is a wonderful story wonderfully told, and the soundtrack fits in perfectly. That doesn't change my earlier opinion, but the soundtrack does indeed suit it's intended purpose.

Adding Unix to Active Directory

2012-05-23T16:18:00.002-04:00

Thanks to the Many Roots of All Evil Rants blog for pointing out Likewise-Open to me. (I tried to post this thanks as a comment, but commenting is either broken or they've given up on notices that the blog owner has comments set to be reviewed.)

Somehow I had never come across Likewise-Open, but basically it takes all of the fiddly bits of adding a *nix machine to Active Directory. That is correct! No more do you have to mess with NTP, Kerberos, Samba, and Winbind! Huzzah! Finally, someone wrapped all of that crap up!

In the article, the poster (who is unnamed, otherwise I'd use it) noted that he was unable to ssh using domain credentials, but could locally. I found this to be true initially, but it appears that he may have been suffering at the hands of syntax. I was able to get ssh to work from a *nix host using any of these syntaxes:

ssh 'domain\user'@sshserver

ssh "domain\user"@sshserver

ssh domain\\user@sshserver

As we're a Windows shop, I'm using mRemoteNG as a connection-manager, and it wraps the username in doublequotes automatically, so simply domain\user works there.

Being a Ubuntu server (I'm moving to 12.04 Precise Pagolin), I prefer to keep to the sudo mentality, so I've added this line to my /etc/sudoers:

%ernstinfo\\domain^admins ALL=(ALL) ALL

This could be done better in terms of granularity, but that's a yak for another day...

-Waldo

On File Paths

2012-05-17T16:54:00.003-04:00

This has bothered me for years, but when Microsoft FINALLY dropped "\Documents and Settings" for "\Users", I rejoiced.

Right up until they forgot to fix "Program Files" too.

Microsoft, here's how to do it without breaking backwards-compatibility: C:\Programs. Then, an alias named "Program Files" points to Programs. And add a warning to MSDN and the Packaging and Install tools to stop using retarded conventions.

(Incidentally, I did this when installing Strawberry Perl because it can't handle living under a directory with a space. On one hand, I can't blame them, but on the other ...really, guys?!

Besides, you don't need a directory off of the root of the drive. Stop with that C:\Strawberry nonsense.

-Waldo

Dealing with MicroHDMI

2012-05-17T13:23:00.001-04:00

My new laptop for work is great, with one significant exception: the only digital external display port is MicroHDMI.

While I'm sure that the engineers were well-intentioned, I can't believe that they used it much, as it's a pretty flaky connection. When you plug it in, it feels like it has made a secure connection, but nothing registers until you fiddle with it.

After a few minutes of frustrated fiddling, and 30 seconds rummaging through my desk, I found the perfect solution:

The humble binder-clip.

If it looks like it's pushing the cable at a severe angle, that's because it is. Move the clip so much as an eighth-inch away from the connection and it disconnects.

I intend to write more fully on my Samsung Series 9 (2012 model, NP900X3B) later, but I must say that I am reasonably happy with it. The trackpad is a bit touchy compared to Apple's, and the keyboard takes a little getting used to, but it performs solidly. I do with that Samsung had selected Mini-DisplayPort / Thunderbolt instead.

-Waldo