But I’m ambivalent. It would be great if everyone could work at an exciting job where every day was an adventure and the work actually gave you energy instead of taking it away. There are plenty of passionate bloggers out there writing about their work and a whole start up junkie culture of serial entrepreneurs and others who love what they do. They found their passion and they capture the experience of it in moments of inspired writing that can make the rest of us alternately envious and skeptical. But for most people, work pays the bills and we can’t lose sight of that. So much of happiness in life is about managing expectations and some of the gushing war stories we read can lead us to set impossible expectations for ourselves.

Vacation can be a time not just to put energy back into a life drained by work, but to gain perspective on why we do what we do. There have to be moments of passion and accomplishment in our work lives or we will be bored and frustrated. But it’s not a nonstop joyride. We selectively remember the good times–or, if we have lost our way, we dwell on the negative and make our experiences miserable. But as we are away from the day-to-day responsibilities of a job, we can take some time to think about the many reasons we do the work we do.

For many of us, it is about the money. Money can’t buy happiness, but lack of money can make you unhappy because of all the things that go along with it. Especially with the overall economy in the state it is in these days, anyone who can afford to take a vacation and be paid for it should count themselves lucky. Freelancers, contract workers–you have to buy your vacations, so they are even more valuable.

I think most people who work in technology are looking for the satisfaction that comes from solving difficult technical challenges. But nobody wants a job that is a non-stop problem ticket. If you tell yourself you live to solve problems…you should reflect on what that kind of work life would really be like.

Most of our work satisfaction comes from the people we work with and being part of a team. There will be moments of great success and hopefully times when the thing you are working on does something amazing. But most of the time work is an excuse to spend time with people who share similar interests. Helping others with your knowledge and insights, doing the little things from day to day where you feel like you added something and it was appreciated–these are the real things that keep you going. The real “battery recharge” comes from the simple appreciation of those around you who acknowledge what you do and make you feel like a real contributor.

So this is is a very “touchy-feelie” post; I know. And everyone will have different motivators and depending on their perspective, may agree or disagree with how I feel about work. But I hope–for those who can take a vacation this summer–you will use that time primarily to have fun–and if you find yourself worrying about work, think about renewing your interest, not about trying to find energy that has been “lost.” I think for most of us the energy comes from within and our challenge is to find it, not replace it.

But it is replaced by the always-on culture–and we’re signing up for it willingly because it makes us feel important. I can not only be notified of a problem on my phone, I can even launch an ssh session back into a server and do something about it. While the kids are busy playing in the park, Daddy (or Mommy) can be working away on their little device. When things get really complicated, it’s easy to take the laptop out of the minivan, dial in, and continue working. Maybe it only takes 15 minutes. We tell ourselves that availabilty and convenience to work lets us balance our work and life.

Entrepreneurs tell themselves many lies. Sometimes, they learn from their mistakes and work out a balance or blend that is honest and realistic. I think it is all about taking responsibility for your time, recognizing the value of it, and investing in solutions that make the work in your life easier.

I can’t make the case here that buying Heroix software for networking and application monitoring is part of a work life balance strategy, but I think it is helpful to apply a broader perspective to how you approach solving your challenge to keep systems working. Maybe the investment in mastering open-source monitoring solutions is worth it. Maybe spending more time training other people to be as proficient as you are is a better use of time than designing or purchasing what you believe is the perfect solution that will give you the proactive alerts and warning you need. But if your support strategy is simply to make sure someone is always on call to deal with crisis, that is essentially a strategy to keep having crises.

“I would not trust a sysadmin that knows so few.”

Give me a break. The role of a sysadmin these days is far more broadly-defined than to expect that any one person would become expert in every tool that is out there. Environments change, staff are turned over, and the most important skill is to be able to find help when you need it. A company may not be able to afford to hire a Linux guru–just as they may not be able to afford a database expert who can check off a list of every imaginable technology he or she has mastered. Even when such people are found, they may not fit the culture of a company or, they may have an attitude of entitlement and superiority that makes them difficult to manage.

I think the key for all of us is to focus on continuing to learn. I was not really a sysadmin; when I look at a real sysadmin’s list of resources like Hal Pomeranz’s Index of Presentations and Articles, it puts my scrappy but resourceful IT career in perspective. There are many people we can learn from, and it is always about how you use the tools, skills, technology and your own creativity to achieve results that matter in a given context. It’s not about collecting awards or being able to prove you know more than the next person.

Emphasize That You “Get Stuff Done”…You can be the most brilliant engineer/marketer/whatever on the planet, but if you don’t have a tendency to get a lot of stuff done, you’re not an attractive recruit…If the startup team hires you, they want to know that you’re going to put a dent in their workload–not just come up with great ideas for other people to work on.

It’s good advice to keep reminding yourself of, even after you are hired and start work. Yes, you are smart and have great ideas, but you are not alone. In technology, we have a tendancy to look for the optimal solution and to believe that there can be some great tool that will solve all our problems. But you need to roll up your sleaves and do something more than talk about great ideas because:

• it’s arrogant and presumptuous–it sounds like you think you know better and are putting down the efforts of people who have been competently trying to solve the same challenges for months before you arrived.
• it’s lazy–talking about what could be done starts to sound like ways for you to avoid work when no progress is being made on actually doing anything.
• it’s tiring–chances are, many of your ideas were tried and rejected before because no on had time to do the detail work. That’s why they were able to justify hiring someone.

So the question to ask yourself is always, “What can I do right now?”–and then, go do it! In a startup, that’s what’s expected.

Most start up companies I worked for did everything in-house, but we also engaged consultants in ways that became longer and more expenseive than intended. A big company might outsource to save money by handing off “routine” tasks like server maintenance and management. Small startups might outsource because they don’t know whether the task is going to be needed for more than a short period of time. Rather than hire a team of software engingeers, manage them for a year, then lay them off, you might engage a consulting company in the early growth stages as you better understand your needs.

But when the revenue stalls…a day of reckoning comes. There are many “what exactly does that person do?” questions. It is terribly frustrating for IT departments that are well-run because, if everyone is doing their jobs well, the IT function is very low profile. Management can’t see all the problems you have prevented, they just see heads.

If you have outsourcing arrangements, you can shift the discussion to how your team can save the company money. In one organization, we were spending thousands of dollars per month for a support company’s “care and feeding” of our servers. I rationalized and defended that choice because I knew that if we did bring the function in house, it would consume all of my time and/or require an investment of time upfront that I did not have. But it became untenable as we realized even adding virtual servers and putting them “under management” added hundreds of dollars per month. All our open source initiatives carried an additional price tag.

In a large distributed organization, it’s the same story. Why can’t we just use Wordpress for those blogs? Because it would cost too much. Although Wordpress is free, if we set it up on a server that is managed by the IT department, they will internally charge us for a server because we would now be sharing the servers they maintain. But why should I share equally when the real amount of work required is so minimal?

If you know what you are doing, insourcing can allow you to take control of costs. I know what the “care and feeding” of a linux server is and I’m ok with the minimal automated backups I can set up. If you don’t know what you’re doing…you should invest in tools that help make the insourcing manageable.

Heroix Longitude or another monitoring solution has to be a part of your insourcing strategy. If you have a person who has the time and is excited to learn and manage nagios or another open-source solution, that’s great. But don’t underestimate the time and effort involved. You are making an investment of time, if not money, and you should be careful about positioning any person in your organization as “he’s got time to do all that.” You might just get your best person laid off.

Propose a solution that lays out the costs as 3 plans:

You have to fill in the details. You might use the same approach to justify an open source toolset, but use our ROI calculator and don’t “cheat” on costs by assuming you can make up for money with time that nobody really has available.

Insourcing can be an opportunity for change…and as the podcast I linked to above notes, change is usually good because it forces a re-evaluation of the cost justifications. When times are good, “care and feeding” sounds like a necessary expense. But when you start to ask detailed questions about how your money is being spent, you may discover opportunties to save money and create value.

• Use a wiki to write down your overall strategy and solicit feedback from other team members when you are thinking about what to monitor in the first place. You might start with an outline of the key business tasks behind a web site, then validate that with the product manager and other business people, then get very specific instructions from the tech team about what to monitor. Then, follow up and write down exactly what you are monitoring and how to tell what the alerts you set up mean. Leave a trail of documentation.
• Incorporate this documentation into your monitoring by customizing the alert emails. We tend to want to keep those emails short so they can be read on a pager, but consider including a link back to your documentation or even setting up a separate alert that is triggered by a sequence of “correlated events.” The type of alerts could be turned on ONLY when you are not available (e.g. on vacation) and configured to go to a wider audience.
• Don’t over do it. The problem with setting up elaborate monitoring schemes and alerting too many people is the risk that too much information will mean people ignore it or assume someone else is going to do something about it. So use your special alert emails sparingly and disable them when you are around.
• Consider configuring some alerts to go directly to status pages. Heroix Longitude allows you to publish reports directly to a portal. People will likely ignore these things during “good times,” but think about how your less experienced colleages might have to react in an emergency. Can you set up alerts to send emails with links to your best efforts at “forensics?” When you are first setting things up, you may do this for your own use, but then you’ll tend to prune things down to filter out the noise. Save your original work and re-activate it in situations where you are asking others to help cover for you.

The key to making all of this work is to do it from the beginning, not as your last task before leaving for two weeks. The reality is that your elaborate coverage plan might break down immediately if it has not been used in practice. And if you are always the one who jumps in and solves the problems…no one else is ever going to really learn how to use the systems you’ve created. It’s just like a backup or disaster recovery plan–unless you actually use it and practice it, it’s just a bunch of ideas on paper. So take the time not only to plan well, but to bring other people into your coverage plan from the beginning. Examples:

• Notify the DBA of any database-related alerts. But don’t just cc that person on every message that goes out. Send one alert that links to your page and reports, then, during a post-mortem, go over what the email meant and validate that the DBA would have been able to do something useful if you had not been around.
• Don’t automate manager alerts. I recall one software product we used for monitoring that had different templates like “email to Boss.” The template was something like “We are busy addressing the problem, don’t worry.” That is a recipe for trouble. Don’t alert people unless you want them to take action. You can use reports and documentation with links to generated history and status to inform–but don’t bypass the people you want to take action. Instead, send an email alert with the links I described earlier; the person on call can then forward that email on to a manager with a legitimate/authentic message like “I received this alert and am investigating; see links below for background.”

The title of this post sounded a bit ominous…”when you are Gone” means, hopefully, “on vacation.” But its a good survival strategy in general to always be thinking about creating effective shared responsibility. It’s a mistake to think indispensability equals job security because when the company hits a business crisis, everyone is replaceable. You will find yourself becoming a DBA because that role was cut or your boss will suddenly have to become an expert on systems administration. The best security is to stay ahead of problems and make everyone indispensible.

Fiction:

WWW:Wake by Robert J. Sawyer

Angels & Demons by Dan Brown

The Space Between Us by Thrity Umrigar

Star Trek (the novelization of the new movie)

Non-Fiction:

Physics for Future Presidents: The Science Behind the Headlines by Richard A. Muller

Blink: The Power of Thinking Without Thinking by Malcolm Gladwell

Lipstick Jihad: A memoir of Growing Up Iranian in America and American in Iran by Azadeh Moaveni

Gang Leader for A Day by Sudhir Venkatesh

The Ascent of Money by Niall Ferguson

The Agony and the Agony: Raising Your Teenager Without Losing Your Mind by Betty Londergan

Tuesday With Morrie & other titles by Mitch Albom

Travels with Charley by John Steinbeck

Here are some classics available for Stanza on the iPhone, a free book reader rated “totally awesome” by one of our engineers:

The Count of Monte Cristo by Alexandre Dumas

The First Men in the Moon by H. G. Wells

The Princess of Mars by Edgar Rice Burroughs

The Captains Daughter by Aleksandr Pushkin

Robinson Crusoe by Daniel Defoe

Taras Bulba by Nikolai Gogol

Kim by Rudyard Kipling

These and lots of other titles are available for free or cheap!

• passport
• bike pedals
• bike computer
• traveler’s checks
• maps, compass, equipment
• clothes and supplies

Or does it look like this:

• laptop
• satellite phone
• laminated password cheat-sheet
• internet access/cell phone adapter for laptop

Or, put another way…do you vacation shop at REI or Radio Shack? Now it’s one thing if you love your work and the idea of remaining connected while enjoying travel in remote places appeals to you. But if you “have” to remain connected because your company can’t survive without you…that is either a problem of perception (no person is irreplaceable) or planning (why can’t someone cover for you?).

I remember one vacation where my IT manager quit while I was away…I ended up talking people through rebooting servers and editing systems configuration files over the phone. I call that process the “10-baud” connection:

So type ‘vi httpd.conf.’ OK, what directory are you in? Type ‘pwd.’ Yes, then hit ‘enter.’…call me back when you get to the colo…

That’s no way to live. There is no excuse, unless you work in a life-and-death industry, to be that tethered to work. When people start talking about getting you a satellite phone…you need to nip that conversation in the bud with some advance planning.

• Document your procedures. A wiki can be helpful for this as you write things down while you do them and create a ‘cookbook’ for later. Use this approach in general–not just to prepare some massive document before you leave.
• Rotate coverage of all key systems and develop a practice of delegation. When other people are “primary” on the pager, do they have the authority and experience to truly cover for you? How can you develop that?

Tomorrow, I’ll develop some more specific ideas on how you can leverage your monitoring tools–in advance–to prepare for situations when you can’t personally be there to handle the unexpected crisis that is sure to develop as you board that 6-hour flight out of the country.

When a company is expanding rapidly…or a new service is being launched, people understand there can be some hiccups, but they need to know your and your company are working to fix the problems. When you can say, “yes, we are aware of that and addressing it right now,” or “we had a problem for 15 minutes last night at 3:15am and took the following steps…” it boosts your personal credibility and the credibility of your company. It’s not just “covering yourself;” it’s part of the nature of rapid growth and uncertainty. You cannot eliminate risk, but how you mange it demonstrates what customers can expect from you in the future.

So, if you just have ping monitors set up that tell you all your servers are pingable from another machine on the local network–and then email you an alert if somethings goes down…ask the following questions:

• What happens if your monitoring server can’t connect to the mail server to send the pages or email alerts?
• What happens if your database server becomes overwhelmed with connections and can’t serve the web application?
• Is there a single point of failure you are not monitoring?
• If you have load balanced servers…what happens when a node has a performance problem but does not completely fail?

But those are just questions to start you thinking. What you really need to do is stop thinking about monitoring in terms of point tests and start thinking in terms of functionality. Start from use cases and then develop a testing plan that goes deep enough into the detail to monitor these user transactions with your business:

• Can a visitor from outside our firewall successfully log in to their account on our service?
• Can they navigate to particular services that are only visible to logged in users?
• If we are sending automated updates by email…do those emails actually get sent?
• Are new visitors able to register new accounts? Are we actually seeing a reasonable number of new members joining every day?

To answer these questions, you need to monitor transactions and construct correlated events.

Correleated events means you monitor a series of dependent transactions…the simplest example is to set up an external monitor of your site, sitting outside the firewall that would test the following sequence of transactions and then alert you with only the first failure:

• Is the public router pingable?
• Can I ssh into a particular server?
• Is the web service responding at all?
• Is a web page delivering expected content (versus an error message or long response delay)?
• Can I log in to a particular part of the web site?
• Does a database query indicate we are recording an expected number of new registrants or tracking database rows based on some minimal user activity?
• Do the smtp logs contain indications that emails are being sent every day?

Transactions are the detail of how you get beyond the simple status of “pingability.” I’ve talked about using web content monitors before, but if you have more granular testing capabilities you can use transaction monitors like the following ones in the Heroix Longitude product to answer those questions:

• Use the FileContent monitor to inspect a log file. There is a wealth of information you can collect by monitoring syslogs and Windows event logs, but for a specific functionality test, you can focus on searching for a specific phrase in the logs that indicates a nightly process completed successfully.
• Use the SQLQuery monitor to count rows in a table or report the most recent new row added to a table. How many new shopping carts were created in the past 24 hours? What is the current maximum userid?

Ideally, these kinds of tests should be a part of your development process–don’t stop at QAing a build, find out what daily tests should be done to ensure the application if really working. Make ongoing testing and verification a part of your deployment process. Recognize that the process needs to be constantly improved–although you can start by monitoring each node in your network based on a template of what should be monitored on that server, you should then step back and design some functionality-based monitors to monitor your applications from the perspective of users who don’t know or care what a ping is.

Heroix calls these kinds of tests, “transactions,” the most basic being the HTTPUrl transaction. This transaction checks the content of a web page to match or fail specific content based on a regular expression you provide. This is the transaction monitor I used above coupled with a web page script.

It seems a basic concept, but I’ve found even this monitor is often not used in practice. At one company our web systems were monitored by an IT department unfamiliar with our new line of business. They used What’s Up Gold to monitor their entire network, but they had only deployed the default configurations that essentially amounted to little more than a ping or http connect monitor. Similarly, a 3rd-party vendor we engaged to monitor our servers produced interesting reports about low latency and “100% uptime,” but when we said, “no, we actually know it’s not 100% because we were down yesterday,” things became complicated and we debated over “server uptime” vs. “application uptime.”

Most monitoring–out of the box–is just server monitoring. And it’s amazing how well servers perform when their applications are broken and they can just sit there idle. Once you have found a real problem, it is easy to “point-source” a solution after the fact to watch one particular thing, but that is a very reactive mode to live in. Similarly, once the IT group figures out how to do the monitor they didn’t know you needed, sure, they can turn that on. But if systems engineers have a complete toolset available, they can be much more proactive about helping you monitor your applications.

Here’s a list of some of the transactions Heroix can monitor out of the box. I’ll follow up more this week about how these types of monitors can be used in real situations: