I think I should add a little clarification here.  What I am complaining about is not the web interface that Nagios has had forever, where you can acknowledge alerts, view check results, force rechecks, or any of that; I have no problems with that interface.  Instead, I’m talking about the fact that all Nagios configuration is handled through text files.  For experienced *nix administrators, text files are great.  Sed, grep, and vim allow you to manage hundreds of text files without much difficulty.  But whenever I had to show anyone else how to make a change I would cringe, because most people don’t enjoy editing hundreds of text files.  That is the problem that has been solved.  The two most popular solutions are Centreon and GroundWork.  We chose to go with Centreon, so that’s what I’m more familiar with, but from what I can tell, both products provide basically the same functionality.

The key improvement is that Centreon relies on a MySQL database to store configuration data.  On top of that, there’s a web interface (written in PHP, so imminently hackable) that allows you to do all of the necessary management.  Using a database allows the front end to let you do things like bulk changes to tens of hundreds of services at once.  It lets you define a service, and then link it to multiple hosts.  Centreon supports everything that Nagios supports, such as templates that inherit settings, escalations, dependencies, and event handlers.  In addition it supports a lot of other things, like setting up host templates with a bunch of services automatically attached.  It integrates with LDAP if you want to authenticate your users against your AD controller.  It also has built-in performance graphing:

And best of all, it makes configuration easier for those who aren’t totally in love with the command line!  I believe this opens Nagios up to a whole new range of potential users, and to me that’s really exciting.  There’s still a learning curve associated with Nagios itself — learning what sorts of things it can monitor and how to make it react the way you want it to — but at least now it’s more accessible.

We’ve been running Centreon on our production Nagios server for 3 months now, and so far we’re very pleased.  How about you?

Like Nagios, Cacti is capable of monitoring your servers, as well as processor, memory, network, and disk utilization on your networking devices. After initial installation, adding hosts to be monitored can be completed using nothing but the web interface.

Installing Cacti is fairly straightforward. There are even customized distributions, such as CactiEZ, which is a CentOS derivative, to make for an even easier install. Cacti can even be installed on a Windows host.

Cacti is incredibly useful right out of the box, but with a few plugins Cacti’s functionality can really begin to shine. The following is a handful of plugins that I think no Cacti installation should be without. The CactiEZ distribution includes a lot of these plugins out of the box. All of these plugins can be downloaded at http://cactiusers.org/downloads/.

CLOG – CLOG provides all of your Cacti logs in an easy-to-use interface built into the Cacti web GUI. This can be immensely helpful in troubleshooting issues within your Cacti installation.

Discovery – This plugin can be used to track down SNMP-enabled hosts on your network that haven’t been entered into Cacti yet.

NTop - While this plugin only provides your ntop data within an iframe, it allows you a single interface to get a full overview of exactly what’s going on in your network.

Thold - “Thold” is short for threshold. This plugin allows you the ability to configure thresholds and alerting. If you want to get an email or text message when a disk is nearly full, or a network link is reaching max capacity, this plugin is what you need.

Weathermap - This plugin is really cool. It allows you to draw maps of your network. Once the maps are drawn, you can assign graphs to the different links. These links will be updated in real time, changing colors based on utilization. This gives you a quick overview of how the network is performing.

These are just a few of the many plugins out there that make Cacti such a great tool.

More interestingly, perhaps, many of the participants in the survey reported that they are switching to OSS for non-financial reasons such as:

• Better security/bug fixing
• Faster time to development
• Improved reliability
• Easier to maintain the software
• Better match of software to organizational/business needs
• Auditability of the software
• Better quality software than proprietary systems

At AppliedTrust we have experience with a wide range of Open Source Software.  Many of the servers we support are running on the Linux kernel, many of the webservers are running Apache, PHP, and MySQL.  We use Cacti and Nagios for monitoring, and Mantis, PMWiki, and MediaWiki for tracking bugs and technical issues.  It makes sense that at a time when companies are looking to cut costs and conserve money, many would look to enhancing their use of free and Open Source products.  I firmly believe that Open Source makes sense, not only from a financial perspective, but also because of the power and flexibility it gives you over your IT destiny.  When a client wants to add a piece of functionality to Mantis, not only is that a straightforward task, but they know that it will be easy to find programmers that can do the work.

I think we should all remember, however, that OSS is not a panacea.  For instance, Linux web servers usually provide great return on investment, but it’s important to integrate them properly into your environment.  If you use Active Directory to authenticate your Windows hosts, set up your PAM modules to authenticate against AD on your Linux hosts as well.  MySQL is easy to set up and maintain, but it’s important to remember to secure it properly by creating accounts and restricting access appropriately.  Staying abreast of vulnerability announcements and installing security updates is just as important as it is with proprietary software.  Luckily, administration in the Open Source world isn’t any harder than in the proprietary world – it’s just different.  And if you need a few pointers, there are some great resources available (warning, shameless plug!) such as the Unix and Linux System Administration Handbook, 4th Edition.

In any event, it’s an exciting time to be in the IT industry, and it’s especially exciting as a consultant to be able to point out such a wide variety of choices to our clients.  The number of solutions to any given problem seems to multiply with each passing month, and that can be no bad thing.

TCP header, Lego (tm) style

The older I get, the more lessons I seem to learn (or, not learn) over and over.  Have you ever seen TCP offload work correctly?  Of course not!  I’ve been bitten by a TCP offload (aka TCP Offload Engine or TOE) problem in just about every environment I’ve touched in the last 20 years, and sadly this week was no exception.

To make a long story short, we have a production vmware ESXi 4.1 host with both Linux (CentOS) and Windows Server 2008 guests.  No problems were reported (or measured) with the Linux guests, but the Win 2008 guests suffered from extremely choppy network connections, for common services like Remote Desktop and backups (including lost connections).  As you probably know, I’m big into actually investigating the underlying cause of a problem rather than randomly throwing darts at it, and as such I grabbed some packet traces with wireshark.  Check this out:

wireshark analysis of poor TCP connection

Ouch! That is super ugly (this is across a LAN, btw)!  How can you screw up a single TCP connection so badly in 6 feet of cable?  Probably not the cable (or network), sherlock.  It appears this is a “known problem.”    While this problem (described in the vmware article as “Network performance is very slow and connections drop intermittently”) seems contrained in the article to vmware guests running on a Windows host, I can attest to this occuring on both ESXi 4.0 Update 1 and ESXi 4.1 hosts with Windows guests.  After following the instructions in this vmware article to remedy the situation by disabling TCP offload on the Win 2008 guests, they exhibit downright snappy network performance.  Check out the improved trace results:

wireshark analysis after disabling TCP offload

Moral of the story:  TCP offload always sucks.  Turn it off on Windows Server 2008 vmware guests.

Writing a book of this magnitude is an intense process that I learned all about. The steps to produce the book, from inception to dead trees, include:

• Brainstorm and agree on full topic list
• Brainstorm and agree on contributing authors
• Assign chapters to authors and contributors
• Write chapter, distribute for review
• Integrate reviewed comments from all other authors, distribute to external reviewers
• Integrate external review comments
• Repeat for all 32 chapters
• Edit chapters
• Index chapters individually
• Engage artist (Lisa Haney) for new chapter cartoons, dividers and cover art
• Engage outside organizations (IBM, Sun, HP) for test equipment
• Regular (semi-weekly) meetings with authors, occasional meetings with publisher
• Read and revise page proofs, searching for any obvious errors or inconsistencies
• Deliver final manuscript to publisher and wait patiently

One of the biggest challenges in producing this edition was the distributed collaboration effort. We Skyped regularly to stay in sync. Evi was around for much of the development, but we also corresponded with her while she was sailing in the Caribbean and the Pacific. We used a subversion repository for the Adobe FrameMaker source files to avoid stomping on each other’s work. I’d say this was met with mixed success; Frame’s binary files are hard to merge, despite Garth’s valiant efforts at a scripted solution.

Special thanks to our named and unnamed contributors whose efforts are highly appreciated and certainly worthy of recognition. This is the best edition yet!

Goodie #1: Learn why version control is important for all businesses across the board.

Goodie #2: Get some assistance in deciding “Git or Subversion? Git or Subversion? Git…?”

Goodie #3 (otherwise known as the cherry on top): Meet Jim Turpin, one of our fabulous network engineers, who embodies the concept of multi-discipline to a T both inside and outside of the office.

Click here to read Q3 2010, and, as always, enjoy the treat!

We’d love to hear from you, so please post your comments and questions here.

Twitter is well known for great application-layer monitoring and instrumentation, so this gap in monitoring is a surprise. It exposes a common misconception among social software companies – that their server and network infrastructure is “covered” by their hosting provider.  As web applications scale to even 1/1000 the size of Twitter, software becomes critically interdependent on the underlying network. Infrastructure should be instrumented and monitored at least as closely as the software that depends on it.

For more The Barking Seal articles on monitoring and troubleshooting, see:

I should emphasize that this was our experience, and your mileage may vary. AppliedTrust is not a web design firm – we pride ourselves on infrastructure (servers, networks, security, performance, etc.). Still, I am tremendously impressed with Drupal and you are probably making a mistake if you are building a complex web site and haven’t considered it.

Looking back, since 2001 we have transitioned from static HTML (managed with GoLive), to Joomla, to WordPress (which we continue to use for this blog), to Drupal. Each transition has been a marked improvement, and today, I can’t imagine using anything except Drupal (for complex sites) or WordPress (for simple ones). In closing, here is a visual history of AppliedTrust’s web platform “evolution”:

1) Set goals: Both in exercise and in information security, it is good to set goals. For example, before I can write up a training plan for myself, I need to know what race I’m training for, and what my target pace is. Similarly, before I can write up my information security plan, I need to know what information I need to protect and how much protection I need (is this credit card data, or is it records of what color paint my store sold last year?)

2) Stick to the plan: Sometimes on Saturday morning at 6 a.m. I don’t really want to go for a run, but I know that the only way to reach my goal is to throw my legs over the side of the bed and stand up. I also know that the best infosec plan in the world does no good if it doesn’t get followed. It may not be fun to conduct that periodic audit again, and it may be frustrating to have to patch those darn servers in the middle of the night so you don’t impact the production systems, but you’ve got to do it. A plan in a drawer is no plan at all!

3) Make the plan doable: I could probably run my race a lot faster if I was willing to quit my job and train full time, but that’s just not practical. I need to keep perspective on the rest of my life and make the plan something that I can accomplish. The same is true for the security plan. It’s not reasonable to expect your team to install each and every patch within 24 hours of release. Save that extreme stuff for the really critical items that only come up once in a while. Be reasonable, and you’ll make everyone’s life easier. No one wants a security approach that flies in the face of usability. But there are some things that just can’t slip – and make sure you know what those are. Passwords, for example: I know it’s easier to remember five-letter passwords with no complexity requirements, but if you let that happen, you may as well forget the rest of the plan.

4) Celebrate your successes: I run with a group twice a week, and the coach keeps track of our times throughout the season(s). When I hit a personal record, she knows it and we celebrate the accomplishment. Do the same thing with security! Did your annual assessment just come back with 20% fewer recommendations? Did you just pass a penetration test with flying colors? Great! Celebrate! There’s always another mitigation recommendation you can implement, but don’t forget you’ve done many already, and congratulate yourself on a job well done.

The Deepwater Horizon rig was equipped with a vessel management system (VMS), which records dozens of different metrics about the conditions on the rig and in the well. These VMS logs would contain valuable details about the blowout, much like an airplane “black box” is essential in understanding a plane crash.

Steven Newman, the CEO of Transocean, said during a recent senate hearing, “There is some delay in the replication of our data, so our operational data, our sequence of events ends at 3 o’clock in the afternoon on the 20th. And so the VMS system, along with the logs of the VMS system, would have gone down with the vessel.”  The blowout and massive explosion happened at 10, taking eleven lives and seven hours of VMS data to the bottom of the ocean. Representative Bruce Braley from Iowa followed up with “So you have no mirrored backup data device so that that information is recorded at some other location than on the rig itself?”.  Newman replied, “We do not have real-time off-rig monitoring of what’s going on on the vessel”.

The costs to synchronize this data back to shore closer to “real-time” are nothing compared to the catastrophe at hand.  If an IT Disaster Recovery risk analysis had been performed, this replication delay would have stood out like a sore thumb.  We can be certain that new congressional regulations will be established to ensure that VMS data is replicated back to shore in a timely manner, but what about the rest of us?  Now is a perfect time to take a look at your business and make sure that critical data is being backed up appropriately to an off-site location.