Before I go any further, let me confess that the blog post itself was another failure in that saga. I began writing it the day I discovered the a forgotten boot DVD in the optical drive was the cause of the server not coming back up, and I continued adding to the post over the next several days. Because I didn’t give a contiguous block of time to the writing, I left out some details. Furthermore, I posted it too soon because I was absent minded as I was saving the post and accidentally published it instead of saved it. I had to quickly unpublish it, however it was too late; an email notification went out to my subscribers so the unfinished article was read by a few people. Then when they clicked to go to the blog, they were met with a 404 error. Finally, I continued to write the post, scheduled it to be published the next day, and then completely forgot to polish it off because I staggered away from my computer to hit the treadmill, shower, and then collapse into bed. I woke up the next day, my computer still on, my desk still in “work mode” with items scattered all over it, and the WordPress control panel still open with the post being edited. The failure train was still going full steam.

If you’ve read the previous blog post, you might want to read it again because it’s now a little better presented with some key points added that I had left out.

The conclusions that I came to as a result of the Circus of Calamity that happened over Mother’s Day weekend are nothing new to me, and very likely nothing new to you. However I think they bear enough fruit to be written down. I’d like to codify my thoughts on technical debt in the system administration world and how to avoid it or deal with it if you’re currently in over your head. Mayhaps this will be the first effort in a larger work.

Without giving too much thought to the order in which I think the following tenants are valued at, here are some of the lessons that stand out to me.

Lessons learned:

Devote contiguous time to projects. In this case, my client gives me an hour cap. I can work on their systems for a certain amount of hours per month. That naturally lends itself to non contiguous blocks of time as I hit my maximum and then wait for the month to roll over. However, even with that limit, it’s best to spend it all in a largely uninterrupted segment of time. Regardless of if you are salaried, full time contracted, or an independent hired gun that works for whoever needs you whenever they can pay you, the concept remains the same.

If you have something to do, devote as much contiguous time to performing a task as you can. In my cast I tend to carve my day up thusly: four or five hours on one client’s systems, then another four or five hours on another client, and then perform three or four hours of tasks that I need to do on my own systems, bookkeeping, and general business busywork. This leeds to long days of frequent context shifting. In my last post’s situation, that led to errors like leaving a DVD in the optical drive of a server as well as not copying over the client’s password store on a schedule. The last post of mine was even a victim; as I context shifted too frequently, I forgot that I still had some polishing to do on it.

Don’t break your thoughts up. Think on a specific task, task set, and/or client for as long as possible with as few interruptions and context changes as possible.

Fix problems as they come. “I’ll get to that later” is death. The problem with the ILO needed to be addressed immediately. The problem with the BIOS clock resetting after power state changes needed to be addressed immediately. The problem with the hard drive controller failing needed to be addressed immediately. These were emergency level things that got distracted by the peculiarities of being an independent consultant and working for a place that has very small project budgets. Or perhaps that had nothing to do with it. Perhaps I could have pushed harder and taken more initiative. I know this office well and could have ordered the BIOS battery and contacted a local contractor to walk in and replace it. Act first, bill later! After all the years working with this group, I’m fairly certain that such an emergency action would be accepted and paid with no question.

Regardless of my specific situation, the idea is that, with few exceptions, one needs to address problems that crop up at that time, and not later. This is a sister concept to devoting contiguous time to a project. Keep on with solving a problem, and its directly related troubles for as long as possible without interruption. That can mean hours, or days, or longer as possible. This can lead to a rabbit-hole scenario where one simple change then leads to a huge infrastructure change. However, simply glossing over an issue adds another mound of debt to the overall systems debt. Don’t know why DNS queries are taking five seconds to resolve? Eh, it’s just a few seconds to wait and we’ve got bigger issues to solve with the payroll application. However, when a problem crops up as a result of DNS not being as smooth as one would expect, now you’ve got to face the DNS problem with another problem on your back. That pressure may lead to greater problems by encouraging you to implement half-measure solutions for the DNS resolution delay, which then causes another problem, which then causes another, and another… etc. and etc.

This is rather hard for consultants like me, however, so this bad tendency is strengthened. Consultants are paid by the hour in most cases, so the pressure to deliver without drawing out billable time is great. If a flat-rate project quote is made, you’re always one step away from hitting a mine field if you go too deeply into ancillary systems. I’ve lost my shirt as a result of a flat-rate project quote that ended up with scope-creep. While once in a while chasing down each problem to its root turns out great because you get extra business to fix those systems, most of the time it causes much more pain and suffering in the form of unpaid invoices, broken systems, and angry business owners.

This also seems rather tough for any IT person in general. At any given moment we’ve got large amounts of projects and rooms full of executives all competing for our time. Each one thinks they’re the most important person and project. Each one wants to be completed yesterday. There comes a point when being driven by the tyranny of the urgent has to stop, one way or another. The balance of when to chase down a problem to completion is a fine one, but I think we should collectively assume that a problem should be fixed immediately and require strong evidence to the contrary before abandoning the pursuit.

Get rest and stay healthy. I’ve been grinding hard for far too long. Starting a business is no joke. Keeping the business alive with paying clients all the while sharpening your skills and keeping potential clients engaged just in case existing business leaves is even less funny. I’ve been working so many hours in a week for three straight years that I’m aging myself prematurely. I’d love it if I could take a vacation or relax more, but the truth of the matter is that the clients I’ve picked up haven’t been the most lucrative and there have been some billing and invoicing… issues. I don’t have the time or the money to do much else with life except clatter away in front of a computer.

I’ve been departing in the evenings to get back to a hobby that has always interested me: weight lifting. That’s helped, and I’m contentedly regaining strength and getting re-bitten by the lifting bug and the addiction to “the pump”, but this is a fairly new re-committment. The plain facts are that I’m tired and zapped of mental energy. I’ve been grinding and it shows. The dumb mistakes I made for the client in the last post are in some part a result of mental exhaustion. I’ve lost some of the love for information technology that I used to have. I’ve visibly aged ten years in just three and I don’t have much material gain to show for it. I’m tired, and I did a disservice to my best client because of it. Shame on me.

(That said, if anyone knows of a business known for paying market hourly rates, and on time, that needs an independent system administrator with my skills based in Phoenix, Arizona, I’ve now got some time that I can book for a new client. Please, no employment positions at this point. Only consultant / contractor.)

Kanban can help! Kanban – I lurves it. I’m not about to suggest that it is the solution to everyone’s problems, but for those of us who are more tactile and visual, this kind of project and task management system can really make a difference. Kanban, in the simplest explanation that I’ve come up with so far, is a means of visualizing work and encouraging a limit to concurrent work.

In the cases of carving out contiguous time, fixing problems at the earliest possible moment after discovery, and even staying rested, kanban can be very useful since it forces you to be constantly aware of what work is currently being performed and what work is waiting to be performed. I use it to break larger projects into smaller chunks. Typically if a task would take more than four hours then it needs to be broken down into more than one ticket. However, in some cases I simply write “Work on X project, 4 hours” and that’s enough. But I’m digressing into the specifics of kanban when that’s not the point here.

Kanban as a means of staying on target and being ever aware of what context you’re currently in can be a huge boon to the hamstrung IT person. It’s especially helpful if you work with others and keep the kanban board highly visible. That way people always know what you’re working on. It’s very helpful to have management buy-in to that kind of system. Why? Imagine you’re in an environment where everyone thinks their projects are of utmost importance. If you limit your concurrent working projects to one or two like a good kanban system suggests, you can point to the board, specifically the area that is dedicated to tasks that you are currently working on this very moment, and force a choice. When someone complains about their stuff not getting done, then, with the blessings of leadership, you can require that the person who wants their project to be given top priority, contact the task owner of the current project that’s being worked on and explain to them why that has to be shelved and how long it will take for you to get back to work on it.

This can really help. Everyone understands sticky notes on a white board. If Gilles in accounting thinks that his project is the most important thing, tell him he can move James’s ticket from the currently worked-on project over to the holding tank. You know James. Six feet eight inches of security guard whose beard has more muscle than your entire body? Good luck, Gilles! But seriously, putting things into perspective for various project owners can seriously help you block off contiguous amounts of time and stay on track.

Sadly, in my case as an independent consultant, I can’t make Client A call Client B and explain why Client B needs to let up on using my time. It doesn’t work that way. However, I can still use kanban to aid in my own self discipline of keeping track of what I’m working on and when.

Digging out of the Hole

Some people have asked how I’ve dug out of that hole with the client. I’m still working on it. Meanwhile, in addition to that client, I’ve got a few other projects that I’m trying to sew up, so it’s a delicate balance of time and guarding contiguous working hours on each project. I can tell you that when I do get out of the hole of technical debt, it will be in large part due to kanban and a good set of targeted goals.

I need to first create a large vantage point for all my major task spheres. Currently I have three clients that I’m working with. One is a bit of a deadbeat. One is low priority. One is high priority (the one with the crashing server). In my personal life, I’ve got a few large goals to be concerned with as well. I’ll make those large circles of potential task lists and then figure out what is most important to me at this point in my life.

I can tell you that the high priority client (who also pays on time and is in good standing) will be at the top of the list of work projects. My low priority client will be a close second because the project is smaller and close to completion. It will be a great relief to get them finished. The client who is late paying is down on the list of priorities. Even them paying up their overdue invoices won’t get them past third spot until I see a history of on-time payments and not wanting cut-rate hourly rates.

Within each task sphere will be a list of tasks ordered by importance. Of highest importance for the client with the failing server will be to get new equipment in and start the migration. That much is obvious. I’ll need to address each quirk and hiccup along the way, such as ILOs dropping off the network and the like. Furthermore, I’ll need to guard contiguous time. Instead of dividing a day in three parts, (four or five hours for one client, four or five hours for a second client, and then a handful of hours for business management), I prefer to block off multiple days in a row for each client and task. That looks like this: Monday and Tuesday for one client, Wednesday for another, Thursday for business management, Friday and Saturday for another client (Yes, I work six days a week. It ain’t easy being self employed).

This should facilitate a steady march towards normalcy and healthy systems.

Parting Thoughts

1. Don’t assume something isn’t important. Assume it’s important and require a lot of evidence to prove that it’s not. Give a serious effort at fixing any problem at the earliest possible moment.
2. Dedicate contiguous time to completing a task and don’t dare multitask.
3. Prioritize based on danger and worth.
4. Chill out and get some exercise. BRO, DO YOU EVEN LIFT?

Not exactly ground breaking advice, but perhaps you need to hear it. I know I do. Preach it. Got any other tips and ideas? Any stories of failure and phoenix-like recovery from ashes? Let me know in the comment below or send me a guest post!

At 11:32PM, Saturday May 11th, I got an email from MX Toolbox notifying me that a SBS 2008 machine that I support had gone unresponsive. It’s 600 miles away from me in another state. This was not a strange occurrence with this server.

A Cluster of Prior Failures

Five years ago a small office with a minimal budget needed a SBS implementation. I recommended an HP ML 115 G5 with four hard drives and onboard RAID provided by an NVIDIA chipset. I have regretted that decision for all five years. Here’s a post of mine concerning that chipset and the troubles I’ve had with it.

In short, I have poor insight into and control over the entire server’s health. Some examples include:

• I couldn’t update the hard drives’ firmware, which was a big deal because the serial numbers of those hard drives fell into a set of drives that have a known problem with suddenly going offline. The firmware update has to be applied through HP’s support tools, which are not supported on the ML 110/115. After much research and seeking help from HP, I was told that, in essence, I was left out to dry.
• The ML 110/115 does not support the ProLiant Support Pack nor does that model support the Insight Control Manager. Keeping drivers updated and staying abreast of the various components’ health was virtually impossible.
• There was also no HP ILO CLI interface available which made doing things like firmware updates especially difficult remotely.
• The on-board storage controller had poor support form Nvidia, and offered very slim storage management features or reporting on hard drive health.

For years I hit the management ceiling with that box which probably cost my client more of my time and theirs than had a more robust server been purchased for twice the hardware cost. And then what I had been dreading for years finally happened…

Two Months Ago

“Did you reboot the server?” That’s never a question you want to hear, especially when you did not reboot a server. I VPN’d into that office’s network and checked for the presence of the server on the network. Yes, the server was down. One power cycle later, the OS loaded just fine.

I checked the event logs and it turns out there was a massive flurry of parity errors that came out of nowhere. The server froze as a result. The controller was apparently dying. After a reboot, the data appeared fine, and there were no more parity errors coming from the Nvidia storage driver. I knew something had to be done, but being remote and working with an office that has a shoestring budget (and can often only afford used shoestrings) made the options few and unattractive.

What’s worse, as I started investigating things further, I noticed that the ILO Advanced card that was in the server was no longer showing on the network. Aaaaand the BIOS clock would reset to July 2009 after being shut down (BIOS battery dying) causing strange problems with Active Directory and other applications running on the network that relied on accurate time (read: everything). AAAaaaaaand the two mirror sets (one for the system volume and one for the email server’s databases) had split apart and could not be re-synced because the Nvidia storage management software no longer recognized that any hard drives were connected.

The options, as I saw them, were for the business to either buy a new RAID controller, BIOS battery, and perhaps ILO card (and then scramble to perform the complex surgery remotely on their own, or pay a local consultant to coordinate with me, or pay to ship me on site) or get a new server altogether (and pay a local consultant to coordinate with me, or… you get the idea). Either way, it started to look more and more like a total forklift migration was necessary.

Two Months Later

Yes, it’s been about two months and the server is still riding in the same perilous state. Split mirrors, bi-monthly freezes that require a power cycle to recover from, and a lot of hoping and praying that data is not corrupted. Welcome to the world of supporting small business IT where people re-use tea bags and don’t run heat or AC in order to save money and keep the business open.

That Saturday night, it was getting late and I was thinking about bed. I checked my email one last time for anything pressing when I saw a MX Toolbox alert. This is never good. I scanned the email, saw what host was causing the alert, and knew that I was dead in the water. I could get into that client’s network via both a SonicWall VPN and unattended TeamViewer installations that existed on most of the workstation PCs. However, it was all futile because I didn’t have hardware level access to the server as a result of the ILO’s failure. The office has a Lantronix Spider KVMoIP device that was being used to work on a workstation migration for one employee, and was therefore not hooked up to the main office server. That was two layers of out of band management that was not doing any good for the most important technology asset in the building.

All of this meant that someone would have to show up at the office to power cycle the PC. The technical debt and compound interest of failure had already mounted fairly high by that point, considering the state of the server. However, things were about to get comical.

I’ll Gladly Pay You Tomorrow for Out of Band Management Today

What happened in the next 24 hours was a morbid comedy of oversights and compounded problems that ended in a whiplash inducing facepalm.

First, I needed to email three people who would most likely be in the vicinity of that office so I could coordinate with one of them to drop by on their Sunday morning and power cycle the server. Except the server is what does email for the organization so I can’t send to their organization email addresses (this is a Microsoft SBS machine). I only know of one employee’s non-work address, and I also happen to know the gmail address of another employee’s son.

I email those two people and tell them of the situation. As it turns out, two key workers are out, traveling to a convention in Texas. That makes access to email even more vital than normal. Everyone knows the situation and there’s not much more I can do so I get to bed. It’s not until about 2PM on Sunday, Mother’s Day here in the USA, that I hear back from one worker who has just enough time to skip by the office and power cycle the server.

Myself, I’m in the midst of a Mother’s Day dinner with my own family so I had ditched my phone… just moments before the employee called me from the remote office. I missed the call and the employee left a voicemail expressing a state of confusion over which server to power cycle. The organization is small and only has two servers. One is the SBS machine and the other is a HP MicroServer that is used as a network monitoring station and catchall for various extraneous services. I had assumed that over the years everyone had each server’s role understood by sight so I simply asked him to power cycle the SBS server, expecting that it would be known which piece of hardware that was. The fellow power cycled both servers since he couldn’t get in touch with me directly.

Okay, no big deal. The MicroServer is just running CentOS and OpenNMS. They’re resilient and can handle a sudden shutdown. As I listened to that voicemail, I checked to see if I could remotely connect to the server that had been down all night. I couldn’t. Great. Time to call the office and talk to the person who was on site and see what else could be done. Except the voicemail had been left over an hour ago and the employee had naturally left shortly after power cycling the server. I called his cell phone back, but he’s didn’t pick up. I left a voicemail.

A little later that Sunday I get in touch with another employee in the area who lives closer. He’s on his way out to pick up Mother’s Day dinner for his wife and can swing by to check out the server. First, I have him power cycle it again. Maybe the first guy just clicked the power button and didn’t hold it in? I held out hope for such a simple explanation. However, after I instructed this second person on how to make sure the server had shut down and then powered up, I waited for the duration of the standard bootup but nothing was showing up. It became apparent that the server was not coming back online.

“Do you know where the Spider is?” I asked hopefully. “No, I dunno where the other guy put it.” Gah! The Spider is a well known piece of equipment in that office, and it’s very rare that it can’t be found. I was about to concede defeat for that Sunday when, after some searching, the employee found the Spider. A few minutes of scrambling around and he had the thing hooked up to the server. Except… now I couldn’t get to the Spider. The fellow had to leave to pick up dinner and I wasn’t about to ruin his family’s Mother’s Day so I told him I’d see what I could do remotely, expecting nothing to be successful.

In the process of hooking up the Lantronix Spider, the employee had pulled the network cable out of the server and put it into the Spider. Then from the spider’s cascade port (it’s essentially a one port switch) he had connected a patch cable to the server’s LAN port. That made me wonder… perhaps it was a port on the ProCurve switch that was bad? That would explain both the server and now the Lantronix Spider being inaccessible. Or maybe the port spontaneously shut down as a result of some bug. Crazier things have happened.

I browsed to the switch’s management interface. “Please enter your username and password!” Okay, no problem! “Wait… I can’t remember what the password is… NOOOOOO!” The organization uses KeePass to store important passwords and software keys. The KeePass file is on the server. The server that is down.

But wait! I have a copy of the keepass databases on my own storage. Once a month or so I copy the files to my local storage so that I have an in-sync copy just in case. Whew! I find the switch’s login credentials and begin inspecting things. I looked, hoping for some bad news concerning the switch’s health (at least that would mean the server was okay), but the switch looked perfect. Nothing was amiss.

I’ve always been told to troubleshoot network problems from the lowest layer first. I had pretty much ruled out the physical layer. Layer 2 seemed healthy. Not much that can go wrong on a small, single subnet LAN. Layer three, IP… IP addresses… I gritted my teeth. I knew what the problem was. The Lantronix Spider is set to pick up an address via DHCP. Specifically it’s a DHCP reservation on the network’s DHCP server. The server that’s down. I wanted the network layer benefits of a static IP address, however I also wanted it to be easily portable between networks. My original idea was that the Spider could be used to support PCs on other LANs, like perhaps workers that were based in their home office that didn’t come into the organization’s building very often. With the Spider getting an IP address via DHCP, I could just tell someone to take it home with them and I’d only be left with walking them through configuring port forwarding, or getting TeamViewer set up on a PC on their LAN so I could get in and access the Spider via a local web browser. Except now the Spider was barking out forlorn DHCP discover packets and not getting any response back.

I fired up Network Monitor on an office PC to be sure. Yep, there it was. A DHCP discover request broadcasting every sixty seconds or so. Okay, I can handle this. The small office has a SonicWall firewall that has DHCP services on it. I only need to enable them, check its list of leases to find what IP address it was given, and I’ll be good! I mosey my web browser on over to the firewall’s administrative page. I stare at it. It wants the password for the admin user. “Password… password… I had to change it a few weeks ago. What did I choose…”

Oh well, I’ll look in the organization’s copied password file that I keep on my local storage! Yay foresight!! I found the firewall admin password and entered it. “Password Failure. Please Retry.” What?! Then I remembered that I had changed the firewall password due to security policy about two weeks ago. However, I hadn’t copied the organization’s password file to my local storage in a month. I had the old password in my copy of the password file, but not the new one. The new one was on the server that was currently down. Backups are taken every few hours, but a restoration needs to be done on functioning hardware. Super.

So that means I did it again. I couldn’t log in to the interface because I didn’t have the long password committed to memory. For super important passwords like that, I do keep a disaster recovery hard copy around. It’s essentially a few pages spelling out the most important usernames and password for the organization. However, only two people have that physical copy of information. While I could call them up and have them read off the password to me, I wasn’t ready to do that.

Instead, I turned to the HP MicroServer running CentOS 6. I have OpenNMS installed on it and have plans to install some ticketing software and maybe smokeping or M/Monit. Now, however, it’s going to be an impromptu DHCP server. Fortunately I can remember the password for the MicroServer! A quick ‘yum install dhcpd’ later and… “Couldn’t resolve host ‘centos-distro.cavecreek.net’” WHAT DEVILRY IS THIS?! But of course; DNS for the network is performed by the SBS server… which is down. After facepalming, I changed resolv.conf to point to OpenDNS and continued my march towards a functioning DHCP server on the network. After a few minutes I have dhcpd running and it quickly hands out a lease to the Spider.

And it was then that I saw it. After logging into the Spider, I viewed the remote console and saw a Windows installation screen on the server. Suddenly, I remembered what happened. In the process of preparing for a migration away from the failing hardware, I needed to experiment with making an unattended installation file. I had a remote worker put the SBS 2008 install CD in the main server’s tray. Of course, rebooting caused the server to boot into the high boot priority CD drive. I sat in horror, thinking about my cascade of failures. Nevertheless, that wasn’t the time to flail in self loathing. I simply needed to hit “cancel” and get out of the installation welcome screen to boot from the hard drive.

Except the Spider was unable to interact with the server as a remote keyboard or mouse. I’ve used the Spider on that very server in the past, and it worked great at all stages of the boot process. In the years that I’ve worked with that office I’ve had to check BIOS settings, ILO firmware settings, and storage controller settings, all using either the Spider or the ILO itseld. But now, for some unexplained reason, the Spider was not able to input anything. I couldn’t move the mouse, I couldn’t press keys. So I sat and stared at the remote video in complete disbelief.

It was a simple matter of leaving a voicemail for someone and telling them to remove the disc from the DVD drive the next time they were in the office. The next morning the worker that I left a message for did just that, power cycled the server, and it booted up as normal. Life continued.

I was abashed.

More about my conclusions concerning the situation later. In the mean time, got a similar story to share? Let me know in the comments below or contact me and you can write a guest blog post about it.

Talk #1: Pinterest and scaling

Older code at Pinterest is Python but the newer stuff is in the Java stack. Use Ostrich for Java processes. Python is using Sentry, StatsD, distributed tracing system is an internal tool called Kafka.

Application metrics are in Varnish / MySQL / Nginx / Redis. There are some tools like Percona’s toolkit.

If Pinterest is creating codes and needs metrics, they don’t like gauges. Gauges are a “right now” view. That’s not as useful because you can miss a lot if there is a giant spike between collection periods. They use “counters”, and while you might not know what happened between two sample periods, but you absolutely know that something happened and how much happened.

The nice thing about a counter is that if there’s a collection period downtime, counters in your code are always putting up numbers.

They choose to measure requests per second to get ideas of which systems are unbalanced. Read vs write requests, and of course errors. They measure latency, but also payload size. They saw latency issues in memcache, but they noticed that payload was huge which explained the latency.

• Ganglia is used for Cluster Metrics and Host Metrics
• OpenTSDB for high cardinality and alerting data (Some metrics are really unique and ephemeral, like memcache slabs and EC2 autoscaling.
• Kafka for event data
• Graphite for Application Metrics. Simple incremental monitoring for new applications as they’re made.

They use cgroups that allow them to run collectors in isolation. Kernel processes are separated from user space processes and user code is protected from kernel code. So there’s a cgroup for monitoring that has constrained resources. That way monitoring doesn’t take down the entire system. In the worst case scenario of bad monitoring code, then the monitoring gets killed with the OOM controller. Not the core services that you’re offering.

They use Ganglia because it requires the least amount of touch. It’s set up and they’ve had very few problems with it. They use RRDcache to buffer the stats to disk and don’t use solid states. They don’t use EBS because they don’t want to. They use local ephemeral storage.

Graphite and StatsD are used. The talk on this was very vast and over my head. Sorry.

OpenTSDB. They have 60 billion points on a 10 node cluster. They keep network performance data, memcacheD ketama ring and slab stats, HBase dynamic metrics.

Their graphite front end is written in Flask. Other tools like D3 and rickshaw for UI elements. JSON is good. Data sources need to speak JSON to their front ends (or so people say).

They use R for really complex, weird data queries. If you have data of any kind, you can look at it with R in esoteric ways, it’s just a little more difficult. Pinterest has a data science team that handles it.

Talk 2: Sensu as a monitoring system

Joe Miller (www.getpantheon.com) speaking. Sensu is a monitoring tool / framework. They do PaaS with Drupal.

Sensu has been called “the monitoring router.” It’s really a framework for building a monitoring tool. It’s Ruby 2.0 based, RabbitMQ for messaging, and Redis for basic amount of state storage. JSON is everywhere. You can simply re-use Nagios plugins. It is packaged with all the gems and Ruby packages that are needed.

Leverage your existing config management system to attach checks to your infrastructure. They tend to focus on Chef. There is a Puppet project.

The GUIs for Sensu are slim. There are two options. The Sensu-dashboard (which is included in the omnibus package that has everything in it). It has no user authentication and is stateless. No roles. The other option is Sensu-admin which is a separate package that has users, roles, scheduled downtimes, etc. It’s a Rails app.

Sensu runs on Checks, Events, and Handlers. Checks create events and are fed into handlers. Handlers are passed data simply through STDIN. There are handlers available to send information to logstache, graylog, and many more. It’s all in the community. Another way to diagram how Sensu works is that there is a Sensu-Server on one end, and Sensu-Clients on the other end with rabbitMQ in the middle. If you have a client, a node that needs to be monitored, it gets subscribed to a type, like webserver. It is monitored however webservers are defined. Messages passed from client to server, and then the server can fire off actions like alerts.

You don’t have to discover new systems, you don’t have to add it to any other monitoring system, it gooks into config management. The config files specific to Sensu are all JSON.

Some examples: Because every sensu client heartbeats via rabbit to the server, if it disappears for 180 seconds it generates an event that is sent to the default handler. However, for one large company, it will run a decommission script to remove that server from configuration management tools, and other systems that keep track of the system. Obviously, that’s intended for large cloud instances with lots of ethereal, come-and-go, servers.

18 month old project. Seems very malleable and able to morph with rapidly changing environments, or rather, make a static and rigid environment more flexible with the help of config management.

Talk #3

Linux Enablement for Calxeda ARM Servers by Rob Herring. He works for Calxeda, an ARM processor licesnsee that makes server deployments. They are making a “EnergyCore” arch for processors, quad core Cortex based ARM processors, with a fifth core dedicated to the IPMI, OOB management, etc. It includes a Fabric Switch on board with 10Gbit links, and of course an IO controller is included. It’s all the functionality of a server on one SOC. No South Bridge, North Bridge, etc.

The amazing part is that they are making four-node cards, so four SOCs, that are baked into one card that is about the size of a PCI card. The system board is essentially passive to connect all the nodes together. The first generation processor is called the ECX-1000, AKA “High Bank”, and it has a 1.4GHz quad core CortexA9. The next level up is the ECX-2000 CortexA15 that has virtualization support, KVM and Xen.

There are some example systems that are on the market. There are already example systems that are using massive amounts of these cards, including systems from HP and Boston Servers.

The Linux kernel has increasing ARM support, and major distros are including support for it, but that might be the next hurdle. The hardware is useful and can be used for low power, highly distributed applications, but the distribution support might need to come up more and more. UEFI, ACPI, and LPAE need to grow on Linux for ARM. A demonstration included running OpenStack on a Calxeda box and spinning up a new VM.

Talk #4

Mike McLane – Performance tuning as a web hosting provider. Works on the Genesis Team that is in GoDaddy to performance optimize and tune the services. GoDaddy started as web nodes attached to NAS.  Then went to multiple web nodes attached to NASs with a hardware load balancer in front.

They use nodes with lots of RAM and local disk space, but relatively conservative CPUS. CentOS 6.current is used. Apache 2.2.current but moving to Apache 2.4.current. All customer content is served via NAS (NFS) over local storage. GD is not using RHEL / CentOS 3 as is rumored. It’s  6.current.

They run mod_fcgid for FastCGI. mod_fastcgi was not ideal for scalable web hosting. They do run mod_rewrite, which is to dispel a myth running on the internet concerning GoDaddy not using mod_rewrite.

The use NAS because it was comparable to DAS / local storage for user experience so the didn’t change anything.

GD decided to take a close look at performance but they didn’t have quite the amount of data that they would need to make deep, scientific analysis of the large systems. They did queries for the top fastest loading pages, top slowest, 95th percentile, fastest 250 pages, etc. all measured across all customers, but it ended up being all over the place and just a bunch of numbers. Very hard to make hard conclusions from it.

They created the CEM (Customer Experience Monitor) that was basically a vanilla WordPress install on each and every host in the environment and then they’d load it and measure the performance. It wasn’t the best judge of performance. They watched all the numbers and none of the numbers really pointed to any one thing that was causing slowness. They used the flot javascript library to plot their data.

The problem was later found to be PHP serialization that if two requests came in at the exact same time, two processes would be spun up but the requests would be processed one after the other. It turns out there was a one second sleep in mod_fcgid that would be triggered to keep the box from being thrashed under certain circumstances. mod_fscgid author was notified and he made a nice patch, but apparently it’s not in the stable branch right at the moment.

That was a good first major win for tuning the GD stack. Then they implemented graphite to gauge the performance of the machines across the board. They sometimes use Apache bench to dump the stats to a CSV and then plot it out. Good ol’ iperf for bandwidth measurement.

It’s important to test performance form cold, warm and hot starts if you have different levels of cache. Cache can skew things.

They noticed that certain Xeon processors had problems with cstates. Even if power savings mode was disabled in the BIOS, some CentOS kernels would have stability and performance issues surrounding power state options.

Talk #5: Logging Love w/ Rashid Khan Elasticsearch Developer

Worked at a newspaper, they logged direct to disk, he then created Kibana. If you store locally, you have to ssh into each server, or use a really bad for loop for ssh. Then you get smarter and do a central syslog server. But then you end up making a massive grep/sed/awk spaghetti script. Log search scripts would break as things changed.

Then he started looking at commercial log analyzers, but they were punishing and hard. Enter: Logstash and Elasticsearch. Logstash is like an event pipe that works over the network. It takes things in with various inputs, amqp, eventlog, exec, irc, tcp, redis, twitter, udp, xmpp, etc. and etc.

Log files suck. Dates in particular. So many different types of date possibilities. Logstash can take in different date inputs and normalize. Logstash can use filters like grep, geoip, grep, metrics, multiline. Then outputs can be things like circonus, cloudwatch, elasticsearch, more and more and more. Rashid uses Elastic search to store logstash information in. It’s schema free, clusters very well, JSON over HTTP, does search and aggregation.

However, he needed an interface to manage all the data gathered and outputted by Logstash and held by elastic search. enter Kibana! It’s been re-written a number of times, PHP first, then Ruby to match Logstash, except Logstash is Ruby on Java through jruby and Kibana didn’t work well that way for a number of reasons. Now it’s all simple HTML and JavaScript interfacing with the ElasticSearch RESTful API.

Talk #6

Der.Hans talking about SSH and Bash tricks.

Set your shell’s prompt to something informative. $PS1 is the variable. \u@\h is a minimal set of information, that shows username and hostname. Sometimes you need the FQDN instead though. If you have your own account just use your shell config files on the remote machines, or you can use SSH config files. However, SSH config files runs before SSH launches your shell, so config files run with system default shell, not your user account’s shell.

In the case of not being able to modify any of those files, you can call ssh with a script that set up the remote environment on a temporary basis.

Obfuscate the names in known_hosts. If you put command= in your auth keys file, whatever command you put after the equal sign is run when you open a connection to that machine.

Set PermitRootLogin=”no” – never log in as root. You can pipe straight to ssh so you can tar on one machine and pipe it to ssh. Fuse-sshfs can mount a remote system as a local system.

Parallel SSH! for i in $( cat hostnames.txt ); do ssh $i some_command; done. However, Shmux is better! Also, ClusterSSH, parallel SSH, Mussh, Massh. sssl runs on a port and listens for incoming connections, determines what is intended for HTTP and what is for SSH and then sends the traffic on to the proper daemon.

SSH can perform local and remote tunnels, etc. etc. Check the -L and -R switches. Lots of examples, but I was paying attention and not writing.

Talk #7

Mike Dorman CentOS & Xen at GoDaddy. A couple years ago Go Daddy released a product called cloud servers. It turned out to not be such a great fit for GD’s target group. GD is more focused on the SMB that needs to have a simple website and have a simple online store. It’s been EOL’d and was completely turned off last week.

The development group for that products has come back together and they’re revamping it to an internal private cloud platform that will be sued internally. The vision is kind of like Google where they have their infrastructure that runs everything concerning the company, public and private. Maybe in the future GD will get off of the physical focus of dropping in more metal, and be more cloud based.

To provision a new server, there are server build times of 10, 12, to 15 weeks because of “The Checklist” that five, six, or seven groups have to look at and approve. The idea is to have a cloud platform to make it faster for servers to spin up and develop internal products and projects.

One of the troubles with monitoring it all is that the dynamic creation and destruction of machines is hard with traditional tools. They’ve also been trying to decouple the idea of physical servers from services. Physical servers are like puppies. You name them, care about them, and nurse them back to health when they’re sick. Virtual servers are like cattle. They’re given numbers, they’re driven hard, and if they get sick you shoot them in the head.

GoDaddy went with CloudStack at the outset of their decision to go cloud. Clustering and HA functions gave them trouble. It was essentially a forked version of cloudstack with code that couldn’t be submitted back up to the core project for various reasons.

They looked at CloudStack, and they ran into the problem that is in the OpenStack community about “clouds of clouds” or “pods of pods” where you can’t really grow a cloud too big. There were some management problems with it as well where it was not operationally simple to deal with. There are also concerns about the OpenStack community looking at the big players, their aren’t a lot of good contributions back into the core. A lot of the main users of OpenStack are essentially using forked versions of the project.

So, GD developed their own internal management stack and called it Norm, after the robot from some cartoon. It’s basically CloudStack, rebuilt into something very different.

So why not use XenServer as the hypervisor? It worked okay. You do have official Microsoft support if you want to run MSFT products in XenServer. However, it looks like Linux, but it can’t be treated like Linux. It has the network layer configured through a different configuration system. Management of XenServer couldn’t be done like a regular Linux machine.

Now they use stock CentOS and open source Xen. There is more of a community around CentOS, but the XenServer community isn’t comparable in a support sense. Patch management is easier because none of the XenServer patches are able to be patched through regular management systems like SpaceWalk. They never found a way to update XenServers to scale.

So, now after trying all the major cloud stacks, they’re building from CentOS and Xen raw and real.

The second CentOS Dojo is happening May 10th, 2013 in Scottsdale Arizona at the GoDaddy offices in the Scottsdale Airpark. Quoting from the CentOS website:

The CentOS Dojo’s are a one day event, organised around the world, that bring together people from the CentOS Communities to talk about systems administration, best practises in linux centric activities and emerging technologies of note. The emphasis is to find local speakers and tutors to come together and talk about things that they care about most, and to share stories from their experiences working with CentOS in various scenarios.

The location is at GoDaddy.com Studios 14455 N Hayden Rd #219 Scottsdale, AZ 85260:

View Larger Map

I will probably live blog the event as it happens. Talks are still being registered, so if you’ve got a talk that you want to give and are going to be in the Phoenix area, get in contact with CentOS project leader Karanbir Singh.

Some of the talks that I’m most interested in include:

• Metrics at Pinterest by Jeremy Carroll
• Leveraging CentOS and Xen for the Go Daddy private cloud by Mike Dorman
• Logging Love: Build a log analysis system with Logstash, Elasticsearch and Kibana by Rashid Khan

Check back at the official CentOS Dojo Phoenix page to see how the talks start lining up. Is anyone out there going? Let me know in the comments below.

I like strong passwords. No, I like strong pass phrases. It’s brutal to remember CX}bk<b_-K97 which is only a 12 character 96-bit password. However, it’s easier to remember doubledoubletoilandtrouble because it’s real words, and a literary reference on top of it (MacBeth, for those interested). It’s a 208 bit password with 26 characters. If you have password complexity requirements we could use Doubledoubletoilandtrouble1 to include an upper-case letter and a number (which brings it up to 216 bits) or if we also need a special character we can tack on an exclamation point to bring us up to 224 bits.

I typically use pass phrases for myself and clients. It’s gotten me into a bit of trouble in the past. In short, I changed the password on a public facing phone system interface to one that was longer than the system could handle. However, I didn’t know that and the system didn’t validate the input so I ultimately ended up locking everyone out of the system and a full factory reset was required.

I’ve got numerous other anecdotes concerning long passwords either breaking something or simply not being accepted. (I’m looking at you Chase Manhattan! Actually, let me re-scope that. I’m looking at you banking and financial sector!)

This story begins with a fresh installation of Windows 7 and some activation warnings. A client of mine had a Windows 7 desktop that was installed weeks earlier, but not activated. There is no KMS system in the building so MAK keys are used for this small deployment. I had to log into the Microsoft Volume Licensing Service Center to retrieve the client’s MAK key for Windows 7. For those not in the know, logging into any Microsoft online system requires a Microsoft Account (Formerly Windows Live ID, formerly Microsoft Passport). So I went to log in. With the 29 character password that I chose for that client’s ID.

Microsoft account passwords can contain up to 16 characters. If you’ve been using a password that has more than 16 characters, enter the first 16.

I was flabbergasted. I counted out the first 16 characters of the 29 character password and pasted them into the password dialog. I was able to log in. I searched online for more information and came back with this FAQ: Why can’t my Microsoft account password have more than 16 characters?

So I am assuming that all along, for the nearly ten years I’ve been using Microsoft’s Passport / Live ID / Microsoft Account sign-in service, only 16 characters have ever been accepted while the remaining characters were discarded. I cannot say that this wasn’t disclosed because it never occurred to me to verify that my 16+ character passwords, which were accepted without hesitation, were actually being honored to the last character.

Now I’m wondering what other accounts might be doing the same thing; accepting long passwords but silently dropping characters past a certain point. Is it so hard to calculate and store hashes from long passwords? Password policies seem to be encouraging complexity, not strength. Complexity that encourages post it notes to be placed under someone’s keyboard. Or a sheet full of administrator accounts and passwords kept on top of a filing cabinet… but I’m trying to forget that ever happened.

What do you think about all of this? Did Microsoft disclose the discarding of characters and most of us missed it? Should they have made a hard warning that required less than 17 characters?

What’s your take on Windows 8? I just read a story saying it had flopped and was pulling down global PC sales.

I wrote out a response to him using 500 words in one comment. Then I came back and added a few hundred more words in a second comment. After my third comment and a total of 1,000 words, I decided that perhaps my thoughts might be worthy of copying and pasting them into a blog post. So with very little revision, here are my shoot-from-the-hip thoughts on Windows 8. They may not make total sense, and might ramble off track, but hey… that’s me every day.

It’s an OS designed for tablets. Microsoft is banking on a very real possibility that consumer PCs will eventually be largely tabletized. They want to make an iPad killer without making an iPad. They’re designing their OS to turn PCs into tablets, even if they’re not. The “Metro Design”, the first vestiges of which was seen in Windows Phone 7, is purpose built for tactile interaction with fingers or styluses.

They also realize they need to get into the app store game, so they’re making the Microsoft app store which is a separate issue, but Windows 8 marks the app store’s debut into the desktop OS.

The UI changes a lot of what people have understood about the human operating system interaction for Windows. It’s like the big bridge-burning between Mac OS 9 and OS X. Things changed – drastically.

IMO I don’t see any killer features in Windows 8 that make it a must have. There’s lots of stick and no carrot. It is a huge shift in how one interacts with their Windows PC, and there’s not a lot of help. There’s a lot of floundering and misdirection that I’ve seen while interacting with it. It would take a few hours of video training to get any one person set up to know enough of the flicks, gestures, motions, and hotspots that would get the most out of the Windows 8 changes.

Sadly, that means that Grandma and Grandpa are screwed. Actually, no, *I’m* screwed because anyone not particularly computer savvy, or not interesting in the time investment to learn the new UI/UX (BTW, UI stands for User Interface and UX stands for User Experience) are going to be calling me up when I’m not really an end-user supporting person, so I’m mostly just Googling and clicking blindly to figure user support questions out.

IMO Windows 8 is good for the people that fit the Venn diagram of “Geeky” and “Tablet Users.” Get a Surface Pro (The price on those is absurd though) or some kind of purpose-build tablet PC, and then let Windows 8 happen all over you. Don’t argue, don’t whine – just let it have it’s way with you. Watch videos on YouTube, search for information, read a Windows 8 book, and just go with the flow. Then, yes, THEN you will become one with the PC and go all Borg-like with the human-computer relationship.

Or you’ll get mad and buy a Mac.

For example, my step-grandmother has a 6 year old Dell laptop. It had Vista on it (Note: I’m not a Vista basher and rather liked it. Most whining against Vista were for peripheral reasons, not the OS’s fault). She got a virus as a result of a cleverly disguised advertisement / popup that said “Oh noes you’re haxored! Buy this Super AntiMalrus 2000-and-late software to fixinate it!”

She had to ship the laptop to me from New Jersey to Arizona. Fortunately she’s just an email user that also uses websites to manage finances and prescriptions, so there was no heavy application dependency that tied her to Windows. She opined about maybe buying a Mac. I put the brakes on that idea and told her to save her money. Instead I dropped Ubuntu 8 on it (this was years ago).

Before I go further I should say I’m not a brand apologist. I don’t care about Windows this, Mac that, or Linux all up in yo’ grill. I use what works when it fits. I’d install CP/M on a crate of peaches if it worked right for the scenario. I installed Linux / Ubuntu for her because it fit for her scenario. And no, Linux isn’t virus proof, it’s just a smaller target at the moment. The instant it really does become the mythical “Year of the Linux” and market share shifts, it’ll get bombed and invaded like it had American oil within its sovereign borders.

Nevertheless, the point I’m getting at, is that Windows 8 is now such a burden of new interfaces and interactions that I’m going to avoid it for anyone that I don’t think can give the attention to it that it deserves. Also, for corporate installations, I’m going to have to consider my options. Things need to be up to date, and from a backend perspective (Active Directory, group policy, and other things that I’ll not mention for the sake of brevity) are all what matter most to me, but I can’t roll that out to business customers if it will drag their workday down.

It’s a tough choice – and in some cases a different OS altogether might make more sense. I think iPads are going to eat Windows 8′s lunch in many instances. I think Windows 7 will be the new XP and last will into Windows 9 and 10′s lifespans.

Oh, and let me prophesy for a bit: Windows will become a cloud-based OS on a subscription basis by version 10. Office is going that way and 2013 is likely to be the last “off the shelf” version.

Personally, I’m okay with Windows 8. I’ve ran the gamut of OSs from Macs pre-X to Windows to Linux to toying with OpenSolaris and the BSDs and now Mac OS X. I don’t have a Windows machine anymore except for a Windows 7 virtual machine I keep around for testing and a few apps that don’t exist for Mac. I’m used to change, and I think Windows 8 is… okay. From a certain perspective anyway.

However, I’m tired of keeping up and learning new things every day, and can’t fit Windows 8 into the learning rotation so I’m ignoring it. Most of my work these days is with 75 / 20 Linux Servers / Windows Servers. The last 5% is desktop support issues and my clients are still on XP and 7 and are likely to stay there. I’ve had all of two fleeting support cases with Windows 8 and it was basically “Will XYZ app work on Windows 8?” “I dunno, try it.” “Yeah it works.” “Cool.”

I would like a Surface Pro, but they’re too expensive for me, business write-off or not. I’m just not interested in learning it from the ground up – it’s not where my money is being made, or will be made in the near future.

TL;DR It’s not the steaming heap of filth that people think it is. It’s just different, which may be good or bad depending on your life and needs.

Let’s face it. NFS is a magical thing. It allows you to centralize your storage, share volumes across systems, and all while maintaining sane permissions and ownership. Unfortunately, it can also be a bit of a fickle beast. Let’s say you just had your volume configured and you set up the mounts. You go and run this command:

mount -t nfs 10.10.10.1:/vol1/fs1 /data

Works like a champ, you now have your data partition mounted over NFS. So you add this line to your /etc/fstab and make it mount automagically.

10.10.10.1:/vol1/fs1 /data nfs defaults 0 0

A few weeks go by and you apply a kernel update. No big deal, you apply the updates and during your next maintenance window reboot to apply the new kernel. Then you start to see applications failing and notice the volume isn’t actually mounted. This is an unfortunate result of the automounter subsystem.

It’s like this. At boot time the root partition gets mounted, automounter reads the /etc/fstab file, and boots any filesystem that doesn’t have noauto as a mount option. We’re still very early in the boot process so the network isn’t up yet, so naturally any network filesystems fail. The real problem here is that at no point does automounter go back and attempt to remount those systems. So your NFS mount points fail because there is no network, and done is done.

The developers were nice enough to provide a fix for this. There exists a mount option called _netdev. If we quote directly from the man page (sourced from RHEL 6.4):

_netdev
The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

This is awesome, and exactly what we want. So you modify your entry in fstab to look like this:

10.10.10.1:/vol1/fs1 /data nfs defaults,_netdev 0 0

You’ve been bitten by NFS mounting in the past so you throw this in your test environment and reboot immediately. After the system comes up you notice a problem. Your NFS volumes are still unmounted. You see, there’s a bit of a hitch. Automounter followed the same procedure that it did before, except this time it didn’t even attempt to mount /data. The _netdev option doesn’t tell the system to mount the filesystem when network comes up, it says don’t attempt to mount it at all if the network isn’t up. There is still a missing piece to the puzzle. If you look at your init scripts there is a service called netfs. If you read the script you can see in the chkconfig header this description:

# description: Mounts and unmounts all Network File System (NFS), \
# CIFS (Lan Manager/Windows), and NCP (NetWare) mount points.

This is exactly what you need. It is a service whose sole purpose is to read your /etc/fstab and mount network filesystems. All you have to do is enable it

chkconfig netfs on

and watch the magic happen. Now your mount boot process should look something like this:

1. Automounter reads /etc/fstab
2. Ignores /data since it has _netdev option set
3. Mounts all other filesystems
4. Finishes mount jobs and allows system to continue booting
5. Network comes up
6. Service netfs started
7. netfs reads /etc/fstab and finds an nfs filesystem
8. netfs mounts /data

What’s funny is that while I was researching this problem I never stumbled across netfs as a service. I had even gone so far as to start planning out my own custom init script that would do exactly this, except specifically for my mount points instead of generalizing. It’s nice to see that I was on the right track, but even better that the tools already existed.

I have a NTFS formatted hard drive whose partition table suddenly went AWOL. I removed the drive from the PC, attached it to a SATA-to-USB adapter and connected the drive to another PC. My intention was to use a recovery tool to either rebuild the partition table or recover all the files and transfer them to another PC.

The problem is that when I attached the drive to another Windows machine, Windows did not recognize it and wanted to initialize it. Then, when attempting to initialize the drive (all initialization does is write a few bytes worth of a signature to the MBR) I received the error “Virtual Disk Manager: The device is not ready.”

My Solution

I know you’re not going to like how simple this is, but try it anyway. Move your disk to another controller of some kind. In my case the disk was being used on a SATA-to-USB adapter. Once I switched it to a direct SATA interface, the drive was able to work. I had to use the internal SATA ports on a separate computer.

While it was connected via USB I even attempted to inspect the disk with Western Digital drive tools as well as diskpart, but it was simply not visible beyond being seen as some kind of USB attached storage device that could not be initialized. There are other reasons for this error to be seen, but one of the simpler fixes is to try the drive on a different interface of some kind.

I could reproduce the applet error in a browser, so started using tail -f to watch some of the log files that I thought were the best candidates for error and status messages. After watching eight files and seeing no errors, I began questioning my assumptions. Elsewhere there was a folder of daemon logs; 49 log files to be exact. I was not about to watch them individually, nor did I really want to take note of their size before and after re-creating the error to deduce which ones might have information being dumped to them.

The solution is obvious! Continue using tail but with multiple source files. As man tail puts it: “If more than a single file is specified, each file is preceded by a header consisting of the string “==> XXX <==” where XXX is the name of the file unless -q flag is specified.”

And the -q switch performs thusly:

-q Suppresses printing of headers when multiple files are being examined.

So to watch every log file in a directory, I simply used shell globbing:

tail -f ./*

And from there ran the web app, got the error entries that helped me troubleshoot further, and learned which log files they wrote to. From there I was able to scope my watchfulness down to just a subset of the available log files with this command:

tail -f collectd.log eventd.log link.log outage.log

It should be noted that when tailing multiple logs, the latest log file to have an entry written to it will rotate to the bottom of your terminal. In the above example of four logs, the terminal output looks like this:

==> collectd.log  eventd.log  link.log  outage.log <==

Notice nothing was written to eventd.log, link.log and outage.log. But then, minutes later, events were written to collectd.log which caused the following output to appear at the bottom-most lines of the open terminal:

 >collectd.log<=
2013-03-27 14:01:09,579 WARN  [CollectdScheduler-50 Pool-fiber1] CollectionResourceWrapper: getAttributeValue: can't find attribute called ifHighSpeed on node[197].interfaceSnmp[ncmac0-000074f767a4]
2013-03-27 14:01:09,579 WARN  [CollectdScheduler-50 Pool-fiber1] ThresholdingSet: passedThresholdFilters: can't find value of ifHighSpeed for resource node[197].interfaceSnmp[ncmac0-000074f767a4]
2013-03-27 14:01:09,579 WARN  [CollectdScheduler-50 Pool-fiber1] CollectionResourceWrapper: getAttributeValue: can't find attribute called ifHighSpeed on node[197].interfaceSnmp[ncmac0-000074f767a4]
2013-03-27 14:01:09,579 WARN  [CollectdScheduler-50 Pool-fiber1] ThresholdingSet: passedThresholdFilters: can't find value of ifHighSpeed for resource node[197].interfaceSnmp[ncmac0-000074f767a4]
2013-03-27 14:01:09,581 WARN  [CollectdScheduler-50 Pool-fiber1] CollectionResourceWrapper: getAttributeValue: can't find attribute called ifHighSpeed on node[197].interfaceSnmp[ppp0]
2013-03-27 14:01:09,581 WARN  [CollectdScheduler-50 Pool-fiber1] ThresholdingSet: passedThresholdFilters: can't find value of ifHighSpeed for resource node[197].interfaceSnmp[ppp0]
2013-03-27 14:01:09,581 WARN  [CollectdScheduler-50 Pool-fiber1] CollectionResourceWrapper: getAttributeValue: can't find attribute called ifHighSpeed on node[197].interfaceSnmp[ppp0]
2013-03-27 14:01:09,581 WARN  [CollectdScheduler-50 Pool-fiber1] ThresholdingSet: passedThresholdFilters: can't find value of ifHighSpeed for resource node[197].interfaceSnmp[ppp0]

Yes, when tailing multiple log files, it can get very, very messy if the log files are VERY active. In that case it might be best to run an individual tail command multiple times using a terminal multiplexer like screen or tmux.

Got any better ideas for watching multiple log files in real time? Let me know below.

Side note: I try to take every platform that I work with at face value and work with its native tools. I do not like using cygwin on Windows machines. I also know there’s a curl binary for Windows. Natheless, I like to look for solutions to problems using packages native to the OS I’m using, and then fan out from there as absolutely necessary. I find things to be much more stable and portable that way. Moving on…

My Scenario

In my use case, I have a client who uses a hosted wiki from a major online office provider. The only ways that you can back up the wiki to a local file is to either manually log in to a dashboard and click a URL to download the backup file, or you can script the retrieval of a HTTP URL to get the file. Of course, being a good-lazy admin, I want to script this.

My Options

There are some really ugly options that I’ll refer to briefly. One of them is to simply script the HTTP communication directly using telnet. Like,  telnet http://url ; GET /path HTTP/1.1 type stuff. Fortunately I’m not a substance abuser, and if I was I wouldn’t be able to afford the quantities of methamphetamines necessary to make that option seem rational.

Another option is to use .NET libraries directly from within PowerShell. For example: (New-Object System.Net.WebClient).DownloadFile(
"http://clientsubdomain.cloudofficewikithing.com/mybackup","c:\backupfile.zip") You know your Windows scripting efforts are hitting Saw-levels of grotesque when you’re routinely calling .NET namespaces in your scripts.

The more sane options is to use tried and true BITS, but from within PowerShell itself. However, first, one has to import the BitsTransfer module. Under normal circumstances on a recent version of Windows, it will be an available module, just not imported. To check if it’s available, run the following command:

PS C:\Users\user> Get-Module -ListAvailable
 
ModuleType Name ExportedCommands
---------- ---- ----------------
Manifest BitsTransfer {}
Manifest PSDiagnostics {}

If you see the BitsTransfer module, you can import it thusly:

Import-Module bitstransfer

To see the commands available for bits, check out:

PS C:\Users\user> get-command *bits* | Where-Object {$_.commandtype -eq "cmdlet"}
 
CommandType     Name                                                Definition
-----------     ----                                                ----------
Cmdlet          Add-BitsFile                                        Add-BitsFile [-BitsJob]  [-Source] <S...
Cmdlet          Complete-BitsTransfer                               Complete-BitsTransfer [-BitsJob]  [-V...
Cmdlet          Get-BitsTransfer                                    Get-BitsTransfer [[-Name] ] [-AllUsers...
Cmdlet          Remove-BitsTransfer                                 Remove-BitsTransfer [-BitsJob]  [-Ver...
Cmdlet          Resume-BitsTransfer                                 Resume-BitsTransfer [-BitsJob]  [-Asy...
Cmdlet          Set-BitsTransfer                                    Set-BitsTransfer [-BitsJob]  [-Displa...
Cmdlet          Start-BitsTransfer                                  Start-BitsTransfer [-Source]  [[-Desti...
Cmdlet          Suspend-BitsTransfer                                Suspend-BitsTransfer [-BitsJob]  [-Ve...

The command in question that can do our file retrieval bidding is start-bitstransfer. In my case, it was as simple as this:

Start-BitsTransfer -Source http://clientsubdomain.cloudofficewikithing.com/mybackup -Destination C:\backups\

I was able to schedule that to run at the right times and then get my automated backup file downloaded each night.

I will not feign that start-bitstransfer or the rest of the BITS commands can suffice for all scenarios that wget or curl would eat for lunch. However at least consider those PowerShell command before you go about shimming other tools into a Windows install. Have any better methods for retrieving URLs at the command line in Windows? Let me know below.