I Hope Solution

Book Review: Time Management for Systems Administrators

2009-11-06T14:44:03Z

Recently I took on a new position at work. Instead of just banging out help desk tickets and incoming phone calls, I’m assisting customers on longer-term projects. It’s an entirely different workflow, and it’s helpimg me appreciate this book even more. I’m glad I procrastinated writing this post, as it helped clarify my thoughts to simmer with the information contained within. I can confidently give this book a rating of

Multiple Thumbs Up

although it’s not without it’s flaws.

Pros:

This book is what it says it is on the tin: a time-management system specifically geared to the workflows of systems administrators.

It’s a shorter book, yet covers the topic more than adequately.

Both the tips and the overall system are chock-full of utility.

Cons:

The folksy style is off-putting, for myself at least. Geeks drink Mountain Dew! And it’s funny! *rolls eyes*

This book can be repetitive. I recognize that repetition is a hallmark of pedagogy, but the eighth time I heard the filling-the-gas-tank example I very nearly endangered the material well-being of my copy. (But now I know and use the point of that example often. So, it worked while it annoyed.)

The last con I see is more subtle; you might consider it a con as such.

Books in the time management genre (e.g. Getting Things Done
) tend to espouse a somewhat-eastern zen-like worldview (taking the “be a systems administrator NINJA!” metaphor too far, perhaps?). This book, though milder in that respect, does have some (pro-yoga) moments. (I can’t stand yoga, in part because I am an inflexible jerk.) For me, that’s a con; it may not be a con for you.But I am the one writing this review, so I call it a con. And not the kind of con that prevented me from enjoying and recommending this book.

Disclaimer:

Dear FTC: I bought this book. Nobody is compensating me for this review. If I’m lucky, I’ll be able to get a latte with the proceeds from the amazon affiliate links I’ve placed here. The fact that I felt the need to mention any of this makes you jerks.

Emacs vs. VI Keybindings

2009-08-31T13:26:37Z

Chances are, if you’ve spent any significant amount of time with POSIX-compliant systems, you’ve made an effort to learn either emacs or vi. (Unless crontab -e shunts you to nano, and you’ve not moved on.) I suspect that those who stick with emacs are generally programmers, while those who stick with vi are generally sysadmins (although I do recognize that plenty of programmers spend their days in vi – this is a hasty generalization made for the sake of argument). Since both editors are so different in approach, it makes sense to get deeply involved in one to the exclusion to the other. Being proficient in one is eventually more productive then only half-knowing both.

That said, I suggest learning both sets of keybindings.

The reason I suggest this is because many other programs implement either emacs or vi keybindings. By default, bash uses some emacs keybindings for moving around the command-line, although it can be configured to us vi-like bindings too. The aforementioned frankeneditor nano also has emacs-like bindings. Pagers such as more, less, and most utilize vi keybindings. As a systems administrator, at some point you’re going to find yourself on an unfamiliar box, with a less than familiar tool. I guarantee that pulling the old “I wonder if this tool uses vi commands for navigation” trick will result in a percentage increase in productivity that, while completely cooked up off the top of my head, is nevertheless real and measurable.

So. If you’ve slavishly commited yourself to one editor and one set of keybindings to the exclusion of the other: Be Ye Not So Stupid.

Paid per email and why it won’t work

2009-08-22T17:55:33Z

A while back, people in IT kicked the tires on paying per email sent. Nothing ever came of it but the idea is coming back in the form of Yahoo sponsored CentMail. I initially heard of this thanks to a well written article by MacGregor Campbell over at the New Scientist. According to the article Yahoo is experimenting with Pay per email system, but with the cost of an email be only one US cent ($.01US) and that money going the sender’s charity of choice. The eliminates the original complaint of Pay per Email; different ISPs charging different rates. It also gets more money into charities which is always a good idea.

Problem is, it won’t work.

Simple Mail Transport Protocol, or SMTP, is the mechanism by which is email is sent. It’s simple and it works. There really isn’t a lot to it. You send a plain text stream that contains your message header and the email body to your domain’s SMTP server and it sends it out to the recipient’s mail server. I’m simplifying it a bit, but not by much. And due to its relatively straightforward design, SMTP is subject to abuse. We’ve all gotten the viagra and porn spam in our inbox. And there are methods to deal with spam like Domain Keys, SPF records, RBLs and server side spam filter software. And that all works, but it adds unnecessary levels of complexity to the email process. The CentMail idea will add just another layer to that.

I don’t work with that aspect of email much anymore, but my coworkers who work the phones do. Each of them is helping one of our customers get out of email purgatory. A customer will send some email that one of the ISP doesn’t like and it’ll get put on the ISPs blacklist. The customer then calls us and one of the guys in Support has to fight with the ISP to get the customer’s Domain removed from the blacklist. The problems lay in that there is no standard on what classifies spam and how to handle it. Each email provider, be it an ISP like Charter or Comcast, or a webmail provider like Google and Yahoo have different requirements and have different levels of pain involved with getting a domain whitelisted.

And that’s not including the RBLs, or Remote Blacklists. These RBLs are another level of spam prevention and another level of complexity. They work by keeping a list of spam related IP address and Domains. Your mail client checks each incoming email against each RBL and drops mail that is on the list. Getting off the RBLs can be even more painful, ranging from paying a fee to clearing out an entire C-block of IPs.

The CentMail website, which is a horrible pun by the way, states “Anyone you email can automatically verify your donation and confirm you’re not a spammer. Since spammers send millions of emails every day, it is prohibitively expensive for them to donate even just 1¢ per email.” That won’t stop the spammers. I’m guessing that Yahoo is assuming that there will be checks on both ends of the email sending process. But unless they intentionally modifiy SMTP to be non standard compliant, all a spammer has to do is use an MTA that doesn’t check for the Cyberstamp to get their mail out.

From the process described on the New Scientist article, it sounds like there will be an addition to the email’s header that will contain the Cyberstamp signature, and a link will be embedded that will allow the sender to verify that your did donate that penny to the Humane Society. That’s great until the Spammers figure out a way to spoof the headers and make that link redirect the user to a phish site. There are enough people out there using unpatched versions of Windows XP that this would be presenting another vector for hackers to compromise new systems.

On that same vein, what is to prevent a receiver from filtering email that contains a cyberstamp to a charity that they find objectionable? Lets say I send you an email and I donate that penny to AIDS research or Planned Parenthood. Now lets say you work for a conservative company that finds both of objectionable. What is stopping them from just dropping those emails? This adds another layer of potential problems for CentMail.

And what’s keeping companies from creating their own systems or not even adopting it? This system would only work if everyone, and I mean everyone was on board with it. And the disturbing trend of ISPs hijacking DNS indicates that the ISPsa at least have no desire for adopting a standard that someone else created.

All in all, I don’t see Cyberstamps stopping the spam problem with the way email is currently setup. The RBLs and the spam filters work well. I’ve gotten maybe on 1 spam in the past 3 months on one of my email accounts, which is a lot less then what i get in my physical mailbox. In order for Centmail to work, you’d need a major upheval in the status quo and I don’t see it happening. Cyberstamps version 2 is doomed to fail even before its released.

Accessing remote file systems with sshfs

2009-08-11T05:44:50Z

Like anyone has data stored accross multiple servers, keeping it organized can be a headache. At my house, I’m using a server running Samba to store my data for my laptop and desktop. That way, when I edit file X, I know it’ll be the same on either system.

Samba works great at home. However, opening a Samba share to entire world is a Bad Idea™.I wanted to be able to access some of my files off of my server at work that was secure and didn’t require setting up too much extra software. And this is where sshfs comes into play. Sshfs is a Filesystem client based off of the SSH Protocol. I picked sshfs over something like webdav because the server in question doesn’t have Apache installed and all of the servers I work on run ssh.

Setting up and using sshfs is pretty straight forward. First, install sshfs on the client. That’s one of the best part of sshfs; as long as the server has ssh running, you don’t need have to do anything on the server. All of my boxes are running a varient of Debian, so I just used aptitude to install sshfs.

sudo aptitude install sshfs

I have not tried installing this on other linux distros, but I’m going to guess that using their package managers will give you similiar results. If you want to do things the hard way, source can be downloaded here. You’ll need FUSE2.2 and the latest version of glibc devel installed also. Your package manager of choice should take care of the dependencies for you.

Once you’ve got the software installed, you’ll need to add your normal user to the fuse group so they can mount and unmount the sshfs shares. You can use your GUI of choice, of the following CLI command:

sudo adduser user fuse

That’s it! That’s all you need to do to setup sshfs. Like with any file system, make a directory to mount the system to and then run the following command:

sshfs remoteuser@remotesystem:/path/to/dir localdir

You’ll now be able to access that directory remotely using sshfs. When running that above command, you may get the following error:

fusermount: failed to open /dev/fuse: Permission denied

Try running the command in a fresh shell. Chances are the current shell hasn’t picked up on the refreshed permissions. To unmount the sshfs directory, run the following command:

fusermount -u mountpoint

Like any filesystem, sshfs shares can be automagically mounted via the server’s fstab. Please note that if you do want to have the sshfs automount due to fstab, you will need to have SSH keys setup between the client and the server. Below is the syntax you’d want to use:

sshfs#USERNAME@REMOTE_HOST:REMOTE_PATH MOUNT_POINT fuse SSHFS_OPTIONS 0 0

It’s a pretty straightforward solution to a problem that had been bugging me for a while. I am able to access my remote files on a server with the ease of a samba share without having to install any extra software on the server itself. To me, something like sshfs is more convenient than webdav or sftp, because it allows the remote system to integrate seamlessly into my current system. I hope you find sshfs as useful as I have.

Naming Variables Sanely

2009-08-01T21:49:46Z

I had a simple goal. Well, someone else had a simple goal, but it was my task to fulfill it: raise the maximum number of files that could be listed in a directory in an ftp client. Pure FTP by default limits this to 2000, a reasonable limit in my opinion. But no, them what pays the bills needed it higher.

It took me fifteen minutes to find the variable that needs to be set in Pure FTP’s config file (and not because of a deficiency in my google-fu):

LimitRecursion

Oh, the things that I googled! The terms that I grepped! Things like “max” and “num” and “files”. I finally stumbled upon “limit” and found the bugger. Yet fifteen minutes! Far too much time for such a simple task.

Naming that variable LimitRecursion makes sense when you’re programming. You’re writing an ftp server, and suddenly you realize “hey, this directory with a crapton of files is nomming my resources, thanks to recursion. Let’s limit the amount of recursion that can occur and keep the server happy in the process. Let’s see, I’m trying to limit recursion… hmm… what should I name my variable…”

But when you answer LimitRecursion you are wrong sir. Because this program will be used by people who don’t give a fig how you keep the server up (limiting recursion) but do give a fig about the maximum number of files they can display.

When you name a variable, or create an API, please, for the love of Pete and end users and sysadmins everywhere, NAME IT LIKE YOU’RE AN END USER. By doing so, you GREATLY lessen your chances of being injured by an end user with a voodoo doll.

Disabling Update Notifier Pop-up in Ubuntu 9.04

2009-07-30T08:52:21Z

It seems that some ad-wizard at Canonical decided that the tray icon that notifies you of available updates in Ubuntu was simply too well liked. Thus, in 9.04, they decided to make the update manager automatically pop open any time there are updates found no matter what you are doing. Luckily, for those of us that actually enjoy an unobtrusive user interface, restoring the update manager to it’s original tray icon glory is a single command away. Simply run the following and your web 1.0 auto pop-up annoyances with Update Notifier will decrease a full 100%!

gconftool -s --type bool /apps/update-notifier/auto_launch false

To make the change take effect you will need to restart the Update notifier, or log out and log back in.

Backing up postgreSQL DBs with bash and cron

2009-07-21T15:59:34Z

I’ve been working with postgres in my free time for a while now. I have been playing with fire, so to speak. I haven’t had any sort of backup solution on my postgres server. So after a bit of googling and reading postgres docs, I’ve came up with a very basic backup script.

What it does is backup each database separately and zips them up in order to save on space. It puts an hour – minute stamp on the database file and backs the DB up to a directory that is the current date. I’m going to add more to the script later, like only keeping so many backups, etc. But at this point, I just wanted to get a script running that creates backups just in case the server’s main drive dies or I botch something.

So without further ado, here’s the code of the script. I do have to give credit to the author of this post, as I heavily modified that script to do what I needed it to do. The big thing was the sed-awk line, as my sed-fu is quite poor.

#! /bin/bash
# PGSQL backup script

# logfile loc
logfile="/var/log/pgbackup.log"
# backup loc
backup_dir="/backup/pg_backups/"
touch $logfile
timeslot=`date +%H-%M`
monthday=`date +%F`

# mount /backup, create the dir and get a list of dbs
/bin/mount /backup
/bin/mkdir $backup_dir/$monthday
databases=`psql -U postgres -q -c "\l" | sed -n 4,/\eof/p | grep -v rows\) | awk {'print $1'}`

#backup the dbs
for i in $databases; do
        timeinfo=`date '+%T %x'`
        echo "Backup complete at $timeinfo for time slot $timeslot on database: $i " >> $logfile
	/usr/bin/pg_dump -U postgres $i | gzip > "$backup_dir/$monthday/postgre-$i-$timeslot-db.gz"
done

#unmount the backup drive
/bin/umount /backup

And that’s that. It’s a pretty simple script, but it gets the job done. I’m obviously running the script on the machine that postgres is on and the server I’m running it on is Ubuntu LTS with a backup drive in the server, so your milage may vary depending on your distro and your setup. You may want to test the script first before setting it as a cron on your DB server. Speaking of the cron, here’s the line you’ll want to set in your root crontab.

00 06 * * * /root/bin/pgbackup.sh > /dev/null 2>&1

I have the cron set to run at 6am since I’m asleep at 6am, so no one will be accessing the DBs. And now my DBs are safe from me and my postgres nubness.

The importance of documentation

2009-07-16T17:43:08Z

A few jobs ago I was a Work Study in college, and I was the head of IT’s assistant. He was in charge of the PBX phone system along with the network and workstations.

We had to do some desk juggling and rather than change assignments in the PBX we decided to just move the cables that connected the patch panel from the desk sto the PBX. Easy right? Nope. My boss’s predecessor when he set up the PBX system, along with all of the other patch panels to labs and offices didn’t document anything. He also used 15 foot cables when a 1 foot cable would have worked, but that’s a rant for another day.

My boss and I spent 20 hours over the course of a weekend documenting which ports on the patch panel matched which port in which office. 40 man-hours wasted because my boss’s predescor didn’t take a few minutes to document as he set everything up.

This kind of behavior is hard for me to wrap my head around. And my PBX example above isn’t an isolated experience. It seems that non-documentation like this occurs in IT departments across the country. It appears to be a mixture of pure laziness this and a “If I know everything, I’m irreplaceable” mentality.

SPOILERS: Everyone is replaceable.

The replacement may not be better, but everyone is replaceable. Case in point, Michael Jordan, the greatest basketball player of all time, retired from the Bulls twice. After both times, the Bulls had to find another guy to start at Shooting Guard. Were either of them as good as Jordan? No, not at all, but they were his replacement.

My advice is this: You’re not going to be at your current job forever, so do yourself and everyone else a favor and document everything. I use a mediawiki install for personal stuff, but even if its just a collection of Word docs in a network share, find something that works for you and run with it.

Deleting a Crapton of Small Files from a Directory Using Find

2009-07-13T12:16:54Z

One of the fun situations you find yourself in when administrating people’s email (at least, I find myself in this situation frequently enough, and I am an uninteresting person, so this is what passes for a fun situation in my world) is the dreaded I-have-magically-hit-my-email-quota problem. The conversation usually runs something like this:

User: HALP I cannot send any email on this address!

Me: checks logs The mail logs are telling me that it can’t write to file due to quota issues. Looks like you’ve hit your $BIGNUM quota.

User: I don’t know why the system quota is saying I have $BIGNUM of mail when $MAILCLIENT informs me I have $MUCHSMALLERNUM of mail.

Me: cd’s into user’s maildir directory, starts running ll -ah like a boss

User: I mean, holy crap, I friggin’ hate email, why does the world have to hate me, I have $VERYIMPORTANTATTACHMENT to send to $BIGCLIENT and then this happens. Why don’t computers run on happiness and unicorn farts?

Me: Ah, it looks like this email address has $BIGNUM – $MUCHSMALLERNUM worth of mail in its .spam directory.

User: I have a dot spam directory? How come I can’t see it in $MAILCLIENT?

Me: You know, I could explain the concept of how hidden directories work in linux, and bemoan the fact that the mailserver uses them and most mail clients don’t, but in lieu of that, how about I delete all of that spam for you, thereby fixing your quota issue?

User: in a monotone Thank you sir may I have another

Here is the bash one-liner for removing the mails from this directory:

for file in $(find -type f -maxdepth 1); do rm -fv $file; done

“But sir! Sir!” you find yourself saying, “why on earth are you not using xargs? Surely piping find’s output to xargs is more memory-efficient than a naive for loop!”

While it is true that a for loop is less memory efficient than piping find to xargs, I always use it in this situation. The reason is simple; if a directory is $BIGNUM large, and most of the files in it are measured in tens of kilobytes, your find command will more often than not send too many arguments xarg’s way. Your time is too precious a commodity to see whether or not this is one of the situations when the stars align and the mail quota is small enough that the directory doesn’t have too many files for xargs. Just use the for loop. If the server is reasonably fast, it will handle it gracefully.

Every once in a while you’ll run into an enlightened user who realizes, just as you’re about to go and mow through the .spam directory, that hey, there was an important email last week that somehow never got through. They’d like to be able to poke at the last week’s .spam emails. Which means you can’t just delete all the mails willy-nilly. This is why God invented the -mtime flag.

for file in $(find -type f -maxdepth 1 -mtime +7); do rm -fv $file; done

That -mtime +7 causes find to snag only the files that haven’t been modified in the last seven days.

I really can’t get over how useful a little command like find can be. I mean, I’ve tried to get over it. Brought it into therapy a couple of times. Still can’t shake how useful it is. I usually refer to this find tutorial when I trip over the syntax.

One final tip: before you start deleting data with reckless abandon, I highly recommend substituting “do echo $file;” for "do rm -fv $file;" in the above one-liners. It never hurts to make sure that you’re actually grabbing the files you think you are before you delete them.

Recompiling Apache to use mpm_worker on a CPanel Server

2009-07-15T18:27:50Z

Arguably, the most important service that runs on a server is its webserver, and for most of the world this webserver is Apache. One of Apache’s main selling points, its modularity, is also its biggest drawback if not configured properly. Apache 2.0 and newer can be compiled with Multi Process Modules, or MPMs. The default MPM is prefork. Prefork is based off of the aging Apache 1.3 and works fine, but does have its disadvantages: it doesn’t handle large amounts of traffic gracefully nor is it really designed to handle newer hardware.

This is where mpm_worker comes in. Worker is a threaded MPM that is designed for newer hardware and handles larger amounts of traffic better. While prefork is the Apache default MPM, Debian based distrubutions run worker by default. Information regarding worker is available at the Apache Documentation.

“But,” you’re asking yourself, “I run a CPanel dedicated server and/or virtual private server at my webhost. Can I use worker?” Absolutely. CPanel provides the aptly named EasyApache script that allows for recompiles of Apache and PHP using an easy to navigate interface.

Before I get into the nitty gritty, a few caveats:

If you’re in a shared hosting or reseller environment using CPanel, you’re probably out of luck. Most hosting providers do not allow for software changes in such situations.
The Webhost Manager does have a web interface for EasyApache. I wouldn’t use it, because if your browser crashes during the EA, it’ll Break Everything™.
Unless you are running a module that requires non threaded Apache, upgrading to worker will not cause any problems. Upgrading to Worker will not affect your PHP code, or your databases or your Google Analytics.

First thing, You’ll want to check to see what version of Apache you’re running, because if you’re running worker already then you’re already set. You can figure out the version of Apache by running the following command:

/usr/local/apache/bin/httpd -V

Which will give output like the following:

The line you’re looking for is the “APACHE_MPM_DIR”. As the screen shot shows, this server has Apache running with prefork, so it needs an upgrade. You launch the EasyApache script with the following command.

/scripts/easyapache

If your server is in a virtual environment using Xen, the EasyApache script will require a flag when it is executed. This flag will be provided when the script is ran. The technical reason is because the Xen console doesn’t support curses, which the EasyApache script uses.

A warning, I highly recommend running the EasyApache in a screen. As stated earlier, stopping an EA while its doing its thing will Break Everything™.

You’ll want to Customize based off of current profile and make sure that your compile Apache 2.0 or 2.2. Unless you’re upgrading PHP also, the next couple of windows can be skipped. On the Short Options List, pick the Exhaustive Options List. Once you’re in the exhaustive list, scroll until you hit the MPMs. It’ll look like this.

Unselect any selected MPMs and select Worker. More often then not, no MPM will be selected, which is fine because Prefork will be installed if nothing will be selected. The screen shot is with Apache 2.2 select, this screen will be different if you’re compiling Apache 2.0.

Once you select Worker, the following screen will pop up:

Select OK and hit enter. The warning is there because the non prefork or worker MPMs are experimental will probably not work most of the time. Worker is completely stable, so its safe to ignore this warning.

After that warning, navigate to the end and click save and build. With that, you have to wait for EasyApache to do its thing. This normally takes about 30 minutes, but can take longer depending on server load and how many modules are selected in EasyApache.

WARNING: Do not close your shell window until the EasyApache is done, or it will break things in interesting and fun ways. You DID remember to run this in a screen, right?

Once it’s done, you’ll want to check that it finished correctly. I usually do this by restarting Apache. Generally, if the EasyApache failed, Apache will error on restart. After Apache restarts, you’ll want to check to make sure that Worker was installed, running the version check command from earlier. If everything went according to plan, you should get something like this:

If you’ve got something like that, then everything worked correctly. Now we’ll want to check to make sure Apache has its optimization directives in place. Sometimes EasyApache includes these directives, other times it doesn’t and I’m not really sure why. You’ll want to search for a block of text towards the top of the config at /usr/local/apache/conf/httpd.conf like the following. If its not there, add it. This block of text will be before the vhosts.

Timeout 300
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5
<IfModule worker.c>
ServerLimit 16
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 500
</IfModule>

Once you’ve added these directives, you’ll want to restart Apache so they can take effect. These directives will probably need to be scaled depending on your server’s traffic load and how much resources the server has, but they do work well as a starting point.

And you’re done! Apache is now running the mpm_worker module. If your server has a high traffic load, the difference should be immediate. Slower traffic servers may not notice it until later.