linfactory, a dmourati Production

7 Thunderbird Extensions will allow you to replace Outlook

2008-02-07T11:11:00.000-08:00

Suseblog.com contained an interesting article called 7 Thunderbird Extensions will allow you to replace Outlook. Outlook is one of the worst pieces of junk I have ever had to use. These tips really help. Slowly but surely, I'm freeing myself from the shackles of Microsoft in the workplace.

For a way out of using Microsoft's Visio, check out Gliffy.

Upgrading to RAID 1

2008-02-05T00:43:00.000-08:00

Linuxconfig put together a great document on how to upgrade an existing system from a non-RAID setup to software RAID 1.

http://www.linuxconfig.org/Linux_Software_Raid_1_Setup

Why Sysadmins Suck at Documentation

2008-01-30T21:33:00.000-08:00

I just read an interesting post titled System Administration and Documentation at The Branch. Allen writes that documentation is missing from the standard identify, troubleshoot, resolve methodology applied by most sysadmins.

Sysadmins suck at documentation. We can't be bothered. We think documentation is beneath us. We are the gatekeepers of information.

Want your application in the DMZ to be able to talk to the backend? Better go see your sysadmin (or netadmin) because without them you are dead in the water.

"Open a ticket!," we bark.

Not writing documentation gives us power. Power to hold over our users and our colleagues. Job security. Power to be needed.

Without us, the workplace grinds to a halt. Need a new user account?

"File a ticket, our sysadmin is out today. He will get to it when he gets back from the conference."

Hours go by, work piles up. The ticket queue is now full with requests from the past days office absence.

"Ha, all those users are hopeless without me. Let's see who I've kept waiting while I was enjoying my precious PTO."

"Hmm, Fran needs a new printer...low priority, she can wait. James can't get his application to talk though the firewall, what a loser. Aha, here's one, install this new Wiki package for inline editting. That sounds like fun, off I go working on that. Where's that coffee?"

Meanwhile, Fran and James are stuck. They are patient people, but they filed their issues yesterday, while we were out. By lunchtime, they are wondering what is up. If they are feeling up to it, they may swing by your office to "get an update."

"Update, you shout, ya, I'm working on it."

"Okay, do you have any ETA, I'm kinda stuck without that printer," Fran says.

"I'll get to it."

Meanwhile, more work piles in. You're off in Wiki-land reading *gasp* documentation.

"How do I install this plugin. What version am I running, let's see. Gosh, these developers suck, why couldn't they make this easier to follow..."

By funneling all this work through us, we have succeeded in creating a log jam. The work we find interesting is often time-consuming and often not time-critical. This makes for a bad situation.

The users suffer. They are stuck because we have taken away their ability to be productive. Fran still needs that printer, James still needs connectivity.

If any of the above scenarios sound familiar it is because they are true. This is how things go in a real office. What can we learn from this pattern? What is the way out? How can a guy take a day off without grinding things to a halt?

Documentation.

Organizations don't get very far with just one admin. They usually add help over time to meet the growing needs. This could be for IT, networking, security, systems administration, database administration or any related job description. The point is, we are part of a larger team.

So, how do we leverage this team to make our lives easier.

Documentation.

Let's go back to the Wiki example and forget about Fran and James. We'll come back to them.

So, the trusty sysadmin has the time now to figure out all about the fancy Wiki upgrade he wants to make. Download this plugin, check. Note the version number, check, unzip the file, check. Go to the page and try the new feature. Hey, it works, cool done, right?

Wrong.

To be truly done you have to move the ball forward.

What does that mean?

You have to make sure that the same task can be performed easier the next time.

How do we do that?

You guessed it, documentation.

The irony was we were just mucking around in a Wiki. What a perfect place to write down the steps on how we did the upgrade.

It can start out simple. Just write down the steps. This is a great start and will provide 80% of what you need. (See the 80/20 rule for more info).

But, how can we get it to 100%? The answer is with more care. We have to write down not only the what but the why. Why are we installing this new feature? Who needs it? What are its benefits.

The how is usually easier. We can link to the Wiki where we downloaded the plugin. We can copy/paste the instructions we used to unzip the file and the steps we took during installation. We can link to our new page showing the newly installed functionality.

What are the benefits of all this work?

Allen touches on it when he talks about "less experienced, and thus cheaper labor" but misses the key word:

Delegation.

Documentation enables delegation.

Solving a problem the first time is always the hardest. If we take the time to write down what we've done, we create power through delegation.

Now, back to Fran and her printer.

Hers is not the first printer set up in this office is it?

Of course not.

Rewind back to day 1 when we setup the office. What did we do to setup the printers?

Well, something like this:

Pick a model
Give it a name
Give it an IP address
Set it up in the domain
Add a share
Install some drivers
Test Print

If we had written those steps down the first time around, we got 80% of what we need to delegate that task to the new IT guy or other team member. Heck, even the DBA could setup a printer if given the right documentation.

Over time, we fill in that 80% to get it to 100%. Edit, link, re-write, clarify. It will get there. So the DBA couldn't quite get the printer setup correctly following your instructions? You missed something. "Edit the wiki, you tell him." That grows the documentation and makes it that much easer for the next printer, and the next one.

In the end game, Fran may be the one setting up her printer herself. How cool is that? Self service.

Same thing goes for James and his firewall issue. This was not the first firewall change made in the company history. No, someone in some back room somewhere banged out many a cryptic invocation to make that firewall do their bidding for their own needs. It has happened many times.

Is there documentation on that?

Probably not.

"Security considerations!," your firewall guy will chime. I don't trust anyone else making these changes so I'm not writing anything down for you guys. This is simplemindedness. Allen also brings up my favorite scenario, "getting hit by the proverbial bus." So, what happens what that indispensable firewall guy doesn't show up to work tomorrow?

With a little bit of coaching, the IT guy, DBA, sysadmin, or whoever else can make that change and James can be on his way.

Let's tackle self-service on the firewall another day though.

Introducing Gliffy

2008-01-30T11:50:00.000-08:00

Gliffy

Dovecot

2008-01-17T16:25:00.000-08:00

Dovecot is an IMAP server for Linux/UNIX-like systems, written with security
primarily in mind. It also contains a small POP3 server. It supports mail
in either of maildir or mbox formats.

Lets install it:

[root@vm01-lin-01 ~]# yum -y install dovecot

And start the service:

[root@vm01-lin-01 ~]# /etc/init.d/dovecot start
Starting Dovecot Imap: [ OK ]

Cool. All set.

P2V Migration

2008-01-17T00:45:00.000-08:00

Today I completed work for a P2V (Physical to Virtual) migration. The physical box was a Dell running FC6. The virtual machine was also running FC6 but under VMWare running under windows.

The steps I took were as follows.

Inventory physical box

Accounts
Scripts
Services
RPMs

Create Virtual Machine

Boot VM
Insert CD-ROM
Install Vanilla OS
Patch

Migrate

RPMs
Home directory
Services

Test

Verify servies
Check filesystem

In all, it was a pretty complicated move. Luckily, I've done this before and knew what I was up against.

Disabling Snapshots on a NetApp Filer

2007-12-18T11:04:00.000-08:00

Network appliance makes a series of devices called NetApp filers. These are network-attached storage devices used to share a filesystem across the network.

These filers support a feature known as snapshot. A snapshot is a point-in-time picture of the disk. The snapshot can be used as a backup or for any other copy and keep procedure.

In some cases, these snapshots can be undesirable. To disable them, log on to filerview.

Go to the:

Volumes->Snapshots->Manage

menu.

Select the checkboxes next to the snapshots you want to delete.

Then click delete.

A dialog box asking you to confirm comes up.

Confirm the deletion.

You are done.

How to Configure IPMI on a Dell PowerEdge running Red Hat Enterprise Linux

2007-12-14T00:32:00.000-08:00

http://dmourati.blogspot.com/2007/12/how-to-configure-ipmi-on-dell-poweredge.html

Working with Cron

2007-12-12T13:54:00.001-08:00

In today's installment dmourati will show you how to setup cron to do your bidding. With a little advanced planning, you can have cron run jobs for you while you sleep freeing you up to do other, more interesting work.
cron

Cron is a daemon to execute scheduled commands. This can be setup on a system-wide basis, or for individual user accounts. Today we will setup a system level cron.

Configuring a Cron Job

To set up a system cron job, make a new file in /etc/cron.d. This directory is used to store cron entries. Here's an example:

[root@lncdw2 cron.d]# cat ds-log-syncer
30 00 * * * root /usr/local/sbin/ds-log-syncer.sh > /dev/null 2>&1

This entry starts off with the standard cron format:

minute hour day month day-of-week user script

A "*" above just means every.

So, this can be read as follows

At 30 minutes past midnight, every day, the root user should run the following script /usr/local/sbin/ds-log-syncer.sh and redirect the output to /dev/null (throw it away).

Script

Here's the shell script that the cron entry refers to:

#!/bin/bash
#Streaming logs:
#From DS:/var/log/streaming/* to
#lncdw2:/var/log/streaming/
#Httpd logs:
#From DS:/var/log/httpd/* to lncdw2:/var/log/httpd/

#Setup directories
mkdir -p /var/log/streaming/Three/threeprodds1/ /var/log/streaming/Three/threeprodds2/ /var/log/streaming/Three/threeprodds3/ /var/log/streaming/Three/threeprodds4/ /var/log/streaming/Three/threeprodds5/
mkdir -p /var/log/httpd/Three/threeprodds1/ /var/log/httpd/Three/threeprodds2/ /var/log/httpd/Three/threeprodds3/ /var/log/httpd/Three/threeprodds4/ /var/log/httpd/Three/threeprodds5/

#Get Three streaming logs
echo "Synchronizing Three streaming logs" | logger -s -t ds-log-syncer.sh
rsync -avze ssh rhythm@threeprodds1:/var/log/streaming/ /var/log/streaming/Three/threeprodds1/&
rsync -avze ssh rhythm@threeprodds2:/var/log/streaming/ /var/log/streaming/Three/threeprodds2/&
rsync -avze ssh rhythm@threeprodds3:/var/log/streaming/ /var/log/streaming/Three/threeprodds3/&
rsync -avze ssh rhythm@threeprodds4:/var/log/streaming/ /var/log/streaming/Three/threeprodds4/&
rsync -avze ssh rhythm@threeprodds5:/var/log/streaming/ /var/log/streaming/Three/threeprodds5/&

#Get Three httpd logs
echo "Synchronizing Three streaming logs" | logger -s -t ds-log-syncer.sh
rsync -avze ssh rhythm@threeprodds1:/var/log/httpd/ /var/log/httpd/Three/threeprodds1/&
rsync -avze ssh rhythm@threeprodds2:/var/log/httpd/ /var/log/httpd/Three/threeprodds2/&
rsync -avze ssh rhythm@threeprodds3:/var/log/httpd/ /var/log/httpd/Three/threeprodds3/&
rsync -avze ssh rhythm@threeprodds4:/var/log/httpd/ /var/log/httpd/Three/threeprodds4/&
rsync -avze ssh rhythm@threeprodds5:/var/log/httpd/ /var/log/httpd/Three/threeprodds5/&

As you can see, this script is used to synchronize log files down to this local system. Some "logger" statements have been added to generate useful log messages on this host as well.

Running It

The cool thing is that once you set this up, there is nothing left to do. Sit back, relax, and enjoy synchronized logs.

For more information on rsync over SSH as used in the script, stick around linfactory.

Monitoring a File's Age with Nagios

2007-12-11T14:45:00.000-08:00

Sometimes when running Linux you need to know how long a file has been sticking around. One great example is the presence and age of a lock file. In this case, I'm running a repository syncing tool called cobbler and I want to make sure it is running successfully

Nagios

Nagios is an open source monitoring software package. I've been using Nagios since its earlier name Netsaint was still in effect. Nagios will be the subject of another post here on linfactory so stick around.

Plugins

Plugins do the heavy lifting for Nagios. These plugins are small pieces of code that do something useful like check a ping timeout or in this case, a file's existence and age.

Check_file_age

I found a stock plugin called check_file_age. Here's how it works:

-sh-3.00$ /usr/lib/nagios/plugins/check_file_age --help
check_file_age (nagios-plugins 1.4.2) $Id: check_file_age.pl,v 1.2 2003/10/21 15:56:35 tonvoon Exp
The nagios plugins come with ABSOLUTELY NO WARRANTY. You may redistribute
copies of the plugins under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
Copyright (c) 2003 Steven Grimm

Usage:
check_file_age [-w ] [-c ] [-W ] [-C ] -f
check_file_age [-h | --help]
check_file_age [-V | --version]

File must be no more than this many seconds old
File must be at least this many bytes long

Send email to nagios-users@lists.sourceforge.net if you have questions
regarding use of this software. To submit patches or suggest improvements,
send email to nagiosplug-devel@lists.sourceforge.net.
Please include version information with all correspondence (when possible,
use output from the --version option of the plugin itself).

Test Invocation

[nagios@lnciserver1 ~]$ /usr/lib/nagios/plugins/check_file_age -f /var/lib/cobbler/lock -w 3600 -c 7200
/var/lib/cobbler/lock: File not found

So, the check_file_age plugin tells us that this file is missing.

Let's look at the return code.

[nagios@lnciserver1 ~]$ echo $?
3

A non-zero return code indicates a problem. In this case the 3 corresponds to "UNKOWN." Nagios doesn't know what to make of this situation. I need to tweak this plugin. No problem, this is open source after all. I need to allow for the absence of the file as this is okay in my situation. That would just indicate the cobbler has cleaned up successfully and deleted its lock file.

Modification

For the cobbler case above, we need this situation to retun a 0.

Here's the change.

-sh-3.00$ diff -c /usr/lib/nagios/plugins/check_file_age /usr/lib/nagios/plugins/check_file_age2
*** /usr/lib/nagios/plugins/check_file_age 2006-03-06 23:57:22.000000000 +0000
--- /usr/lib/nagios/plugins/check_file_age2 2007-10-19 23:26:24.000000000 +0000
***************
*** 74,80 ****
# Examine the file.
unless (-f $opt_f) {
print "$opt_f: File not found\n";
! exit $ERRORS{'UNKNOWN'};
}

$st = File::stat::stat($opt_f);
--- 74,80 ----
# Examine the file.
unless (-f $opt_f) {
print "$opt_f: File not found\n";
! exit $ERRORS{'OK'};
}

$st = File::stat::stat($opt_f);

Invoking the Plugin

Now that we have a modified plugin in place, we'll run it to see what is up with our cobbler server.

Invocation Take Two

[nagios@lnciserver1 ~]$ /usr/lib/nagios/plugins/check_file_age2 -f /var/lib/cobbler/lock -w 3600 -c 7200
/var/lib/cobbler/lock: File not found
[nagios@lnciserver1 ~]$ echo $?
0

All good!

Fun with Fdisk

2007-12-10T14:49:00.001-08:00

When managing disks under Linux, one of the most powerful and lowest level tools you will need to use is called fdisk. Fdisk has been around for a long time and even has use in the Microsoft world. Today, however, we will deal with linux fdisk. The Linux fdisk is a used to manipulate the partition table of disks.

Partition Table

Hard disks can be divided into one or more logical disks called partitions. This division is described in the partition table found in sector 0 of the disk. Fdsik manpage.

Example Partition Table

This is the partition table for a production Linux database server. The server has two disks. First, let us examine the first disk, referred to in Linux parlance as /dev/sda.

To do so, run the fdisk command and pass the argument /dev/sda to tell fdisk to look at the first disk. At the prompt, hit "p" to print the table. Pressing "m" will show you all the options.

[root@lncproddb2 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 8844.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 72.7 GB, 72746008576 bytes
255 heads, 63 sectors/track, 8844 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 8844 70935007+ 8e Linux LVM

Above we see the partition table for /dev/sda on production host lncproddb2. The /dev/sda device or disk contains two partitions /dev/sda1, and /dev/sda2. The first partition, /dev/sda1 starts at the beginning of the disk, or cylinder 1 and goes to cylinder 13. It is of type 83, or Linux.

The second partition on /dev/sda is /dev/sda2. It begins on the following cylinder, number 14, and goes to the end of the disk. The /dev/sda2 partition is of type 8e known as Linux LVM which stands for Logical Volume Manager. The disk has a total of 8844 cylinders. We can also see above the this is a 72.7 GB SCSI disk.

Scanning the Bus

Some Linux machines will have more than one disk. In this case, it is helpful to scan the bus to see how many disks are there and what they are called. You can scan the bus with the fdisk -l command.

[root@lncproddb2 ~]# fdisk -l

Disk /dev/sda: 72.7 GB, 72746008576 bytes
255 heads, 63 sectors/track, 8844 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 8844 70935007+ 8e Linux LVM

Disk /dev/sdb: 218.9 GB, 218909114368 bytes
255 heads, 63 sectors/track, 26614 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Here we see the /dev/sda again as described above. There is a second hard disk, /dev/sdb that is available on this system as well.