1
2
3
4
5
6
7
8
9
10
11
12

kattoo@roadrunner /files/toSave/home/kattoo/Downloads $ perl -ane '$c{$F[0]}++; END {print map {$_ . "\t->\t" . $c{$_} . "\n"} (sort {$c{$b} <=> $c{$a}} keys %c)[0..9]}' sakana.fr-28-02-2010.log 
XX.XX.XX.XXX	->	163
XX.XXX.XXX.XXX	->	150
XXX.XXX.XX.XX	->	136
XXX.XX.XXX.XXX	->	134
XX.XXX.XXX.XX	->	116
XX.XXX.XXX.XXX	->	110
XX.XX.XXX.XX	->	104
XX.XXX.XXX.XXX	->	90
XX.XXX.X.XXX	->	75
XXX.XXX.XX.XXX	->	72
kattoo@roadrunner /files/toSave/home/kattoo/Downloads $

Short explanation of the command line switches :

• -a : turns on autosplit, the different fields will then be accessible in the @F variable
• -e : specifies the code to execute
• -n : wraps that code into a loop for each text line of the file to process from the standard input or the files specified on the command line

This is equivalent to Nils’ approach in awk but you know, I’ve always been partial to Perl

Feel free to suggest any improvement… I know there must be many !

Suggested reading

• Programming Perl (3rd Edition) (on Amazon) : The reference guide for Perl programming, written by Perl Gods (Larry Wall, Tom Christiansen, John Orwant).
• Use command-line Perl to make UNIX administration easier (on techrepublic) : Short intro about perl 1-liners

Perl : Counting occurences of IP addresses in Apache logs is a post from: Tech@Sakana - A sysadmin's blog

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

login@servername:/> sar
 
AIX servername 3 5 00CFE81B4C00    01/25/10
 
Configuration du système : lcpu=20 ent=8,00
 
00:00:01    %usr    %sys    %wio   %idle   physc   %entc
00:05:01       2       2       0      96    0,40     5,0
00:10:01       3       2       0      94    0,48     6,0
00:15:00       1       1       0      98    0,21     2,6
00:20:01       1       1       0      98    0,23     2,9
00:25:01       2       1       1      97    0,25     3,2
00:30:01       1       1       0      98    0,23     2,9
00:35:00       2       1       1      96    0,31     3,8
00:40:01       1       1       0      98    0,21     2,6
00:45:01       1       1       0      98    0,22     2,7
00:50:00       1       1       0      98    0,23     2,8
00:55:00       1       1       0      98    0,21     2,6
01:00:00       1       1       0      98    0,21     2,6
01:05:00       2       1       0      96    0,35     4,4

The columns are the usual for this type of tools :

1. %usr : the percent of time spent in user land. The higher the more CPU intensive processes you are running.
2. %sys : the percent of time spent in kernel. This is mostly due to I/O intensive processes and/or processes performing a lot of syscalls.
3. %wio : I’ve got no idea about that one … If you’ve got an explanation for this, feel free to hit the comments. I suspect this is the time spent waiting for I/Os to complete, or waiting after a busy device. This could be a sign of I/O (disk, network) bottlenecks
4. %idle : the percent of time spent doing … nothing. The higher this number, the more underused your machine
5. physc : no idea neither !
6. %entc : the number of CPU used to absorb the load

If you’re looking for a top like tool, topas is the standard AIX equivalent.

AIX : Use sar to check cpu usage is a post from: Tech@Sakana - A sysadmin's blog

In this script, we use the JSON Twitter API. Perl has a JSON module to make parsing or JSON streams a bliss. I also used formats for easy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#! /usr/bin/perl
 
use strict;
use warnings;
 
use JSON;
use Encode;
 
my $json = new JSON;
my $json_stream = join "", map {decode("UTF-8", $_)} <>;
 
my $mentions = $json->decode($json_stream);
 
foreach my $mention (@$mentions) {
	my $screen_name = encode("UTF-8", $mention->{user}->{screen_name});
	my $text = encode("UTF-8", $mention->{text});
	format =
@<<<<<<<<<<<<<<    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$screen_name,      $text
                   ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ~~
				   $text
.
	write;
}

You’ll have to run it as demonstrated below :

1
2
3
4
5
6
7
8
9

kattoo@roadrunner /files/toSave/home/kattoo $ curl -sS -u YourLoginHere:YourPasswordHere http://twitter.com/statuses/mentions.json | twitter-mentions.pl 
divarvel           @killermouse0 mon /boot est pas dans une
                   partition à part, et pour une install en
                   chroot je ne sais pas quoi faire (un autre
                   /boot ?)
fenice             @killermouse0 merci, c'est sympa. Tu tourne
                   sous Gentoo ?
[...]
kattoo@roadrunner /files/toSave/home/kattoo $

That’s all ! Of course this piece of code could use a lot of improvements, but this is intended as a kickstart sample… The curl part could have been directly integrated into the Perl code, but I like to keep it simple.

Now off to integrate this in my Conky layout

And by the way, you’ll find my twitter account on the right, in the “Follow me” section. Feel free to add me for updates !

Worth reading

• Twitter : Send free SMS messages to your friends : now you can do this from the command line
• Scripting Twitter with cURL
• Scripting Twitter with Perl + LWP : a script in Perl doing the same thing (much simpler).
• Perl+Twitter : Getting @mentions from command line

Perl+Twitter : Getting @mentions from command line is a post from: Tech@Sakana - A sysadmin's blog

df command

The df (disk free) command will report how much of each file system is used. You can combine this with with the sort command to get the fullest file systems at the end :

1
2
3
4
5
6
7
8
9
10
11
12

kattoo@roadrunner ~/Downloads $ df | sort -n -k 5,5
Filesystem           1K-blocks      Used Available Use% Mounted on
shm                    2029696         0   2029696   0% /dev/shm
/dev/sda6              1968588      3232   1865356   1% /tmp
udev                     10240       228     10012   3% /dev
/dev/sda12           367628876  28899532 338729344   8% /files
/dev/sda7              1989820    246300   1743520  13% /var
/dev/sda9              1999932    260996   1738936  14% /usr/portage
/dev/sda5               244176     52264    191912  22% /
/dev/sda8              9990188   3161340   6828848  32% /usr
/dev/sda11            99947736  32910756  67036980  33% /files/vservers
/dev/sda10             3989912   1699372   2290540  43% /usr/portage/distfiles

The sort arguments are :

• -n : order numerically
• -k 5,5 : order by the 5th field

du command

When you found out that a specific file system is full, you need to quickly analyze which folders are the biggest.
You can use the du (disk usage) command. This command will report the space used by a directory and sub-directories. I usually run something like :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

kattoo@roadrunner /usr/portage $ du -sk * | sort -n | tail -20
5711	x11-misc
6056	app-misc
6224	dev-ruby
6320	app-dicts
7219	profiles
7428	net-analyzer
7686	app-text
8112	media-plugins
8777	kde-base
8883	media-libs
8943	sys-apps
9764	dev-util
9986	media-sound
10955	dev-libs
11295	dev-python
11479	dev-java
11645	net-misc
17553	dev-perl
112556	metadata
1705976	distfiles

This will show the 20 largest files or directories. du will display the size of each directories and sub-directories by default. The -s switch will make du display only the grand total for each arguments.

According to the results, you might want to go on repeating this command in the sub-directories, until you’ve narrowed down enough to find the culprit.

find command

The file system full error is more often than not due to a faulty script or program which logs are running wild due to a bug. This will usually lead to a huge file somewhere in a directory, hogging all the available space on the file system.

An easy way to spot it is to use the find command. The following snippet for example will search for files bigger than 10MB in the current directory and sub-directories and print the 10 largest ones :

1
2
3
4
5
6
7
8
9
10
11
12

kattoo@roadrunner /usr/portage $ find . -type f -size +10000000c -exec ls -l \{} \; | sort -n -k 5,5 | tail -10
-rw-rw-r-- 1 portage portage 23416703 Oct  1 10:07 ./distfiles/samba-3.0.37.tar.gz
-rw-rw-r-- 1 portage portage 28749072 Jan 18  2008 ./distfiles/extremetuxracer-0.4.tar.gz
-rw-rw-r-- 1 portage portage 46905557 Oct 16 17:20 ./distfiles/firefox-3.5.4.source.tar.bz2
-rw-rw-r-- 1 portage portage 46914620 Dec  2 05:32 ./distfiles/firefox-3.5.6.source.tar.bz2
-rw-rw-r-- 1 portage portage 50380241 Oct  2 11:47 ./distfiles/VirtualBox-3.0.8-53138-Linux_amd64.run
-rw-rw-r-- 1 portage portage 50595281 Nov 17 10:44 ./distfiles/VirtualBox-3.0.12-54655-Linux_amd64.run
-rw-rw-r-- 1 portage portage 59368714 Aug  8 02:10 ./distfiles/gcc-4.3.4.tar.bz2
-rw-rw-r-- 1 portage portage 61494822 Sep 10 00:34 ./distfiles/linux-2.6.31.tar.bz2
-rw-rw-r-- 1 portage portage 125384668 Oct  1 13:58 ./distfiles/qt-x11-opensource-src-4.5.3.tar.gz
-rw-rw-r-- 1 portage portage 314942420 Jun 18  2008 ./distfiles/sauerbraten_2008_06_17_ctf_edition_linux.tar.bz2
kattoo@roadrunner /usr/portage $

I also like to filter out already compressed files, in order to collect the biggest files which I could compress to save some space with something like :

1

find . -type f -size +10000000c \! -name "*.Z" \! -name "*.gz" \! -name "*.bz2" -exec ls -l \{} \;

Tip : Set this as an alias in your profile by adding this to your .bashrc (if you’re using bash … otherwise check your shell documentation) :

1

alias bigfiles='find . -type f -size +10000000c \! -name "*.Z" \! -name "*.gz" \! -name "*.bz2" -exec ls -l \{} \; | sort -n -k 5,5 | tail -10'

This will give you the 10 largest not-yet-compressed files in a single handy command.

Caveat

“Too many arguments” error :

If you are working with directory holding an important number of files, you might get a “Too many arguments” error when using a star expansion (like in the du -sk * example above).

This is actually not the du command complaining, but the shell which gets too much data when expanding the star. When this happens, you are usually better off using the find command as explained above.

Another possibility is to pipe the output of find to the xargs command. Basically xargs will take everything on the standard input and give it as arguments to the specified command.

Deleting / compressing a file which is still open
Also beware of this : if you delete or compress a file which is still being open by a process, then the space this file use won’t get freed before the process actually closes the file (I’ll explain why in a different post, this is an interesting topic on its own ).

If you deleted or compressed the file (so basically the big file disappeared or was replaced by a compressed version), but the space doesn’t get freed (which you can check with df), then you can bet that a process is still holding the file open. You can spot this with tools like lsof or fuser. Those tools vary greatly according to which variant of Unix you’re running. On IBM’s AIX, fuser had a handy -d option to spot files on the file system which have a link count of 0 and it will report the PID of the attached processes.

Better check those tools’ man pages before you run into this situation !

A word to the wise

Those recipes will help you to find a way out when you are already facing a file system full problem. The best is of course to avoid them in the first place … The following ideas are a good start :

• Set alerts to have an early warning and let you deal with it before applications start crashing. Tools like Nagios would be your friends here, but home made scripts running from cron and sending emails might be enough.
• Check the trends : You can use tools like cacti to graph the space occupation of your file systems over the time. This will let you anticipate when you’ll need to add more disks or if your log rotation and/or file archiving policies are adequate.

More ideas ? Tips to share ? Hit the comments !

Linux / Unix : Disk usage and identifying biggest files is a post from: Tech@Sakana - A sysadmin's blog

This posts lists a few ideas that you might find useful to implement when in such an environment.

1. Try to avoid emails when the job processed correctly : It is not useful to swamp the tech support with emails requiring no action. Of course you’ll have to be able to differentiate between a script which ran fine and sent no email, and one which didn’t send an email because it didn’t run at all
2. Keep the subject of the email specific : It could be for example the name of the script and the status (OK or KO). This helps for faster (human or rule based) processing of the emails
3. Clearly state the priority of the email in the subject. This helps to decide efficiently in which order to deal with problems when many are occurring simultaneously.
4. Clearly mention which script produced the email : It IS annoying to have to find out first WHAT broke and ONLY THEN start analyzing and recovering …
5. The body of the email should clearly state the error condition.
6. When possible, the body of the email should give the recovery procedure. If this is not possible, it should point to the documentation

So let’s say those are my “best practices” based on field experience. Have more tips or rules ? Share them in the comments !

Monitoring batches in IT environment : efficiently using emails is a post from: Tech@Sakana - A sysadmin's blog

You can control this behaviour with this about:config property middlemouse.contentLoadURL. Set to true, the middle-click on the page will load the URL (if it is one) in the page, set to false it won’t.

Thanks Ayman !

Firefox (Linux) : “Middle-click on page loads URL” annoyance is a post from: Tech@Sakana - A sysadmin's blog

So here is a simple snippet demonstrating how to get the price of a stock :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#! /usr/bin/perl
 
use strict;
use warnings;
use Finance::Quote;
 
my $q = Finance::Quote->new();
 
my @tickers = ("CS.PA", "DEXB.PA", "AF.PA", "VIE.PA", "KEYYO.PA");
my $s = $q->fetch("yahoo_europe", @tickers);
 
my $stocks;
 
while (my ($k, $v) = each %$s) {
	my $stock;
	my $label;
 
	($stock, $label) = (split $;, $k)[0, 1];
	$stocks->{$stock}->{$label} = $v;
}
 
my @data = ();
foreach (@tickers) {
	if (not $stocks->{$_}->{success}) {
		push @data, { name => $_, price => "N/A"};
	} else {
		push @data, $stocks->{$_};
	}
}
my @sorted = sort { $a->{name} cmp $b->{name} } @data;
@data = undef;
 
print sprintf "% -20s %s\n", "STOCK", "PRICE";
foreach (@sorted) {
	print sprintf "% -20s % 7.2f\n", $_->{name}, $_->{price};
}

The execution will provide something like that :

1
2
3
4
5
6
7
8

kattoo@roadrunner ~/bin $ ./get-stocks.pl 
STOCK                PRICE
AIR FRANCE - KLM       11.35
AXA                    16.64
DEXIA                   4.87
KEYYO                   3.30
VEOLIA ENVIRONN.       22.70
kattoo@roadrunner ~/bin $

That’s pretty simple so I won’t go into more details (feel free to ask if you need clarifications).

I’ve already added it into my conky layout with an execpi directive, and it works like a charm

Perl : Using the Finance::Quote module to get your stock prices is a post from: Tech@Sakana - A sysadmin's blog

1
2

roadrunner ~ # equery belongs /etc/init.d/samba [ Searching for file(s) /etc/init.d/samba in *... ]
net-fs/samba-3.0.37 (/etc/init.d/samba)

equery is part of the gentoolkit package, which provides useful tools to maintain/investigate your gentoo packages.

1
2
3

roadrunner ~ # equery belongs /usr/bin/revdep-rebuild 
[ Searching for file(s) /usr/bin/revdep-rebuild in *... ]
app-portage/gentoolkit-0.2.4.5 (/usr/bin/revdep-rebuild)

Gentoo Linux / Portage : How to know which package provided an installed file is a post from: Tech@Sakana - A sysadmin's blog

1. Clone the disk image with a similar command line (you will have to adjust your paths) :
   ^?View Code TEXT

   1 2 3 4 5 6 7 8

   kattoo@roadrunner /files/toSave/home/kattoo $ VBoxManage clonehd /files/vservers/{VM1/disk1.vdi,VM2/disk2.vdi} VirtualBox Command Line Management Interface Version 3.0.10 (C) 2005-2009 Sun Microsystems, Inc. All rights reserved.   0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Clone hard disk created in format 'VDI'. UUID: 84956ae4-6c85-41cf-aa23-a265029d99a8 kattoo@roadrunner /files/toSave/home/kattoo $

2. Setup a new virtual machine through the virtualbox GUI or otherwise, with the same parameters as the original one and attach this new disk

Note : This command doesn’t take care of potential snapshots you could have on the source disk image. It will only clone the base image, so you might have to merge the snapshots into the base image before cloning if you want those to be integrated in the copy.

References :

• VirtualBox manual : vboxmanage clonehd

VirtualBox : Cloning a virtual hard disk is a post from: Tech@Sakana - A sysadmin's blog

Step 1 : install the packages

(This step is optional if you already have them installed on the machine)

1
2
3
4
5
6

kattoo@backup3:~$ pfexec pkg install SUNWsmbskr
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  2/2       35/35      2.0/2.0 
 
PHASE                                        ACTIONS
Install Phase                                  90/90

At this point, you’ll need to perform a reboot (if you can’t afford that, check in the references for a link to the full procedure).

Step 2 : Join a WORKGROUP

1
2
3
4
5

kattoo@backup3:~$ pfexec smbadm join -w SAKANA
After joining SAKANA the smb service will be restarted automatically.
Would you like to continue? [no]: yes
Successfully joined SAKANA
kattoo@backup3:~$

Step 3 : Set passwords for users who will access the shares

Add the following line into /etc/pam.conf :

1
2

kattoo@backup3:~$ grep smb /etc/pam.conf 
other 	password required 	pam_smb_passwd.so.1 nowarn

Now reset the passwords for the users who will be access the shares :

1
2
3
4

kattoo@backup3:~$ pfexec passwd kattoo
New Password: 
Re-enter new Password: 
passwd: password successfully changed for kattoo

Step 4 : Share the filesystems

You can then share any zfs filesystem with the following :

1

kattoo@backup3:~$ pfexec zfs set sharesmb=on backup

To check what is exported, you can do this :

1
2
3
4
5
6
7
8
9
10

kattoo@backup3:~$ sharemgr show -pv
default nfs=()
zfs
    zfs/backup smb=()
	  backup=/backup
	  backup_roadrunner=/backup/roadrunner
	  backup_thekla=/backup/thekla
    zfs/storage smb=()
	  storage=/storage
kattoo@backup3:~$

Here, the backup_roadrunner and backup_thekla were automatically exported because the property sharesmb was inherited from backup.

References :

• Genunix : Getting Started With the Solaris CIFS Service

OpenSolaris : Exporting a ZFS filesystem with CIFS is a post from: Tech@Sakana - A sysadmin's blog