free - free and used memory stats

free command is the most obvious choice for a first command when it comes to your RAM.

Simply run it without any parameters, and it will show you something like this:

ubuntu# free
             total       used       free     shared    buffers     cached
Mem:       4051792    4024960      26832          0      63768    3131532
-/+ buffers/cache:     829660    3222132
Swap:      4096492      43212    4053280

For this exercise, you're only interested in the "total" column of the first line. 4051792 confirms that my home PC seems to have around 4Gb of memory available for Ubuntu to use.

Using dmesg to check memory size as recognized by Linux kernel

dmesg command shows you the last status messages reported by your OS kernel, and since every boot procedure includes scanning the hardware and confirming the devices and resources recognized by the kernel, you can see some basic information by using dmesg.

For our purpose, we need to filter out the memory stats:

ubuntu# dmesg | grep Memory
[   18.617904] Memory: 4043492k/5242880k available (2489k kernel code, 150360k reserved, 1318k data, 320k init)

Once again, the overall amount of memory confirms that 4Gb of RAM were still found during the last time my PC booted up.

Using /proc/meminfo to confirm the RAM size

/proc/meminfo is one of the special files managed by Linux kernel. It's a clear text presentation of the most vital memory stats of your system (this means you can do something like cat /proc/meminfo to see all the parameters)

This is what you need to do to get the total size of your physical memory:

ubuntu# grep MemTotal /proc/meminfo
MemTotal:      4051792 kB

That's it for today, enjoy!

Basic arithmetic calculations in Unix scripts

In most scenarios, your script will need only basic arithmetic operations: summing or subtracting numbers, multiplying or dividing them. Using only these operations, you'll be able to implement all sorts of counters and calculate percentages or any other progress indicators you may need.

So far, I've only shown you this way of evaluating arithmetic expressions in Unix shell (yes, it's not a script - you can just type commands one after another right in your shell prompt):

ubuntu$ START=1
ubuntu$ FINISH=5
ubuntu$ ELAPSED=`expr $FINISH - $START`
ubuntu$ echo $ELAPSED
4

But an even better way to evaluate arithmetic expressions is to use the built-in functionality of double brackets. Just continue typing in your shell:

ubuntu$ ELAPSED2=$(($FINISH - $START))
ubuntu$ echo $ELAPSED2
4

You see, everything that is inside double brackets gets evaluated and then this calculated value is assigned to the variable. I'm using a different variable - ELAPSED2 - to clearly show that this calculation has nothing to do with the original ELAPSED variable.

Assignment operator

If you wonder about the theory behind such a way of evaluating expressions, I should talk a bit about using variables in Unix shells.

As you probably remember, when you're assigning your variable a new value, you're simply specifying the name of the variable to the left of the expression, then use the = sign (it's not just a symbol but an assignment operator by the way), and to the right of this operator goes the expression which represents the new value:

ubuntu$ MYVAR=123

Variable substitution operator

As you also remember, when you want to access the value of the variable, for example to print it out, you use the variable name with a leading $ sign. That's because $, once again, isn't just a fancy symbol, but in fact a special character in Bash - it's a variable substitution operator, which means your Unix script substitutes the variable name you're specifying in the script with the value of the variable:

ubuntu$ echo $MYVAR
3

Using the substitution operator ($) before the name of a variable is crucial. If you omit the $ sign, your shell will think you have simply passed it a text string, and will not recognize it to be a name of the variable:

ubuntu$ echo MYVAR
MYVAR

Expression evaluation explained

Coming back to arithmetic evaluations, you can probably see now that a line similar to this:

ubuntu$ ELAPSED=$(($FINISH - $START))

uses the variable substitution operator ($) and it's therefore only logical that the further specified expression gets evaluated and the ELAPSED variable gets assigned with the resulting value.

Once again, using $ sign is crucial as without it your shell will not understand you:

ubuntu$ ELAPSED=(($FINISH - $START))
sh: syntax error near unexpected token `('

So, what do you think? I hope this makes sense to you, and will be sure to use this technique from now on - once you get used to it, it's really easy to write and read shell scripts based on this functionality.

Let me know if there's anything else you'd like to hear an explanation of! I'll do my best!

See also:

• Basics of BASH scripting
• Using variables in Unix scripts

How to find files owned by a user

If you know the username, this is the command you might use to locate all the files which belong to it:

ubuntu$ find /var -user www-data
/var/cache/apache2/mod_disk_cache
/var/lib/awstats
…

In this case, we're looking for all the files and directories owned by a webserver used www-data under /var.

Find files owned by a Unix group

If you're doing a broader search, you may be interested in identifying all the files owned by a Unix group. Here's how you can do it:

ubuntu$ find /usr -group staff
/usr/local/lib/site_ruby
/usr/local/lib/site_ruby/1.8
/usr/local/lib/site_ruby/1.8/x86_64-linux
/usr/local/lib/python2.5
/usr/local/lib/python2.5/site-packages
/usr/local/lib/python2.4
/usr/local/lib/python2.4/site-packages
/usr/local/share/fonts

In this example, we're using find to locate all the files in /usr owned by the staff group.

Locate files by UID or GID

If you're more comfortable dealing with Unix user IDs (UIDs) and group IDs (GIDs), you can use them with find command as well. In this example, I'm looking for the temporary files created by myself (my UID on that system is 1000):

ubuntu$ find /var/tmp -uid 1000
/var/tmp/photos.rar
/var/tmp/mysql.tar.gz
…

I presume this post answers your questions on the topic, but do let me know if there's anything else you'd like me to explain!

See also:

• Unix commands: find
• How to find an owner of a Unix file
• Finding a location of a directory
  

Going further, I'd like to show you some basics of working with time and date in your scripts - generating all sorts of timestamps and timing some parts of your script for reporting purposes.

The history of (Unix) time

In case you didn't know, time in Unix starts with a Unix epoch, sometimes also referred to as POSIX epoch. The Unix epoch is the time 00:00:00 UTC on January 1, 1970.

Most of Unix-like system today store and calculate this time according to Unix epoch, which means they all have an internal counter which counts the number of seconds elapsed since midnight of January 1, 1970.

As you can imagine, this is a pretty big number (it passed 1,000,000,000 seconds back in 2001, just so that you know), and reporting in directly would not be very useful to real humans. Because of this, most of Unix time and date reporting commands and functions do the conversion from Unix time into something more meaningful to the end user.

While for the vast majority of time and date related tasks this conversion is good, a task of timing certain events can sometimes benefit from using raw second counters and simply subtracting them when needed.

Getting Unix time and date

In Unix shells, the easiest way to get a current time and date is to use the date command:

ubuntu$ date
Tue Jun 10 10:46:07 IST 2008

date is a very smart command, and apart from this default behavior it supports template system for printing the current time and date - so you can use it to report only specific part of the time and date like the current hour or the day of a week or the year.

Read the man page for date (type man date in your shell prompt) to learn all the details, but this is how you use date output templates:

ubuntu$ date "+%b %d, %Y"
Jun 10, 2008

In this example, the %b parameter in this template represents the short name of the current month, %d is the day of the month, and %Y is the four-digit representation of the current year.

Timing parts of your Unix script

If you're after timing some parts of your script, you'll need to use two variables: one for saving the time before the start of an observed part of the script, and another one for saving the time after the same piece of code.

These two variables will be used to subtract the time and report the elapsed time in seconds, so in order to do this we'll need the date command to report time in seconds. That's why I've given you an introduction to Unix time earlier: date reports the number of seconds since the Unix epoch:

ubuntu$ date +%s
1213091896

If you run it just a few seconds later, you'll see a different number:

ubuntu$ date +%s
1213091922

That's why, if such values are saved and then subtracted, we'll get the elapsed time in seconds.

Here's a simple script showing how this is done:

#!/bin/bash
#
START=`date +%s`
echo "Script start time (Unix epoch): $START"
#
echo "- sleeping for 3 seconds..."
sleep 3
echo "- sleeping for 2 seconds more..."
sleep 2
#
FINISH=`date +%s`
echo "Script finish time (Unix epoch): $START"
#
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"

And if you run it, you will see this output:

ubuntu$ /tmp/time-example.sh
Script start time (Unix epoch): 1213092131
- sleeping for 3 seconds…
- sleeping for 2 seconds more…
Script finish time (Unix epoch): 1213092131
Elapsed time: 5

That's all I wanted to show you today. In future posts, I'll show you a few more things you can do regarding timing and timestamps in Unix. Until then - good luck with your scripting, and feel free to ask if you need any more help!

See also:

• Unix time - a great article on Wikipedia
• date command in Unix
  
• Working with variables in Unix shell

Why compare directories?

First of all, let's agree on why you may need to compare directories. There's a few possible reasons:

• identifying if some files are missing from one of the directories - can be useful when you want to make sure two directories with configuration files for a certain package are identical - files can be different, but the same files are present in the same locations for both directories
• confirming if files in two directories are the same - a typical task when comparing your actual data against a backup copy. When something goes wrong, this is one of the first things you do to make sure all the important files are not only present, but are actually the same as they have been when you took the last backup copy
• highlighting textual differences between files in directories - this is a useful exercise when you're looking at two similar directories and expect only minor changes between the files - version numbers, different file or directory names hardcoded in various scripts, etc.

Test setup for comparison exercises

For today's post, I've created a set of directories and files to show how you can compare them. Here is the setup:

ubuntu$ find /tmp/dir1 /tmp/dir2
/tmp/dir1
/tmp/dir1/file1
/tmp/dir1/file2
/tmp/dir1/dir11
/tmp/dir1/dir11/file11
/tmp/dir1/dir11/file12
/tmp/dir2
/tmp/dir2/file1
/tmp/dir2/dir11
/tmp/dir2/dir11/file11
/tmp/dir2/dir11/file12
/tmp/dir2/file3

As you can see, I've got two directories: /tmp/dir1 and /tmp/dir2, with a dir11 subdirectory in each of them. There's also a few files here and there, some of them missing from one of the directories specifically to be highlighted by our comparison exercises.

Basic diff usage for comparing directories

The easiest way to get started is to simply invoke diff command and specify two directories as command line parameters. Here's what you will probably see:

ubuntu$ diff /tmp/dir1 /tmp/dir2
Common subdirectories: /tmp/dir1/dir11 and /tmp/dir2/dir11
diff /tmp/dir1/file1 /tmp/dir2/file1
1d0
< New line
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

This output confirms that /tmp/dir1 and /tmp/dir2 both contain a dir11 directory, and also shows that /tm/dir1/file1 and /tmp/dir2/file1 are actually different files even though they have the same name. By default, diff compares such files and you can see the result of each comparison in the output. Also included are pointers to the files which are present only in one of the compared directories: you can see that file2 can only be found in /tmp/dir1 and file3 was present only in /tmp/dir2.

Find which files are missing in one of the directories

From the example below, it is easy to deduct that the command line for identifying files missing in one of the directories will be this one:

ubuntu$ diff /tmp/dir1 /tmp/dir2 | grep Only
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

Highlight the different files, not the differences

If you're only interested in files which exist in both directory structures, but are different - you can use a special command line option. It will simply point the files out, without getting into any further details. You'll probably notice how this output is very similar to the default one:

ubuntu$ diff –brief /tmp/dir1 /tmp/dir2
Common subdirectories: /tmp/dir1/dir11 and /tmp/dir2/dir11
Files /tmp/dir1/file1 and /tmp/dir2/file1 differ
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

Note how instead of showing the difference between file1 in /tmp/dir1 and /tmp/dir2, this time you only get told that these two files are different.

How to recursively compare directories

If you're dealing with a complex directory structure, you'll be glad to know that –recursive parameter for the diff command compares not only the immediate directories pointed to from the command line, but also walks through the full tree of subdirectories:

ubuntu$ diff –recursive –brief /tmp/dir1 /tmp/dir2
Files /tmp/dir1/dir11/file12 and /tmp/dir2/dir11/file12 differ
Files /tmp/dir1/file1 and /tmp/dir2/file1 differ
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

Feel better now? Many directory comparison tasks can be accomplished using the diff command, but if you're stuck with a particular problem which can't be solved using my examples - please leave a commend and I'll come up with a solution.

See also:

• Comparing text files in Unix
• Listing directories in a directory

How to find the username using user id (uid)

Simply use the getent command. Most common use for it is to query the passwd source for a username, like this:

ubuntu$ getent passwd greys
greys:x:1000:113:Gleb Reys,,,:/home/greys:/bin/bash

however, if you query for a user id instead (1000 in this case), getent will work just as good:

ubuntu$ getent passwd 1000
greys:x:1000:113:Gleb Reys,,,:/home/greys:/bin/bash

That's all there is to it! Enjoy!

What is a UUID exactly?

UUID is a Universally Unique IDentifier. It's a identification code given to each storage device you have on your system, aimed to help you uniquely identify each device no matter what.

UUIDs can be used to identify DVD drives, removable media (USB flashsticks) and each partition on any of your hard drives.

This is how a typical UUID looks:

c73a37c8-ef7f-40e4-b9de-8b2f81038441

Why use UUID?

Reason 1: Truly unique identification

UUID is the only way to guarantee you recognize the same drive or partition no matter what. For example, if you introduce to your system another hard drive, this might upset quite a few things, starting with the way your system boots up (or stops booting up upon the new drive introduction). Using UUID helps remedy most of such things.

Reason 2: Device names are not always persistent

Automatically assigned device names in your system are not consistent, they are according to the order of loading the kernel modules up during (most usually) the startup time, and thus the names will look different if you boot with one of your USB flashsticks attached and then reboot after you plug it out.

Generally, I've also found UUIDs really useful for mounting my removable media - I have a USB reader, one of the 24-in-1 kinds - it supports various types of cards and I use UUID to always mount the same card at the same location.

Reason 3: Most critical functionality of your Ubuntu system already depends on UUIDs

GRUB - your boot loader - relies on UUIDs as well. If you look into /boot/grub/menu.lst file, you'll find something similar to this:

title Ubuntu hardy (development branch), kernel 2.6.24-16-generic
root (hd2,0)
kernel /boot/vmlinuz-2.6.24-16-generic root=UUID=c73a37c8-ef7f-40e4-b9de-8b2f81038441 ro quiet splash
initrd /boot/initrd.img-2.6.24-16-generic
quiet

List UUIDs for all your devices

If you are using one of the recent releases of Ubuntu (UUIDs have been there since Edgy), you can use the blkid command to get a list of all the drives and partitions along with their UUIDs:

ubuntu# blkid
/dev/sda1: UUID="2220CF8220CF5B83″ TYPE="ntfs"
/dev/sda2: UUID="48E81F29E81F14B2″ LABEL="DRIVE-D" TYPE="ntfs"
/dev/sdb1: UUID="c73a37c8-ef7f-40e4-b9de-8b2f81038441″ SEC_TYPE="ext2″ TYPE="ext3″
/dev/sdb5: TYPE="swap" UUID="abe7529e-dcd5-4afc-b714-05569dbcd30b"
/dev/sdb6: UUID="f34c8c7c-a020-4a14-8c97-257180240250″ SEC_TYPE="ext2″ TYPE="ext3″
/dev/sdb7: UUID="8fa274ca-5b22-411f-b5da-7469c1f276da" SEC_TYPE="ext2″ TYPE="ext3″
/dev/sdc1: UUID="1e36f323-c4e5-4f55-ba0a-838643550bf9″ TYPE="ext3″ SEC_TYPE="ext2″
/dev/sdc2: UUID="83aa92e4-4df4-4aab-80f3-9bbb447e0459″ TYPE="ext3″ SEC_TYPE="ext2″

As you can see, I've still got a few NTFS partitions as I'm slowly migrating my data from Windows after my switch to Ubuntu desktop a couple months ago.

Get UUID for a particular device

If you know a device name and just want to confirm the UUID for it to later use it in /etc/fstab, here's how you can do it using vol_id command:

ubuntu# vol_id -u /dev/sdb1
c73a37c8-ef7f-40e4-b9de-8b2f81038441

That's all I can think of so far. I know a few more things about UUID which I'll share in a separate post, but it's a start.

Have you got any more great ideas and tips for UUID? Let me know and I'll be sure to share them with others in the future posts.

See also:

• How to mount an ISO image
• Find large files and directories in Unix
• Using UUIDs - Ubuntu Help wiki
• How to find the default block size in Unix

Defining a variable in Unix shell

To specify a value for a variable, you need to decide on the variable name - can be any word or combination of English alphabet symbols and digits, and specify the value.

In Bourne shell (sh), Bourne Again Shell (bash) and Korn Shell (ksh), here's how you would define a new variable and assign it a value:

CONFIG_FILE="/etc/myfile"

In C-Shell (csh), it's done like this:

setenv CONFIG_FILE "/etc/myfile"

Basic variables usage

To access the value of a variable, you need to use the same variable name, but with a dollar sign in front of it, like this:

$CONFIG_FILE

Important: to set a new value to the variable, you use just the variable name like CONFIG_FILE, but to access this value later you need to use the $CONFIG_FILE form.

The most basic way to use variables is to assign them constant values at the beginning of your script. This is a good way to define locations of standard files or directories your script will work with, etc:

#!/bin/sh
#
CONFIG_FILE=/etc/myfile
MY_DIR=/etc
echo $CONFIG_FILE

Using output of Unix commands to set variables

One of the best things about shell scripting is that it's very easy to use any Unix command to generate the output and use it to set the variable.

In this example, I'm running a date command and saving its output as values for my variables:

ubuntu$ cat /tmp/1.sh

#!/bin/sh
#
STARTED=`date`
sleep 5
FINISHED=`date`
#
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

If I run this simple script, I see the following:

ubuntu$ /tmp/1.sh
Script start time: Wed May 7 04:56:51 CDT 2008
Script finish time: Wed May 7 04:56:56 CDT 2008
The same approach can be used for practically any scenario.

Here's an example of using uname command to extract some useful information about our system:

#!/bin/sh
#
STARTED=`date`
NODE=`uname -n`
OS=`uname -o`
CPU=`uname -p`
FINISHED=`date`
#
echo "Nodename: $NODE"
echo "OS type: $OS"
echo "Processor: $CPU"
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

And this is how it works:

ubuntu$ /tmp/1.sh
Nodename: ubuntu
OS type: GNU/Linux
Processor: unknown
Script start time: Wed May  7 05:05:31 CDT 2008
Script finish time: Wed May  7 05:05:31 CDT 2008

That's it for today! Let me know what next topic about Unix shell scripting you'd like to see covered, and I'll do my best to explain it in the coming posts.

Showing processes tree with ptree

In Solaris, there's quite a few commands which make the life of any system administrator much easier, they're the process commands (p-commands). One of them which I particularly like is the ptree command which shows you a list of processes.

As you run the command, you get a hierarchical list of all the processes running on your Solaris system, along with process IDs (PIDs). To me, this is a very useful command, because it shows you how exactly each process relates to others in your system.

Here's a fragment of the ptree output:

bash-3.00$ ptree
7     /lib/svc/bin/svc.startd
  250   /usr/lib/saf/sac -t 300
    268   /usr/lib/saf/ttymon
  260   -sh
    5026  -csh
9     /lib/svc/bin/svc.configd
107   /usr/lib/sysevent/syseventd
136   /usr/lib/picl/picld
140   /usr/lib/crypto/kcfd
159   /usr/sbin/nscd
227   /usr/sbin/rpcbind
234   /usr/lib/nfs/statd
235   /usr/sbin/keyserv
236   /usr/lib/netsvc/yp/ypserv -d
  237   rpc.nisd_resolv -F -C 8 -p 1073741824 -t udp
241   /usr/lib/nfs/lockd
247   /usr/lib/netsvc/yp/ypbind
263   /usr/lib/utmpd
286   /usr/sadm/lib/smc/bin/smcboot
  287   /usr/sadm/lib/smc/bin/smcboot
  288   /usr/sadm/lib/smc/bin/smcboot

Processes tree with pstree

In most Linux distributions, you can find a pstree command, very similar to ptree.

That's how you may use it (-p is an option to show PIDs and -l uses long output format):

ubuntu$ pstree -pl
init(1)─┬─NetworkManager(5427)
        ├─NetworkManagerD(5441)
        ├─acpid(5210)
        ├─apache2(6966)─┬─apache2(2890)
        │               ├─apache2(2893)
        │               ├─apache2(7163)
        │               ├─apache2(7165)
        │               ├─apache2(7166)
        │               ├─apache2(7167)
        │               └─apache2(7168)
        ├─atd(6369)
        ├─avahi-daemon(5658)───avahi-daemon(5659)
        ├─bonobo-activati(7816)───{bonobo-activati}(7817)
…

Showing processes tree with ps –forest

ps command found in Linux has a –forest option, which shows you a tree structure of processes.

The best in my experience is to use it like this:

ubuntu$ ps -aef –forest
UID        PID  PPID  C STIME TTY          TIME CMD
…
107       5473     1  0 10037  4600   0 Apr28 ?        00:00:02 /usr/sbin/hald
root      5538  5473  0  5511  1288   0 Apr28 ?        00:00:00  \_ hald-runner
root      5551  5538  0  6038  1284   0 Apr28 ?        00:00:01      \_ hald-addon-input: Listening on /dev/input
107       5566  5538  0  4167   992   1 Apr28 ?        00:00:00      \_ hald-addon-acpi: listening on acpid socke
root      5600  5538  0  6038  1272   1 Apr28 ?        00:00:15      \_ hald-addon-storage: polling /dev/scd0 (ev
root      5476     1  0 10272  2532   0 Apr28 ?        00:00:00 /usr/sbin/console-kit-daemon
root      5627     1  0 12728  1176   1 Apr28 ?        00:00:00 /usr/sbin/sshd
root      9151  5627  0 17536  3032   0 10:53 ?        00:00:00  \_ sshd: greys [priv]
greys     9162  9151  0 17538  1892   1 10:54 ?        00:00:00      \_ sshd: greys@pts/3
greys     9168  9162  0  5231  3820   1 10:54 pts/3    00:00:00          \_ -bash
greys     9584  9168  0  3802  1124   0 11:27 pts/3    00:00:00              \_ ps -aeF –forest

This output is for demonstration purpose only, and so I've taken the first lines of the output out because they weren't serving the purpose of this example very well.

For thins fragment of the output you can see how you get all the vital information about each process. I really like this way of running the ps command.

That's it for today! Do you know any other neat way of looking at processes tree? Let me know!

For all of you who were waiting for the next release of Ubuntu, it's finally here! I've been using the beta of Ubuntu 8.04 (Hardy Heron) for the past month or so, and found it an excellent release to work with.

Ubuntu 8.04 links:

• Ubuntu 8.04 TLS download
• What's new in Ubuntu 8.04
• Upgrading to Ubuntu 8.04