UNIX BASH scripting

Unix - Append 0 to single digit date

2013-05-08T10:33:00.004+05:30

Input file file.txt has dates in month/day/year format.

$ cat file.txt 
3/4/2013
3/10/2013
10/4/2013
12/10/2012

Required: Add prefix 0 to first and second field if its a single digit.

Awk solution:

$ awk 'BEGIN {FS=OFS="/"} 
    { 
 if (length($1) == 1) $1="0"$1
 if (length($2) == 1) $2="0"$2
        { print }
}' file.txt

Output:

UNIX time command output redirect to file

2013-02-28T08:54:00.000+05:30

As you know the 'time' command run programs and summarize system resource usage. Here is a way how you can redirect 'time' command output to file. I would recomend you to go through this page to understand more about 'time' command.

e.g. We are executing 'prog.sh' along with 'time' command.

$ time ./prog.sh 
Initiating merge
Merge completed

real	0m8.803s
user	0m0.010s
sys	0m0.000s

#Trying to redirect the output to a file

$ time ./prog.sh > out.txt

real	0m8.804s
user	0m0.020s
sys	0m0.000s

#out.txt content:

$ cat out.txt 
Initiating merge
Merge completed

#So it only redirected the STDOUT of the script executed, but the 'time' command outut is not redirected.
#This is becuase the command time sends it's output to STDERR (instead of STDOUT)
#To capture output of 'time' command:

$ { time ./prog.sh ; } 2> out.txt
Initiating merge
Merge completed

#Now out.txt content:

$ cat out.txt 

real	0m8.303s
user	0m0.010s
sys	0m0.000s

#And to capture output of script as well as time command:

$ { time ./prog.sh ; } &> out.txt

#'out.txt' now has both the outputs.

$ cat out.txt 
Initiating merge
Merge completed

real	0m8.303s
user	0m0.020s
sys	0m0.000s

As I have mentioned in my earlier post on UNIX 'time' command, there's two types of time command available:
1) Shell's in-build time: Gives only scheduler information
2) /usr/bin/time: Gives more information, also allows formatting the output

The second (/usr/bin/time) one accepts output redirection without code block:

$ /usr/bin/time ./prog.sh &> newout.txt

$ cat newout.txt 
Initiating merge
Merge completed
0.00user 0.01system 0:08.30elapsed 0%CPU (0avgtext+0avgdata 5728maxresident)k
0inputs+8outputs (0major+719minor)pagefaults 0swaps

Bash - convert delimited file to html table

2013-02-15T22:27:00.003+05:30

Here is a simple UNIX bash script that can be used to convert a simple delimited file to a HTML table format. Default delimiter if not mentioned as part of the command line argument to this script will be comma.

#!/bin/sh
#convert_2_html_table.sh 
#Converts a delimited file to a HTML table
#Jadu Saikia http://unstableme.blogspot.in

NOARG=64

#usage function
f_Usage () {
echo "Usage: $(basename $0) -d <delimiter> -f <delimited-file>"
}

#command line args
while getopts d:f: OPTION
do
    case $OPTION in
        d)  DELIMITER=$OPTARG ;;
        f)  INFILE=$OPTARG ;;
    esac
done

#Less than 2 command line argument, throw Usage
[ "$#" -lt 2 ] && f_Usage && exit $NOARG

DEFAULTDELIMITER=","
#If no delimiter is supplied, default delimiter is comma i.e. ,
SEPARATOR=${DELIMITER:-$DEFAULTDELIMITER}

if [ -f "${INFILE}" ]
        then
                printf "<table border=\"1\">"
                sed "s/$SEPARATOR/<\/td><td>/g" $INFILE | while read line
                        do
                                printf "<tr><td>${line}</td></tr>"
                done
                printf "</table>"
                echo
fi

e.g. Input file:

$ cat data.txt 
First Name:Last Name:Points
Alex:Hall:45
Niraj:Kumar:290
Brian:Smith:100

Executing it:

$ ./convert_2_html_table.sh -d ":" -f data.txt  > data.html

Beginning of line in UNIX screen session

2012-11-25T23:03:00.002+05:30

As you know every "screen" command begins with "Ctrl-a", then how to go to beginning of the line when you are working under a "screen" UNIX session (which we achieve by "Ctrl-a" in a normal UNIX session) ?

Solution:
Under a "screen" UNIX session you can do "Ctrl-a a" to move to the beginning of the line.

So in order to print ^A (hex: \x01) in a UNIX "screen" session, usual "Ctrl-v Ctrl-a" will not work, you will type "Ctrl-v Ctrl-a a" to print ^A.

Related posts:
- Save command without executing it in Bash history
- Ctrl s freezes UNIX shell and vi editor
- UNIX - prevent exit of shell with Ctrl-d
- Print UNIX command line history with time-stamp of command execution

Transpose using UNIX paste command

2012-11-21T08:24:00.001+05:30

I have already posted about performing transpose of a matrix using Awk in my earlier posts post1 and post2 . This example shows a not so efficient way of achieving the same.

Input file:

$ cat file.txt 
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
32420 testuser 20   0 1219m 969m  49m R   34 12.1   1043:14 worker

Required output:

PID 32420
USER testuser
PR 20
NI 0
VIRT 1219m
RES 969m
SHR 49m
S R
%CPU 34
%MEM 12.1
TIME+ 1043:14
COMMAND worker

$ for i in $(seq 12); do awk -v X=$i '{print $X}' file.txt | paste - - ; done

UNIX - append text to filename using Awk

2012-11-14T22:18:00.000+05:30

I have some files in my current directory whose file-name is of this pattern:

$ ls -1
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt

Requirement: Append a text "MY-2" as the 6th field (dot delimited) of the filename. E.g.

 
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt

should be renamed to

log.1024.94.1326776200.1326776300.MY-2.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt

A bash script using awk to achieve this:

for file in $(ls)
 do 
  newfilename=$(echo $file | awk 'BEGIN {FS=OFS="."} {$6="MY-2" OFS $6} {print}')
  mv -v $file $newfilename
done

All the files are renamed to:

$ ls -1
log.1024.94.1326776200.1326776300.MY-2.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.MY-2.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.MY-2.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt

Feel free to post (as comment below) any alternative to this, much appreciated and a big thank you in advance.
Related posts:

- Padding zeros in filename using Bash in UNIX
- Rename file to uppercase except extension - Bash
- Rename multiple files using UNIX rename command

Time bucketing using Awk in Unix

2012-10-30T21:11:00.002+05:30

Each line of input file file.txt is having UNIX epoch timestamp and some value.

$ cat file.txt 
Epoch,Value
1351605000,120
1351605060,130
1351605120,340
1351605180,200
1351605240,120
1351605300,890
1351605360,124
1351605420,450
1351605480,120
1351605540,120
1351605600,200
1351605660,120
1351605720,340
1351605780,670
1351605840,990

The UNIX epoch timestamps are of 1 minute interval. Converting the UNIX epoch timestamps to human readable format using awk 'strftime' function:

$ awk 'BEGIN {FS=OFS=","} NR != 1 {$1=strftime("%c",$1)} {print}' file.txt

Output:

Epoch,Value
Tue 30 Oct 2012 01:50:00 PM GMT,120
Tue 30 Oct 2012 01:51:00 PM GMT,130
Tue 30 Oct 2012 01:52:00 PM GMT,340
Tue 30 Oct 2012 01:53:00 PM GMT,200
Tue 30 Oct 2012 01:54:00 PM GMT,120
Tue 30 Oct 2012 01:55:00 PM GMT,890
Tue 30 Oct 2012 01:56:00 PM GMT,124
Tue 30 Oct 2012 01:57:00 PM GMT,450
Tue 30 Oct 2012 01:58:00 PM GMT,120
Tue 30 Oct 2012 01:59:00 PM GMT,120
Tue 30 Oct 2012 02:00:00 PM GMT,200
Tue 30 Oct 2012 02:01:00 PM GMT,120
Tue 30 Oct 2012 02:02:00 PM GMT,340
Tue 30 Oct 2012 02:03:00 PM GMT,670
Tue 30 Oct 2012 02:04:00 PM GMT,990

Required: Group each 5 minutes entries together to one time bucket and perform a corresponding sum(Value) of the entries. E.g. All entries from '01:50:00 PM' (1351605000) till '01:54:00 PM' (1351605240) should be grouped together to '01:50:00 PM' (1351605000) time bucket, '01:55:00 PM' (1351605300) till '01:59:00 PM' (1351605540) to '01:55:00 PM' (1351605300) bucket and so on.

$ awk 'BEGIN {FS=OFS=","}  NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt

Output:

 
1351605000,910
1351605300,1704
1351605600,2320

Converting UNIX epoch timestamps to human readable format:

$ awk 'BEGIN {FS=OFS=","}  NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt  | awk '
BEGIN {FS=OFS=","} {$1=strftime("%c",$1)} {print}'

Output:

Tue 30 Oct 2012 01:50:00 PM GMT,910
Tue 30 Oct 2012 01:55:00 PM GMT,1704
Tue 30 Oct 2012 02:00:00 PM GMT,2320

Related posts:
- Grouping files using Awk in Bash shell
- Awk convert epoch to date format
- SUM and GROUP BY using Awk in UNIX post1 post2
- COUNT number of occurrences using Awk
- Awk Associative array

Awk - separate last field with spaces

2012-09-30T14:06:00.001+05:30

Each line of following file 'file.txt' actually has total 4 fields/columns. As you can see some of the values of the last field (4th field, header : Description) has spaces in them.

$ cat file.txt 
ID Name Active Description
2312 DEMO-1 1 Demo for VOD LIVE
1245 DEMO-4 1 LIVE
1002 CUST2 0 VOD Event
19000 DEMO-2 1 VOD
189 DEMO-3 1 Demo for LIVE

Required output:
We need to separate the 4 fields with a comma (,) delimiter such that all the values after 3rd field is considered as 4th field/column. i.e. Required output is something like this:

ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE

One way to achieve this is to add comma (,) to each of the first 3 fields of every line/row.

$ awk ' { for ( i=1;i<=3;i++ ) {$i=$i","} print }' file.txt

Output:

ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE

Any other alternative ? Feel free to post in the comment section, much appreciated. Thanks.

And to print the last field of the above input file one solution using UNIX cut command:

$ cut -d" " -f4- file.txt

Output:

Description
Demo for VOD LIVE
LIVE
VOD Event
VOD
Demo for LIVE

Compare two numeric fields of two files - UNIX awk

2012-09-26T23:15:00.002+05:30

Following example will show how we can use UNIX paste and join command with awk to find difference of two numeric columns of two similar files.

e.g. Input file:

$ cat file1.txt 
Aug-1,100
Aug-2,220
Aug-4,230
Aug-5,100

$ cat file2.txt 
Aug-1,100
Aug-2,120
Aug-4,400
Aug-5,250

Required output:
As you can see both file1.txt and file2.txt has got first field (comma separated) in the same order. We will have to find out the corresponding (2nd field of file1.txt - 2nd field of file2.txt) and the output required is something like this:

Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Solution#1:

$ paste -d "," file1.txt file2.txt
Aug-1,100,Aug-1,100
Aug-2,220,Aug-2,120
Aug-4,230,Aug-4,400
Aug-5,100,Aug-5,250

$ paste -d "," file1.txt file2.txt | awk 'BEGIN {FS=OFS=","} {print $1,$2,$NF,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Solution#2:

$ join -t, -1 1 -2 1 file1.txt file2.txt 
Aug-1,100,100
Aug-2,220,120
Aug-4,230,400
Aug-5,100,250

$ join -t, -1 1 -2 1 file1.txt file2.txt | awk ' BEGIN {FS=OFS=","} {print $0,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Print double quotes in unix Awk

2012-07-23T10:20:00.000+05:30

I have already described in one of the earlier posts on how to print string within single quote in Awk print statement. Here is how we can print strings without "double quotes" in Awk.

$ cat file.txt 
6289693505455 Plan_DAIL_30D_AA
6289693505475 Plan_DAIL_30D_AA
6289693505462 Plan_DAIL_30D_AB

Output required:

Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

i.e. Print the first field of the file within "double quotes".
Here are some of the alternatives:

$ awk '{print $2,$1}' file.txt 
Plan_DAIL_30D_AA 6289693505455
Plan_DAIL_30D_AA 6289693505475
Plan_DAIL_30D_AB 6289693505462

$ awk '{print $2,"\""$1"\""}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Assigning the quotes sequence to a variable x 
$ awk -v x="\"" '{print $2,x$1x}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Using octal code of double quotes
$ awk '{print $2,"\042"$1"\042"}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Using ASCII code of double quotes
$ awk '{print $2,"\x22"$1"\x22"}'  file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

Unix sort file ignoring first line

2012-07-12T10:30:00.000+05:30

Input file:

$ cat file.txt 
chrom:index:forward:reverse
chr01:13:1:2
chr03:12:1:4
chr01:3445:1:6
chr02:2311:3:1
chr13:23432:4:7
chr01:212:5:2
chr02:345:12:6
chr01:45:45:0

Output required:

 
chrom:index:forward:reverse
chr01:13:1:2
chr01:45:45:0
chr01:212:5:2
chr01:3445:1:6
chr02:345:12:6
chr02:2311:3:1
chr03:12:1:4
chr13:23432:4:7

i.e. Sort file.txt ignoring the header line (1st line).
Sort first field in ascending order (string)
Sort second field in ascending order (numeric)
One of the solution would be:

$ head -1 file.txt ; tail -n +2 file.txt | sort -t : -k1,1 -k2,2n

As you can see two commands are combined together to satisfy the output.
i) First part is printing the first line of file.txt
ii) Second part is printing the lines except first line and then sorting the lines based on first field (string) and second field (numeric sort).
Related posts:
- Unix sort file based on last field
- Sort strings by length using Awk
- Unix sort date in ddmmyyyy format using Awk

Run UNIX bash loop with nohup

2012-02-18T19:58:00.002+05:30

Suppose we have to run following UNIX bash shell loop construct under 'nohup'

$ for i in $(seq 20); do echo $i;./somescript.sh $i; done

Running the above loop under nohup directly will not work as 'nohup' expects a single-word command and its arguments. This is how we can run a UNIX bash shell loop construct using 'nohup'.

$ nohup sh -c 'for i in $(seq 20); do echo $i;./somescript.sh $i; done' &

From man page of 'sh'

-c               Read commands from the command_string operand instead of from the
standard input.  Special parameter 0 will be set from the command_name operand
and the positional parameters ($1, $2, etc.)  set from the remaining argument operands.

Print line number with unix less command

2011-10-28T09:32:00.002+05:30

Less command introduction:
As you know Unix less command writes the contents of a file onto the screen a page at a time and this is one of the utilities using which one can view the content of a file without opening it in an editor.
Press the [space-bar] if you want to see another page, and type [q] if you want to quit reading.

This is how we can print line number with Unix less command.

1)
From man page of CAT(1):

       -n, --number
              number all output lines

So,

$ cat -n file.txt | less

will print the line number in-front of each line.

2)
From man page of LESS(1):

       -N or --LINE-NUMBERS
              Causes a line number to be displayed at the beginning of each line in the display.

So,

$ less -N file.txt

will do the same as 1) above.

3) One can set the following to print line number with less by default:

$ export LESS='-RS#3NM~g'

Awk - convert epoch to date in the same file

2011-06-27T22:07:00.004+05:30

Example 1)
Input file:

$ cat test1.txt
2|Z|1219071600|AF|0
3|N|1219158000|AF|89
4|N|1220799600|AS|12
1|Z|1220886000|AS|67
5|N|1220972400|EU|23
6|R|1221058800|OC|89

Required output:

2|Z|Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

i.e. convert the UNIX epoch values on 3 rd field of the above file to standard human readable date format using Awk.

The Awk solution using "strftime" function:

$ awk 'BEGIN {FS=OFS="|"}{$3=strftime("%c",$3)} {print}' test1.txt

More about strftime format specifiers can be found here

I have created a python program to achieve the above output and the script is here

Example 2)
Input file:

$ cat test2.txt
2|Z|time:1219071600|AF|0
3|N|time:1219158000|AF|89
4|N|time:1220799600|AS|12
1|Z|time:1220886000|AS|67
5|N|time:1220972400|EU|23
6|R|time:1221058800|OC|89

Required output:

2|Z|time:Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|time:Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|time:Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|time:Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|time:Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|time:Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

The awk solution:

$ awk 'BEGIN {FS=OFS="|"}{$3="time:"strftime("%c",substr($3,6,10))} {print}' test2.txt

Filter line using Awk split function

2011-06-20T22:09:00.003+05:30

Input file "file.txt" has data in the following format:

$ cat file.txt
232323 90 /T/1382/8558/365p/133 100 234 679
S/1234 90 /N/1389/5000/365s/5000 800 134 679
792363 80 /T/1381/9858/365q/133 100 234 779
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Required output: From the above file print only the lines whose 3rd field has the value "5000" as the 4th field (separated by the delimiter "/") in it. i.e. required output:

S/1234 90 /N/1389/5000/365s/5000 800 134 679
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Using awk:

$ awk '{
    split($3,arr,"/")
    if(arr[4] == 5000) {
        print $0
        }
}' file.txt

The awk function split(s,a,sep) splits a string "s" into an awk array "a" using the delimiter "sep".

A simple UNIX bash script to solve this:

$ while read line
    do 
        thirdf=$(echo "$line" | awk '{print $3}')
        fourthf=$(echo "$thirdf" | awk -F "/" '{print $4}')
        [ "$fourthf" -eq 5000 ] && echo $line
done < file.txt

Bash insert newline after every 3 lines

2011-05-24T20:15:00.004+05:30

-----------
Input file:
-----------

$ cat file.txt
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80
NK 80

---------
Required:
---------
Insert a newline after every 3 lines of the above file. i.e. required output:

FR 24
AA 33
EE 34

EE 46
BE 30
AA 31

DE 90
AL 10
AA 50

FR 67
EE 94
AA 80

NK 80

----------
Solutions:
----------
a) Using awk:

$ awk '!( NR % 3 ) {$0 = $0"\n"} 1' file.txt

Which is same as:

$ awk '!( NR % 3 ) {$0 = $0"\n"} {print}' file.txt

b) A UNIX bash script to achieve this:

#!/bin/sh
c=0
while read line
    do
    ((s=c%3))
    if [ "$s" -eq 0 ]; then
        echo -e "\n$line"
    else
        echo "$line"
    fi
    ((c=c+1))
done < file.txt | sed '1d'

Following if-else block

 
    if [ "$s" -eq 0 ]; then
        echo -e "\n$line"
    else
        echo "$line"
    fi

can also be written as:

[ "$s" -eq 0 ] && echo -e "\n$line" || echo "$line"

---------------------------
Any other alternative ?
---------------------------
Feel free to comment any other alternative to achieve this, much appreciated, thanks.

Related posts:
- UNIX Bash while loop sum issue explained
- Difference between awk NR and FNR

Awk - replace n-th occurrence of pattern

2011-05-08T19:33:00.003+05:30

Input file:

$ cat file.txt
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80

Required:
Replace the "third" occurrence of first field "AA" with text "XX" in the above file.

Awk Solutions:

$ awk '$1=="AA" {
    count++
    if(count == 3){
        sub("AA","XX",$1)
    }
}
{print}' file.txt

Output:

FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
XX 50
FR 67
EE 94
AA 80

In-case you want to add 100 to the 2nd field of that line where first field "AA" has occurred for the 3rd time, here is a way:

$ awk '$1=="AA" {
    count++
    if(count == 3){
        $2=$2+100
    }
}
{print}' file.txt

Output:

FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 150
FR 67
EE 94
AA 80

Awk - print up-to nth occurrence of pattern

2011-04-28T22:20:00.004+05:30

Input file:

$ cat file.txt
FR 24
AA 33
AA 36
EE 34
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67

Required: Print the portion of the above file up-to the 3rd occurrence of the pattern "AA" as the first field.

$ awk '
    $1=="AA" {c++} {print $0}
    c==3 {exit}
' file.txt

Output:

FR 24
AA 33
AA 36
EE 34
BE 30
AA 31

Related posts:
- Print first occurrence of a set using Awk

Unix Bash script - Check if integer or not

2011-02-03T10:25:00.004+05:30

Please find below two UNIX bash scripts to test if the entered input is integer or not. Request all to suggest if any other alternatives to check this. Thanks in advance.
1) Script 1:


$ cat check-integer.sh
#!/bin/sh
#Check if input is integer or not

[ -z $1 ] && echo "No input, exiting .." && exit
[[ $1 = *[![:digit:]]* ]]  && echo "Not Integer" || echo "Integer"

Executing/testing it:


$ ./check-integer.sh     
No input, exiting ..

$ ./check-integer.sh 12
Integer

$ ./check-integer.sh 12.1
Not Integer

$ ./check-integer.sh testvar
Not Integer

2) Script 2:


#!/bin/sh
#Check if input is integer or not

[ -z $1 ] && echo "No input, exiting .." && exit
result=$(echo $1 | egrep ^[[:digit:]]+$)
if [ "$result" = "" ] ; then
    echo "Not Integer"
else
    echo "Integer"
fi

Executing/testing it:


$ ./check-integer.sh    
No input, exiting ..

$ ./check-integer.sh 12
Integer

$ ./check-integer.sh 12.1
Not Integer

$ ./check-integer.sh testvar
Not Integer

$ ./check-integer.sh 12U   
Not Integer

$ ./check-integer.sh U12
Not Integer

The if else in the second script can also be written using && and ||, something like:


[ "$result" = "" ] && echo "Not Integer" || echo "Integer"

Vim abbreviation not working - solution

2011-01-06T10:19:00.001+05:30

Abbreviations in vi editor are the shortcuts for strings. For example, if you frequently need to write the text "Change review not needed", you can have a shortcut like "crn" for this. So every-time you require to type "Change review not needed", you type crn (then a space) and your abbreviation will be expanded (in insert mode).

With my vi editor version:


VIM - Vi IMproved 7.1 (2007 May 12, compiled Jan  8 2009 02:24:14)

I created an abbreviation on my .vimrc:


$ tail -4 ~/.vimrc
:se shiftwidth=4
:se ts=4
:se paste
:ab crn Change review not needed

This abbreviation ("crn") was not working as expected and later I came to know its because of the "set paste" option set on my .vimrc. Vi abbreviations are disabled if the 'paste' option is on. I removed the ":se paste" from my .vimrc and the above abbreviation worked.

Another alternative to set an abbreviation in vim


:inoremap crn Change review not needed

This also require the 'paste' option to be unset/not set.

To unset an abbreviation in vi editor, type in ex mode:


:una crn

To see what all abbreviations are already set, type in ex mode:

:ab

For help on vi abbreviation type, type in ex mode:


:h iab
:h ab

Vi editor - directory modification time changes

2011-01-02T00:27:00.003+05:30

Why last modified time of a directory changes when any file inside the directory is viewed (i.e. opened in vi editor but not saved) ?
e.g.


#Created a directory named 'testdir'
$ mkdir testdir
$ ls -lrt | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:37 testdir

#Created a file named 'testfile.txt' inside directory 'testdir'
$ cd testdir/
$ touch testfile.txt
$ ls -l
total 0
-rw-r--r-- 1 root root 0 Dec  5 14:38 testfile.txt

#Modification time of the directory 'testdir' is changed to 'Dec  5 14:38' as expected.
$ ls -lrt ../ | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:38 testdir

#Opened the file 'testfile.txt' in Vi editor, but I quit using "!q", so nothing saved.
$ vi testfile.txt

#No Change in modification time of file 'testfile.txt'
$ ls -l
total 0
-rw-r--r-- 1 root root 0 Dec  5 14:38 testfile.txt

#But modification time of the directory 'testdir' is changed.
$ ls -lrt ../ | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:40 testdir

Its because when we open a file in vi, a hidden swap file (with .filename.swp extension) is created. The swap file is used for recovery of the vi session in case vi crashes (e.g. when system reboots, suppose you accidentally close your vi session with 'Ctrl q' etc). This swap file is deleted when we save and exit the file in vi editor. The creation of the swap file and later deletion of this swap file on normal vi exit is the cause of the change in modification time of a directory.

Bash - Concatenate with underscore in between

2010-12-28T10:28:00.006+05:30

Input file:


$ cat file.txt
X 2 10
X 2 11
Y 5 12
Z 3 11
X 6 78

Required: For each of the lines of the above file, we are required to create sub directories of the following naming convention inside the directory outdir/.

i.e. for line
X 2 10
we need to create the sub directory:
outdir/b_config_X_2

The bash script I wrote:


$ cat createdir.sh
#!/bin/sh

while read line
    do
        eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
        echo "$AID,$BID,$CID"
        dir=b_config_$AID_$BID
        mkdir -p outdir/$dir
done < file.txt

Executing it:


$ ./createdir.sh
X,2,10
X,2,11
Y,5,12
Z,3,11
X,6,78

Lets see what got created under outdir/


$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_2
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_3
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_5
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_6

To make it work, I did the following change in the above script:


dir=b_config_$AID\_$BID

So the modified script:


$ cat createdir.sh
#!/bin/sh

while read line
    do
        eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
        echo "$AID,$BID,$CID"
        dir=b_config_$AID\_$BID
        mkdir -p outdir/$dir
done < file.txt

After execution, it created the correct sub directories:


$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_2
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_6
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Y_5
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Z_3

The other alternative to concatenate values of two variables with underscore in between is:


dir=b_config_${AID}_${BID}

What is the use of the following Awk eval function ?


eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')

is a replacement of:


AID=$(echo $line | awk '{print $1}')
BID=$(echo $line | awk '{print $2}')
CID=$(echo $line | awk '{print $3}')

or

AID=$(echo $line | cut -d " " -f1)
BID=$(echo $line | cut -d " " -f2)
CID=$(echo $line | cut -d " " -f3)

Awk - Sum of multiple columns of file

2010-12-12T21:44:00.002+05:30

Input file:


$ cat file.txt
500:120:100
100:120:700
200:900:125
120:120:900

Required:
Compute the sum of individual columns of the above file. i.e. required output:


920:1260:1825

Awk solution - 1:


$ awk 'BEGIN {FS=OFS=":"}                                                   
    NR == 1 { n1 = $1; n2 = $2; n3 = $3;  next }
    { n1 += $1; n2 += $2; n3 += $3 }
END { print n1, n2, n3 }' file.txt

Output:


920:1260:1825

Awk solution - 2:


$ awk -F ":" '
{ for (i=1; i<=NF; ++i) sum[i] += $i; j=NF }
END { for (i=1; i <= j; ++i) printf "%s ", sum[i]; printf "\n"; }
' file.txt

Output:


920 1260 1825

And the solution for finding sum of numbers in each row of a file (i.e. horizontal sum) is here

A newbie tutorial on UNIX Awk utility

2010-11-22T10:02:00.006+05:30

Sharing a few Awk one liners (very basic and random) which will help the Awk newbies to gain a quick confidence on one of the beautiful UNIX tools - Awk. Some of my recommendations for learning UNIX Awk are mentioned at the bottom of this post. Have a great time with Awk.

Meaning of some of the Awk Built-in Variables used below:


NF : Number of fields in current line/record
NR : Ordial number of current line/record
FS : Field Separator (Also -F can be used)
OFS : Output Field Separator (default=blank)
FILENAME : Name of current input file

All of following Awk one liners are based on the following input file 'test1.txt'


$ cat test1.txt
Continent:Val
AS:12000
AF:9800
AS:12300
NA:3400
OC:12000
AF:500
AS:1000

A)
#Print 'line number' NR and 'Number of fields' NF for each line


$ awk -F ":" '{print NR,NF}' test1.txt
1 2
2 2
3 2
4 2
5 2
6 2
7 2
8 2

B)
#Print first field, colon delimited


$ awk -F ":" '{print $1}' test1.txt
Continent
AS
AF
AS
NA
OC
AF
AS

C)
#Same as B, but excluding the 'first line' (NR!=1)


$ awk -F ":" 'NR!=1 {print $1}' test1.txt
AS
AF
AS
NA
OC
AF
AS

D)
#Same as B but only for line number 1 (NR==1)


$ awk -F ":" 'NR==1 {print $1}' test1.txt
Continent

E)
#Print first and second field


$ awk -F ":" 'NR!=1 {print $1,$2}' test1.txt
AS 12000
AF 9800
AS 12300
NA 3400
OC 12000
AF 500
AS 1000

F)
#Setting output field separator as pipe


$ awk -F ":" 'BEGIN{OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

G)
#FS and OFS can be included in BEGIN section


$ awk 'BEGIN{FS=":"; OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

H)
#Anything on BEGIN executes first


$ awk 'BEGIN{FS=":"; OFS="|"; print "Con|SomeVal"} NR!=1 {print $1,$2}' test1.txt
Con|SomeVal
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

I)
#Printing FILENAME, will be printed for all the lines


$ awk -F ":" '{print FILENAME}' test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt

#Same as above but printing only last instance using END clause


$ awk -F ":" ' END {print FILENAME}' test1.txt
test1.txt

J)
#Revisiting NF, number of fields in each line


$ awk -F ":" '{print NF}' test1.txt
2
2
2
2
2
2
2
2

K)
#Printing the last field of the file, same as printing $2 as there are only 2 fields


$ awk -F ":" '{print $NF}' test1.txt
Val
12000
9800
12300
3400
12000
500
1000

L)
#Matching, printing lines begin with "AS"


$ awk -F ":" '/^AS/' test1.txt
AS:12000
AS:12300
AS:1000

#'Not condition' of the L


$ awk -F ":" '!/^AS/' test1.txt
Continent:Val
AF:9800
NA:3400
OC:12000
AF:500

#Direct matching, first field as "AS"


$ awk -F ":" '$1=="AS"' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

#$0 prints the full line, same as {print}


$ awk -F ":" '$1=="AS" {print $0}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print}' test1.txt
AS:12000
AS:12300
AS:1000

M)
# 'Or' condition


$ awk -F ":" '$1=="AS" || $1=="OC" {print}' test1.txt
AS:12000
AS:12300
OC:12000
AS:1000

# 'Or' and 'AND' together


$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print}' test1.txt
AS:12000
AS:12300
OC:12000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print $2-10000}' test1.txt
2000
2300
2000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print ($2-($2%100))}' test1.txt
12000
12300
12000

N)
#Partial Matching


$ awk -F ":" '$1 ~ /AS/ {print}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1 ~ /A/ {print}' test1.txt
AS:12000
AF:9800
AS:12300
NA:3400
AF:500
AS:1000

O)
#Reading from STDOUT


$ cat test1.txt | awk -F ":" '!/Continent/ {print $1}' | sort | uniq
AF
AS
NA
OC

P)
#Add value 1000 to the 2nd field, where first field is "AF" and then print the output file


$ awk -F ":" '$1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF 10800
AS:12300
NA:3400
OC:12000
AF 1500
AS:1000

#As no OFS is mentioned above, by default, OFS is blank, specifying it now


$ awk -F ":" 'BEGIN {OFS=":"} $1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF:10800
AS:12300
NA:3400
OC:12000
AF:1500
AS:1000

Q)
#Sum of 2nd fields, exclude first line


$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum}' test1.txt
51000

#If END is not mentioned ?


$ awk -F ":" 'NR!=1 {sum+=$NF} {print sum}' test1.txt
12000
21800
34100
37500
49500
50000
51000

#Average of 2nd field, as first field is excluded, (NR-1) instead of NR for total number of items


$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum/(NR-1)}' test1.txt
7285.71

$ awk -F ":" 'NR!=1 && $1=="AS" {sum+=$NF} END {print sum}' test1.txt
25300

R) 'Group by' and 'Count' using associative array in awk


$ awk ' BEGIN {FS=OFS=":"}
NR==1 {print "Continent:Count:Sum(val)"}
NR>1 {a[$1]++;b[$1]=b[$1]+$2} END{for (i in a) print i,a[i],b[i]}' test1.txt

Continent:Count:Sum(val)
OC:1:12000
NA:1:3400
AF:2:10300
AS:3:25300

#Count of each of the continents(1st field)


$ awk -F ":" '
NR==1 {next}
NR>1 {a[$1]++} END{for (i in a) print i,a[i]}' test1.txt
OC 1
NA 1
AF 2
AS 3

#An alternative


$ awk -F ":" 'NR!=1 {print $1}' test1.txt  | sort | uniq -c
   2 AF
   3 AS
   1 NA
   1 OC

#Reading from STDOUT


$ cat test1.txt | awk 'BEGIN {OFS=":"} {print NR-1,$0}'
0:Continent:Val
1:AS:12000
2:AF:9800
3:AS:12300
4:NA:3400
5:OC:12000
6:AF:500
7:AS:1000

S)
#Accessing external variable in awk:


$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

#Suppose value of filterval is "AS"


$ filterval="AS"
$ echo $filterval
AS

#Accessing value of variable filterval inside Awk


$ awk -F ":" -v con=$filterval '$1==con {print $2}' test1.txt
12000
12300
1000

#Set 2nd value as 0 where first field is "AS"


$ awk -F ":" 'BEGIN {OFS=":"} $1=="AS" {$2=0} {print}' test1.txt
Continent:Val
AS:0
AF:9800
AS:0
NA:3400
OC:12000
AF:500
AS:0

My recommendation for AWK:
- A complete tutorial on AWK from vectorsite
- Awk One line page
- Peteris Krumins has done a great job by explaining each and every one lines in details, a must read : part1 part2

Related posts from my blog:
- A post on awk substr function
- Difference between awk NR and FNR variables

Sort file based on last field - UNIX

2010-11-17T20:20:00.002+05:30

From SORT(1) man page:


-n, --numeric-sort
compare according to string numerical value

-k, --key=POS1[,POS2]
start a key at POS1, end it at POS2 (origin 1)

-t, --field-separator=SEP
use SEP instead of non-blank to blank transition

1) File with same number of fields in each line


$ cat file.txt
IN,90,453
US,12,120
NZ,89,200
WI,20,500
TS,12,124


$ sort -t, -n -k3 file.txt
US,12,120
TS,12,124
NZ,89,200
IN,90,453
WI,20,500

2) File with uneven number of fields in each line


$ cat file.txt
IN,90,453
US,12,1,120
NZ,89,200
WI,500
TS,12,124

Solution : The idea is join the last field(column) of each line at the beginning of each of the lines (with a different delimiter, in this case I am using pipe "|")


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt
453|IN,90,453
120|US,12,1,120
200|NZ,89,200
500|WI,500
124|TS,12,124

Then sort on 1st field with pipe ("|") delimited.


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1
120|US,12,1,120
124|TS,12,124
200|NZ,89,200
453|IN,90,453
500|WI,500

Then discard the first field i.e. print the 2nd field with pipe ("|") delimited.


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | awk -F "|" '{print $NF}'
US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500

Which is same as:


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | awk -F "|" '{print $2}'

or

$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | cut -d"|" -f2

A question to unstableme readers:
- Is there a better way to directly mention the last field number (as counting field number incase line has huge number of fields is never going to be easy) ? Thanks in advance, much appreciated.

Related posts:
- UNIX sort strings by string length using awk in Bash
- Changing temporary directory for UNIX sort command
- Sort date in ddmmyyyy format using awk