UNIX BASH scripting

Print line number with unix less command

2011-10-28T09:32:00.002+05:30

Less command introduction:
As you know Unix less command writes the contents of a file onto the screen a page at a time and this is one of the utilities using which one can view the content of a file without opening it in an editor.
Press the [space-bar] if you want to see another page, and type [q] if you want to quit reading.

This is how we can print line number with Unix less command.

1)
From man page of CAT(1):

       -n, --number
              number all output lines

So,

$ cat -n file.txt | less

will print the line number in-front of each line.

2)
From man page of LESS(1):

       -N or --LINE-NUMBERS
              Causes a line number to be displayed at the beginning of each line in the display.

So,

$ less -N file.txt

will do the same as 1) above.

3) One can set the following to print line number with less by default:

$ export LESS='-RS#3NM~g'

Awk - convert epoch to date in the same file

2011-06-27T22:07:00.004+05:30

Example 1)
Input file:

$ cat test1.txt
2|Z|1219071600|AF|0
3|N|1219158000|AF|89
4|N|1220799600|AS|12
1|Z|1220886000|AS|67
5|N|1220972400|EU|23
6|R|1221058800|OC|89

Required output:

2|Z|Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

i.e. convert the UNIX epoch values on 3 rd field of the above file to standard human readable date format using Awk.

The Awk solution using "strftime" function:

$ awk 'BEGIN {FS=OFS="|"}{$3=strftime("%c",$3)} {print}' test1.txt

More about strftime format specifiers can be found here

I have created a python program to achieve the above output and the script is here

Example 2)
Input file:

$ cat test2.txt
2|Z|time:1219071600|AF|0
3|N|time:1219158000|AF|89
4|N|time:1220799600|AS|12
1|Z|time:1220886000|AS|67
5|N|time:1220972400|EU|23
6|R|time:1221058800|OC|89

Required output:

2|Z|time:Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|time:Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|time:Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|time:Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|time:Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|time:Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

The awk solution:

$ awk 'BEGIN {FS=OFS="|"}{$3="time:"strftime("%c",substr($3,6,10))} {print}' test2.txt

Filter line using Awk split function

2011-06-20T22:09:00.003+05:30

Input file "file.txt" has data in the following format:

$ cat file.txt
232323 90 /T/1382/8558/365p/133 100 234 679
S/1234 90 /N/1389/5000/365s/5000 800 134 679
792363 80 /T/1381/9858/365q/133 100 234 779
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Required output: From the above file print only the lines whose 3rd field has the value "5000" as the 4th field (separated by the delimiter "/") in it. i.e. required output:

S/1234 90 /N/1389/5000/365s/5000 800 134 679
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Using awk:

$ awk '{
    split($3,arr,"/")
    if(arr[4] == 5000) {
        print $0
        }
}' file.txt

The awk function split(s,a,sep) splits a string "s" into an awk array "a" using the delimiter "sep".

A simple UNIX bash script to solve this:

$ while read line
    do 
        thirdf=$(echo "$line" | awk '{print $3}')
        fourthf=$(echo "$thirdf" | awk -F "/" '{print $4}')
        [ "$fourthf" -eq 5000 ] && echo $line
done < file.txt

Bash insert newline after every 3 lines

2011-05-24T20:15:00.004+05:30

-----------
Input file:
-----------

$ cat file.txt
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80
NK 80

---------
Required:
---------
Insert a newline after every 3 lines of the above file. i.e. required output:

FR 24
AA 33
EE 34

EE 46
BE 30
AA 31

DE 90
AL 10
AA 50

FR 67
EE 94
AA 80

NK 80

----------
Solutions:
----------
a) Using awk:

$ awk '!( NR % 3 ) {$0 = $0"\n"} 1' file.txt

Which is same as:

$ awk '!( NR % 3 ) {$0 = $0"\n"} {print}' file.txt

b) A UNIX bash script to achieve this:

#!/bin/sh
c=0
while read line
    do
    ((s=c%3))
    if [ "$s" -eq 0 ]; then
        echo -e "\n$line"
    else
        echo "$line"
    fi
    ((c=c+1))
done < file.txt | sed '1d'

Following if-else block

 
    if [ "$s" -eq 0 ]; then
        echo -e "\n$line"
    else
        echo "$line"
    fi

can also be written as:

[ "$s" -eq 0 ] && echo -e "\n$line" || echo "$line"

---------------------------
Any other alternative ?
---------------------------
Feel free to comment any other alternative to achieve this, much appreciated, thanks.

Related posts:
- UNIX Bash while loop sum issue explained
- Difference between awk NR and FNR

Awk - replace n-th occurrence of pattern

2011-05-08T19:33:00.003+05:30

Input file:

$ cat file.txt
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80

Required:
Replace the "third" occurrence of first field "AA" with text "XX" in the above file.

Awk Solutions:

$ awk '$1=="AA" {
    count++
    if(count == 3){
        sub("AA","XX",$1)
    }
}
{print}' file.txt

Output:

FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
XX 50
FR 67
EE 94
AA 80

In-case you want to add 100 to the 2nd field of that line where first field "AA" has occurred for the 3rd time, here is a way:

$ awk '$1=="AA" {
    count++
    if(count == 3){
        $2=$2+100
    }
}
{print}' file.txt

Output:

FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 150
FR 67
EE 94
AA 80

Awk - print up-to nth occurrence of pattern

2011-04-28T22:20:00.004+05:30

Input file:

$ cat file.txt
FR 24
AA 33
AA 36
EE 34
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67

Required: Print the portion of the above file up-to the 3rd occurrence of the pattern "AA" as the first field.

$ awk '
    $1=="AA" {c++} {print $0}
    c==3 {exit}
' file.txt

Output:

FR 24
AA 33
AA 36
EE 34
BE 30
AA 31

Related posts:
- Print first occurrence of a set using Awk

Unix Bash script - Check if integer or not

2011-02-03T10:25:00.004+05:30

Please find below two UNIX bash scripts to test if the entered input is integer or not. Request all to suggest if any other alternatives to check this. Thanks in advance.
1) Script 1:


$ cat check-integer.sh
#!/bin/sh
#Check if input is integer or not

[ -z $1 ] && echo "No input, exiting .." && exit
[[ $1 = *[![:digit:]]* ]]  && echo "Not Integer" || echo "Integer"

Executing/testing it:


$ ./check-integer.sh     
No input, exiting ..

$ ./check-integer.sh 12
Integer

$ ./check-integer.sh 12.1
Not Integer

$ ./check-integer.sh testvar
Not Integer

2) Script 2:


#!/bin/sh
#Check if input is integer or not

[ -z $1 ] && echo "No input, exiting .." && exit
result=$(echo $1 | egrep ^[[:digit:]]+$)
if [ "$result" = "" ] ; then
    echo "Not Integer"
else
    echo "Integer"
fi

Executing/testing it:


$ ./check-integer.sh    
No input, exiting ..

$ ./check-integer.sh 12
Integer

$ ./check-integer.sh 12.1
Not Integer

$ ./check-integer.sh testvar
Not Integer

$ ./check-integer.sh 12U   
Not Integer

$ ./check-integer.sh U12
Not Integer

The if else in the second script can also be written using && and ||, something like:


[ "$result" = "" ] && echo "Not Integer" || echo "Integer"

Vim abbreviation not working - solution

2011-01-06T10:19:00.001+05:30

Abbreviations in vi editor are the shortcuts for strings. For example, if you frequently need to write the text "Change review not needed", you can have a shortcut like "crn" for this. So every-time you require to type "Change review not needed", you type crn (then a space) and your abbreviation will be expanded (in insert mode).

With my vi editor version:


VIM - Vi IMproved 7.1 (2007 May 12, compiled Jan  8 2009 02:24:14)

I created an abbreviation on my .vimrc:


$ tail -4 ~/.vimrc
:se shiftwidth=4
:se ts=4
:se paste
:ab crn Change review not needed

This abbreviation ("crn") was not working as expected and later I came to know its because of the "set paste" option set on my .vimrc. Vi abbreviations are disabled if the 'paste' option is on. I removed the ":se paste" from my .vimrc and the above abbreviation worked.

Another alternative to set an abbreviation in vim


:inoremap crn Change review not needed

This also require the 'paste' option to be unset/not set.

To unset an abbreviation in vi editor, type in ex mode:


:una crn

To see what all abbreviations are already set, type in ex mode:

:ab

For help on vi abbreviation type, type in ex mode:


:h iab
:h ab

Vi editor - directory modification time changes

2011-01-02T00:27:00.003+05:30

Why last modified time of a directory changes when any file inside the directory is viewed (i.e. opened in vi editor but not saved) ?
e.g.


#Created a directory named 'testdir'
$ mkdir testdir
$ ls -lrt | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:37 testdir

#Created a file named 'testfile.txt' inside directory 'testdir'
$ cd testdir/
$ touch testfile.txt
$ ls -l
total 0
-rw-r--r-- 1 root root 0 Dec  5 14:38 testfile.txt

#Modification time of the directory 'testdir' is changed to 'Dec  5 14:38' as expected.
$ ls -lrt ../ | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:38 testdir

#Opened the file 'testfile.txt' in Vi editor, but I quit using "!q", so nothing saved.
$ vi testfile.txt

#No Change in modification time of file 'testfile.txt'
$ ls -l
total 0
-rw-r--r-- 1 root root 0 Dec  5 14:38 testfile.txt

#But modification time of the directory 'testdir' is changed.
$ ls -lrt ../ | tail -1
drwxr-xr-x 2 root root      4096 Dec  5 14:40 testdir

Its because when we open a file in vi, a hidden swap file (with .filename.swp extension) is created. The swap file is used for recovery of the vi session in case vi crashes (e.g. when system reboots, suppose you accidentally close your vi session with 'Ctrl q' etc). This swap file is deleted when we save and exit the file in vi editor. The creation of the swap file and later deletion of this swap file on normal vi exit is the cause of the change in modification time of a directory.

Bash - Concatenate with underscore in between

2010-12-28T10:28:00.006+05:30

Input file:


$ cat file.txt
X 2 10
X 2 11
Y 5 12
Z 3 11
X 6 78

Required: For each of the lines of the above file, we are required to create sub directories of the following naming convention inside the directory outdir/.

i.e. for line
X 2 10
we need to create the sub directory:
outdir/b_config_X_2

The bash script I wrote:


$ cat createdir.sh
#!/bin/sh

while read line
    do
        eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
        echo "$AID,$BID,$CID"
        dir=b_config_$AID_$BID
        mkdir -p outdir/$dir
done < file.txt

Executing it:


$ ./createdir.sh
X,2,10
X,2,11
Y,5,12
Z,3,11
X,6,78

Lets see what got created under outdir/


$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_2
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_3
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_5
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_6

To make it work, I did the following change in the above script:


dir=b_config_$AID\_$BID

So the modified script:


$ cat createdir.sh
#!/bin/sh

while read line
    do
        eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
        echo "$AID,$BID,$CID"
        dir=b_config_$AID\_$BID
        mkdir -p outdir/$dir
done < file.txt

After execution, it created the correct sub directories:


$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_2
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_6
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Y_5
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Z_3

The other alternative to concatenate values of two variables with underscore in between is:


dir=b_config_${AID}_${BID}

What is the use of the following Awk eval function ?


eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')

is a replacement of:


AID=$(echo $line | awk '{print $1}')
BID=$(echo $line | awk '{print $2}')
CID=$(echo $line | awk '{print $3}')

or

AID=$(echo $line | cut -d " " -f1)
BID=$(echo $line | cut -d " " -f2)
CID=$(echo $line | cut -d " " -f3)

Awk - Sum of multiple columns of file

2010-12-12T21:44:00.002+05:30

Input file:


$ cat file.txt
500:120:100
100:120:700
200:900:125
120:120:900

Required:
Compute the sum of individual columns of the above file. i.e. required output:


920:1260:1825

Awk solution - 1:


$ awk 'BEGIN {FS=OFS=":"}                                                   
    NR == 1 { n1 = $1; n2 = $2; n3 = $3;  next }
    { n1 += $1; n2 += $2; n3 += $3 }
END { print n1, n2, n3 }' file.txt

Output:


920:1260:1825

Awk solution - 2:


$ awk -F ":" '
{ for (i=1; i<=NF; ++i) sum[i] += $i; j=NF }
END { for (i=1; i <= j; ++i) printf "%s ", sum[i]; printf "\n"; }
' file.txt

Output:


920 1260 1825

And the solution for finding sum of numbers in each row of a file (i.e. horizontal sum) is here

A newbie tutorial on UNIX Awk utility

2010-11-22T10:02:00.006+05:30

Sharing a few Awk one liners (very basic and random) which will help the Awk newbies to gain a quick confidence on one of the beautiful UNIX tools - Awk. Some of my recommendations for learning UNIX Awk are mentioned at the bottom of this post. Have a great time with Awk.

Meaning of some of the Awk Built-in Variables used below:


NF : Number of fields in current line/record
NR : Ordial number of current line/record
FS : Field Separator (Also -F can be used)
OFS : Output Field Separator (default=blank)
FILENAME : Name of current input file

All of following Awk one liners are based on the following input file 'test1.txt'


$ cat test1.txt
Continent:Val
AS:12000
AF:9800
AS:12300
NA:3400
OC:12000
AF:500
AS:1000

A)
#Print 'line number' NR and 'Number of fields' NF for each line


$ awk -F ":" '{print NR,NF}' test1.txt
1 2
2 2
3 2
4 2
5 2
6 2
7 2
8 2

B)
#Print first field, colon delimited


$ awk -F ":" '{print $1}' test1.txt
Continent
AS
AF
AS
NA
OC
AF
AS

C)
#Same as B, but excluding the 'first line' (NR!=1)


$ awk -F ":" 'NR!=1 {print $1}' test1.txt
AS
AF
AS
NA
OC
AF
AS

D)
#Same as B but only for line number 1 (NR==1)


$ awk -F ":" 'NR==1 {print $1}' test1.txt
Continent

E)
#Print first and second field


$ awk -F ":" 'NR!=1 {print $1,$2}' test1.txt
AS 12000
AF 9800
AS 12300
NA 3400
OC 12000
AF 500
AS 1000

F)
#Setting output field separator as pipe


$ awk -F ":" 'BEGIN{OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

G)
#FS and OFS can be included in BEGIN section


$ awk 'BEGIN{FS=":"; OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

H)
#Anything on BEGIN executes first


$ awk 'BEGIN{FS=":"; OFS="|"; print "Con|SomeVal"} NR!=1 {print $1,$2}' test1.txt
Con|SomeVal
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

I)
#Printing FILENAME, will be printed for all the lines


$ awk -F ":" '{print FILENAME}' test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt

#Same as above but printing only last instance using END clause


$ awk -F ":" ' END {print FILENAME}' test1.txt
test1.txt

J)
#Revisiting NF, number of fields in each line


$ awk -F ":" '{print NF}' test1.txt
2
2
2
2
2
2
2
2

K)
#Printing the last field of the file, same as printing $2 as there are only 2 fields


$ awk -F ":" '{print $NF}' test1.txt
Val
12000
9800
12300
3400
12000
500
1000

L)
#Matching, printing lines begin with "AS"


$ awk -F ":" '/^AS/' test1.txt
AS:12000
AS:12300
AS:1000

#'Not condition' of the L


$ awk -F ":" '!/^AS/' test1.txt
Continent:Val
AF:9800
NA:3400
OC:12000
AF:500

#Direct matching, first field as "AS"


$ awk -F ":" '$1=="AS"' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

#$0 prints the full line, same as {print}


$ awk -F ":" '$1=="AS" {print $0}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print}' test1.txt
AS:12000
AS:12300
AS:1000

M)
# 'Or' condition


$ awk -F ":" '$1=="AS" || $1=="OC" {print}' test1.txt
AS:12000
AS:12300
OC:12000
AS:1000

# 'Or' and 'AND' together


$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print}' test1.txt
AS:12000
AS:12300
OC:12000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print $2-10000}' test1.txt
2000
2300
2000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print ($2-($2%100))}' test1.txt
12000
12300
12000

N)
#Partial Matching


$ awk -F ":" '$1 ~ /AS/ {print}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1 ~ /A/ {print}' test1.txt
AS:12000
AF:9800
AS:12300
NA:3400
AF:500
AS:1000

O)
#Reading from STDOUT


$ cat test1.txt | awk -F ":" '!/Continent/ {print $1}' | sort | uniq
AF
AS
NA
OC

P)
#Add value 1000 to the 2nd field, where first field is "AF" and then print the output file


$ awk -F ":" '$1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF 10800
AS:12300
NA:3400
OC:12000
AF 1500
AS:1000

#As no OFS is mentioned above, by default, OFS is blank, specifying it now


$ awk -F ":" 'BEGIN {OFS=":"} $1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF:10800
AS:12300
NA:3400
OC:12000
AF:1500
AS:1000

Q)
#Sum of 2nd fields, exclude first line


$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum}' test1.txt
51000

#If END is not mentioned ?


$ awk -F ":" 'NR!=1 {sum+=$NF} {print sum}' test1.txt
12000
21800
34100
37500
49500
50000
51000

#Average of 2nd field, as first field is excluded, (NR-1) instead of NR for total number of items


$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum/(NR-1)}' test1.txt
7285.71

$ awk -F ":" 'NR!=1 && $1=="AS" {sum+=$NF} END {print sum}' test1.txt
25300

R) 'Group by' and 'Count' using associative array in awk


$ awk ' BEGIN {FS=OFS=":"}
NR==1 {print "Continent:Count:Sum(val)"}
NR>1 {a[$1]++;b[$1]=b[$1]+$2} END{for (i in a) print i,a[i],b[i]}' test1.txt

Continent:Count:Sum(val)
OC:1:12000
NA:1:3400
AF:2:10300
AS:3:25300

#Count of each of the continents(1st field)


$ awk -F ":" '
NR==1 {next}
NR>1 {a[$1]++} END{for (i in a) print i,a[i]}' test1.txt
OC 1
NA 1
AF 2
AS 3

#An alternative


$ awk -F ":" 'NR!=1 {print $1}' test1.txt  | sort | uniq -c
   2 AF
   3 AS
   1 NA
   1 OC

#Reading from STDOUT


$ cat test1.txt | awk 'BEGIN {OFS=":"} {print NR-1,$0}'
0:Continent:Val
1:AS:12000
2:AF:9800
3:AS:12300
4:NA:3400
5:OC:12000
6:AF:500
7:AS:1000

S)
#Accessing external variable in awk:


$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

#Suppose value of filterval is "AS"


$ filterval="AS"
$ echo $filterval
AS

#Accessing value of variable filterval inside Awk


$ awk -F ":" -v con=$filterval '$1==con {print $2}' test1.txt
12000
12300
1000

#Set 2nd value as 0 where first field is "AS"


$ awk -F ":" 'BEGIN {OFS=":"} $1=="AS" {$2=0} {print}' test1.txt
Continent:Val
AS:0
AF:9800
AS:0
NA:3400
OC:12000
AF:500
AS:0

My recommendation for AWK:
- A complete tutorial on AWK from vectorsite
- Awk One line page
- Peteris Krumins has done a great job by explaining each and every one lines in details, a must read : part1 part2

Related posts from my blog:
- A post on awk substr function
- Difference between awk NR and FNR variables

Sort file based on last field - UNIX

2010-11-17T20:20:00.002+05:30

From SORT(1) man page:


-n, --numeric-sort
compare according to string numerical value

-k, --key=POS1[,POS2]
start a key at POS1, end it at POS2 (origin 1)

-t, --field-separator=SEP
use SEP instead of non-blank to blank transition

1) File with same number of fields in each line


$ cat file.txt
IN,90,453
US,12,120
NZ,89,200
WI,20,500
TS,12,124


$ sort -t, -n -k3 file.txt
US,12,120
TS,12,124
NZ,89,200
IN,90,453
WI,20,500

2) File with uneven number of fields in each line


$ cat file.txt
IN,90,453
US,12,1,120
NZ,89,200
WI,500
TS,12,124

Solution : The idea is join the last field(column) of each line at the beginning of each of the lines (with a different delimiter, in this case I am using pipe "|")


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt
453|IN,90,453
120|US,12,1,120
200|NZ,89,200
500|WI,500
124|TS,12,124

Then sort on 1st field with pipe ("|") delimited.


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1
120|US,12,1,120
124|TS,12,124
200|NZ,89,200
453|IN,90,453
500|WI,500

Then discard the first field i.e. print the 2nd field with pipe ("|") delimited.


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | awk -F "|" '{print $NF}'
US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500

Which is same as:


$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | awk -F "|" '{print $2}'

or

$ awk 'BEGIN {FS=","; OFS="|"}{print $NF,$0}' file.txt | sort -n -t"|" -k1 | cut -d"|" -f2

A question to unstableme readers:
- Is there a better way to directly mention the last field number (as counting field number incase line has huge number of fields is never going to be easy) ? Thanks in advance, much appreciated.

Related posts:
- UNIX sort strings by string length using awk in Bash
- Changing temporary directory for UNIX sort command
- Sort date in ddmmyyyy format using awk

Vi open file at required line number

2010-11-15T18:19:00.003+05:30

Sometime we are required to open a file in vim and then need to go to a particular line number in order to check something or to edit something. This is how we can open a file in vim and directly put the cursor on the required line of our interest.

Way1:
Using vim -c option:

From VIM(1) man page:


-c {command}
{command} will be executed after the first file has been read.  {command} is interpreted as an
 Ex command.  If the {command} contains spaces it must be enclosed in double quotes (this
depends on the shell  that is used).  Example: Vim "+set si" main.c

Following command will open file 'run_parallel_qs.py' in vim with the cursor on line number 23.


$ vim -c 23 run_parallel_qs.py

Way2:
One can use the + to specify the cursor starting point.


$ vim run_parallel_qs.py +23

which is same as


$ vim +23 run_parallel_qs.py

Without line number, i.e. the following command will open the file in vim and will put the cursor on last line


$ vim  run_parallel_queries.py +

And as we know just opening a file in vi/vim, the cursor will be on line number one.

Related UNIX vim editor tips:
- Open file under cursor in Vi Editor
- A short tutorial on using tab window in Vi editor
- Pull word under cursor in Vim
- Start a new shell from Vi editor
- How to edit remote file using ssh in Vi editor

Check equality of multiple numbers - awk

2010-10-22T14:04:00.005+05:30

My input file 'file.txt' contains 4 values of a certain metric for each of the following 'Continents'.


$ cat file.txt
Continent Val1 Val2 Val3 Val4
AS 440518 440518 440516 440516
AF 253317 253317 253315 253317
EU 245397 245397 245397 245397
OC 226410 226410 226410 226410
NA 221961 221961 221962 221961

Required : I was required to find out only those 'Continents' for which 'all' values are 'same'.

Solutions:

1) Using awk:


$ awk '
    /^Continent/ {print $1; next}
    $2==$3 && $3==$4 && $4==$5 {print $1}
' file.txt

Output:


Continent
EU
OC

2) Wrote this python program using python 'sets' (Unordered collections of unique elements) to achieve the same. Something like:


from sets import Set
for line in open("file.txt"):
    if line.startswith('Continent'):
        print line.split()[0]
    firstfield = line.split()[0]
    remaining = line.split()[1:]
    vals = Set(remaining)
    if len(vals) == 1:
        print firstfield

Executing it:


$ python printequal.py
Continent
EU
OC

3) Any other solution using Bash, Awk or any other scripting languages ? Readers, please put your solutions here in the comment section. Much appreciated.

Related:
- Bash function to compare multiple numbers equality

Print future date using unix date command

2010-10-20T14:28:00.005+05:30

I have already discussed in one of my earlier post about how we can print future and past dates using UNIX date command, revisiting it again to show a very interesting date that we are going to witness today.

From DATE(1) man pages, some of the options used in this example are:


%H     hour (00..23)
%M     minute (00..59)
%d     day of month (e.g, 01)
%m     month (01..12)
%Y     year

This is an interesting time in 2010, 8:10 PM of 20th Oct 2010, it going to be 20:10 20/10 2010. Lets try to print this using UNIX date command.

Hope you found it interesting !

Awk - Print particular instances of a file

2010-10-12T22:23:00.004+05:30

I am requesting everyone to provide a better alternative (in any scripting language) to this problem. Thanks in advance.

My input file has the following format:
- A instance is a combination of 'h' 'v' and one or more 'i' lines.
- All lines starting with 'h' are header lines and the 3rd field in that line is the 'header number'.
- All lines starting with 'v' are version lines and the second field in that line is the 'version number'.


$ cat file.txt
h,1,100
v,1
i,rt,200
i,rt,210
i,rt,810
h,1,101
v,5
i,rt,500
i,rt,700
h,1,100
v,2
i,rt,100
i,rt,910
h,1,500
v,1
i,rt,190
h,1,100
v,1
i,rt,900
i,rt,210
h,1,300
v,1
i,rt,800
i,rt,210

Required:
- Print all the 'i' lines associated with header number '100' and version number '1'
i.e. required output:


i,rt,200
i,rt,210
i,rt,810
i,rt,900
i,rt,210

The quick solution I can think about is to associate 'header number' and 'version number' with all 'i' lines


$ awk -F "," '$1=="h" {h_value=$NF}
    $1=="v" {v_value=$NF}
    $1=="i" {print h_value,v_value,$0}
' file.txt

Output:


100 1 i,rt,200
100 1 i,rt,210
100 1 i,rt,810
101 5 i,rt,500
101 5 i,rt,700
100 2 i,rt,100
100 2 i,rt,910
500 1 i,rt,190
100 1 i,rt,900
100 1 i,rt,210
300 1 i,rt,800
300 1 i,rt,210

And then print only the lines with header number=100 and version number=1.


$ awk -F "," '$1=="h" {h_value=$NF}
    $1=="v" {v_value=$NF}
    $1=="i" {print h_value,v_value,$0}
' file.txt | awk '$1==100 && $2==1 {print $NF}'

Output:


i,rt,200
i,rt,210
i,rt,810
i,rt,900
i,rt,210

I am sure there can be a better solution to this problem. Readers, please put your solutions here in the comment section. Much appreciated.

Related posts:
- Count instances without specific line in UNIX using Python
- Print last instance of a file in UNIX
- Print first few instances of a file - python

UNIX look command to print lines

2010-09-05T18:02:00.002+05:30

Let me introduce a simple UNIX/Linux command which is very useful but not that popular.
Its called 'look'. And as the name suggests, this command displays lines beginning with a given string.

Usage:


look string <file>

The look utility displays any lines in 'file' which contain 'string' as a prefix. This can be a simple alternative of using


$ grep "^string" <file>

If file is not specified, the file /usr/share/dict/words is used, only alphanumeric characters are compared and the case of alphabetic characters is ignored.
And I feel this lookup of dictionary words from 'usr/share/dict/words' can be a very common use of this utility.
e.g.


$ look advis
advisability
advisability's
advisable
advise
advised
advisedly
advisement
advisement's
adviser
adviser's
advisers
advises
advising
advisor
advisories
advisors
advisory

In order to ignore the case of alphabetic characters you can use its -f option. (similar to grep -i)

Exit Status:
The 'look' utility exits 0 if one or more lines were found and displayed, 1 if no lines were found, and >1 if an error occurred.

Relates posts:
- Generate random word in UNIX/Linux from dict/words
- UNIX - grep and print control characters from file
- Perform exact match with UNIX grep command
- UNIX grep command - print only matched string and not line
- Highlight match with color in grep

UNIX - handle Kill when no PID

2010-08-18T10:30:00.002+05:30

I was grepping for a particular process say 'Sync' in my process listing


$ ps -ef | grep Sync
testuser  26057     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 10
testuser  26066     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 14
testuser  26067     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 22
testuser  26266 23299  0 06:21 pts/3    00:00:00 grep Sync

The 'grep' from the above listing can be removed by using a regular expression:


$ ps -ef | grep [S]ync
testuser  26057     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 10
testuser  26066     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 14
testuser  26067     1  0 06:21 ?        00:00:00 /opt/nms/ety_master Sync -c 22

I wrote a one liner which basically extracts the PIDs from the above output and then kill them.


$ kill -9 $(ps -ef | awk '/[S]ync/ {print $2}')

ps -ef | grep [S]ync
is same as
ps -ef | awk '/[S]ync/'

Now if there is no 'Sync' process running , the above command is going to throw a message like this:


kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

To handle this, here is the solution:


$ ps -ef | awk '/[S]ync/ {print $2}' | xargs -r kill -9

From XARGS(1) man page:


--no-run-if-empty, -r
If the standard input does not contain any nonblanks, do not run the command.
Normally, the command is run once even if there is no input.  This option is a GNU extension.

UNIX - Print first and last line of file - sed, awk

2010-08-15T15:24:00.003+05:30

Input file:


$ cat file.txt
50
57
52
60
52
90

Required output: Print the first and the last line of the above file.


50
90

Few alternatives using bash, sed and awk :


$ head -1 file.txt; tail -1 file.txt

$ sed -n '1p;$p' file.txt

$ awk 'NR==1; END{print}' file.txt

And a simple python program to achieve the same:


$ cat print-first-last-line.py
fp = open('file.txt','r')
data = fp.readlines()
fp.close()
print data[0],
print data[len(data)-1],

Exact match with UNIX grep and awk

2010-06-29T10:01:00.004+05:30

Input file:


$ cat file.txt
1 172.17.4.1
2 172.17.4.5
3 172.17.4.8
4 172.27.4.19
5 172.24.4.12

Now:


$ grep '172.17.4.1' file.txt
1 172.17.4.1
4 172.17.4.19
5 172.17.4.12

Are you trying to grep exact '172.17.4.1' ?

From grep man pages:

-w, --word-regexp

Select only those lines containing matches that form whole words. The test is that the matching substring must either be at the beginning of the line, or preceded by a non-word constituent character. Similarly, it must be either at the end of the line or followed by a non-word constituent character. Word-constituent characters are letters, digits, and the underscore.


$ grep -w '172.17.4.1' file.txt
1 172.17.4.1

Another alternative:


$ grep '\<172.17.4.1\>' file.txt
1 172.17.4.1

Normal awk match:


$ awk '$2 ~ /172.17.4.1/ {print $1}' file.txt
1
4
5

To make it exact match,
From gawk man pages


\y
matches the empty string at either the beginning or the end of a word.

So,


$ awk '$2 ~ /\y172.17.4.1\y/ {print $1}' file.txt
1

For exact match in awk, you can use this though


$ awk '$2=="172.17.4.1" {print $1}' file.txt

Insert text at top of file in UNIX

2010-06-27T12:13:00.004+05:30

Purpose : Insert text at the top and bottom of a file in UNIX.

I had a lot of .in extension files in my current working directory.

One of them:


$ cat a.in
k|1|4|rty|3.4|67|1
k|2|4|rty|1.4|67|1
k|3|4|rty|3.7|62|2

Required: I had to insert two lines at the top of each files and a line at the bottom of each files.

e.g. After modification above file should look like this:


$ cat a.in
h1|t|4
h2|r|0
k|1|4|rty|3.4|67|1
k|2|4|rty|1.4|67|1
k|3|4|rty|3.7|62|2
#End

Solutions:

Unix 'Sed' utility is suitable for adding/appending, changing or inserting lines to files. You can refer one of my earlier post on this.

The bash script using 'sed':


#!/bin/sh

FILES=$(echo *.in)

for file in $FILES
    do
        echo "Modifying file : $file"

        sed '1i\
h1|t|4' $file | sed '2i\
h2|r|0' | sed '$a\
#End' > $file.tmp

        mv $file.tmp $file
done

With newer version of 'sed', the use of temporary files in the above bash script can be avoided by using '-i' option. Please refer example


-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)

A plain bash script without using 'sed' utility would be something like this:


for file in $(ls *.in)
    do
        echo "Modifying file : $file"
    echo "h1|t|4" >> $file.tmp
    echo "h2|r|0" >> $file.tmp
    cat $file >> $file.tmp
    echo "#End" >> $file.tmp
    mv $file.tmp $file
done

Awk way of inserting a line of text in any line number (say insert text 'first line' as line number 1 of file.txt)


$ awk 'NR==1{print "first line"}1' file.txt

Remove leading zero from line - awk, sed, bash

2010-04-12T09:48:00.002+05:30

Input file:


$ cat ids.txt
00009
01902
34390
00190
00001

Required: Remove the leading zero's from each of the lines in the above file.

Using sed:


$ sed 's/^[0]*//' ids.txt

Output:
9
1902
34390
190
1

So in vi ex mode, the command for the same will be:


:1,$ s/^[0]*//

Other alternatives:


$ awk '{printf "%d\n",$0}' ids.txt
$ awk '{print $1 + 0}' ids.txt
$ cat ids.txt  | bc

Using bash parameter substitution:


$ shopt | grep extglob
extglob         off

# Set this option on
$ shopt -s extglob
$ i=000100
$ echo ${i##+(0)} 
100

extglob shell option in bash:
If set, the extended pattern matching features are enabled

Now if you are thinking how we can add zero's at beginning of a bash variable, here is the way using printf:


$ printf "%010d\n" 00005
0000000005

You might look at this post which shows how we can make some numbers equal width by padding the number with leading zeroes.

Some related posts:
Replace leading zero's with blank using sed
Remove white space in vi editor

Simple bash script to parse log file

2010-04-07T08:54:00.004+05:30

A simple bash script; can be useful for bash newbies.

Input file 'log.txt' is a log file and is of the following format:


$ cat log.txt
2010-04-06 08:06:01 INFO Start ....
2010-04-06 08:06:02 INFO Consuming file trp.1270540317.0.in
2010-04-06 08:06:02 INFO Consuming file trp.1270540326.2.in
2010-04-06 08:06:03 INFO Consuming file trp.1270540341.6.in
2010-04-06 08:06:03 INFO Consuming file trp.1270540367.0.in
2010-04-06 08:06:04 INFO End ....

Required: For each file (trp.<epoch>.<id>.in) entry in the above file, calculate the difference in seconds between this "epoch" time-stamp and the time the file is processed.
e.g.
for this entry:


2010-04-06 08:06:02 INFO Consuming file trp.1270540317.0.in

Calculate this:

(2010-04-06 08:06:02)-1270540317

The bash script:


#!/bin/sh

QFILE=log.txt

grep "INFO Consuming file" $QFILE | while read line
    do
        filename=$(echo "$line" | awk '{print $NF}')
        PT=$(date +%s -u -d "${line%INFO*}")
        AT=$(echo "$line" | awk '{print $(NF-2)}' FS=\.)
        ((diff_sec=PT-AT))
        echo "$filename $diff_sec"
done

Output:


$ ./show-diff.sh
trp.1270540317.0.in 845
trp.1270540326.2.in 836
trp.1270540341.6.in 822
trp.1270540367.0.in 796

Some related posts:

Vi editor copy and paste block example

2010-04-05T09:00:00.007+05:30

"file.txt" is of the following format:

File "dummy.txt" contains the following lines:


$ cat dummy.txt
item2
item5
item9
item1

Required: In Vi, Copy the above lines from "dummy.txt" and paste to "file.txt" in the following format:

The solution:

In vim:

In "dummy.txt", press 'Ctrl v' to go to the "Visual Block" mode.
Select the required block from "dummy.txt" using the keyboard arrow keys and then press "y" for copy or "x" to cut the selected block to buffer.
In "file.txt" move cursor to the location you want the paste the copied block (in this example its Row 3 Column 23) and in command mode press P (capital P)
Done !