UNIX BASH scripting

Construct range from numbers - awk

noreply@blogger.com (Jadu Saikia) — Sat, 07 Nov 2009 09:36:00 +0000

Required:
With the numbers between 100 and 139 whose last digit is in between 0-3, construct the following output:

Step by step solution:
1) Numbers between 100 and 139 whose last digit is between 0-3


$ seq 100 139 | grep '[0-3]$'

Output:

2) Make them a single line with comma separated


$ seq 100 139 | grep '[0-3]$' | paste -sd,

Output:


100,101,102,103,110,111,112,113,120,121,122,123,130,131,132,133

3) Split the above into multiple sub-lines with each line containing 4 numbers


$ seq 100 139 | grep '[0-3]$' | paste -sd, | awk -F, '
{ for(i=1;i<=NF;i++)
{printf("%s%s",$i,i%4?",":"\n")}
}'

Output:


100,101,102,103
110,111,112,113
120,121,122,123
130,131,132,133

4) Print the first and last field


$ seq 100 139 | grep '[0-3]$' | paste -sd, | awk -F, '
{ for(i=1;i<=NF;i++)
{printf("%s%s",$i,i%4?",":"\n")}
}' | awk -F, '{print $1"-"$NF}'

Output:

I am sure there must be better ways to achieve this, please comment.

Related post:

- Break a line into multiple lines using awk and sed

Bash - numbering lines in file using awk

noreply@blogger.com (Jadu Saikia) — Mon, 02 Nov 2009 17:56:00 +0000

Input file 'file.txt' contains names of few students.


$ cat file.txt
Sam G
Ashok Niak
Rosy M
Peter K
Sid Thom
Rasi Yad
Papu S
Niaraj J
Aloh N K
Nipu H
Quam L

Required output:

For the entries of the above file,
- add a serial number to each line
- Also add 'House' number such that all the students are group into total 4 houses in the following fashion:


Sl No,Name,House
1,Sam G,House1
2,Ashok Niak,House2
3,Rosy M,House3
4,Peter K,House4
5,Sid Thom,House1
6,Rasi Yad,House2
7,Papu S,House3
8,Niaraj J,House4
9,Aloh N K,House1
10,Nipu H,House2
11,Quam L,House3

The awk solution using awk NR variable:


$ awk '
BEGIN {OFS=","; print "Sl No,Name,House"}
{print NR,$0,"House"((NR-1)%4)+1}
' file.txt

Lets format the output for a better look:


$ awk '
BEGIN {
    FORMAT="%-8s%-18s%s\n" ;
    {printf FORMAT,"Sl No","Name","House"}
}
{printf FORMAT,NR,$0,"House"((NR-1)%4)+1}
' file.txt

Output:


Sl No   Name              House
1       Sam G             House1
2       Ashok Niak        House2
3       Rosy M            House3
4       Peter K           House4
5       Sid Thom          House1
6       Rasi Yad          House2
7       Papu S            House3
8       Niaraj J          House4
9       Aloh N K          House1
10      Nipu H            House2
11      Quam L            House3

Read about text alignment using awk printf function here

A Bash script for the same will be something like this:


#!/bin/sh
i=0
while read
   do
       echo "$((i+1)),$REPLY,House$((i++ % 4 + 1))"
done < file.txt

Output:


$ sh numbering.sh
1,Sam G,House1
2,Ashok Niak,House2
3,Rosy M,House3
4,Peter K,House4
5,Sid Thom,House1
6,Rasi Yad,House2
7,Papu S,House3
8,Niaraj J,House4
9,Aloh N K,House1
10,Nipu H,House2
11,Quam L,House3

Now a question:
What is that '$REPLY' in the above script ?

Answer: '$REPLY' is the default value when a variable is not supplied to read.

So the above script is same as:


#!/bin/sh
i=0
while read line
   do
       echo "$((i+1)),$line,House$((i++ % 4 + 1))"
done < file.txt

In general, numbering of the lines of a file can be done in several ways viz

Using UNIX/Linux nl(1) command - number lines of files


$ nl file.txt
     1  Sam G
     2  Ashok Niak
     3  Rosy M
     4  Peter K
     5  Sid Thom
     6  Rasi Yad
     7  Papu S
     8  Niaraj J
     9  Aloh N K
    10  Nipu H
    11  Quam L

Using awk NR:


$ awk '{print "\t"NR"\t"$0}' file.txt
        1       Sam G
        2       Ashok Niak
        3       Rosy M
        4       Peter K
        5       Sid Thom
        6       Rasi Yad
        7       Papu S
        8       Niaraj J
        9       Aloh N K
        10      Nipu H
        11      Quam L

Using sed syntax:


$ sed = file.txt | sed 'N;s/\n/\t/'
1       Sam G
2       Ashok Niak
3       Rosy M
4       Peter K
5       Sid Thom
6       Rasi Yad
7       Papu S
8       Niaraj J
9       Aloh N K
10      Nipu H
11      Quam L

Extract range of lines using sed awk bash

noreply@blogger.com (Jadu Saikia) — Sat, 31 Oct 2009 17:57:00 +0000

Below are few different ways to print or extract a section of a file based on line numbers.

Lets try to extract lines between line number 27 and line number 99 of input file 'file.txt'

Using sed editor:


$ sed -n '27,99 p' file.txt > /tmp/file1

Which is same as:


$ sed '27,99 !d' file.txt > /tmp/file2

Awk alternative : you can make use of awk NR variable


$ awk 'NR >= 27 && NR <= 99' file.txt > /tmp/file3

Using Linux/UNIX 'head' and 'tail' command:


$ head -99 file.txt | tail -73 > /tmp/file4

Which is basically:


$ head -99 file.txt | tail -$(((99-27)+1)) > /tmp/file5

In vi editor, we can use the following command in ex mode (open the main file 'file.txt' in vi):


:27,99 w! /tmp/file6

i.e. Write lines between line number 27 and line number 99 of main file 'file.txt' to file '/tmp/file6'

Perl alternative would be:


$ perl -ne 'print if 27..99' file.txt > /tmp/file7

And the solution using python:


$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2

>>> fp = open("/tmp/file8","w")
>>> for i,line in enumerate(open("file.txt")):
...     if i >= 26 and i < 99 :
...             fp.write(line)
...
>>>

So the contents of all the output files produced (i.e /tmp/file[1-8]) will be the same (i.e. line number 27 to line number 99 of 'file.txt')

Bash while loop sum issue explained

noreply@blogger.com (Jadu Saikia) — Fri, 30 Oct 2009 03:58:00 +0000

On one of my directory I had a lot of log files and I had to find the count of the total number of lines which starts with 's' (i.e. ^s).
My first approach was:


$ ls | xargs -i grep -c ^s {} | awk '{sum+=$0} END {print sum}'
190978

And I got my result. Then I thought of performing the same using bash scripting for and while loop and this is what I tried.


#!/bin/sh

sum=0
DIR=~/original
for file in $(ls $DIR)
  do
      Slines=$(grep -c ^s $DIR/$file)
      ((sum+=Slines))
      #You can also use
      #sum=$(expr $sum + $Slines)
      #sum=`expr $sum + $Slines`
done
echo $sum

Executing it:


$ ./usingfor.sh
190978

Cool, correct result.

And then I modified the above script for bash while loop:


#!/bin/sh
sum=0
DIR=~/original
ls $DIR | while read file
  do
      Slines=$(grep -c ^s $DIR/$file)
      ((sum+=Slines))
done
echo $sum

Executing it:


$ ./usingwhile.sh
0

Oops!!! what went wrong ?

In Bash shell, piping directly to bash while loop causes the bash shell to function in a sub shell.
So in the above example the scope of the 'sum' variable is limited to the sub-shell of the while loop and so the modified value of 'sum' is not reflected when we exit the loop. Value of sum is still 0 (local value) as we initialized it to 0 at the beginning of the script.

The solution of this variable scoping problem with while and direct piping will be:

Remove the direct pipe and feed the list of file names under '~/original' directory as stdin to the while loop as shown below (Basically create a temp file with the file names of the directory '~/original')


#!/bin/sh
sum=0
DIR=~/original
ls $DIR > /tmp/filelist

while read file
  do
      Slines=$(grep -c ^s $DIR/$file)
      ((sum+=Slines))
done < /tmp/filelist
echo $sum

Executing it:


$ ./usingwhile_1.sh
190978

And the result is correct.

Grep and print control characters in file - unix

noreply@blogger.com (Jadu Saikia) — Wed, 28 Oct 2009 04:16:00 +0000

One of my input file had some control characters (^B i.e. hex \x02)

On my Ubuntu 8.04.3 and GNU grep version of


$ grep --version
GNU grep 2.5.3

I can grep for any control characters like this:


$ grep '[[:cntrl:]]' /tmp/file.txt
$ grep '[[:cntrl:]]' /tmp/file.txt | less

Also if you know what to grep for, say in above example the control character is ^B (hex \x02); then you can directly grep for it like this


$ grep ^B /tmp/file.txt

* ^B to be typed as ctrl V and ctrl B

And to match any non printable characters, here is another way using grep


$ grep '[^[:print:]]' /tmp/file.txt

To display non printable characters, here is a way using GNU cat command (My cat version : cat GNU coreutils 6.10)


$ cat -v -e -t /tmp/s

Output:

Find n-th occurrence of pattern - vim tip

noreply@blogger.com (Jadu Saikia) — Mon, 26 Oct 2009 17:47:00 +0000

Question: In vi editor, how can I find or locate the nth occurrence of a particular search pattern ?

Answer: With new vim editor, once you search a pattern say /queryname , type 4n in command mode which will leap to the 4th occurrence of the word 'queryname' from where you are.

So to find or locate the 10th occurrence of a particular pattern, go to the top of the file (:1), search for the pattern (/pattern) and then in command mode type 10n.

Linux shuf command - generate random permutations

noreply@blogger.com (Jadu Saikia) — Sat, 24 Oct 2009 16:19:00 +0000

shuf - generate random permutations

Lets discuss the command line options available with Linux/UNIX 'shuf' command

From SHUF(1) man page:
1) -e, --echo
treat each ARG as an input line


$ shuf -e 3 5 6 7
7
6
5
3

$ shuf -e 3 5 6 7
7
5
6
3

$ shuf -e 3 5 6 7
3
6
5
7

2) -i, --input-range=LO-HI
treat each number LO through HI as an input line

To shuffle the numbers between 100 and 200


$ shuf -i 100-200

Also, 'shuf' command can be used along with UNIX/Linux 'seq' or 'jot' command to perform the same as shuf "-i" option.


$ shuf -e $(seq 100 200)
$ shuf -e $(jot 100 100)

3) -n, --head-lines=LINES
output at most LINES lines


$ shuf -i 100-200 -n 3
118
133
117

$ shuf -i 100-200 -n 3
193
188
145

To print a random word in Linux/UNIX


$ shuf -n 1 /usr/share/dict/words
disrupted

$ shuf -n 1 /usr/share/dict/words
festered

Note: /usr/share/dict/words is a standard file on UNIX like operating system and is a newline delimited list of dictionary words.

4) -o, --output=FILE
write result to FILE instead of standard output


$ shuf -n 3 /usr/share/dict/words -o /tmp/dict.txt
$ cat /tmp/dict.txt
heartlands
temple
unsatisfied

Also you can use UNIX/Linux redirection for the same


$ shuf -n 3 /usr/share/dict/words > /tmp/dict.txt

You can shuffle the lines of file and print the output to standard output like this


$ shuf < /tmp/file.txt

Related post:

- Generate random words in Linux in bash

Exponential value is awk sum output

noreply@blogger.com (Jadu Saikia) — Sun, 18 Oct 2009 18:40:00 +0000

In one of my Debian box with mawk 1.3.3 (mawk is an interpreter for the AWK Programming Language), if I try to add the 2nd fields of the following file using awk:


$ cat data.txt
a:99540232
b:89795683
a:08160808
c:0971544
d:99500728
a:12212539898
d:98065599
e:92640031
a:3129013
c:4085555

The output:


$ awk -F ":" '{sum+=$NF} END {print sum}' data.txt
1.27084e+10

So, awk is giving sum output as exponential format as seen above.

To get the above sum output in integer, here is a way:


$ awk -F ":" '{sum+=$NF} END { printf ("%0.0f\n", sum)} ' data.txt
12708429091

But on my Ubuntu 8.04.3 with awk version:


$ awk --version | head -1
GNU Awk 3.1.6

$ awk -F ":" '{sum+=$NF} END {print sum}' data.txt
12708429091

$ awk -F ":" '{sum+=$NF} END { printf ("%d\n", sum)} ' data.txt
12708429091

$ awk -F ":" '{sum+=$NF} END { printf ("%0.0f\n", sum)} ' data.txt
12708429091

Awk - split file vertically on columns

noreply@blogger.com (Jadu Saikia) — Thu, 15 Oct 2009 18:02:00 +0000

I have already put a post on - how we can split a file into multiple sub-files based on different conditions (that was basically a horizontal splitting of file); lets see how we can split a file vertically.

Input file 'file.txt' is a csv file:


$ cat file.txt
A,B,C,D,E,F,G,H,I
1,2,3,4,5,6,7,8,9
I,II,III,IV,V,VII,VIII,IX
a,b,c,d,e,f,g,h,i

Required:

Split the above file into two sub-files such that 1st 3 columns are written to sub-file1 and rest of the columns to sub-file2.
i.e.

sub-file1 content will be


A,B,C
1,2,3
I,II,III
a,b,c

And sub-file2 content will be


D,E,F,G,H,I
4,5,6,7,8,9
IV,V,VII,VIII,IX
d,e,f,g,h,i

Well, this is a pretty simple task using Linux/UNIX cut command


#Printing first 3 columns of 'file.txt'
$ cut -d"," -f1-3 file.txt
or
$ cut -d"," -f-3 file.txt

and

#Printing from 4th column till end
$ cut -d"," -f4-9 file.txt
or
$ cut -d"," -f4- file.txt

Awk solution:


$ awk -F "," '
{
  for(i=1;i<=NF;i++) {
    if(i <= 3) {
      printf "%s,", $i >> "sub-file1"
      if(i==3){
        printf "\n" >> "sub-file1"
      }
    } else {
      printf "%s,", $i >> "sub-file2"
      if(i==NF){
        printf "\n" >> "sub-file2"
      }
    }
  }
}' file.txt

Sub-files generated after running the above awk script:


$ cat sub-file1
A,B,C,
1,2,3,
I,II,III,
a,b,c,

$ cat sub-file2
D,E,F,G,H,I,
4,5,6,7,8,9,
IV,V,VII,VIII,IX,
d,e,f,g,h,i,

Insert lines from files using Linux paste command

noreply@blogger.com (Jadu Saikia) — Wed, 14 Oct 2009 04:33:00 +0000

I have already put a post on some good uses of Linux/UNIX 'paste' command; lets check another practical one using paste command.

Input files:


$ cat contestant.txt
Christopher
Williams
Darwin
Ajay
Brain
Amay
Jiten
Lila

$ cat leader.txt
Mr B
Mrs C
Mrs A

Output required:

For every single line of 'leader.txt'; insert 3 lines from file 'contestant.txt'; so that the output looks like this:


Mr B
Christopher
Williams
Darwin
Mrs C
Ajay
Brain
Amay
Mrs A
Jiten
Lila

The step by step solution using Linux/UNIX paste command


$ cat contestant.txt | paste - - -
Output:
Christopher     Williams        Darwin
Ajay    Brain   Amay
Jiten   Lila

$ cat contestant.txt | paste - - - | paste leader.txt -
Output:
Mr B    Christopher     Williams        Darwin
Mrs C   Ajay    Brain   Amay
Mrs A   Jiten   Lila

$ cat contestant.txt | paste - - - |paste leader.txt - |tr "\t" "\n"
Output:
Mr B
Christopher
Williams
Darwin
Mrs C
Ajay
Brain
Amay
Mrs A
Jiten
Lila

Another similar one liner for the same:


$ < contestant.txt paste - - - | paste leader.txt - | tr "\t" "\n"

Grouping files using awk in Bash shell

noreply@blogger.com (Jadu Saikia) — Fri, 09 Oct 2009 03:56:00 +0000

My directory contains a set of log files with filename of the following pattern:
debug.vendor-name.some-serial-number.epoch-time-stamp. device-class.log

where:
epoch-time-stamp
is the UNIX time stamp when the log file is generated.

device-class
first 4 character of this number represent the service-name of the device and next 6 character is for device class name


$ ls -1
debug.cisco.0001.1254059837.svc1class2.log
debug.cisco.0001.1255058827.svc1class3.log
debug.cisco.0001.1255058827.svc2class3.log
debug.cisco.0001.1255058837.svc1class2.log
debug.cisco.0001.1255059834.svc2class3.log
debug.cisco.0002.1255059819.svc1grade2.log
debug.cisco.0002.1255059849.svc1class1.log
debug.cisco.0002.1255059849.svc2class1.log
debug.juniper.0001.1255059831.svc1class2.log

Lets try to group similar files (under different conditions) and count number of files in each of the groups.

One: Group based on vendor-name(2nd field)


$ ls | awk -F "." '{count[$2]++}END{for(j in count) print j,"["count[j]"]"}'

Output:
cisco [8]
juniper [1]

Two: Group based on vendor-name(2nd field) and serial-number(3rd field)


$ ls | awk -F "." '{count[$2" "$3]++}END{for(j in count) print j,"["count[j]"]"}'

Output:
cisco 0002 [3]
juniper 0001 [1]
cisco 0001 [5]

Three: Group based on vendor-name(2nd field) , serial-number(3rd field) and UNIX-time-stamp(4th field) in hour bucketing*


$ ls | awk -F "." '{count[$2" "$3" "$4-($4%3600)]++}
END{for(j in count) print j,"["count[j]"]"}'

Output:
juniper 0001 1255057200 [1]
cisco 0001 1254056400 [1]
cisco 0002 1255057200 [3]
cisco 0001 1255057200 [4]

*hour bucketing :
e.g: 'Fri Oct 9 09:51:55 UTC 2009' and 'Fri Oct 9 09:01:55 UTC 2009' will fall to the same bucket of Fri Oct 9 09:00:00 UTC 2009

Four: Group based on vendor-name(2nd field) and first 4 characters of device-class (5th field)


$ ls | awk -F "." '
{ $5 = substr($5, 0, 4) }
{count[$2" "$5]++}
END{for(j in count) print j,"["count[j]"]"}'

Output:
juniper svc1 [1]
cisco svc1 [5]
cisco svc2 [3]

Five: Group based on
vendor-name(2nd field),
serial-number(3rd field) ,
UNIX-time-stamp(4th field) in hour bucketing
and first 4 characters of device-class (5th field)


$ ls | awk -F "." '
{ $5 = substr($5, 0, 4) }
{count[$2" "$3" "$4-($4%86400)" "$5]++}
END {for(j in count) print j,"["count[j]"]"}'

Output:
cisco 0002 1255046400 svc1 [2]
cisco 0002 1255046400 svc2 [1]
juniper 0001 1255046400 svc1 [1]
cisco 0001 1255046400 svc1 [2]
cisco 0001 1255046400 svc2 [2]
cisco 0001 1254009600 svc1 [1]

Hope you find it useful.

Related post:

- SQL Sum of and group by using awk
- Group by Clause functionality using awk
- Associative array in awk

Extract sub-string from variable in bash

noreply@blogger.com (Jadu Saikia) — Wed, 07 Oct 2009 04:43:00 +0000

Suppose:


$ mypath=/dir1/dir2/dir3/dir4

$ echo $mypath
/dir1/dir2/dir3/dir4

Now, if you need to print the parent path from the above path (i.e. print '/dir1/dir2/dir3')


$ dirname $mypath
/dir1/dir2/dir3

$ parentpath=$(dirname $mypath)

$ echo $parentpath
/dir1/dir2/dir3

Using Sub-string Removal ways in Bash shell

${string%substring}
It deletes shortest match of $substring from 'back' of $string.


$ echo ${mypath%/*}
/dir1/dir2/dir3

or

$ printf '%s\n' "${mypath%/*}"
/dir1/dir2/dir3

If you need to print the last directory name from the above mypath, here are few ways:

Using Sub-string Removal ways in Bash shell
${string##substring}
It deletes the "longest" match of $substring from 'front' of $string.


$ echo ${mypath##*/}
dir4

Another way using awk:

$ echo $mypath | awk '{print $NF}' FS=\/
dir4

Similar post:

- Truncate string using bash script

Directory size excluding sub-directories - Linux

noreply@blogger.com (Jadu Saikia) — Fri, 02 Oct 2009 14:51:00 +0000

Directory '/home/user/work/demo/' contains a few regular files and two directories say "part2"(size=41236 KB) and "libs"(size=20620 KB).


$ du ~/work/demo/
41236   /home/user/work/demo/part2
20620   /home/user/work/demo/libs
87640   /home/user/work/demo/

From Linux/UNIX DU(1) command man page:

-s, --summarize
display only a total for each argument.

So, the following command is going to display the total size of the directory '/home/user/work/demo/'


$ du -s ~/work/demo/
87640   /home/user/work/demo/

Now, if you need to find the size of the '/home/user/work/demo/' directory excluding the size of the sub-directories, there is a command line option with DU(1):

-S, --separate-dirs
do not include size of sub-directories

So


$ du -S ~/work/demo/
41236   /home/user/work/demo/part2
20620   /home/user/work/demo/libs
25784   /home/user/work/demo/

Now,


$ du -S --max-depth=0 ~/work/demo/
25784   /home/user/work/demo/

or

$ du -S ~/work/demo/ | awk 'END {print}'
25784   /home/user/work/demo/

Subtract from total amount - Awk example

noreply@blogger.com (Jadu Saikia) — Wed, 30 Sep 2009 04:42:00 +0000

Input file:


$ cat expense.txt
Particulars,Item1,Item2,Item3
BudgetAmount,12000,4560,5000
Expense@2006,1800,3000,250
Expense@2007,2210,2100,3000
Expense@2008,100,1500,320
Expense@2009,0,100,20

Output required:

For all the items, calculate the amount left after expense i.e.

For an item:
Amount Left = (BudgetAmount - (Expense@2006 + Expense@2007 + Expense@2008 + Expense@2009))

i.e. required output:


BudgetAmount,12000,4560,5000
Expense@2006,1800,3000,250
Expense@2007,2210,2100,3000
Expense@2008,100,1500,320
Expense@2009,0,100,20
Amount Left,7890,-2140,1410

Microsoft Excel representation of the above:

The awk program:


$ awk 'BEGIN { FS=OFS="," }
$1 == "BudgetAmount" {
         bI1 = $2
         bI2 = $3
         bI3 = $4
         print
}
/^Expense@/ {
         bI1 -= $2
         bI2 -= $3
         bI3 -= $4
         print
}
END {
         print "Amount Left",bI1, bI2, bI3
}' expense.txt

Related post:

- Bash script for sequential subtraction of numbers

If else examples in awk - bash

noreply@blogger.com (Jadu Saikia) — Tue, 29 Sep 2009 03:51:00 +0000

Input file: Each line of 'num.txt' contains 2 numbers (say A and B).


$ cat num.txt
34,140
190,140
89,120
110,110
210,115

Required: Calculate and print percentage (A/B)*100 with the following conditions:

- If percentage is less than 100, print the calculated actual percentage
- If percentage is more than 100, print the percentage as 100

First solution:


$ awk '
BEGIN {FS=OFS=","}
{if($1>$2) {print $0,100}
else {print $0,($1/$2)*100}
}' num.txt

Output:
34,140,24.2857
190,140,100
89,120,74.1667
110,110,100
210,115,100

Lets do some text alignment and formatting using awk.


$ awk '
BEGIN {FS="," ;  {printf "%-10s%-8s%s\n","A","B","% age"}}
{if($1>=$2) {printf "%-10s%-8s%s\n",$1,$2,100}
else {printf "%-10s%-8s%2.2f\n",$1,$2,($1/$2)*100}
}' num.txt

Output:
A         B       % age
34        140     24.29
190       140     100
89        120     74.17
110       110     100
210       115     100

Or a different look of the above script:


$ awk '
BEGIN {
    FS="," ; FORMAT="%-10s%-8s%s\n" ;
    {printf FORMAT,"A","B","% age"}
}
{
    if($1>=$2) {printf FORMAT,$1,$2,100}
    else {printf FORMAT,$1,$2,($1/$2)*100}
}' num.txt

Output:
A         B       % age
34        140     24.2857
190       140     100
89        120     74.1667
110       110     100
210       115     100

Another way of writing if else in AWK.


$ awk '
{printf("%-10s%-8s%2.2f\n",\
$1,$2, ($1<=$2) ? ($1/$2)*100 : 100)
}' FS="," num.txt

Output:
34        140     24.29
190       140     100.00
89        120     74.17
110       110     100.00
210       115     100.00

Related post:

- Calculate percentage using awk in bash
- Align text with awk printf function

Insert after certain characters - awk and sed

noreply@blogger.com (Jadu Saikia) — Thu, 24 Sep 2009 18:20:00 +0000


$ add="20010db885a3000000008a2e03707334"
$ echo $add
20010db885a3000000008a2e03707334

Required output: Insert a colon ':' after every 4 characters in the above line.
So the output required:


2001:0db8:85a3:0000:0000:8a2e:0370:7334

Using awk:


$ echo $add | awk -F "" '
{for(i=1;i<=NF;i++){printf("%s%s",$i,i%4?"":":")}}'|awk '{sub(/:$/,"")};1'

Note: Mind the use of "" as the field separator.

Using sed:


$ echo $add | sed 's/..../&:/g;s/:$//'

Related post:

- Break a line into multiple lines using awk and sed

Replace asterisk using sed - bash

noreply@blogger.com (Jadu Saikia) — Tue, 22 Sep 2009 17:44:00 +0000

For example:


$ countries="**India **South Africa **Sri Lanka **West Indies"
$ echo $countries
**India **South Africa **Sri Lanka **West Indies

Output required:
Replace "**" with a newline, so that above line becomes:


**India
**South Africa
**Sri Lanka
**West Indies

The sed replacement:


$ echo $countries | sed 's! **!\n**!g'
sed: -e expression #1, char 12: Invalid preceding regular expression

So you would need to escape the asterisk above.
i.e.


$ echo $countries | sed 's! \*\*!\n**!g'

Another way using sed:


$ echo $countries | sed 's! \*\*!\
\*\*!g'

Similarly:


$ echo "a,b,c,d" | sed 's!,!\n!g'

Output:
a
b
c
d

Related post:

1)
Suppose your i/p line is:


1 b 3 4 e 6 g 8 i j k

And you wish to split the above line into multiple lines (each line with say 3 entries)

i.e.


1 b 3
4 e 6
g 8 i
j k

here is a post

2)
One more related post of breaking a line into multiple lines based on length

Print except few columns using awk - bash

noreply@blogger.com (Jadu Saikia) — Mon, 21 Sep 2009 15:06:00 +0000

Input file:


$ cat details.txt
AX|23.45|1932323|A|VI|-|Y|0
TY|93.45|2932323|B|VI|-|Y|1
RE|63.25|8932323|A|VI|0|N|1
AY|83.85|0932323|C|VI|-|Y|0

Required:
Print all columns from the above file except column number 2 and 7.
i.e. required output:


AX|1932323|A|VI|-|0
TY|2932323|B|VI|-|1
RE|8932323|A|VI|0|1
AY|0932323|C|VI|-|0

Basically for the above file we have to print column # 1,3,4,5,6,8

i.e.


$ awk '
        BEGIN{FS=OFS="|"}{print $1,$3,$4,$5,$6,$8}
' details.txt

But if number of fields is very large on the input file, the above method is not going to be so useful. So here is another technique.


$ awk '
    BEGIN{FS=OFS="|"}
    { for (i=1; i<=NF;i++)
         if( i==2 || i==7 ) continue
         else
         printf("%s%s", $i,(i!=NF) ? OFS : ORS)}
' details.txt

And if you want to exclude a range of column numbers (say exclude column 3 to column 6) here is my earlier post

An additional tip:
Suppose you need to generate the print 'statement' for printing a number of consecutive fields for an awk program, here is a quick way:


$ seq -s ",$" 1 8 | sed 's/.*/{print $&}/'

Output:

{print $1,$2,$3,$4,$5,$6,$7,$8}

Or you can use the the for loop mentioned above.

Find process running time in UNIX

noreply@blogger.com (Jadu Saikia) — Fri, 18 Sep 2009 18:15:00 +0000

Question. How to find out how long a process is running in an UNIX system ?

Ans: Here are some tips to find the process' running time in an UNIX system.

----------------
For 2.6 kernels:
----------------

Identify your process Id

and then do a


ls -ld /proc/PID-OF-YOUR-PROCESS

So the modification time listed on the above file(directory) is the time that the process has started.

e.g. I have started a process say "sleep 10000" few minutes back


$ ps -ef | grep "[s]leep 10000"
jsaikia  24375 23306  0 22:13 pts/10   00:00:00 sleep 10000

$ ls -ld /proc/24375
dr-xr-xr-x 6 jsaikia staff 0 2009-09-18 22:14 /proc/24375

So, "2009-09-18 22:14" is the start time of the above sleep process; if I subtract this time from the current time I can find how long this process has been running.

For subtraction you can have a script like this:


#!/bin/sh

T1=$(date +%s -d "$1")
T2=$(date +%s -d "$2")
((diffsec=T1-T2))
echo - \
| awk -v D=$diffsec '{printf "%d:%d:%d\n",D/(60*60),D%(60*60)/60,D%60}'

So that you can execute like this:


$ sh cal-tdiff.sh "$(date)" "2009-09-18 22:14"
0:22:56

----------------
For 2.4 kernels:
----------------
For 2.4 kernels, the modification time on the "/proc/PID-OF-YOUR-PROCESS" will be the current system time (unlike 2.6 kernels where its the actual process start time)

So how to find the running time of a process on a 2.4 kernel UNIX system ?

Here is the way (this is going to work for 2.6 also)

e.g.


$ ps -ef | grep [s]leep
root      7702  7689  0 17:34 pts/0    00:00:00 sleep 100000

so 7702 is the pid of the above process.

$ pd=7702
$ expr $(awk '{print $1}' FS=\. /proc/uptime) - $(awk '{printf ("%10d\n",$22/100)}' /proc/$pd/stat)

The output will show the number of seconds the process(with pid=pd) is running.

Bash script to copy required files

noreply@blogger.com (Jadu Saikia) — Thu, 17 Sep 2009 17:38:00 +0000

Contents of my "inputdir" is a set of files with filename like this:


$ ls -1 inputdir/
log.10.16.1253168140.txt
log.11.5.1253168345.txt
log.11.9.1253168347.txt
log.12.1.1253168347.txt
log.19.1.1253168140.txt

Directory "testcfgs" contains a set of config xmls.


$ ls -1 testcfgs/
cfg_10_16.xml
cfg_10_5.xml
cfg_11_5.xml
cfg_11_9.xml
cfg_12_1.xml
cfg_19_1.xml
cfg_19_2.xml
cfg_91_9.xml

Required:

For each file of name "log.X.Y.timestamp.txt" in "inputdir", copy the corresponding "cfg_X_Y.xml" config file from "testcfgs" to a directory say "requiredcfgs".

A simple practical bash one liner script:


$ for filename in $(ls -1 inputdir/)
> do
> X=$(echo "$filename" | cut -d"." -f2)
> Y=$(echo "$filename" | cut -d"." -f3)
> cp testcfgs/cfg_$X\_$Y.xml requiredcfgs/
> done

The two lines above for finding X and Y value can be replaced by a single line using 'eval with awk', like this:


$ for filename in $(ls -1 inputdir/)
> do
> eval $(echo "$filename" | awk -F "." '{print "X="$2";Y="$3}')
> cp testcfgs/cfg_$X\_$Y.xml requiredcfgs/
> done

Contents of "requiredcfgs" directory after execution of the above bash script.


$ ls -1 requiredcfgs/
cfg_10_16.xml
cfg_11_5.xml
cfg_11_9.xml
cfg_12_1.xml
cfg_19_1.xml

Related post on eval with awk:

- Subdivide an ip address - assign each part to an variable using awk

Linux seq command format option example

noreply@blogger.com (Jadu Saikia) — Tue, 15 Sep 2009 03:51:00 +0000

I have already post on Linux/UNIX seq command, using which we can generate sequence of numbers. Seq is very useful to generate loop arguments in UNIX bash scripting.

One of very useful seq command line option is -f

-f, --format=FORMAT
use printf style floating-point FORMAT

Lets see some simple examples on the same.


$ seq -f "%04g" 3
Output:
0001
0002
0003


$ seq -f "logfile%02g.txt" 10
Output:
logfile01.txt
logfile02.txt
logfile03.txt
logfile04.txt
logfile05.txt
logfile06.txt
logfile07.txt
logfile08.txt
logfile09.txt
logfile10.txt

Now, to create 10 files with names logfile01.txt, logfile02.txt ,....., logfile10.txt


$ touch $(seq -f "logfile%02g.txt" 10)

$ ls -1
logfile01.txt
logfile02.txt
logfile03.txt
logfile04.txt
logfile05.txt
logfile06.txt
logfile07.txt
logfile08.txt
logfile09.txt
logfile10.txt

FIRST INCREMENT LAST
Sequence numbers between 1.0003 and 1.0012 with an increment of .00002


$ seq -f "1.%04g" 3 2 12
1.0003
1.0005
1.0007
1.0009
1.0011

-s is to specify separator between sequence numbers.


$ seq -s "+" -f "1.%04g" 3 2 12
1.0003+1.0005+1.0007+1.0009+1.0011

$ seq -s "+" -f "1.%04g" 3 2 12 | bc
5.0035

Another good command for printing sequential and random data is jot, read here

Related post:
- Ways of writing for loops in bash scripting
- Print text within style box in bash scripting

Replace duplicate line with blank - awk

noreply@blogger.com (Jadu Saikia) — Thu, 10 Sep 2009 17:59:00 +0000

I just received an query as a comment on one of my older post on "removing duplicates based on fields using awk"

Question was:

Any Idea on how to replace duplicate line with blank line instead of deleting them?
e.g.

Input:


test1
test1
test2
test2
test2
test3

Output:


test1

test2


test3

Thought of making it a separate post here.

The solution using awk:


$ awk 'x[$0]++ {$0=""} {print}' file.txt

Split file using awk - few examples

noreply@blogger.com (Jadu Saikia) — Wed, 09 Sep 2009 17:57:00 +0000

One of good use of awk is splitting files (based on different conditions) into sub-files.
Lets see some examples:

Example 1:

Input file log1.txt:
- Line starting with H is the main header line.
- Line starting with "h" and subsequent lines starting with "s" (till the next "h" line) are part of the same entry/section.


$ cat log1.txt
H,555,etho0
h,1,3
s,1233,456456,1212
s,4251,452456,7215
s,6283,851456,1219
h,9,2
s,2233,156456,1912
s,9233,256456,8212
h,2,4
s,4233,456456,1212
s,7251,252456,7215
s,1288,851456,9219
s,9183,851456,6219

Required:

- Split or subdivide the above file into sub files corresponding to each entry (one entry being the section starting with "h" and "s" lines till the next "h" line)
- Each sub-file should contain(start with) the main header line ("H" line).
- The required output is 3 sub files with the following contents and filename convention.


$ cat 555.1.1.log
H,555,etho0
h,1,3
s,1233,456456,1212
s,4251,452456,7215
s,6283,851456,1219

$ cat 555.9.2.log
H,555,etho0
h,9,2
s,2233,156456,1912
s,9233,256456,8212

$ cat 555.2.3.log
H,555,etho0
h,2,4
s,4233,456456,1212
s,7251,252456,7215
s,1288,851456,9219
s,9183,851456,6219

The awk program:


$ awk -F "," '
$1=="H" {mainH=$0;id=$2;next}
/^h/{
       hid=$2;close(id"."hid"."f".log")
       f++
       print mainH > id"."hid"."f".log"
    }
{print $0 > id"."hid"."f".log"}
' log1.txt

Example 2:

Input file:


$ cat log.txt
1252468812,yahoo,3.5
1252468812,hotmail,2.4
1252468819,yahoo,1.2
1252468812,msn,8.9
1252468923,gmail,12
1252468819,live,3.4
1252468929,yahoo,9.0
1252468929,msn,1.2

Required:

a) Split the above files based on the first field (i.e. lines with same first field should go to the same file)

The awk one liner:


$ awk -F "," '{close(f);f=$1}{print > f".txt"}' log.txt

Output:
Above file is splited into the following sub-files.


$ cat 1252468812.txt
1252468812,yahoo,3.5
1252468812,hotmail,2.4
1252468812,msn,8.9

$ cat 1252468819.txt
1252468819,yahoo,1.2
1252468819,live,3.4

$ cat 1252468923.txt
1252468923,gmail,12

$ cat 1252468929.txt
1252468929,yahoo,9.0
1252468929,msn,1.2

b) Send every 3 lines of above file into a sub file.

The awk code:


$ awk '{print >("log_" int((NR+2)/3))}' log.txt

Output:
The sub-files generated.


$ cat log_1
1252468812,yahoo,3.5
1252468812,hotmail,2.4
1252468819,yahoo,1.2

$ cat log_2
1252468812,msn,8.9
1252468923,gmail,12
1252468819,live,3.4

$ cat log_3
1252468929,yahoo,9.0
1252468929,msn,1.2

- Split file based on start pattern using awk
- Awk - Split file into subfiles based on pattern
- Send alternate lines of a file into separate sub-file using awk

Print first character of a field - awk substr

noreply@blogger.com (Jadu Saikia) — Tue, 08 Sep 2009 03:11:00 +0000

Input file:


$ cat file.txt
8965,1212,c32,1
1221,9000,d90,0
1222,7823,,2
9012,1901,c12,7
9012,1342,t90,9

Output Required: If 3rd field is non empty, print only the first character of the value, else(i.e. when the field is blank) print "NA" in the 3rd field.
i.e. required output:


8965,1212,c,1
1221,9000,d,0
1222,7823,NA,2
9012,1901,c,7
9012,1342,t,9

The awk program:


$ awk '
    BEGIN {FS=OFS=","}
   { if ( length($3) ) { $3 = substr($3, 0, 1) }
     else { $3 = "NA" }
     print
   }
' file.txt

Related post:
- Brief about awk substr function
- Blank column in file - awk newbie
- Count non empty field in file using awk

Truncate string using bash script

noreply@blogger.com (Jadu Saikia) — Mon, 07 Sep 2009 03:01:00 +0000

Input file:


$ cat spears.txt
Baby-One-More-Time.mp3
Autumn-Goodbye.mp3
Baby-One-More-Time.mp3
Cant-Make-You-Love-Me.wmv
Crazy.mp3
Crazy---Stop-Remix.mp3
Dont-Go-Knocking-on-My-Door.mp3
Dont-Let-Me-Be-The-Last-to-Know.flv
From-The-Bottom-of-My-Broken-Heart.mp3
Im-Not-a-Girl-Not-Yet-a-Woman.mp3

Required: Truncate the lines of the above file (filename part, and not the extension) to 15 character long.
Also insert a string "..." in between the filename part and extension in case the line is truncated.

The bash script:


#!/bin/sh
#Bash Script to truncate string
#

while read filename
    do
        name=${filename%%.*}
        extn=${filename##*.}
        if [ ${#name} -gt 15 ]
            then
                nfile=$(echo $name | cut -c1-15)
                fullname=${nfile}...${extn}
                echo $fullname
            else
                echo $filename
        fi
    done < spears.txt > spears.txt.truncated

The output file produced after execution of the above bash script:


$ cat spears.txt.truncated
Baby-One-More-T...mp3
Autumn-Goodbye.mp3
Baby-One-More-T...mp3
Cant-Make-You-L...wmv
Crazy.mp3
Crazy---Stop-Re...mp3
Dont-Go-Knockin...mp3
Dont-Let-Me-Be-...flv
From-The-Bottom...mp3
Im-Not-a-Girl-N...mp3