UNIX Command Line

Display ASCII pictures on Unix terminal

2016-09-08T09:17:00.001+05:30

'cowsay' is a program which generates ASCII pictures of a cow with a message.

🐞  jsaikia@unixcl.com:~$ echo "Hello World" | cowsay
 _____________ 
< Hello World >
 ------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

You can change the "Cow" using the following option:

                
🐞  jsaikia@unixcl.com:~$ echo "Hello World" | cowsay -f tux
 _____________ 
< Hello World >
 ------------- 
   \
    \
        .--.
       |o_o |
       |:_/ |
      //   \ \
     (|     | )
    /'\_   _/`\
    \___)=(___/

🐞  jsaikia@unixcl.com:~$ echo "Hello World" | cowsay -f elephant
 _____________ 
< Hello World >
 ------------- 
 \     /\  ___  /\
  \   // \/   \/ \\
     ((    O O    ))
      \\ /     \ //
       \/  | |  \/ 
        |  | |  |  
        |  | |  |  
        |   o   |  
        | |   | |  
        |m|   |m|

Available text characters are:

🐞  jsaikia@unixcl.com:~$ cowsay -l
Cow files in /usr/local/Cellar/cowsay/3.04/share/cows:
beavis.zen blowfish bong bud-frogs bunny cheese cower daemon default dragon
dragon-and-cow elephant elephant-in-snake eyes flaming-sheep ghostbusters
head-in hellokitty kiss kitty koala kosh luke-koala meow milk moofasa moose
mutilated ren satanic sheep skeleton small sodomized stegosaurus stimpy
supermilker surgery telebears three-eyes turkey turtle tux udder vader

'fortune' is yet another implementation of the Unix-style fortune program that displays a random message from a database of quotations. Something like:

🐞  jsaikia@unixcl.com:~$ fortune
It is not every question that deserves an answer.
  -- Publilius Syrus
🐞  jsaikia@unixcl.com:~$ fortune
It is so soon that I am done for, I wonder what I was begun for.
  -- Epitaph, Cheltenham Churchyard

Now, you can make your cow quote the beautiful quotes from 'fortune' and create something like the one I have (I have added this to my .bash_profile):

You can add the following to your .bash_profile or .bashrc to see a new cowsay fortune for every new terminal session:

fortune | cowsay

If you are on Mac OSX, you can get the cowsay and fortune installation details here. If you need help in getting homebrew installed on your Mac OSX, here's the help link.

Happy New Year 2016 in Unix way

2016-01-01T07:58:00.002+05:30

Wishing everyone a great 2016 ahead !

"toilet" gives you color in UNIX terminal ! Read more here

Display cricket score on Unix Terminal

2015-11-25T13:04:00.000+05:30

If your job demands you to be on Unix console all the time and you happen to be one who can not avoid watching the cricket score, something you would like:
The script:

#!/bin/sh
#Get cricket score on your unix console top corner, enjoy cricket while you work ..

columns=$(tput cols)
startpoint=$(($columns-100))
matchurl='http://www.espncricinfo.com/india-v-south-africa-2015-16/engine/match/903607.html'

while :
do
  line=$(wget -q -O- $matchurl  | awk -F '=' '/og:title/ {print $3}' | awk -F '|' '{print $1}' | sed 's/"//g')
  echo -en "\033[s"    #save current screen position & attributes
  tput cup 0 $startpoint
  echo -en "\033[42m$line\033[0m"
  echo -e -n "\033[u"
  sleep 15
done

Output:

Bash - Get function name inside a function

2015-11-06T16:03:00.003+05:30

$FUNCNAME has the function name in a bash script which can be used for useful purpose like the following: Script:test.sh

!/bin/sh

WORKID=$(date +%s) ; mkdir $WORKID
LOGFILE=$WORKID/$(basename $0).log

_f_log() {
        echo "$(date):$@" >> $LOGFILE
}

_f_testFunc() {
        local logfile=$1
        local status=$2
        _f_log "$FUNCNAME:$logfile:$status"

}

_f_testFunc /tmp/a.log restart

Executing it:

$ ./test.sh

Result:

$ cat 1446805714/test.sh.log
Fri Nov  6 15:58:34 IST 2015:_f_testFunc:/tmp/a.log:restart

Unix - run multiple commands on remote machines in parallel

2015-03-07T21:58:00.000+05:30

Assume you have a requirement of running two commands on 10 hosts.

In this post I would like to show you how you can:

1) Run multiple commands on remote host using ssh.

2) Also instead of working on one host at a time serially, how can we execute the commands on multiple hosts in parallel to save the overall execution time.

Example:
Find the number of files under '/sdb/_read/' and '/sdb/_write/' directories on following 10 hosts.

$ cat host.txt 
192.19.66.203
192.19.66.204
192.19.66.205
192.19.66.206
192.19.66.207
192.19.66.208
192.19.66.209
192.19.66.210
192.19.66.211
192.19.66.212

Serial execution:
We can write a bash for loop and work on each host like this:

$ for ip in $(cat host.txt); do echo -n "$ip|"; ssh -2 root@$ip 'R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l) ; echo "$R|$D"'; done

Time taken:

real 0m50.084s
user 0m0.052s
sys 0m0.032s

Parallel execution:
I have already blogged about Unix xargs parallel execution option here, using that:

$ cat host.txt  | xargs -n 1 -P 5 -i ssh -2 root@{} 'I=$(hostname -i);R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l)  ; echo "$I|$R|$D"'

Time taken:

real 0m10.437s
user 0m0.044s
sys 0m0.036s

10 Seconds vs 50 Seconds, wow !

Output:

192.19.66.205|0|4
192.19.66.204|0|0
192.19.66.203|0|0
192.19.66.207|5|0
192.19.66.206|0|0
192.19.66.208|0|0
192.19.66.209|0|7
192.19.66.210|0|0
192.19.66.211|2|1
192.19.66.212|9|0

Note: Utilities like GNU parallel or Parallel SSH are always the best tool in this job.

Related posts:

Happy new year 2015 to all unix lovers

2015-01-01T00:21:00.001+05:30

Wishing all the readers of unixcl.com a Happy and Healthy 2015 ahead, keep rocking !

Unix - Delete blank lines from file

2014-12-20T17:44:00.002+05:30

Here's some ways to remove empty or blank lines from a file in Unix. Simple but definitely useful.

$ grep -v '^$' file.txt
$ grep '.' file.txt
$ sed '/^$/d' file.txt
$ sed -n '/^$/!p' file.txt
$ awk NF file.txt
$ awk '/./' file.txt

In Vi editor, in escape mode type

:g/^$/ d

Execute script on remote server SSH - Unix

2014-12-13T13:49:00.000+05:30

Assume you have a complex bash script named 'run_q.sh' (containing good number of loops, Awk statements, multiple lines of code etc) that you need to run against one or more remote servers via SSH. One way would be to copy (i.e. scp) the script to the remote server(s) and then do something like this: e.g.

$ scp -2 run_q.sh root@192.168.32.8:/tmp/.

$ ssh -2 root@192.168.32.8 sh /tmp/run_q.sh

If its multiple machines, then you can use bash for loop like this:

$ for ip in 192.168.32.8 192.168.32.9 192.168.32.3 ; do echo "Executing on IP: $ip" ; ssh -2 root@$ip 'sh /tmp/run_q.sh' ; done

Now assume you do not want to copy your script to the remote machine(s) and want to execute from your local machine without copying it. Here's a way:

$ cat run_q.sh | ssh -2 root@192.168.32.8 /bin/sh

And if its to be executed on multiple machines, you can use a for loop like this:

$ for ip in 192.168.32.8 192.168.32.9 192.168.32.3 ; do echo "Executing on IP: $ip" ; cat run_q.sh | ssh -2 root@$ip /bin/sh ; done

I am sure you liked this trick ! Feel free to comment below. Thanks.

Related posts:

Diff remote files using SSH in Unix

Unix xargs parallel execution of commands

2014-04-27T09:11:00.002+05:30

Xargs has option that allows you to take advantage of multiple cores in your machine. Its -P option which allows xargs to invoke the specified command multiple times in parallel. From XARGS(1) man page:

-P max-procs
   Run up to max-procs processes at a time; the default is 1.  If max-procs is 0, xargs will run as many processes as possible at a time.   Use  the  -n  option
   with -P; otherwise chances are that only one exec will be done.

-n max-args
    Use at most max-args arguments per command line.  Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the  -x  
    option is given, in which case xargs will exit.

-i[replace-str]
    This option is a synonym for -Ireplace-str if replace-str is specified, and for -I{} otherwise.  This option is deprecated; use -I instead.

Let me try to give one example where we can make use of this parallel option avaiable on xargs. e.g. I got these 8 log files (each one is of 1.5G size) for which I have to run a script named count_pipeline.sh which does some calculation around the log lines in the log file.

$ ls -1 *.out
log1.out
log2.out
log3.out
log4.out
log5.out
log6.out
log7.out
log8.out

The script count_pipeline.sh takes nearly 20 seconds for a single log file. e.g.

$ time ./count_pipeline.sh log1.out

real 0m20.509s
user 0m20.967s
sys 0m0.467s

If we have to run count_pipeline.sh for each of the 8 log files one after the other, total time needed:

$ time ls *.out | xargs -i ./count_pipeline.sh {}           

real 2m45.862s
user 2m48.152s
sys 0m5.358s

Running with 4 parallel processes at a time (I am having a machine which is having 4 CPU cores):

$ time ls *.out | xargs -i -P4 ./count_pipeline.sh {} 

real 0m44.764s
user 2m55.020s
sys 0m6.224s

We saved time ! Isn't this useful ? You can also use -n1 option instead of the -i option that I am using above. -n1 passes one arg a time to the run comamnd (instead of the xargs default of passing all args).

$ time ls *.out | xargs -n1 -P4 ./count_pipeline.sh

real 0m43.229s
user 2m56.718s
sys 0m6.353s

Related posts:
- UNIX handle kill when no PID available

Unix - merge multiple consecutive lines

2013-07-26T09:34:00.004+05:30

Input file:

$ cat infile.txt 
aid=33
pw=3
nn=90
aid=32
pw=30
nn=70
aid=56
pw=3
nn=93

Required:
Combine or merge every three consecutive lines of the above file so that the output becomes:

aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93

Awk solution: If line number is divisible by 3 then put a new line(\n) else put a comma(,) i.e.

$ awk '{printf("%s%s", $0, (NR%3 ? "," : "\n"))}' infile.txt 
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93

Another way using Awk:

$ awk 'NR%3{printf $0",";next;}1' infile.txt 
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93

Using UNIX paste command:

$ paste -d"," - - - < infile.txt 
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93

A bash command line solution:

$ while read line1; do read line2; read line3; echo "$line1,$line2,$line3"; done < infile.txt 
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93

Unix - Append 0 to single digit date

2013-05-08T10:33:00.004+05:30

Input file file.txt has dates in month/day/year format.

$ cat file.txt 
3/4/2013
3/10/2013
10/4/2013
12/10/2012

Required: Add prefix 0 to first and second field if its a single digit.

Awk solution:

$ awk 'BEGIN {FS=OFS="/"} 
    { 
 if (length($1) == 1) $1="0"$1
 if (length($2) == 1) $2="0"$2
        { print }
}' file.txt

Output:

UNIX time command output redirect to file

2013-02-28T08:54:00.000+05:30

As you know the 'time' command run programs and summarize system resource usage. Here is a way how you can redirect 'time' command output to file. I would recomend you to go through this page to understand more about 'time' command.

e.g. We are executing 'prog.sh' along with 'time' command.

$ time ./prog.sh 
Initiating merge
Merge completed

real	0m8.803s
user	0m0.010s
sys	0m0.000s

#Trying to redirect the output to a file

$ time ./prog.sh > out.txt

real	0m8.804s
user	0m0.020s
sys	0m0.000s

#out.txt content:

$ cat out.txt 
Initiating merge
Merge completed

#So it only redirected the STDOUT of the script executed, but the 'time' command outut is not redirected.
#This is becuase the command time sends it's output to STDERR (instead of STDOUT)
#To capture output of 'time' command:

$ { time ./prog.sh ; } 2> out.txt
Initiating merge
Merge completed

#Now out.txt content:

$ cat out.txt 

real	0m8.303s
user	0m0.010s
sys	0m0.000s

#And to capture output of script as well as time command:

$ { time ./prog.sh ; } &> out.txt

#'out.txt' now has both the outputs.

$ cat out.txt 
Initiating merge
Merge completed

real	0m8.303s
user	0m0.020s
sys	0m0.000s

As I have mentioned in my earlier post on UNIX 'time' command, there's two types of time command available:
1) Shell's in-build time: Gives only scheduler information
2) /usr/bin/time: Gives more information, also allows formatting the output

The second (/usr/bin/time) one accepts output redirection without code block:

$ /usr/bin/time ./prog.sh &> newout.txt

$ cat newout.txt 
Initiating merge
Merge completed
0.00user 0.01system 0:08.30elapsed 0%CPU (0avgtext+0avgdata 5728maxresident)k
0inputs+8outputs (0major+719minor)pagefaults 0swaps

Bash - convert delimited file to html table

2013-02-15T22:27:00.003+05:30

Here is a simple UNIX bash script that can be used to convert a simple delimited file to a HTML table format. Default delimiter if not mentioned as part of the command line argument to this script will be comma.

#!/bin/sh
#convert_2_html_table.sh 
#Converts a delimited file to a HTML table
#Jadu Saikia http://unstableme.blogspot.in

NOARG=64

#usage function
f_Usage () {
echo "Usage: $(basename $0) -d <delimiter> -f <delimited-file>"
}

#command line args
while getopts d:f: OPTION
do
    case $OPTION in
        d)  DELIMITER=$OPTARG ;;
        f)  INFILE=$OPTARG ;;
    esac
done

#Less than 2 command line argument, throw Usage
[ "$#" -lt 2 ] && f_Usage && exit $NOARG

DEFAULTDELIMITER=","
#If no delimiter is supplied, default delimiter is comma i.e. ,
SEPARATOR=${DELIMITER:-$DEFAULTDELIMITER}

if [ -f "${INFILE}" ]
        then
                printf "<table border=\"1\">"
                sed "s/$SEPARATOR/<\/td><td>/g" $INFILE | while read line
                        do
                                printf "<tr><td>${line}</td></tr>"
                done
                printf "</table>"
                echo
fi

e.g. Input file:

$ cat data.txt 
First Name:Last Name:Points
Alex:Hall:45
Niraj:Kumar:290
Brian:Smith:100

Executing it:

$ ./convert_2_html_table.sh -d ":" -f data.txt  > data.html

Beginning of line in UNIX screen session

2012-11-25T23:03:00.002+05:30

As you know every "screen" command begins with "Ctrl-a", then how to go to beginning of the line when you are working under a "screen" UNIX session (which we achieve by "Ctrl-a" in a normal UNIX session) ?

Solution:
Under a "screen" UNIX session you can do "Ctrl-a a" to move to the beginning of the line.

So in order to print ^A (hex: \x01) in a UNIX "screen" session, usual "Ctrl-v Ctrl-a" will not work, you will type "Ctrl-v Ctrl-a a" to print ^A.

Related posts:
- Save command without executing it in Bash history
- Ctrl s freezes UNIX shell and vi editor
- UNIX - prevent exit of shell with Ctrl-d
- Print UNIX command line history with time-stamp of command execution

Transpose using UNIX paste command

2012-11-21T08:24:00.001+05:30

I have already posted about performing transpose of a matrix using Awk in my earlier posts post1 and post2 . This example shows a not so efficient way of achieving the same.

Input file:

$ cat file.txt 
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
32420 testuser 20   0 1219m 969m  49m R   34 12.1   1043:14 worker

Required output:

PID 32420
USER testuser
PR 20
NI 0
VIRT 1219m
RES 969m
SHR 49m
S R
%CPU 34
%MEM 12.1
TIME+ 1043:14
COMMAND worker

$ for i in $(seq 12); do awk -v X=$i '{print $X}' file.txt | paste - - ; done

UNIX - append text to filename using Awk

2012-11-14T22:18:00.000+05:30

I have some files in my current directory whose file-name is of this pattern:

$ ls -1
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt

Requirement: Append a text "MY-2" as the 6th field (dot delimited) of the filename. E.g.

 
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt

should be renamed to

log.1024.94.1326776200.1326776300.MY-2.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt

A bash script using awk to achieve this:

for file in $(ls)
 do 
  newfilename=$(echo $file | awk 'BEGIN {FS=OFS="."} {$6="MY-2" OFS $6} {print}')
  mv -v $file $newfilename
done

All the files are renamed to:

$ ls -1
log.1024.94.1326776200.1326776300.MY-2.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.MY-2.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.MY-2.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt

Feel free to post (as comment below) any alternative to this, much appreciated and a big thank you in advance.
Related posts:

- Padding zeros in filename using Bash in UNIX
- Rename file to uppercase except extension - Bash
- Rename multiple files using UNIX rename command

Time bucketing using Awk in Unix

2012-10-30T21:11:00.002+05:30

Each line of input file file.txt is having UNIX epoch timestamp and some value.

$ cat file.txt 
Epoch,Value
1351605000,120
1351605060,130
1351605120,340
1351605180,200
1351605240,120
1351605300,890
1351605360,124
1351605420,450
1351605480,120
1351605540,120
1351605600,200
1351605660,120
1351605720,340
1351605780,670
1351605840,990

The UNIX epoch timestamps are of 1 minute interval. Converting the UNIX epoch timestamps to human readable format using awk 'strftime' function:

$ awk 'BEGIN {FS=OFS=","} NR != 1 {$1=strftime("%c",$1)} {print}' file.txt

Output:

Epoch,Value
Tue 30 Oct 2012 01:50:00 PM GMT,120
Tue 30 Oct 2012 01:51:00 PM GMT,130
Tue 30 Oct 2012 01:52:00 PM GMT,340
Tue 30 Oct 2012 01:53:00 PM GMT,200
Tue 30 Oct 2012 01:54:00 PM GMT,120
Tue 30 Oct 2012 01:55:00 PM GMT,890
Tue 30 Oct 2012 01:56:00 PM GMT,124
Tue 30 Oct 2012 01:57:00 PM GMT,450
Tue 30 Oct 2012 01:58:00 PM GMT,120
Tue 30 Oct 2012 01:59:00 PM GMT,120
Tue 30 Oct 2012 02:00:00 PM GMT,200
Tue 30 Oct 2012 02:01:00 PM GMT,120
Tue 30 Oct 2012 02:02:00 PM GMT,340
Tue 30 Oct 2012 02:03:00 PM GMT,670
Tue 30 Oct 2012 02:04:00 PM GMT,990

Required: Group each 5 minutes entries together to one time bucket and perform a corresponding sum(Value) of the entries. E.g. All entries from '01:50:00 PM' (1351605000) till '01:54:00 PM' (1351605240) should be grouped together to '01:50:00 PM' (1351605000) time bucket, '01:55:00 PM' (1351605300) till '01:59:00 PM' (1351605540) to '01:55:00 PM' (1351605300) bucket and so on.

$ awk 'BEGIN {FS=OFS=","}  NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt

Output:

 
1351605000,910
1351605300,1704
1351605600,2320

Converting UNIX epoch timestamps to human readable format:

$ awk 'BEGIN {FS=OFS=","}  NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt  | awk '
BEGIN {FS=OFS=","} {$1=strftime("%c",$1)} {print}'

Output:

Tue 30 Oct 2012 01:50:00 PM GMT,910
Tue 30 Oct 2012 01:55:00 PM GMT,1704
Tue 30 Oct 2012 02:00:00 PM GMT,2320

Related posts:
- Grouping files using Awk in Bash shell
- Awk convert epoch to date format
- SUM and GROUP BY using Awk in UNIX post1 post2
- COUNT number of occurrences using Awk
- Awk Associative array

Awk - separate last field with spaces

2012-09-30T14:06:00.001+05:30

Each line of following file 'file.txt' actually has total 4 fields/columns. As you can see some of the values of the last field (4th field, header : Description) has spaces in them.

$ cat file.txt 
ID Name Active Description
2312 DEMO-1 1 Demo for VOD LIVE
1245 DEMO-4 1 LIVE
1002 CUST2 0 VOD Event
19000 DEMO-2 1 VOD
189 DEMO-3 1 Demo for LIVE

Required output:
We need to separate the 4 fields with a comma (,) delimiter such that all the values after 3rd field is considered as 4th field/column. i.e. Required output is something like this:

ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE

One way to achieve this is to add comma (,) to each of the first 3 fields of every line/row.

$ awk ' { for ( i=1;i<=3;i++ ) {$i=$i","} print }' file.txt

Output:

ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE

Any other alternative ? Feel free to post in the comment section, much appreciated. Thanks.

And to print the last field of the above input file one solution using UNIX cut command:

$ cut -d" " -f4- file.txt

Output:

Description
Demo for VOD LIVE
LIVE
VOD Event
VOD
Demo for LIVE

Compare two numeric fields of two files - UNIX awk

2012-09-26T23:15:00.002+05:30

Following example will show how we can use UNIX paste and join command with awk to find difference of two numeric columns of two similar files.

e.g. Input file:

$ cat file1.txt 
Aug-1,100
Aug-2,220
Aug-4,230
Aug-5,100

$ cat file2.txt 
Aug-1,100
Aug-2,120
Aug-4,400
Aug-5,250

Required output:
As you can see both file1.txt and file2.txt has got first field (comma separated) in the same order. We will have to find out the corresponding (2nd field of file1.txt - 2nd field of file2.txt) and the output required is something like this:

Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Solution#1:

$ paste -d "," file1.txt file2.txt
Aug-1,100,Aug-1,100
Aug-2,220,Aug-2,120
Aug-4,230,Aug-4,400
Aug-5,100,Aug-5,250

$ paste -d "," file1.txt file2.txt | awk 'BEGIN {FS=OFS=","} {print $1,$2,$NF,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Solution#2:

$ join -t, -1 1 -2 1 file1.txt file2.txt 
Aug-1,100,100
Aug-2,220,120
Aug-4,230,400
Aug-5,100,250

$ join -t, -1 1 -2 1 file1.txt file2.txt | awk ' BEGIN {FS=OFS=","} {print $0,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150

Print double quotes in unix Awk

2012-07-23T10:20:00.000+05:30

I have already described in one of the earlier posts on how to print string within single quote in Awk print statement. Here is how we can print strings without "double quotes" in Awk.

$ cat file.txt 
6289693505455 Plan_DAIL_30D_AA
6289693505475 Plan_DAIL_30D_AA
6289693505462 Plan_DAIL_30D_AB

Output required:

Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

i.e. Print the first field of the file within "double quotes".
Here are some of the alternatives:

$ awk '{print $2,$1}' file.txt 
Plan_DAIL_30D_AA 6289693505455
Plan_DAIL_30D_AA 6289693505475
Plan_DAIL_30D_AB 6289693505462

$ awk '{print $2,"\""$1"\""}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Assigning the quotes sequence to a variable x 
$ awk -v x="\"" '{print $2,x$1x}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Using octal code of double quotes
$ awk '{print $2,"\042"$1"\042"}' file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

#Using ASCII code of double quotes
$ awk '{print $2,"\x22"$1"\x22"}'  file.txt 
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"

Unix sort file ignoring first line

2012-07-12T10:30:00.000+05:30

Input file:

$ cat file.txt 
chrom:index:forward:reverse
chr01:13:1:2
chr03:12:1:4
chr01:3445:1:6
chr02:2311:3:1
chr13:23432:4:7
chr01:212:5:2
chr02:345:12:6
chr01:45:45:0

Output required:

 
chrom:index:forward:reverse
chr01:13:1:2
chr01:45:45:0
chr01:212:5:2
chr01:3445:1:6
chr02:345:12:6
chr02:2311:3:1
chr03:12:1:4
chr13:23432:4:7

i.e. Sort file.txt ignoring the header line (1st line).
Sort first field in ascending order (string)
Sort second field in ascending order (numeric)
One of the solution would be:

$ head -1 file.txt ; tail -n +2 file.txt | sort -t : -k1,1 -k2,2n

As you can see two commands are combined together to satisfy the output.
i) First part is printing the first line of file.txt
ii) Second part is printing the lines except first line and then sorting the lines based on first field (string) and second field (numeric sort).
Related posts:
- Unix sort file based on last field
- Sort strings by length using Awk
- Unix sort date in ddmmyyyy format using Awk

Run UNIX bash loop with nohup

2012-02-18T19:58:00.002+05:30

Suppose we have to run following UNIX bash shell loop construct under 'nohup'

$ for i in $(seq 20); do echo $i;./somescript.sh $i; done

Running the above loop under nohup directly will not work as 'nohup' expects a single-word command and its arguments. This is how we can run a UNIX bash shell loop construct using 'nohup'.

$ nohup sh -c 'for i in $(seq 20); do echo $i;./somescript.sh $i; done' &

From man page of 'sh'

-c               Read commands from the command_string operand instead of from the
standard input.  Special parameter 0 will be set from the command_name operand
and the positional parameters ($1, $2, etc.)  set from the remaining argument operands.

Print line number with unix less command

2011-10-28T09:32:00.002+05:30

Less command introduction:
As you know Unix less command writes the contents of a file onto the screen a page at a time and this is one of the utilities using which one can view the content of a file without opening it in an editor.
Press the [space-bar] if you want to see another page, and type [q] if you want to quit reading.

This is how we can print line number with Unix less command.

1)
From man page of CAT(1):

       -n, --number
              number all output lines

So,

$ cat -n file.txt | less

will print the line number in-front of each line.

2)
From man page of LESS(1):

       -N or --LINE-NUMBERS
              Causes a line number to be displayed at the beginning of each line in the display.

So,

$ less -N file.txt

will do the same as 1) above.

3) One can set the following to print line number with less by default:

$ export LESS='-RS#3NM~g'

Awk - convert epoch to date in the same file

2011-06-27T22:07:00.004+05:30

Example 1)
Input file:

$ cat test1.txt
2|Z|1219071600|AF|0
3|N|1219158000|AF|89
4|N|1220799600|AS|12
1|Z|1220886000|AS|67
5|N|1220972400|EU|23
6|R|1221058800|OC|89

Required output:

2|Z|Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

i.e. convert the UNIX epoch values on 3 rd field of the above file to standard human readable date format using Awk.

The Awk solution using "strftime" function:

$ awk 'BEGIN {FS=OFS="|"}{$3=strftime("%c",$3)} {print}' test1.txt

More about strftime format specifiers can be found here

I have created a python program to achieve the above output and the script is here

Example 2)
Input file:

$ cat test2.txt
2|Z|time:1219071600|AF|0
3|N|time:1219158000|AF|89
4|N|time:1220799600|AS|12
1|Z|time:1220886000|AS|67
5|N|time:1220972400|EU|23
6|R|time:1221058800|OC|89

Required output:

2|Z|time:Mon 18 Aug 2008 03:00:00 PM UTC|AF|0
3|N|time:Tue 19 Aug 2008 03:00:00 PM UTC|AF|89
4|N|time:Sun 07 Sep 2008 03:00:00 PM UTC|AS|12
1|Z|time:Mon 08 Sep 2008 03:00:00 PM UTC|AS|67
5|N|time:Tue 09 Sep 2008 03:00:00 PM UTC|EU|23
6|R|time:Wed 10 Sep 2008 03:00:00 PM UTC|OC|89

The awk solution:

$ awk 'BEGIN {FS=OFS="|"}{$3="time:"strftime("%c",substr($3,6,10))} {print}' test2.txt

Filter line using Awk split function

2011-06-20T22:09:00.003+05:30

Input file "file.txt" has data in the following format:

$ cat file.txt
232323 90 /T/1382/8558/365p/133 100 234 679
S/1234 90 /N/1389/5000/365s/5000 800 134 679
792363 80 /T/1381/9858/365q/133 100 234 779
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Required output: From the above file print only the lines whose 3rd field has the value "5000" as the 4th field (separated by the delimiter "/") in it. i.e. required output:

S/1234 90 /N/1389/5000/365s/5000 800 134 679
136383 90 /K/1382/5000/365p/5000 500 934 979
136383 90 /T/1382/5000/279p/9000 100 134 601

Using awk:

$ awk '{
    split($3,arr,"/")
    if(arr[4] == 5000) {
        print $0
        }
}' file.txt

The awk function split(s,a,sep) splits a string "s" into an awk array "a" using the delimiter "sep".

A simple UNIX bash script to solve this:

$ while read line
    do 
        thirdf=$(echo "$line" | awk '{print $3}')
        fourthf=$(echo "$thirdf" | awk -F "/" '{print $4}')
        [ "$fourthf" -eq 5000 ] && echo $line
done < file.txt