tag:blogger.com,1999:blog-87964931751952668162023-04-06T21:24:48.259+05:30UNIX Command Line"Where there is a shell, there is a WAY !!"
Blog on Awk, Sed, BASH ones liners and scripts.
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.comBlogger425125tag:blogger.com,1999:blog-8796493175195266816.post-10407228582590961932016-09-08T09:17:00.001+05:302016-09-08T09:17:26.743+05:30Display ASCII pictures on Unix terminal<div dir="ltr" style="text-align: left;" trbidi="on">
'cowsay' is a program which generates ASCII pictures of a cow with a message.
<br />
<pre>🐞 jsaikia@unixcl.com:~$ echo "Hello World" | cowsay
_____________
< Hello World >
-------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
</pre>
You can change the "Cow" using the following option:
<br />
<pre>
🐞 jsaikia@unixcl.com:~$ echo "Hello World" | cowsay -f tux
_____________
< Hello World >
-------------
\
\
.--.
|o_o |
|:_/ |
// \ \
(| | )
/'\_ _/`\
\___)=(___/
🐞 jsaikia@unixcl.com:~$ echo "Hello World" | cowsay -f elephant
_____________
< Hello World >
-------------
\ /\ ___ /\
\ // \/ \/ \\
(( O O ))
\\ / \ //
\/ | | \/
| | | |
| | | |
| o |
| | | |
|m| |m|
</pre>
Available text characters are:
<br />
<pre>🐞 jsaikia@unixcl.com:~$ cowsay -l
Cow files in /usr/local/Cellar/cowsay/3.04/share/cows:
beavis.zen blowfish bong bud-frogs bunny cheese cower daemon default dragon
dragon-and-cow elephant elephant-in-snake eyes flaming-sheep ghostbusters
head-in hellokitty kiss kitty koala kosh luke-koala meow milk moofasa moose
mutilated ren satanic sheep skeleton small sodomized stegosaurus stimpy
supermilker surgery telebears three-eyes turkey turtle tux udder vader
</pre>
<br />
'fortune' is yet another implementation of the Unix-style fortune program that displays a random message from a database of quotations. Something like:
<br />
<pre>🐞 jsaikia@unixcl.com:~$ fortune
It is not every question that deserves an answer.
-- Publilius Syrus
🐞 jsaikia@unixcl.com:~$ fortune
It is so soon that I am done for, I wonder what I was begun for.
-- Epitaph, Cheltenham Churchyard
</pre>
Now, you can make your cow quote the beautiful quotes from 'fortune' and create something like the one I have (I have added this to my .bash_profile):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-enNl9fbzmsg/V9Dbyep91QI/AAAAAAAAN_I/DCd_dMB7CzkSRlIDqyjiG79gbAebkHZwQCLcB/s1600/Unix-ASCII-art-command-prompt-fortune-cowsay.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="226" src="https://1.bp.blogspot.com/-enNl9fbzmsg/V9Dbyep91QI/AAAAAAAAN_I/DCd_dMB7CzkSRlIDqyjiG79gbAebkHZwQCLcB/s320/Unix-ASCII-art-command-prompt-fortune-cowsay.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
You can add the following to your .bash_profile or .bashrc to see a new cowsay fortune for every new terminal session:
<pre>fortune | cowsay
</pre>
If you are on Mac OSX, you can get the cowsay and fortune installation details <a href="https://gist.github.com/evnm/1308428#file-brew-install-missing-unix-tools">here</a>. If you need help in getting homebrew installed on your Mac OSX, here's the <a href="https://coolestguidesontheplanet.com/installing-homebrew-on-os-x-el-capitan-10-11-package-manager-for-unix-apps/">help</a> link.
<br /></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-44519218187859129802016-01-01T07:58:00.002+05:302016-01-01T07:58:49.950+05:30Happy New Year 2016 in Unix way<div dir="ltr" style="text-align: left;" trbidi="on">
Wishing everyone a great 2016 ahead !<br />
<br />
<a href="http://3.bp.blogspot.com/-BGg6e7V2ft0/VoXjTmtVuLI/AAAAAAAANlc/qFHuExIlaMs/s1600/Happy%2BNew%2BYear%2B2016%2B-%2BUnix.png" imageanchor="1"><img border="0" height="120" src="http://3.bp.blogspot.com/-BGg6e7V2ft0/VoXjTmtVuLI/AAAAAAAANlc/qFHuExIlaMs/s640/Happy%2BNew%2BYear%2B2016%2B-%2BUnix.png" width="640" /></a><br />
<br />
"toilet" gives you color in UNIX terminal ! Read more <a href="https://www.linux.com/learn/docs/686943-linux-tips-fun-with-figlet-and-toilet">here</a><br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-50059160421166674172015-11-25T13:04:00.000+05:302015-11-25T13:04:29.984+05:30Display cricket score on Unix Terminal<div dir="ltr" style="text-align: left;" trbidi="on">
If your job demands you to be on Unix console all the time and you happen to be one who can not avoid watching the cricket score, something you would like:<br />
The script:
<br />
<pre>#!/bin/sh
#Get cricket score on your unix console top corner, enjoy cricket while you work ..
columns=$(tput cols)
startpoint=$(($columns-100))
matchurl='http://www.espncricinfo.com/india-v-south-africa-2015-16/engine/match/903607.html'
while :
do
line=$(wget -q -O- $matchurl | awk -F '=' '/og:title/ {print $3}' | awk -F '|' '{print $1}' | sed 's/"//g')
echo -en "\033[s" #save current screen position & attributes
tput cup 0 $startpoint
echo -en "\033[42m$line\033[0m"
echo -e -n "\033[u"
sleep 15
done
</pre>
Output:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Jg22qneH9gY/VlVj6mtvBNI/AAAAAAAANj8/i3Mz5UCPn3Y/s1600/Unix-Cricket-score-update-script.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Jg22qneH9gY/VlVj6mtvBNI/AAAAAAAANj8/i3Mz5UCPn3Y/s320/Unix-Cricket-score-update-script.png" /></a></div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com4tag:blogger.com,1999:blog-8796493175195266816.post-19155365109681261002015-11-06T16:03:00.003+05:302015-11-06T16:03:22.829+05:30Bash - Get function name inside a function<div dir="ltr" style="text-align: left;" trbidi="on">
$FUNCNAME has the function name in a bash script which can be used for useful purpose like the following:
Script:test.sh
<pre>
!/bin/sh
WORKID=$(date +%s) ; mkdir $WORKID
LOGFILE=$WORKID/$(basename $0).log
_f_log() {
echo "$(date):$@" >> $LOGFILE
}
_f_testFunc() {
local logfile=$1
local status=$2
_f_log "$FUNCNAME:$logfile:$status"
}
_f_testFunc /tmp/a.log restart
</pre>
Executing it:
<pre>
$ ./test.sh
</pre>
Result:
<pre>
$ cat 1446805714/test.sh.log
Fri Nov 6 15:58:34 IST 2015:_f_testFunc:/tmp/a.log:restart
</pre>
<br /></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com2tag:blogger.com,1999:blog-8796493175195266816.post-5853772285066395112015-03-07T21:58:00.000+05:302015-03-07T21:58:10.826+05:30Unix - run multiple commands on remote machines in parallel<div dir="ltr" style="text-align: left;" trbidi="on">
Assume you have a requirement of running two commands on 10 hosts.<br />
<br />
In this post I would like to show you how you can:<br />
<br />
1) Run multiple commands on remote host using ssh.<br />
<br />
2) Also instead of working on one host at a time serially, how can we execute the commands on multiple hosts in parallel to save the overall execution time.<br />
<br />
<b><u>Example:</u></b><br />
Find the number of files under '/sdb/_read/' and '/sdb/_write/' directories on following 10 hosts.
<br />
<br />
<pre><span style="color: red;">$</span> cat host.txt
192.19.66.203
192.19.66.204
192.19.66.205
192.19.66.206
192.19.66.207
192.19.66.208
192.19.66.209
192.19.66.210
192.19.66.211
192.19.66.212
</pre>
<br />
<b><u>Serial execution: </u></b><br />
We can write a bash for loop and work on each host like this:
<br />
<br />
<pre><span style="color: red;">$</span> for ip in $(cat host.txt); do echo -n "$ip|"; ssh -2 root@$ip 'R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l) ; echo "$R|$D"'; done
</pre>
<br />
Time taken:
<br />
<pre>real 0m50.084s
user 0m0.052s
sys 0m0.032s
</pre>
<br />
<b><u>Parallel execution:</u></b><br />
I have already blogged about Unix xargs parallel execution option <a href="http://www.unixcl.com/2014/04/unix-xargs-parallel-execution-of.html">here</a>, using that:
<br />
<br />
<pre><span style="color: red;">$</span> cat host.txt | xargs -n 1 -P 5 -i ssh -2 root@{} 'I=$(hostname -i);R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l) ; echo "$I|$R|$D"'
</pre>
<br />
Time taken:
<br />
<pre>real 0m10.437s
user 0m0.044s
sys 0m0.036s
</pre>
<br />
10 Seconds vs 50 Seconds, wow !<br />
<br />
Output:<br />
<pre>192.19.66.205|0|4
192.19.66.204|0|0
192.19.66.203|0|0
192.19.66.207|5|0
192.19.66.206|0|0
192.19.66.208|0|0
192.19.66.209|0|7
192.19.66.210|0|0
192.19.66.211|2|1
192.19.66.212|9|0
</pre>
<br />
Note: Utilities like <a href="http://www.gnu.org/software/parallel/">GNU parallel</a> or <a href="https://code.google.com/p/parallel-ssh/">Parallel SSH</a> are always the best tool in this job.<br />
<br />
<b><u>Related posts:</u></b><br />
<br />
<ol style="text-align: left;">
<li><a href="http://www.unixcl.com/2014/04/unix-xargs-parallel-execution-of.html">Unix Xargs paralel execution</a></li>
<li><a href="http://www.unixcl.com/2014/12/execute-script-on-remote-server-ssh-unix.html">Unix - Execute script on remote host using SSH without copying the script.</a></li>
<li><a href="http://www.unixcl.com/2009/01/diff-remote-files-using-ssh-in-linux.html">Diff remote files using SSH in Unix</a></li>
<li><a href="http://www.unixcl.com/2008/01/using-ssh-from-bash.html">Bash and SSH some examples</a></li>
</ol>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com1tag:blogger.com,1999:blog-8796493175195266816.post-13179169359080959262015-01-01T00:21:00.001+05:302015-01-01T00:21:19.977+05:30Happy new year 2015 to all unix lovers<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
Wishing all the readers of unixcl.com a Happy and Healthy 2015 ahead, keep rocking !</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-YFjDu2IrJc8/VKRFJ7-eTgI/AAAAAAAAMKg/bv5ur3Z29rE/s1600/Happy%2BNew%2BYear%2B2015.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-YFjDu2IrJc8/VKRFJ7-eTgI/AAAAAAAAMKg/bv5ur3Z29rE/s1600/Happy%2BNew%2BYear%2B2015.png" height="361" width="400" /></a></div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-42701891259384008862014-12-20T17:44:00.002+05:302014-12-20T17:44:58.073+05:30Unix - Delete blank lines from file<div dir="ltr" style="text-align: left;" trbidi="on">
Here's some ways to remove empty or blank lines from a file in Unix. Simple but definitely useful.
<br />
<pre><span style="color: red;">$</span> grep -v '^$' file.txt
<span style="color: red;">$</span> grep '.' file.txt
<span style="color: red;">$</span> sed '/^$/d' file.txt
<span style="color: red;">$</span> sed -n '/^$/!p' file.txt
<span style="color: red;">$</span> awk NF file.txt
<span style="color: red;">$</span> awk '/./' file.txt
</pre>
<br />
In Vi editor, in escape mode type
<br />
<pre>:g/^$/ d
</pre>
Related posts:
<br />
<br />
<ul style="text-align: left;">
<li><a href="http://www.unixcl.com/2008/12/remove-duplicate-blank-lines-bash-awk.html">Unix - Remove duplicate blank lines from file using Awk</a></li>
<li><a href="http://www.unixcl.com/2009/12/awk-numbering-lines-ignoring-blank.html">Unix - Numbering lines of a file ignoring blank lines using Awk</a></li>
</ul>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com2tag:blogger.com,1999:blog-8796493175195266816.post-43737257596030052672014-12-13T13:49:00.000+05:302014-12-13T13:49:21.830+05:30Execute script on remote server SSH - Unix<div dir="ltr" style="text-align: left;" trbidi="on">
Assume you have a complex bash script named 'run_q.sh' (containing good number of loops, Awk statements, multiple lines of code etc) that you need to run against one or more remote servers via SSH.
One way would be to copy (i.e. scp) the script to the remote server(s) and then do something like this:
e.g.
<br />
<pre><span style="color: red;">$</span> scp -2 run_q.sh root@192.168.32.8:/tmp/.
<span style="color: red;">$</span> ssh -2 root@192.168.32.8 sh /tmp/run_q.sh
</pre>
If its multiple machines, then you can use bash for loop like this:
<br />
<pre><span style="color: red;">$</span> for ip in 192.168.32.8 192.168.32.9 192.168.32.3 ; do echo "Executing on IP: $ip" ; ssh -2 root@$ip 'sh /tmp/run_q.sh' ; done
</pre>
Now assume you do not want to copy your script to the remote machine(s) and want to execute from your local machine without copying it.
Here's a way:
<br />
<pre><span style="color: red;">$</span> cat run_q.sh | ssh -2 root@192.168.32.8 /bin/sh
</pre>
And if its to be executed on multiple machines, you can use a for loop like this:
<br />
<pre><span style="color: red;">$</span> for ip in 192.168.32.8 192.168.32.9 192.168.32.3 ; do echo "Executing on IP: $ip" ; cat run_q.sh | ssh -2 root@$ip /bin/sh ; done
</pre>
I am sure you liked this trick ! Feel free to comment below. Thanks.
<br />
<br />
<b>Related posts:
</b><br />
<br />
<ul style="text-align: left;">
<li><a href="http://www.unixcl.com/2009/01/diff-remote-files-using-ssh-in-linux.html">Diff remote files using SSH in Unix</a></li>
</ul>
<br />
<b><br /></b></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com1tag:blogger.com,1999:blog-8796493175195266816.post-18342037303125749602014-04-27T09:11:00.002+05:302014-04-27T09:11:49.558+05:30Unix xargs parallel execution of commands<div dir="ltr" style="text-align: left;" trbidi="on">
Xargs has option that allows you to take advantage of multiple cores in your machine.
Its -P option which allows xargs to invoke the specified command multiple times in parallel.
From XARGS(1) man page:
<br />
<pre>-P max-procs
Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option
with -P; otherwise chances are that only one exec will be done.
-n max-args
Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x
option is given, in which case xargs will exit.
-i[replace-str]
This option is a synonym for -Ireplace-str if replace-str is specified, and for -I{} otherwise. This option is deprecated; use -I instead.
</pre>
Let me try to give one example where we can make use of this parallel option avaiable on xargs.
e.g. I got these 8 log files (each one is of 1.5G size) for which I have to run a script named count_pipeline.sh which does some calculation around the log lines in the log file.
<br />
<pre>$ ls -1 *.out
log1.out
log2.out
log3.out
log4.out
log5.out
log6.out
log7.out
log8.out
</pre>
The script count_pipeline.sh takes nearly 20 seconds for a single log file. e.g.
<br />
<pre>$ time ./count_pipeline.sh log1.out
real 0m20.509s
user 0m20.967s
sys 0m0.467s
</pre>
If we have to run count_pipeline.sh for each of the 8 log files one after the other, total time needed:
<br />
<pre>$ time ls *.out | xargs -i ./count_pipeline.sh {}
real 2m45.862s
user 2m48.152s
sys 0m5.358s
</pre>
Running with 4 parallel processes at a time (I am having a machine which is having 4 CPU cores):
<br />
<pre>$ time ls *.out | xargs -i -P4 ./count_pipeline.sh {}
real 0m44.764s
user 2m55.020s
sys 0m6.224s
</pre>
We saved time ! Isn't this useful ?
You can also use -n1 option instead of the -i option that I am using above.
-n1 passes one arg a time to the run comamnd (instead of the xargs default of passing all args).
<br />
<pre>$ time ls *.out | xargs -n1 -P4 ./count_pipeline.sh
real 0m43.229s
user 2m56.718s
sys 0m6.353s
</pre>
<br />
Related posts:<br />
- <a href="http://www.unixcl.com/2010/08/unix-handle-kill-when-no-pid.html">UNIX handle kill when no PID available</a> </div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com2tag:blogger.com,1999:blog-8796493175195266816.post-38997930449996713972013-07-26T09:34:00.004+05:302013-07-26T09:34:39.493+05:30Unix - merge multiple consecutive lines<div dir="ltr" style="text-align: left;" trbidi="on">
Input file:<br />
<pre><span style="color: red;">$</span> cat infile.txt
aid=33
pw=3
nn=90
aid=32
pw=30
nn=70
aid=56
pw=3
nn=93
</pre>
Required:<br />
Combine or merge every three consecutive lines of the above file so that the output becomes:
<br />
<pre>aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93
</pre>
Awk solution: If line number is divisible by 3 then put a new line(\n) else put a comma(,) i.e.
<br />
<pre><span style="color: red;">$</span> awk '{printf("%s%s", $0, (NR%3 ? "," : "\n"))}' infile.txt
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93
</pre>
Another way using Awk:
<br />
<pre><span style="color: red;">$</span> awk 'NR%3{printf $0",";next;}1' infile.txt
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93
</pre>
Using UNIX paste command:
<br />
<pre><span style="color: red;">$</span> paste -d"," - - - < infile.txt
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93
</pre>
A bash command line solution:
<br />
<pre><span style="color: red;">$</span> while read line1; do read line2; read line3; echo "$line1,$line2,$line3"; done < infile.txt
aid=33,pw=3,nn=90
aid=32,pw=30,nn=70
aid=56,pw=3,nn=93
</pre>
Related posts:<br />
<ol style="text-align: left;">
<li><a href="http://www.unixcl.com/2008/04/join-multiple-lines-using-awk.html">Join multiple lines using Awk</a></li>
<li><a href="http://www.unixcl.com/2009/05/combine-related-consecutive-lines-awk.html">Combine related consecutive lines using Awk</a></li>
<li><a href="http://www.unixcl.com/2008/07/merging-lines-using-awk-bash.html">Merging lines in UNIX</a></li>
</ol>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com6tag:blogger.com,1999:blog-8796493175195266816.post-81830316341775363712013-05-08T10:33:00.004+05:302013-05-08T10:33:40.513+05:30Unix - Append 0 to single digit date<div dir="ltr" style="text-align: left;" trbidi="on">
Input file file.txt has dates in month/day/year format.
<br />
<pre><span style="color: red;">$</span> cat file.txt
3/4/2013
3/10/2013
10/4/2013
12/10/2012
</pre>
Required: Add prefix 0 to first and second field if its a single digit. <br />
<br />
Awk solution:
<br />
<pre><span style="color: red;">$</span> awk 'BEGIN {FS=OFS="/"}
{
if (length($1) == 1) $1="0"$1
if (length($2) == 1) $2="0"$2
{ print }
}' file.txt
</pre>
Output:
<br />
<pre>03/04/2013
03/10/2013
10/04/2013
12/10/2012
</pre>
Related posts: <br />
- <a href="http://unstableme.blogspot.in/2010/11/newbie-tutorial-on-unix-awk-utility.html">A newbie tutorial on Unix Awk</a><br />
- <a href="http://unstableme.blogspot.in/2009/09/if-else-examples-in-awk-bash.html">Awk if else</a><br />
- <a href="http://unstableme.blogspot.in/2008/07/convert-date-format-using-sed-awk-perl.html">Convert date format in unix using awk and sed</a><br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com6tag:blogger.com,1999:blog-8796493175195266816.post-51180655209298478742013-02-28T08:54:00.000+05:302013-02-28T08:54:02.816+05:30UNIX time command output redirect to file<div dir="ltr" style="text-align: left;" trbidi="on">
As you know the 'time' command run programs and summarize system resource usage. Here is a way how you can redirect 'time' command output to file. I would recomend you to go through <a href="http://www.trentu.ca/academic/physics/batkinson/scalapack_docs/node18.html">this page</a> to understand more about 'time' command.<br />
<br />
e.g. We are executing 'prog.sh' along with 'time' command.
<br />
<pre><span style="color: red;">$</span> time ./prog.sh
Initiating merge
Merge completed
real 0m8.803s
user 0m0.010s
sys 0m0.000s
</pre>
#Trying to redirect the output to a file
<br />
<pre><span style="color: red;">$</span> time ./prog.sh > out.txt
real 0m8.804s
user 0m0.020s
sys 0m0.000s
</pre>
#out.txt content:
<br />
<pre><span style="color: red;">$</span> cat out.txt
Initiating merge
Merge completed
</pre>
#So it only redirected the STDOUT of the script executed, but the 'time' command outut is not redirected. <br />
#This is becuase the command time sends it's output to STDERR (instead of STDOUT)<br />
#To capture output of 'time' command:
<br />
<pre><span style="color: red;">$</span> { time ./prog.sh ; } 2> out.txt
Initiating merge
Merge completed
</pre>
#Now out.txt content:
<br />
<pre><span style="color: red;">$</span> cat out.txt
real 0m8.303s
user 0m0.010s
sys 0m0.000s
</pre>
#And to capture output of script as well as time command:
<br />
<pre><span style="color: red;">$</span> { time ./prog.sh ; } &> out.txt
</pre>
#'out.txt' now has both the outputs.
<br />
<pre><span style="color: red;">$</span> cat out.txt
Initiating merge
Merge completed
real 0m8.303s
user 0m0.020s
sys 0m0.000s
</pre>
As I have mentioned in my earlier <a href="http://unstableme.blogspot.in/2008/03/understand-your-codes-performance-using.html">post</a> on UNIX 'time' command, there's two types of time command available:<br />
1) Shell's in-build time: Gives only scheduler information<br />
2) /usr/bin/time: Gives more information, also allows formatting the output<br />
<br />
The second (/usr/bin/time) one accepts output redirection without code block:
<br />
<pre><span style="color: red;">$</span> /usr/bin/time ./prog.sh &> newout.txt
<span style="color: red;">$</span> cat newout.txt
Initiating merge
Merge completed
0.00user 0.01system 0:08.30elapsed 0%CPU (0avgtext+0avgdata 5728maxresident)k
0inputs+8outputs (0major+719minor)pagefaults 0swaps
</pre>
Related posts:<br />
- <a href="http://unstableme.blogspot.in/2009/01/redirecting-man-pages-to-file-linux.html">UNIX redirect man pages to file</a><br />
- <a href="http://unstableme.blogspot.in/2008/12/redirect-top-command-output-to-file.html">Redirect UNIX top command output to file</a><br />
- <a href="http://unstableme.blogspot.in/2008/08/redirect-both-stderr-and-stdout-to.html">UNIX redirect both stderr and stdout to file</a><br />
- <a href="http://unstableme.blogspot.in/2012/02/run-unix-bash-loop-with-nohup.html">Run UNIX bash loop with nohup command</a></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-46237894527624988632013-02-15T22:27:00.003+05:302013-02-15T22:28:43.509+05:30Bash - convert delimited file to html table<div dir="ltr" style="text-align: left;" trbidi="on">
Here is a simple UNIX bash script that can be used to convert a simple delimited file to a HTML table format. Default delimiter if not mentioned as part of the command line argument to this script will be comma.
<br />
<pre>#!/bin/sh
#convert_2_html_table.sh
#Converts a delimited file to a HTML table
#Jadu Saikia http://unstableme.blogspot.in
NOARG=64
#usage function
f_Usage () {
echo "Usage: $(basename $0) -d <delimiter> -f <delimited-file>"
}
#command line args
while getopts d:f: OPTION
do
case $OPTION in
d) DELIMITER=$OPTARG ;;
f) INFILE=$OPTARG ;;
esac
done
#Less than 2 command line argument, throw Usage
[ "$#" -lt 2 ] && f_Usage && exit $NOARG
DEFAULTDELIMITER=","
#If no delimiter is supplied, default delimiter is comma i.e. ,
SEPARATOR=${DELIMITER:-$DEFAULTDELIMITER}
if [ -f "${INFILE}" ]
then
printf "<table border=\"1\">"
sed "s/$SEPARATOR/<\/td><td>/g" $INFILE | while read line
do
printf "<tr><td>${line}</td></tr>"
done
printf "</table>"
echo
fi
</pre>
e.g. Input file:
<br />
<pre>$ cat data.txt
First Name:Last Name:Points
Alex:Hall:45
Niraj:Kumar:290
Brian:Smith:100
</pre>
Executing it:
<br />
<pre>$ ./convert_2_html_table.sh -d ":" -f data.txt > data.html
</pre>
<a href="http://1.bp.blogspot.com/-r7PsGwwuMWM/UR5n2JMMGHI/AAAAAAAAFSc/-atCvP2OGM4/s1600/Unix-Bash-convert-delimited-file-to-html-table.png" imageanchor="1"><img border="0" src="http://1.bp.blogspot.com/-r7PsGwwuMWM/UR5n2JMMGHI/AAAAAAAAFSc/-atCvP2OGM4/s320/Unix-Bash-convert-delimited-file-to-html-table.png" /> </a><br />
Related posts:<br />
- <a href="http://unstableme.blogspot.in/2007/12/setting-default-value-for-shell.html">Setting default value for Bash shell variable</a><br />
- <a href="http://unstableme.blogspot.in/2008/03/accessing-external-variable-in-awk-and.html">Accessing external variable in SED and AWK</a><br />
- <a href="http://unstableme.blogspot.in/2009/10/bash-while-loop-sum-issue-explained.html">Issue with Bash while loop during SUM</a></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com1tag:blogger.com,1999:blog-8796493175195266816.post-24963971117583860332012-11-25T23:03:00.002+05:302012-11-25T23:03:33.589+05:30Beginning of line in UNIX screen session<div dir="ltr" style="text-align: left;" trbidi="on">
As you know every "screen" command begins with "Ctrl-a", then how to go to beginning of the line when you are working under a "screen" UNIX session (which we achieve by "Ctrl-a" in a normal UNIX session) ? <br /><br />Solution:<br />Under a "screen" UNIX session you can do "Ctrl-a a" to move to the beginning of the line. <br /><br />So in order to print ^A (hex: \x01) in a UNIX "screen" session, usual "Ctrl-v Ctrl-a" will not work, you will type "Ctrl-v Ctrl-a a" to print ^A.<br />
<br />
Related posts:<br />
- <a href="http://unstableme.blogspot.in/2009/08/bash-save-command-without-executing-it.html">Save command without executing it in Bash history</a><br />
- <a href="http://unstableme.blogspot.in/2009/04/ctrl-s-in-linux-shell-vi-hangs.html">Ctrl s freezes UNIX shell and vi editor</a><br />
- <a href="http://unstableme.blogspot.in/2010/01/prevent-exit-of-shell-with-ctrl-d.html">UNIX - prevent exit of shell with Ctrl-d</a><br />
- <a href="http://unstableme.blogspot.in/2009/01/linux-command-line-history-with.html">Print UNIX command line history with time-stamp of command execution </a></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-44557466669535967432012-11-21T08:24:00.001+05:302012-11-21T08:25:42.465+05:30Transpose using UNIX paste command<div dir="ltr" style="text-align: left;" trbidi="on">
I have already posted about performing transpose of a matrix using Awk in my earlier posts <a href="http://unstableme.blogspot.in/2008/05/transpose-using-awk-bash.html">post1</a> and <a href="http://unstableme.blogspot.in/2008/08/row-to-column-transpose-bash-scripting.html">post2</a> . This example shows a not so efficient way of achieving the same. <br/>
<br/>Input file:
<pre><span style="color: red;">$</span> cat file.txt
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32420 testuser 20 0 1219m 969m 49m R 34 12.1 1043:14 worker
</pre>
Required output:
<br />
<pre>PID 32420
USER testuser
PR 20
NI 0
VIRT 1219m
RES 969m
SHR 49m
S R
%CPU 34
%MEM 12.1
TIME+ 1043:14
COMMAND worker
</pre>
<pre><span style="color: red;">$</span> for i in $(seq 12); do awk -v X=$i '{print $X}' file.txt | paste - - ; done
</pre>
Related posts:<br/>
- <a href="http://unstableme.blogspot.in/2008/03/accessing-external-variable-in-awk-and.html">How to access external variable in Awk and SED</a><br/>
- <a href="http://unstableme.blogspot.in/2007/02/generating-loop-arguments-seq-command.html">Generate bash loop arguments using seq command</a><br/>
- <a href="http://unstableme.blogspot.in/2009/09/linux-seq-command-format-option-example.html">UNIX seq command format option example</a><br/>
- <a href="http://unstableme.blogspot.in/2009/01/linux-paste-command-good-examples-uses.html">Some good usage of UNIX paste command</a><br/>
- <a href="http://unstableme.blogspot.in/2012/09/compare-two-numeric-fields-of-two-files.html">Compare two numeric fields of two files in Bash</a><br/>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com3tag:blogger.com,1999:blog-8796493175195266816.post-82001538491042012302012-11-14T22:18:00.000+05:302012-11-14T22:25:10.572+05:30UNIX - append text to filename using Awk<div dir="ltr" style="text-align: left;" trbidi="on">
I have some files in my current directory whose file-name is of this pattern:
<br />
<pre><span style="color: red;">$</span> ls -1
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt
</pre>
Requirement: Append a text "MY-2" as the 6th field (dot delimited) of the filename. E.g.
<br />
<pre>
log.1024.94.1326776200.1326776300.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
</pre>
should be renamed to
<br />
<pre>log.1024.94.1326776200.1326776300.<span style="color: blue;">MY-2</span>.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
</pre>
A bash script using awk to achieve this:
<br />
<pre>for file in $(ls)
do
newfilename=$(echo $file | awk 'BEGIN {FS=OFS="."} {$6="MY-2" OFS $6} {print}')
mv -v $file $newfilename
done
</pre>
All the files are renamed to:
<br />
<pre><span style="color: red;">$</span> ls -1
log.1024.94.1326776200.1326776300.<span style="color: blue;">MY-2</span>.172.16.12.6.1326844995.0.s-1326528000.r-8192.txt
log.1024.94.1326776400.1326776400.<span style="color: blue;">MY-2</span>.172.16.12.5.1326844995.0.s-1326528000.r-2234.txt
log.1024.95.1326776420.1326776460.<span style="color: blue;">MY-2</span>.172.16.12.5.1326844995.0.s-1326528000.r-8192.txt
</pre>
Feel free to post (as comment below) any alternative to this, much appreciated and a big thank you in advance.
<br />
Related posts:<br />
<br />
- <a href="http://unstableme.blogspot.in/2009/06/padding-zeros-in-file-name-using-bash.html">Padding zeros in filename using Bash in UNIX</a><br />
- <a href="http://unstableme.blogspot.in/2009/09/rename-file-to-uppercase-except.html">Rename file to uppercase except extension - Bash</a><br />
- <a href="http://unstableme.blogspot.in/2008/09/bash-rename-command-rename-multiple.html">Rename multiple files using UNIX rename command</a> </div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com4tag:blogger.com,1999:blog-8796493175195266816.post-82474746750298140352012-10-30T21:11:00.002+05:302012-10-30T21:11:30.905+05:30Time bucketing using Awk in UnixEach line of input file file.txt is having UNIX epoch timestamp and some value.
<br />
<pre><span style="color: red;">$</span> cat file.txt
Epoch,Value
1351605000,120
1351605060,130
1351605120,340
1351605180,200
1351605240,120
1351605300,890
1351605360,124
1351605420,450
1351605480,120
1351605540,120
1351605600,200
1351605660,120
1351605720,340
1351605780,670
1351605840,990
</pre>
The UNIX epoch timestamps are of 1 minute interval.
Converting the UNIX epoch timestamps to human readable format using awk 'strftime' function:
<br />
<pre><span style="color: red;">$</span> awk 'BEGIN {FS=OFS=","} NR != 1 {$1=strftime("%c",$1)} {print}' file.txt
</pre>
Output:
<br />
<pre>Epoch,Value
Tue 30 Oct 2012 01:50:00 PM GMT,120
Tue 30 Oct 2012 01:51:00 PM GMT,130
Tue 30 Oct 2012 01:52:00 PM GMT,340
Tue 30 Oct 2012 01:53:00 PM GMT,200
Tue 30 Oct 2012 01:54:00 PM GMT,120
Tue 30 Oct 2012 01:55:00 PM GMT,890
Tue 30 Oct 2012 01:56:00 PM GMT,124
Tue 30 Oct 2012 01:57:00 PM GMT,450
Tue 30 Oct 2012 01:58:00 PM GMT,120
Tue 30 Oct 2012 01:59:00 PM GMT,120
Tue 30 Oct 2012 02:00:00 PM GMT,200
Tue 30 Oct 2012 02:01:00 PM GMT,120
Tue 30 Oct 2012 02:02:00 PM GMT,340
Tue 30 Oct 2012 02:03:00 PM GMT,670
Tue 30 Oct 2012 02:04:00 PM GMT,990
</pre>
Required:
Group each 5 minutes entries together to one time bucket and perform a corresponding sum(Value) of the entries.
E.g. All entries from '01:50:00 PM' (1351605000) till '01:54:00 PM' (1351605240) should be grouped together to '01:50:00 PM' (1351605000) time bucket, '01:55:00 PM' (1351605300) till '01:59:00 PM' (1351605540) to '01:55:00 PM' (1351605300) bucket and so on.
<br />
<pre><span style="color: red;">$</span> awk 'BEGIN {FS=OFS=","} NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt
</pre>
Output:
<br />
<pre>
1351605000,910
1351605300,1704
1351605600,2320
</pre>
Converting UNIX epoch timestamps to human readable format:
<br />
<pre><span style="color: red;">$</span> awk 'BEGIN {FS=OFS=","} NR != 1 {arr[$1-($1%300)]+=$2} END {for (i in arr) print i,arr[i]}' file.txt | awk '
BEGIN {FS=OFS=","} {$1=strftime("%c",$1)} {print}'
</pre>
Output:
<br />
<pre>Tue 30 Oct 2012 01:50:00 PM GMT,910
Tue 30 Oct 2012 01:55:00 PM GMT,1704
Tue 30 Oct 2012 02:00:00 PM GMT,2320
</pre>
Related posts:<br />
- <a href="http://unstableme.blogspot.in/2009/10/grouping-files-using-awk-in-bash-shell.html">Grouping files using Awk in Bash shell</a><br />
- <a href="http://unstableme.blogspot.in/2011/06/awk-convert-epoch-to-date-in-same-file.html">Awk convert epoch to date format</a> <br />
- SUM and GROUP BY using Awk in UNIX <a href="http://unstableme.blogspot.in/2008/09/sum-of-and-group-by-using-awk.html">post1</a> <a href="http://unstableme.blogspot.in/2008/09/group-by-clause-functionality-in-awk.html">post2</a><br />
- <a href="http://unstableme.blogspot.in/2008/05/count-number-of-occurrences-using-awk.html">COUNT number of occurrences using Awk</a><br />
- <a href="http://unstableme.blogspot.in/2008/05/associative-array-in-awk.html">Awk Associative array</a> <br />
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com3tag:blogger.com,1999:blog-8796493175195266816.post-30909555684470456632012-09-30T14:06:00.001+05:302012-09-30T14:08:14.103+05:30Awk - separate last field with spaces<div dir="ltr" style="text-align: left;" trbidi="on">
Each line of following file 'file.txt' actually has total 4 fields/columns. As you can see some of the values of the last field (4th field, header : Description) has spaces in them.
<br />
<pre><span style="color: red;">$</span> cat file.txt
ID Name Active Description
2312 DEMO-1 1 Demo for VOD LIVE
1245 DEMO-4 1 LIVE
1002 CUST2 0 VOD Event
19000 DEMO-2 1 VOD
189 DEMO-3 1 Demo for LIVE
</pre>
Required output:</br>
We need to separate the 4 fields with a comma (,) delimiter such that all the values after 3rd field is considered as 4th field/column.
i.e. Required output is something like this:
<br />
<pre>ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE
</pre>
One way to achieve this is to add comma (,) to each of the first 3 fields of every line/row.
<br />
<pre><span style="color: red;">$</span> awk ' { for ( i=1;i<=3;i++ ) {$i=$i","} print }' file.txt
</pre>
Output:
<br />
<pre>ID, Name, Active, Description
2312, DEMO-1, 1, Demo for VOD LIVE
1245, DEMO-4, 1, LIVE
1002, CUST2, 0, VOD Event
19000, DEMO-2, 1, VOD
189, DEMO-3, 1, Demo for LIVE
</pre>
Any other alternative ? Feel free to post in the comment section, much appreciated. Thanks. </br>
</br>
And to print the last field of the above input file one solution using UNIX cut command:
</br>
<pre><span style="color: red;">$</span> cut -d" " -f4- file.txt
</pre>
Output:
<br />
<pre>Description
Demo for VOD LIVE
LIVE
VOD Event
VOD
Demo for LIVE
</pre>
Related posts:</br>
- <a href="http://unstableme.blogspot.in/2008/01/awk-change-field-separator-add-line.html">Awk change field separator or delimiter of a file</a> </br>
- <a href="http://unstableme.blogspot.in/2010/11/sort-file-based-on-last-field-unix.html">UNIX sort file based on last field</a></br>
- <a href="http://unstableme.blogspot.in/2008/04/print-last-field-grep.html">Different ways to print last field of line in UNIX command line</a></br>
</div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com4tag:blogger.com,1999:blog-8796493175195266816.post-10683079412615427592012-09-26T23:15:00.002+05:302012-09-26T23:15:16.789+05:30Compare two numeric fields of two files - UNIX awk<div dir="ltr" style="text-align: left;" trbidi="on">
Following example will show how we can use UNIX paste and join command with awk to find difference of two numeric columns of two similar files. <br />
<br />
e.g. Input file:
<br />
<pre><span style="color: red;">$</span> cat file1.txt
Aug-1,100
Aug-2,220
Aug-4,230
Aug-5,100
<span style="color: red;">$</span> cat file2.txt
Aug-1,100
Aug-2,120
Aug-4,400
Aug-5,250
</pre>
Required output:<br />
As you can see both file1.txt and file2.txt has got first field (comma separated) in the same order.
We will have to find out the corresponding (2nd field of file1.txt - 2nd field of file2.txt) and the output required is something like this:
<br />
<pre>Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150
</pre>
Solution#1:
<br />
<pre><span style="color: red;">$</span> paste -d "," file1.txt file2.txt
Aug-1,100,Aug-1,100
Aug-2,220,Aug-2,120
Aug-4,230,Aug-4,400
Aug-5,100,Aug-5,250
<span style="color: red;">$</span> paste -d "," file1.txt file2.txt | awk 'BEGIN {FS=OFS=","} {print $1,$2,$NF,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150
</pre>
Solution#2:
<br />
<pre><span style="color: red;">$</span> join -t, -1 1 -2 1 file1.txt file2.txt
Aug-1,100,100
Aug-2,220,120
Aug-4,230,400
Aug-5,100,250
<span style="color: red;">$</span> join -t, -1 1 -2 1 file1.txt file2.txt | awk ' BEGIN {FS=OFS=","} {print $0,$2-$NF}'
Aug-1,100,100,0
Aug-2,220,120,100
Aug-4,230,400,-170
Aug-5,100,250,-150
</pre>
Some similar posts:<br />
- <a href="http://unstableme.blogspot.in/2009/01/linux-paste-command-good-examples-uses.html">A small tutorial on UNIX paste command</a><br />
- <a href="http://unstableme.blogspot.in/2008/03/performing-join-using-awk.html">Performing join using awk</a><br />
- <a href="http://unstableme.blogspot.in/2007/12/command-in-discussion-join-bash.html">Join in bash</a><br />
- <a href="http://unstableme.blogspot.in/2008/06/perform-outer-join-bash-join-command.html">Perform outer join in UNIX Bash</a><br />
- <a href="http://unstableme.blogspot.in/2010/02/bash-find-diff-based-on-field.html">Find diff based on field - UNIX Bash</a></div>
Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com1tag:blogger.com,1999:blog-8796493175195266816.post-29488077184916283512012-07-23T10:20:00.000+05:302012-07-23T10:20:02.676+05:30Print double quotes in unix Awk<div dir="ltr" style="text-align: left;" trbidi="on">
I have already described in one of the earlier <a href="http://unstableme.blogspot.in/2009/11/printing-single-quote-in-awk-bash.html">posts</a> on how to print string within single quote in Awk print statement. Here is how we can print strings without "double quotes" in Awk. <br />
<pre><span style="color: red;">$</span> cat file.txt
6289693505455 Plan_DAIL_30D_AA
6289693505475 Plan_DAIL_30D_AA
6289693505462 Plan_DAIL_30D_AB
</pre>
Output required:
<br />
<pre>Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"
</pre>
i.e. Print the first field of the file within "double quotes". <br />
Here are some of the alternatives:
<br />
<pre><span style="color: red;">$</span> awk '{print $2,$1}' file.txt
Plan_DAIL_30D_AA 6289693505455
Plan_DAIL_30D_AA 6289693505475
Plan_DAIL_30D_AB 6289693505462
<span style="color: red;">$</span> awk '{print $2,"\""$1"\""}' file.txt
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"
#Assigning the quotes sequence to a variable x
<span style="color: red;">$</span> awk -v x="\"" '{print $2,x$1x}' file.txt
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"
#Using octal code of double quotes
<span style="color: red;">$</span> awk '{print $2,"\042"$1"\042"}' file.txt
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"
#Using ASCII code of double quotes
<span style="color: red;">$</span> awk '{print $2,"\x22"$1"\x22"}' file.txt
Plan_DAIL_30D_AA "6289693505455"
Plan_DAIL_30D_AA "6289693505475"
Plan_DAIL_30D_AB "6289693505462"
</pre>
Related posts:
- <a href="http://unstableme.blogspot.in/2008/03/accessing-external-variable-in-awk-and.html">Accessing external variable in Awk and Sed</a></div>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com3tag:blogger.com,1999:blog-8796493175195266816.post-74378151753782902882012-07-12T10:30:00.000+05:302012-07-12T10:30:43.611+05:30Unix sort file ignoring first line<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
Input file:
<br />
<pre><span style="color: red;">$</span> cat file.txt
chrom:index:forward:reverse
chr01:13:1:2
chr03:12:1:4
chr01:3445:1:6
chr02:2311:3:1
chr13:23432:4:7
chr01:212:5:2
chr02:345:12:6
chr01:45:45:0
</pre>
Output required:
<br />
<pre>
chrom:index:forward:reverse
chr01:13:1:2
chr01:45:45:0
chr01:212:5:2
chr01:3445:1:6
chr02:345:12:6
chr02:2311:3:1
chr03:12:1:4
chr13:23432:4:7
</pre>
i.e.
Sort file.txt ignoring the header line (1st line).<br />
Sort first field in ascending order (string)<br />
Sort second field in ascending order (numeric)<br />
One of the solution would be:<br />
<pre><span style="color: red;">$</span> head -1 file.txt ; tail -n +2 file.txt | sort -t : -k1,1 -k2,2n
</pre>
As you can see two commands are combined together to satisfy the output. <br />
i) First part is printing the first line of file.txt <br />
ii) Second part is printing the lines except first line and then sorting the lines based on first field (string) and second field (numeric sort). <br />
Related posts: <br />
- <a href="http://unstableme.blogspot.in/2010/11/sort-file-based-on-last-field-unix.html">Unix sort file based on last field</a> <br />
- <a href="http://unstableme.blogspot.in/2009/08/sort-strings-by-length-using-awk-and.html">Sort strings by length using Awk</a> <br />
- <a href="http://unstableme.blogspot.in/2008/08/sort-date-in-ddmmyyyy-format-awk-and.html">Unix sort date in ddmmyyyy format using Awk</a></div>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com1tag:blogger.com,1999:blog-8796493175195266816.post-34927895655254847712012-02-18T19:58:00.002+05:302012-02-18T20:09:28.294+05:30Run UNIX bash loop with nohupSuppose we have to run following UNIX bash shell loop construct under 'nohup'<br /><pre><span style="color: rgb(255, 0, 0);">$</span> for i in $(seq 20); do echo $i;./somescript.sh $i; done<br /></pre><br />Running the above loop under nohup directly will not work as 'nohup' expects a single-word command and its arguments. This is how we can run a UNIX bash shell loop construct using 'nohup'.<br /><pre><span style="color: rgb(255, 0, 0);">$</span> nohup sh -c 'for i in $(seq 20); do echo $i;./somescript.sh $i; done' &<br /></pre><br />From man page of 'sh'<br /><pre>-c Read commands from the command_string operand instead of from the<br />standard input. Special parameter 0 will be set from the command_name operand<br />and the positional parameters ($1, $2, etc.) set from the remaining argument operands.<br /></pre><br />Related post:<br />- <a href="http://unstableme.blogspot.in/2008/03/setsid-keep-linux-program-running-while.html">Setsid : Run UNIX program in a new session</a><br />- <a href="http://unstableme.blogspot.in/2007/02/generating-loop-arguments-seq-command.html">More about UNIX seq command</a>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com0tag:blogger.com,1999:blog-8796493175195266816.post-32894871182666190892011-10-28T09:32:00.002+05:302011-10-28T09:46:42.097+05:30Print line number with unix less command<span style="font-weight: bold; font-style: italic;">Less command introduction:</span><br />As you know Unix less command writes the contents of a file onto the screen a page at a time and this is one of the utilities using which one can view the content of a file without opening it in an editor.<br />Press the [space-bar] if you want to see another page, and type [q] if you want to quit reading.<br /><br />This is how we can print line number with Unix less command.<br /><br />1)<br />From man page of CAT(1):<br /><pre> -n, --number<br /> number all output lines<br /></pre><br />So,<br /><pre><span style="color: rgb(255, 0, 0);">$</span> cat -n file.txt | less<br /></pre><br />will print the line number in-front of each line.<br /><br />2)<br />From man page of LESS(1):<br /><pre> -N or --LINE-NUMBERS<br /> Causes a line number to be displayed at the beginning of each line in the display.<br /></pre><br />So,<br /><pre><span style="color: rgb(204, 0, 0);">$</span> less -N file.txt<br /></pre><br />will do the same as 1) above.<br /><br />3) One can set the following to print line number with less by default:<br /><pre><span style="color: rgb(204, 0, 0);">$</span> export LESS='-RS#3NM~g'<br /></pre><br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/-B3ZVikgEGpk/TqopZz9IMzI/AAAAAAAADcA/4Tt6V1rLxnU/s1600/Unix%2Bless%2Bcommand%2B-%2Bline%2Bnumber.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 305px; height: 364px;" src="http://4.bp.blogspot.com/-B3ZVikgEGpk/TqopZz9IMzI/AAAAAAAADcA/4Tt6V1rLxnU/s400/Unix%2Bless%2Bcommand%2B-%2Bline%2Bnumber.png" alt="" id="BLOGGER_PHOTO_ID_5668388604419257138" border="0" /></a><br /><span style="font-weight: bold; font-style: italic;">Related posts:</span><br />- <a href="http://unstableme.blogspot.com/2009/11/bash-numbering-lines-in-file-using-awk.html">Numbering lines in a file using Bash awk</a><br />- <a href="http://unstableme.blogspot.com/2010/11/vi-open-file-at-required-line-number.html">Open file at required line number in Unix vi editor</a>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com5tag:blogger.com,1999:blog-8796493175195266816.post-32738956742229265362011-06-27T22:07:00.004+05:302011-06-27T22:31:17.965+05:30Awk - convert epoch to date in the same file<span style="font-style: italic;"><span style="font-weight: bold;"></span></span><span style="font-weight: bold; font-style: italic;">Example 1)</span><br />Input file:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> cat test1.txt<br />2|Z|1219071600|AF|0<br />3|N|1219158000|AF|89<br />4|N|1220799600|AS|12<br />1|Z|1220886000|AS|67<br />5|N|1220972400|EU|23<br />6|R|1221058800|OC|89<br /></pre><br />Required output:<br /><pre>2|Z|Mon 18 Aug 2008 03:00:00 PM UTC|AF|0<br />3|N|Tue 19 Aug 2008 03:00:00 PM UTC|AF|89<br />4|N|Sun 07 Sep 2008 03:00:00 PM UTC|AS|12<br />1|Z|Mon 08 Sep 2008 03:00:00 PM UTC|AS|67<br />5|N|Tue 09 Sep 2008 03:00:00 PM UTC|EU|23<br />6|R|Wed 10 Sep 2008 03:00:00 PM UTC|OC|89<br /></pre><br />i.e. convert the UNIX epoch values on 3 rd field of the above file to standard human readable date format using Awk.<br /><br />The Awk solution using "strftime" function:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> awk 'BEGIN {FS=OFS="|"}{$3=strftime("%c",$3)} {print}' test1.txt<br /></pre><br />More about strftime format specifiers can be found <a href="http://pubs.opengroup.org/onlinepubs/007908799/xsh/strftime.html">here</a><br /><br />I have created a python program to achieve the above output and the script is <a href="http://pythonstarter.blogspot.com/2009/04/apply-operation-on-field-python.html">here</a><br /><br /><span style="font-weight: bold; font-style: italic;">Example 2) </span><br />Input file:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> cat test2.txt<br />2|Z|time:1219071600|AF|0<br />3|N|time:1219158000|AF|89<br />4|N|time:1220799600|AS|12<br />1|Z|time:1220886000|AS|67<br />5|N|time:1220972400|EU|23<br />6|R|time:1221058800|OC|89<br /></pre><br />Required output:<br /><pre>2|Z|time:Mon 18 Aug 2008 03:00:00 PM UTC|AF|0<br />3|N|time:Tue 19 Aug 2008 03:00:00 PM UTC|AF|89<br />4|N|time:Sun 07 Sep 2008 03:00:00 PM UTC|AS|12<br />1|Z|time:Mon 08 Sep 2008 03:00:00 PM UTC|AS|67<br />5|N|time:Tue 09 Sep 2008 03:00:00 PM UTC|EU|23<br />6|R|time:Wed 10 Sep 2008 03:00:00 PM UTC|OC|89<br /></pre><br />The awk solution:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> awk 'BEGIN {FS=OFS="|"}{$3="time:"strftime("%c",substr($3,6,10))} {print}' test2.txt<br /></pre><br />Related posts:<br />- <a href="http://unstableme.blogspot.com/2008/05/awk-substr-function.html">Awk Substr function explained</a><br />- <a href="http://unstableme.blogspot.com/2009/09/print-first-character-of-field-awk.html">Print first character of a field in a file using Awk</a><br />- <a href="http://unstableme.blogspot.com/2009/12/replace-first-5-characters-awk-and-sed.html">Replace first 5 characters of a file using UNIX Sed and Awk</a><br />- <a href="http://unstableme.blogspot.com/2009/11/convert-fixed-length-file-to-csv-awk.html">Convert fixed length file to a CSV file using Awk</a>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com9tag:blogger.com,1999:blog-8796493175195266816.post-82060842836968656522011-06-20T22:09:00.003+05:302011-06-20T22:25:21.377+05:30Filter line using Awk split functionInput file "file.txt" has data in the following format:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> cat file.txt<br />232323 90 /T/1382/8558/365p/133 100 234 679<br />S/1234 90 /N/1389/<span style="color: rgb(0, 0, 153);">5000</span>/365s/5000 800 134 679<br />792363 80 /T/1381/9858/365q/133 100 234 779<br />136383 90 /K/1382/<span style="color: rgb(0, 0, 153);">5000</span>/365p/5000 500 934 979<br />136383 90 /T/1382/<span style="color: rgb(0, 0, 153);">5000</span>/279p/9000 100 134 601<br /></pre><br /><span style="font-weight: bold;">Required output:</span> From the above file print only the lines whose 3rd field has the value "5000" as the 4th field (separated by the delimiter "/") in it. i.e. required output:<br /><pre>S/1234 90 /N/1389/<span style="color: rgb(0, 0, 153);">5000</span>/365s/5000 800 134 679<br />136383 90 /K/1382/<span style="color: rgb(0, 0, 153);">5000</span>/365p/5000 500 934 979<br />136383 90 /T/1382/<span style="color: rgb(0, 0, 153);">5000</span>/279p/9000 100 134 601<br /></pre><br />Using awk:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> awk '{<br /> split($3,arr,"/")<br /> if(arr[4] == 5000) {<br /> print $0<br /> }<br />}' file.txt<br /></pre><br />The awk function split(s,a,sep) splits a string "s" into an awk array "a" using the delimiter "sep".<br /><br />A simple UNIX bash script to solve this:<br /><pre><span style="color: rgb(255, 0, 0);">$</span> while read line<br /> do <br /> thirdf=$(echo "$line" | awk '{print $3}')<br /> fourthf=$(echo "$thirdf" | awk -F "/" '{print $4}')<br /> [ "$fourthf" -eq 5000 ] && echo $line<br />done < file.txt<br /></pre><br />Related posts:<br />- <a href="http://unstableme.blogspot.com/2008/06/add-prefix-to-lines-using-awk-bash.html">Add prefix to lines in a file using awk and bash</a><br />- <a href="http://unstableme.blogspot.com/2011/05/awk-replace-n-th-occurrence-of-pattern.html">Replace n-th occurrence of pattern in a file using Awk</a>Anonymoushttp://www.blogger.com/profile/00278385409494551967noreply@blogger.com3