The UNIX School

Linux Shell - What is IFS?

guru@theunixschool.com (Guru Prasad) — Tue, 19 May 2020 11:15:00 +0530

IFS stands for internal field separator. This is the delimiter used when words are split.

The man page of bash tells :

IFS The Internal Field Separator that is used for word splitting after expansion and to split lines into
words with the read builtin command. The default value is ``<space><tab><newline>''.

The default value of IFS is a space, a tab followed by a newline.

guru@unixschool:~$ echo "$IFS"
  

guru@unixschool:~$ echo "$IFS" | cat -tve
 ^I$
$
guru@unixschool:~$

When we echoed IFS for the first time, we could not see anything becuase they are special characters. On using the tve options of cat, we can see a space, followed by a ^I which is a tab character and then followed by a newline.

1. Variable with space separated values:

guru@unixschool:~$ var="hi 25 hello"

We have declared a variable var with space separated values. Now, lets use for loop at the prompt itself to access the individual values:

guru@unixschool:~$ var="hi 25 hello"
guru@unixschool:~$ for i in $var
> do
>   echo $i
> done
hi
25
hello
guru@unixschool:~$

If you notice, the values got separated with the space as delimiter and the variable i got each value in every iteration.

2. Variable with tab separated values :

guru@unixschool:~$ var="hi      25      hello"
guru@unixschool:~$ echo $var
hi 25 hello
guru@unixschool:~$ echo "$var" | cat -tve
hi^I25^Ihello$
guru@unixschool:~$ 
guru@unixschool:~$ for i in $var
> do
>  echo $i
> done
hi
25
hello
guru@unixschool:~$

As shown above, the values got separated with tab as delimiter . And the value of i got each one in subsequent iterations. Please note when we print $var, we cannot see the tabs, only when you pipe with cat and use the tve option, we can.

3. Variable with values separated by a new line character:

guru@unixschool:~$ var="hi
> 25
> hello"
guru@unixschool:~$ 
guru@unixschool:~$ echo "$var"
hi
25
hello
guru@unixschool:~$ for i in $var
> do
>  echo $i
> done
hi
25
hello
guru@unixschool:~$

Like in earlier cases, the values got separated now with newline as delimiter.
Basically, shell splits the value of variable as what is the value is IFS special variable. Since IFS has space, tab and newline, the variables will get split with any/all of them as a delimiter.

4. Variable with comma separated values:
Similarly, if we want to split a variable on the basis of comma, simply set it to IFS.

guru@unixschool:~$ IFS=","
guru@unixschool:~$ var="hi,hello,world"
guru@unixschool:~$ 
guru@unixschool:~$ for i in $var
> do
>   echo $i
> done
hi
hello
world
guru@unixschool:~$

As seen , the variables now got split with comma as separator the moment we set the IFS to comma.

5. Default IFS vs Custom IFS:
But there is a difference with the default IFS and the IFS value set by the user. In case of default IFS, the shell tries to separate taking into account sequence of spaces, sequence of tabs as one single separator, whereas it is not the case with user set separator.

5a. Variable with more spaces:

guru@unixschool:~$ var="hi 25     hello  "
guru@unixschool:~$ 
guru@unixschool:~$ for i in $var
> do
>   echo [$i]
> done
[hi]
[25]
[hello]
guru@unixschool:~$

If you notice, the shell considered the entire set of spaces as one delimiter. Square brackets are used to capture the variable value as is.

5b. Variable with multiple commas:

guru@unixschool:~$ var="hi,25,,,hello "
guru@unixschool:~$ IFS=,
guru@unixschool:~$ for i in $var
> do
>   echo [$i]
> done
[hi]
[25]
[]
[]
[hello ]
guru@unixschool:~$

As seen above, the shell took each comma as a separate delimiter unlike taking sequence of commas as it happened in case of default IFS. Due to this, we got empty some values, and also notice the space in the last value.

6. Reading a file with custom IFS:
Let us have a multi line file with a colon delimiter :

guru@unixschool:~$ cat user.txt 
guru:x:1001:1001:guru,,,:/home/guru:/bin/bash
oracle:x:1002:1002::/u01/app/oracle:/bin/bash
test:x:1003:1003::/home/test:/bin/bash
guru@unixschool:~$

Now, we will have a script to parse the file using colon as IFS:
6a. Reading into multiple variables:

#!/bin/bash
  
IFS=":"
while read user home_dir shell
do
    echo "User : $user , Home Dir : $home_dir"
done < $HOME/user.txt

Output:

User : guru , Home Dir : /home/guru
User : oracle , Home Dir : /u01/app/oracle
User : test , Home Dir : /home/test

Once the IFS is set to colon, the read started reading each value with colon as delimiter and read the individual values in user, home_dir and shell variables.

6b. Reading into an array:
Instead of reading into multiple variables, we can also read into an array:

#!/bin/bash
  
IFS=":"
while read -a arr
do
    echo "User : ${arr[0]} , Home Dir : ${arr[1]}"
done < $HOME/user.txt

By using -a option of read, the values got read into arrays.

Python - How does an iterator work

guru@theunixschool.com (Guru Prasad) — Thu, 14 May 2020 11:17:00 +0530

In this article, we will discuss how does an iterator work. Python has 2 function related to iterator: iter and next. iter creates an iteration object for the requested input, and the next function returns the next element present in the iterator. next keeps on returning till the last element is reached.

Let us create an iterator for a list and see how the next and iter function works:

>>> l1 = [2,25,33,12]
>>> l1
[2, 25, 33, 12]
>>> it1 = iter(l1)
>>> it1
<list_iterator object at 0x7fc24f93aa58>
>>> next(it1)
2
>>> next(it1)
25
>>> next(it1)
33
>>> next(it1)
12
>>> next(it1)
Traceback (most recent call last):
  File "", line 1, in 
StopIteration
>>>

When we printed it1, it shows it as an list_iterator object. Everytime next is hit, it gave the next element and finally when there are no more elements, it gives StopIteration.

When we hit iter, it actually calls __iter__ method of the corresponding class. Meaning when we call iter on a list, it will call __iter__ of the list class. Similarly, when we hit next on an object, it calls the __next__ of the corresponding object.

Let us write a test class and have __iter__ and __next__ methods: This class takes a list as input and assigns it to data variable. The __iter__method simply returns the reference. The __next__ method returns the next available data element and increments the index by 1 after every iteration. And on finally reaching the maximum, the StopIteration exception is thrown.

Let us run and check:

>>> from iterator import TestIterator
>>> t1 = TestIterator([2, 25, 33])
>>> t1
<iterator.TestIterator object at 0x7f5782d4ea90>
>>> next(t1)
2
>>> next(t1)
25
>>> next(t1)
33
>>> next(t1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guru/iterator.py", line 13, in __next__
    raise StopIteration()
StopIteration
>>>

When the object t1 is created, it is showing as iterator.TestIterator object. On subsequent calls to next function, the __next__ method gets called and we get the subsequent values. Its this simple to have our own iterator.

>>> t1 = TestIterator([2, 25, 33])
>>> t2 = iter(t1)
Iter got called
>>> t2
<iterator.TestIterator object at 0x7f5782d4ec88>
>>>

As seen here, on calling iter, the __iter__ method got called.

bash - 5 examples to do arithmetic operations

guru@theunixschool.com (Guru Prasad) — Tue, 5 May 2020 12:16:00 +0530

While working in bash or while writing bash scripts, many times it so happens we rely on external commands to get our math done. Most of the arithmetic can be handled by the bash itself. In this article, we will see how to do arithmetic operations in bash.

1. Add / Subtract numbers:
One common way to basic arithmetic using bash is by using the let command.

x=20
y=30
let z=x+y
echo "z is $z"

This is purely a shell operation without any external command. This way one can do for subtract(/), multiply(*), divide(/) and remainder(%) operation.

x=20
y=30
((z=x+y))
echo "z is $z"

The other way to do is using the ((expression)) where the expression is an arithmetic one. Using this double brace notation, all the bash arithmetic can be done.
Bash also allows to use +=, -= operators like in many programming languages. So, if we have something like x=x+10, it can be written as

x=40
((x+=10))
echo "x is $x"

2. Unary operator:
Bash has the unary operator ++ and -- . In the below example, the value of x is being incremented by 1. It is equivalent to writing x=x+1.

x=40
((x++))
echo "x is $x"

3. Conditional:
Bash has a C program like syntax when it comes for condition checking using if block.

x=20
y=30
if (( x > y ))
then
    echo "$x is bigger"
else
    echo "$y is bigger"
fi

If you notice above, we did not use $ infront of the variables x or y. Within the bracket notations, we refer to the variable directly.

4. Logical Operator:
Bash provides both the logical AND and logical OR like other programming languages.

x=20
y=30
if ((x > 10)) && (( x > y ))
then
    echo "x is bigger"
else
    echo "y is bigger"
fi

We are comparing more than one condition here using logical &&. The condition will be true only when both the condition are true. Similarly, for logical OR, we use ||.

5. Ternary operator:
Bash has an exact C like ternary operator too.

x=40
y=50
((z=(x>y)?100:200))
echo $z

If x is greater than y, 100 is assigned to z, else 200 is assigned.

Note: Bash does not support floating point operations. To deal with floating point arithmetic, we can make use of the bc command.

Python - How to read a file?

guru@theunixschool.com (Guru Prasad) — Thu, 30 Apr 2020 08:40:00 +0530

In this article, we will see how to read from a file. Let us have file with content as below:

$ cat file
Unix
Linux
AIX

1. Reading line by line:

>>> fd = open('/home/guru/file', 'r')
>>> for line in fd:
...   print(line)
... 
Unix

Linux

AIX

>>>

open function opens a file and returns a file object. Arguments are filename and the mode, 'r' for reading. When the file object is traversed, every iteration reads a line and line gets printed. Notice the blank line after every print. This is because the variable line contains a newline character read from the file and print function adds another newline.

One way to remove the blank line is by suppressing the print function to print without newline. This is done by passing an empty value to end. Since, the default mode is read, it can be removed.

>>> fd = open('/home/guru/file')
>>> for line in fd:
...   print(line, end='')
... 
Unix
Linux
AIX
>>>

Another way to not print empty line is to ensure the newline character is stripped from the line variable using the strip function.

>>> fd = open('/home/guru/file')
>>> for line in fd:
...   print(line.strip())
... 
Unix
Linux
AIX
>>>

2. Reading using readline :

readline function reads one line at a time. Once the end of the file has reached, it returns empty string.

>>> fd = open('/home/guru/file')
>>> fd.readline()
'Unix\n'
>>> fd.readline().strip()
'Linux'
>>> fd.readline().strip()
'AIX'
>>> fd.readline().strip()
''
>>>

3. Read using readlines:

>>> fd = open('/home/guru/file')
>>> for line in fd.readlines():
...   print(line.strip())
... 
Unix
Linux
AIX
>>>

readlines reads the entire file and returns a list.

4. Read a file into a list:

>>> fd = open('/home/guru/file')
>>> l1 = fd.readlines()
>>> l1
['Unix\n', 'Linux\n', 'AIX\n']

Since readlines returns a list, it can directly be assigned to a list variable. However, it has newline characters at the end.

>>> l1 = map(str.strip,fd.readlines())
>>> list(l1)
['Unix', 'Linux', 'AIX']
>>>

Using map, we strip off the newline characters of every element from the list using the strip function.

List comprehension:

>>> fd = open('/home/guru/file')
>>> [ line for line in fd ]
['Unix\n', 'Linux\n', 'AIX\n']
>>>

With newline characters stripped,

>>> [ line.strip() for line in fd ]
['Unix', 'Linux', 'AIX']
>>>

5. Read a file using read function:
read function reads the entire file and returns a string.

>>> fd = open('/home/guru/file')
>>> fd.read()
'Unix\nLinux\nAIX\n'

By stripping the newline characters from the string, it can be converted to a list.

>>> fd.seek(0)
0
>>> fd.read().split('\n')
['Unix', 'Linux', 'AIX', '']
>>>

Notice the extra element, an empty one at the end. It is due to the last newline character. Once read is called, the file object reaches to the end of the file. In order to read again, we have to take the file to the beginning which is done using seek function.

>>> fd.seek(0)
0
>>> fd.read().strip().split('\n')
['Unix', 'Linux', 'AIX']
>>> 
>>> fd.closed
False
>>> fd.close()
>>>

By stripping the newline before doing split, we will get the correct result. closed tells whether a file is closed or not. We always have to manually close the file using close function.

6. Reading with 'with open' :

Python3 came up with a new 'with open' syntax for file handling.

>>> with open('/home/guru/file') as fd:
...   for line in fd:
...     print(line.strip())
... 
Unix
Linux
AIX
>>> fd.closed
True
>>>

The advantage of this is file is automatically closed after reading. This is true even if an exception occurs.

bash - 10 examples to find / replace in a string

guru@theunixschool.com (Guru Prasad) — Tue, 28 Apr 2020 05:47:00 +0530

Many times when we want to replace or extract something, we immediately end up with and awk or a sed oneliner. Keep in mind, the first option should always be a internal shell option, only in the absence of which we should resort to an external command. In this article, we will see 10 different examples where instead of an awk/sed, using bash specific internal will be very beneficial:

1. Capitalize a string: Though there are many ways to capitalize a string, one tends to use toupper function to get it done. Instead, bash can directly handle it.

$ x='hello'
$ echo ${x^^}
HELLO
$ echo "$x" | awk '{print toupper($0)}'
HELLO

This feature of bash(^^) is only available for bash version 4. The two carrot pattern capitalizes the entire string.

2. Capitalize only first character: One common way used to get this is using sed where we match a single character(.) and replace it with uppercase. Bash 4, when used with only carrot(^) capitalizes only first character.

$ x='hello'
$ echo ${x^}
Hello
$ echo "$x" | sed -e "s/./\u&/"
Hello

3. Convert string to lowercase:
Bash has another case modification pattern ,, which will change the case to lowercase. Generally, folks try to use tolower of awk or the tr command for this.

$ x='HELLO'
$ echo ${x,,}
hello
$ echo "$x" | awk '{print tolower($0);}'
hello

4. Convert first character to lowercase:
When the case modification pattern used is a single comma(,), only the first character is converted.

$ x='HELLO'
$ echo ${x,}
hELLO
$ echo "$x" | sed -e "s/./\l&/"
hELLO

5. Replace set of characters to lowercase:
Bash's case modification can take a pattern after the double comma or double caret's. When given,the case conversion is done only on the pattern which is matched. In this example, we want to change the case of all E and O to lowercase.

$ echo ${x,,[EO]}
HeLLo
$ echo "$x" | sed 's/[EO]/\l&/g'
HeLLo

Now if you look at example 3 when we giving without any pattern, it means apply the action on the entire string and hence we got the entire string capitalized.

6. Replace a pattern with another pattern:
Bash has an inbuilt substitution option where we can replace a pattern with another. In this case, to replace 'a' with 'A', we use it like below. Most of the times we go for sed by default when it comes to string repalcement like this:

$ x='Rama and Sita'
$ echo ${x/a/A}
RAma and Sita
$ echo "$x" | sed 's/a/A/'
RAma and Sita

The above replaced only the 1st instance of the match. To replace all instances, the pattern should start with a /

$ echo ${x//a/A}
RAmA And SitA
$ echo "$x" | sed 's/a/A/g'
RAmA And SitA

7. Replace a pattern with another:
Bash allows to use wildcards in the pattern match. In the below example, we want to replace everything starting from a digit with a pattern "old".

$ x='Ram is 24 years old'
$ echo ${x/[0-9]*/ old}
Ram is old
$ echo "$x" | sed 's/[0-9].*/old/'
Ram is old

8. Extract filename from absolute path:
Bash's, remove matching prefix pattern, ##, removes the longest matching pattern from the beginning. The pattern we have given is */ meaning an set of characters followed by /. Since we have asked for longest match, it will remove till the last /.

$ echo "$x" | awk -F'/' '{print $NF}'
test.csv
$ x='/home/guru/test.csv'
$ echo ${x##*/}
test.csv
$ echo "$x" | awk -F'/' '{print $NF}'
test.csv

9. Extract directory from absolute path:
Bash's, remove matching suffix pattern, %, removes the smallest matching pattern from the end. In this case, the pattern /* indicates a slash followed by set of characters. Since we have asked for shortest match, it will remove /test.csv, as a result we get the directory.

$ x='/home/guru/test.csv'
/home/guru/
$ echo ${x%/*}/
/home/guru/
$ echo ${x%/*}
/home/guru

10. Remove a pattern from the string:
Bash's, remove matching suffix pattern, %%, removes the longest matching pattern from the end. In this case, the pattern [0-9]* indicates a number followed by set of characters. Since we have asked for longset match, it will start removing 2 all the way till the end.

$ x='Ram is 24 and Lax is 50 years old'
$ echo ${x%%[0-9]*}
Ram is
$ echo "$x"  | sed 's/[0-9].*//'
Ram is

Python - How to get the NSE daily data feed?

guru@theunixschool.com (Guru Prasad) — Thu, 23 Apr 2020 08:42:00 +0530

These are days when trading decisions are taken more by a software, and people write algorithms to get their trades executed. For all this to happen, you need to have the stock data feed with you. Data is nothing but the daily open,close,high, low and volume for the stocks. Once you have the feed, the kind of things you can do with that data is endless. In this article, we will see how to download the data feed through a Python program.

Pre-requisites:
1. Install pandas if not present already:

pip install pandas

2. Alpha Vantage has exposed free API's for downloading data feed. Install the alpha vantage package:

pip install alpha-vantage

3. You need to register yourself in Alpha vantage and get an API key.

Program:
Below is the Python program, alpha_data.py, to get the feed:

Changes you should do in the script:

1. In Line 1, replace the she-bang line with the appropriate one as present in your system.
2. In Line 7, replace the path in stock_dir to that location in which you want to save your feed files.
3. In Line 8, replace the file nifty50list.dat along with path with a file which contains the list of stock names for which you want to get the data feed. One stock per line. Sample below:

$ head -3 /home/guru/history/config/nifty50list.dat
ADANIPORTS
ASIANPAINT
AXISBANK
$

4. In Line 10, replace the value of key with the value you got during registration with alpha-vantage.

Execute the Python script:
Lets run the program, to begin with you can just have one or two stocks in the config file.

$ python alpha_data.py
Fetching for the stock NSE:ADANIPORTS
Completed for the stock NSE:ADANIPORTS in /home/guru/history/stock/adaniports.csv
Fetching for the stock NSE:ASIANPAINT
Completed for the stock NSE:ASIANPAINT in /home/guru/history/stock/asianpaint.csv
Fetching for the stock NSE:AXISBANK
Completed for the stock NSE:AXISBANK in /home/guru/history/stock/axisbank.csv
$

Checking the content of one of the feed files:

$ head -3 /home/guru/history/stock/axisbank.csv
Date,Open,High,Low,Close,Volume
2000-01-03,5.4394,5.4527,5.3779,5.4,521349.0
2000-01-04,5.4,5.51,5.08,5.38,494000.0
$ 
$ tail -3 /home/guru/history/stock/axisbank.csv
2020-04-20,482.0,483.45,451.25,455.95,42163627.0
2020-04-21,436.0,439.95,412.95,420.65,33572119.0
2020-04-22,418.55,434.55,412.0,431.15,36619611.0
$

Start playing with your data!!!

Note: A sleep of 20 seconds is given because alpha vantage allows only 3 API calls per minute.

Python - Get nth prime number

guru@theunixschool.com (Guru Prasad) — Sat, 11 Apr 2020 16:02:00 +0530

Let us try to solve the 7th problem in Euler's project which is to get the 10,001st prime number. We will start with writing a function to check if a number is prime or not.

from math import sqrt

def isPrime(x):
    if x < 2:
        return False
    if x == 2 or x == 3:
        return True
    for i in range(2, int(sqrt(x)) + 1):
        if x % i == 0:
            return False

    return True

Let us try this in Repl:

>>> isPrime(6)
False
>>> isPrime(13)
True
>>>

Now we will write a function to get the nth prime, i.e, if n is 3, we should get 5 which is the 3rd prime number.

def getNthPrime(x):
    cnt = 0
    num = 2
    while True:
        if isPrime(num):
            cnt = cnt + 1
            if cnt == x:
                return num
        num = num + 1

Testing in REPL:

>>> getNthPrime(3)
5
>>> getNthPrime(5)
11

Similarly to get the 10,001st prime number:

>>> getNthPrime(10001)
104743
>>>

Lets find out the time elapsed to get this:

>>> from timeit import timeit
>>> 
>>> timeit('getNthPrime(10001)', globals=globals(), number = 1)
0.289607039999737
>>>

All it took is just a quarter of a second to get the 10,001st prime.

Using itertools:
getNthPrime function can be completely avoided by using itertools. itertools has 2 important functions:
islice - returns an iterator of a slice of values
count - returns an iterator of values. Similar to range function, but it does not take the end value.
>>> islice((x for x in count() if isPrime(x)),10001)

<itertools.islice object at 0x7fbc350ab8b8>
>>> 
>>> list(islice((x for x in count() if isPrime(x)),10001))[-1]
104743
>>>

Using the count we get an iteration of infinite list of numbers and only the prime ones are filtered. islice takes the 1st 10,001 numbers and stops the iteration.
The result is taken in list context and we print the last element which is the 10,0001st prime.

>>> timeit('list(islice((x for x in count() if isPrime(x)),10001))[-1]', globals = globals(), number = 1)
0.3422034679997523
>>>

Time taken is almost same as what it took in the earlier one.

Perl - Connect to Oracle database and SELECT

guru@theunixschool.com (Guru Prasad) — Tue, 25 Jul 2017 10:19:00 +0530

How to connect to a database from a Perl program? Let us see in this article how to connect to Oracle and read from a table. As a pre-requisite, we need to have the DBI and DBD::Oracle packages installed.
In this article, we are going to see how to read name of a student from the students table.

Example 1:

#!/usr/bin/perl 
use warnings ;
use strict ;
use DBI;

$\="\n";

print "Connecting to DB..";

my $dbh = DBI->connect('dbi:Oracle:xe',  'scott', 'tiger') or
          die "Cannot connect to DB => " . DBI->errstr;
my $sth = $dbh->prepare("select first_name, last_name from students where id = 10000") or
          die "Couldn't prepare statement: " . $dbh->errstr;
$sth->execute();

while (my ($f_name, $l_name) = $sth->fetchrow_array()){
    printf "First Name : %-10s Last Name : %-20s\n" , $f_name,  $l_name;
}
#$sth->finish();
$dbh->disconnect();

The above program when run will print the First Name and last name of the student whose id is 10000.

1. DBI->connect => This is to connect to the Oracle database, where 'xe' is the name of the database, 'scott' is the username, and 'tiger' is the password. This function returns a database handler object.

2. $dbh->prepare => This command prepares a query provided and creates a statement handler object.

3. $sth->execute() => This will submit the query to Oracle and gets it executed. Keep in mind, this just executes the query, does not return the result back though.

4. $sth->fetchrow_array() => This is the one which fetches the result of the last query executed. A while loop is used to parse this result just in case the output returns multiple rows, still it can be handled.
In this case, since the query returns 2 columns, we are collecting it in 2 variables and then printing the values.

5. $sth->finish => This tells Oracle that we are done with fetching the results for the query. If we are reading all the results of the query, then this should not be given. It is needed only in cases when we are reading partial results.

6. $dbh->disconnect => This disconnects the connection with the Oracle. Any subsequent query if needs to be executed, we should connect again. Usually, this is done at the end of the program.

Example 2:
In this, the query is parameterized where the parameter is passed as part of the execute command:

#!/usr/bin/perl 
use warnings ;
use strict ;
use DBI;

$\="\n";

print "Connecting to DB..";

my $dbh = DBI->connect('dbi:Oracle:xe',  'scott', 'tiger') or
          die "Cannot connect to DB => " . DBI->errstr;
my $sth = $dbh->prepare("select first_name, last_name from students where id = ?") or
          die "Couldn't prepare statement: " . $dbh->errstr;
$sth->execute("10000");

while (my ($f_name, $l_name) = $sth->fetchrow_array()){
    printf "First Name : %-10s Last Name : %-20s\n" , $f_name,  $l_name;
}
$sth->finish();
$dbh->disconnect();

Example 3:
In this, instead of collecting the result in 2 scalar variables, it is being assigned to an array, and then we are accessing using the array index positions.

my @data=();
while (@data = $sth->fetchrow_array()){
    printf "First Name : %-10s Last Name : %-20s\n" , $data[0], $data[1];
}

grep vs awk - Part 2

guru@theunixschool.com (Guru Prasad) — Wed, 11 Jan 2017 11:28:00 +0530

In this article, we will see more awk alternatives for the frequently used grep commands. This is in continuation to the grep vs awk - Part 1. Let us consider a sample file as shown below:

$ cat file
Unix

Linux
Uniix
Solaris
AIX
ArchLinux
Ubuntu

1. Search for a pattern present in a variable

$ x="Linux"
$ grep "$x" file
Linux

The variable x contains the search pattern. Using grep, the variable can directly be used.

$ awk -v var="$x" '$0 ~ var' file
Linux

In awk, the variable cannot be used directly. It needs to be passed to awk from shell using the -v option. More about the variable passing to awk.

2. Search for lines beginning with a specific pattern:

$ grep '^S' file
Solaris

The ^ symbol is used to search for lines which begin with. In this example, we are trying to search for lines which are beginning with is S.

$ awk '/^S/' file
Solaris

3. Search for lines ending with a specific pattern:

$ grep 'x$' file
Unix
Uniix
Linux
ArchLinux

The $ symbol is used to search for matches which are ending with.

$ awk '/x$/' file
Unix
Uniix
Linux
ArchLinux

4. To search for a exact word in a file:

$ grep -w Linux file
Linux

The option -w is used in grep for the specific word search. The pattern ArchLinux did not retutrn because 'Linux' is just part of the the word.

$ awk '{for(i=1;i<=NF;i++)if($i=="Linux"){print;}}' file
Linux

In awk, we need to loop over the columns and compare the value against every column.

5. To get count of lines matching pattern:

$ grep -c ix file
2

The -c option of grep gives the count of lines matching the pattern. The "ix" pattern is matched in 2 lines.

$ awk '/ix/{x++}END{print x}' file
2

In awk, whenever the pattern "ix" is matched, a variable x is incremented. At the end of the file processing, x contains the count of lines matching the pattern.

6. Search for a pattern which exactly matches the whole line:

$ grep -x Linux file
Linux

grep has the -x option for matching exact matches.

$ awk '/^Linux$/' file
Linux

In awk, we use the ^ (beginning with) and $(ending with) meta characters to search for the exact match.

7. Search for non-empty lines or lines containing atleast one character:

$ grep . file
Unix
Linux
Uniix
Solaris
AIX
ArchLinux
Ubuntu

The . matches any character. Since empty line does not have any, it does not match.

$ awk NF file
Unix
Linux
Uniix
Solaris
AIX
ArchLinux
Ubuntu

NF indicates number of fields. Since an empty line will have NF as 0, it does not get matched.

8. Extract part of the line instead of the entire line:

$ grep -o "..$" file
ix
ux
ix
is
IX
tu

By default, grep prints the entire line which matches the pattern. Using -o option only a part of string can be extracted. .. extracts 2 characters, ..$ extracts 2 characters from the end of the line.

$ awk 'NF{print substr($0,length-1);}' file
ix
ux
ix
is
IX
tu

awk uses the sub-string function(substr) to do the extraction.

Let us consider another sample file with 2 columns in it:

$ cat file
Unix 2
Linux 3
Uniix 4
Solaris 5

9. Extract the 2nd column from the lines containing a specific pattern :

$ grep Uni file | cut -d" " -f2
2
4

grep extracts the lines containing the pattern, and cut extracts only the 2nd column.

$ awk '/Uni/{print $2}' file
2
4

In awk, searching a pattern and printing a particular column all are done as part of the same command.

10. Extracting first 3 characters from lines containing a specific pattern:

$ grep x file | cut -c 1-3
Uni
Lin
Uni

grep extracts lines containing the pattern 'x' and cut cuts the 1st three columns.

$ awk '/x/{print substr($0,1,3)}' file
Uni
Lin
Uni

To know& learn more about the awk, you can refer to the awk related articles.

How to use arrays in ksh?

guru@theunixschool.com (Guru Prasad) — Wed, 4 Jan 2017 12:16:00 +0530

Array, as in any programming language, is a collection of elements. These elements need not be of the same type. One important difference here is shells support only one-dimensional arrays.

1. Creating an array:
Creating an array is pretty simple.

$ typeset -a arr
$ arr[0]=25
$ arr[1]=18
$ arr[2]="hello"

Using the typeset command, we let the shell know that we are intending to use the variable arr to store a list of elements. The -a indicates arr as an indexed array. Then the elements are assigned one by one using their index positions. Array indexing always starts from 0.

To print the 1st element of the array:

$ echo ${arr[0]}
25

Similarly, to print the 2nd element of the array:

$ echo ${arr[1]}
18

To get the total number of elements in the array:

$ echo ${#arr[*]} 
3

To print all the elements of the array:

$ echo ${arr[*]}
25 18 hello

$ echo ${arr[@]}
25 18 hello

The difference between using the * and @ is same as the difference between using $* and $@ in case of command line arguments.

Declaring an initializing an array at one go:

$ typeset -a arr=(25 18  "hello")

$ typeset -a arr
$ arr=(25 18  "hello")

Array is initialized by having values inside brackets. Every element within the brackets are assigned to individual index positions.
Accessing the array elements one by one using a for loop:

for i in ${arr[*]}
do
    echo $i
done

Similarly, we can access the array elements using for loop by using @ instead of *:

for i in "${arr[@]}"
do
    echo $i
done

Printing array elements using the printf :

$ printf "%s\n" ${arr[*]}
25
18
hello

Using array to store contents of a file
Let us create a file as shown below:

$ cat file
Linux
Solaris
Unix

Dumping the file contents to an array:

$ arr=($(cat file))

With this, every line of the file gets stored in every index position of the array. Meaning, the 1st line of the file will be in arr[0], 2nd line in arr[1] and so on.

$ echo ${arr[0]}
Linux

Similarly, to print the entire file contents:

$ echo ${arr[*]}
Linux Solaris Unix

Getting the total element count of the array in this case gives the line count of the file:

$ echo ${#arr[*]}
3

Perl : Find files using File::Find

guru@theunixschool.com (Guru Prasad) — Wed, 17 Sep 2014 23:14:00 +0530

How to find the list of files in Perl like the UNIX find command? The answer is File::Find CPAN module. Let us see in this article how to use the File::Find module.

File::Find:
File::Find module contains 2 modules: find and finddepth. Both are used to find the files, the difference being the order in which the files and directories are parsed. find has all the options like the Unix find command.

1. Find all .txt files:

use File::Find;

my @files;
my @dirpath=qw(/home/user1/);
find(sub {
           push @files,$File::Find::name if (-f $File::Find::name and /\.txt$/);
      }, @dirpath);

print join "\n",@files;

find function takes 2 arguments:
1. The first argument is a subroutine which is called for each and every file found by the find function.
2. The 2nd argument is list of directories where the find function will search for files.

By default, the find function searches for all files and directories which in Unix find command can be written as:

find .

Every file found by the find command is pushed into the array @files if it is a file (-f) and has .txt extension(\.txt$). The variable $File::Find::name contains the complete path name of the file.

2. Find all .txt or .xml files:

my @files;
my @dirpath=qw(/home/user1/);
find(sub {
      push @files,$File::Find::name if (-f $File::Find::name and /\.(txt|xml)$/);
    }, @dirpath);

print join "\n",@files;

The regex used here checks for either a txt file or a xml file.
The Unix equivalent of this is:

$ find /home/user1 \( -name "*.txt" -o -name "*.xml" \)

3. Find all dirs:

use File::Find;

my @files;
my @dirpath=qw(/home/user1);
find(sub {
        push @files,$File::Find::name if (-d $File::Find::name );
      }, @dirpath);

print join "\n",@files;

The directories alone are filtered by checking for the -d option against the $File::Find::name variable.
The Unix equivalent of this is:

$ find /home/user1 -type d

4. Files modified in the last 1 day:

use File::Find;

my @files;
my @dirpath=qw(/home/user1/);
find(sub {
      push @files,$File::Find::name if (-f $File::Find::name and int(-M $_) < 1);
    }, @dirpath);

print join "\n",@files;

-M filename gives the age of the file which is nothing but the number of days since it is last modified. By checking this value to be less than 1, files modified in the last day can be retrieved.

The Unix equivalent of this is:

$ find /home/user1 -mtime -1

5. Files modified in the last 1 hour:

my @files;
my @dirpath=qw(/home/user1/);
find(sub {
      push @files,$File::Find::name if (-f $File::Find::name and (-M $_) < 1/24);
    }, @dirpath);

print join "\n",@files;

Since -M gives the number of days since the file is last modified, by checking it against 1/24 filters only those files whose age is less than 1 hour.

The Unix equivalent of this is:

$ find /home/user1 -mmin -60

6. Files modified in the last 30 mins:

my @files;
my @dirpath=qw(/home/user1/);
find(sub {
    push @files,$File::Find::name if (-f $File::Find::name and (-M $_) < 0.5/24);   }, @dirpath);

print join "\n",@files;

The Unix equivalent of this is:

$ find /home/user1 -mmin -30

This module, along with the functions, provides some variables as well. The following are those:
$name : the full name of the file
$dir : contains the name of the current directory in which find is finding.

Note: The find2perl command internally uses the File::Find module itself.

sed - 20 examples to remove / delete characters from a file

guru@theunixschool.com (Guru Prasad) — Wed, 13 Aug 2014 23:25:00 +0530

In this article of sed series, we will see the examples of how to remove or delete characters from a file. The syntax of sed command replacement is:

$ sed 's/find/replace/' file

This sed command finds the pattern and replaces with another pattern. When the replace is left empty, the pattern/element found gets deleted.

Let us consider a sample file as below:

$ cat file
Linux
Solaris
Ubuntu
Fedora
RedHat

1. To remove a specific character, say 'a'

$ sed 's/a//' file
Linux
Solris
Ubuntu
Fedor
RedHt

This will remove the first occurence of 'a' in every line of the file. To remove all occurences of 'a' in every line,

$ sed 's/a//g' file

2. To remove 1st character in every line:

$ sed 's/^.//' file
inux
olaris
buntu
edora
edHat

.(dot) tries to match a single character. The ^ tries to match a pattern(any character) in the beginning of the line. Another way to write the same:

$ sed 's/.//' file

This tells to replace a character with nothing. Since by default, sed starts from beginning, it replaces only the 1st character since 'g' is not passed.

3. To remove last character of every line :

$ sed 's/.$//' file
Linu
Solari
Ubunt
Fedor
RedHa

The $ tries to match a pattern in the end of the line.

4. To remove the 1st and last character of every line in the same command:

$ sed 's/.//;s/.$//' file
inu
olari
bunt
edor
edHa

Two commands can be given together with a semi-colon separated in between.

5. To remove first character only if it is a specific character:

$ sed 's/^F//' file
Linux
Solaris
Ubuntu
edora
RedHat

This removes the 1st character only if it is 'F'.

6. To remove last character only if it is a specific character:

$ sed 's/x$//' file
Linu
Solaris
Ubuntu
Fedora
RedHat

This removed the last character only if it s 'x'.

7. To remove 1st 3 characters of every line:

$ sed 's/...//' file
ux
aris
ntu
ora
Hat

A single dot(.) removes 1st character, 3 dots remove 1st three characters.

8. To remove 1st n characters of every line:

$ sed -r 's/.{4}//' file
x
ris
tu
ra
at

.{n} -> matches any character n times, and hence the above expression matches 4 characters and deletes it.

9. To remove last n characters of every line:

$ sed -r 's/.{3}$//' file
Li
Sola
Ubu
Fed
Red

10. To remove everything except the 1st n characters in every line:

$ sed -r 's/(.{3}).*/\1/' file
Lin
Sol
Ubu
Fed
Red

.* -> matches any number of characters, and the first 3 characters matched are grouped using parantheses. In the replacement, by having \1 only the group is retained, leaving out the remaining part.

11. To remove everything except the last n characters in a file:

$ sed -r 's/.*(.{3})/\1/' file
nux
ris
ntu
ora
Hat

Same as last example, except that from the end.

12. To remove multiple characters present in a file:

$ sed 's/[aoe]//g' file
Linux
Slris
Ubuntu
Fdr
RdHt

To delete multiple characters, [] is used by specifying the characters to be removed. This will remove all occurences of the characters a, o and e.

13. To remove a pattern :

$ sed 's/lari//g' file
Linux
Sos
Ubuntu
Fedora
RedHat

Not just a character, even a pattern can be removed. Here, 'lari' got removed from 'Solaris'.

14. To delete only nth occurrence of a character in every line:

$ sed 's/u//2' file
Linux
Solaris
Ubunt
Fedora
RedHat

By default, sed performs an activity only on the 1st occurence. If n is specifed, sed performs only on the nth occurence of the pattern. The 2nd 'u' of 'Ubuntu' got deleted.

15. To delete everything in a line followed by a character:

$ sed 's/a.*//' file
Linux
Sol
Ubuntu
Fedor
RedH

16. To remove all digits present in every line of a file:

$ sed 's/[0-9]//g' file

[0-9] stands for all characters between 0 to 9 meaning all digits, and hence all digits get removed.

17. To remove all lower case alphabets present in every line:

$ sed 's/[a-z]//g' file
L
S
U
F
RH

[a-z] represents lower case alphabets range and hence all lower-case characters get removed.

18. To remove everything other than the lower case alphabets:

$ sed 's/[^a-z]//g' file
inux
olaris
buntu
edora
edat

^ inside square brackets negates the condition. Here, all characters except lower case alphabets get removed.

19. To remove all alpha-numeric characters present in every line:

$ sed 's/[a-zA-Z0-9]//g' file

All alpha-numeric characters get removed.

20. To remove a character irrespective of the case:

$ sed 's/[uU]//g' file
Linx
Solaris
bnt
Fedora
RedHat

By specifying both the lower and upper case character in brackets is equivalent to removing a character irrespective of the case.

How to use SUID for shell scripts in Linux?

guru@theunixschool.com (Guru Prasad) — Tue, 12 Nov 2013 10:25:00 +0530

Question : How to write a shell script which will read the required passwords/connect strings from a config file? Other users should be able to execute the scripts, however they should not be able to read the config file.

Say, I have a config file, myconfig.txt, which contains credentials for an oracle connection:

$ cat myconfig.txt
SQLUSR="scott"
SQLPWD="pwd123"

Shell script,script1.sh, to connect to Oracle and get employee name:

$ cat script1.sh
#!/usr/bin/bash

source myconfig.txt

NM=`sqlplus -s ${SQLUSR}/${SQLPWD}@XE <<EOF
set heading off
select ename from emp where empno=7369;
EOF`

echo Name is $NM

The issue here is: Any user, who has permissions to run this script can get to know the password present in the myconfig.txt either by reading the file directly or by running the shell script in debug mode.

How to prevent the user from reading the password?
By removing the read permission from the group and others on the myconfig.txt file will not help. Because, in that case, the other users will not be able to run the script as well.

Setting the SUID on the Shell script:
After removing the read permission and then applying the suid bit on the shell script will work on say Solaris, will not work in Linux flavors. Because SUID can be applied only on binary executables in Linux.

Solution:
In order to create a binary executable, we need to write our code in either say C, C++, etc..which gives binary executables.
Let us write a C program which reads the myconfig.txt file and sets environment variables for all the entries present in the file. And then invoking the shell script from the C program should suffice:

$ cat getName.c
#include<stdio.h>
#include<string.h>
#include<stdlib.h>

int main()
{
    FILE *fp = fopen("/home/guru/C/myconfig.txt","r");
    char data[128];
    char *d1;
    while(fgets(data,128,fp)!=NULL){
        d1=(char *)malloc(128);
        strcpy(d1,data);
        d1[strlen(d1)-1]='\0';
        putenv(d1);
    }
    fclose(fp);
    system("/home/guru/C/script.sh");
}

Creating an executable for this C program :

$ gcc -o getName getName.c
$ ./getName
NM is SMITH

Remove the read permission on the myconfig.txt file for group and others and applying the SUID on the executable:

$ chmod go-r myconfig.txt
$ chmod u+s getName

Logging as a different user to check :

$ su - guest
$ echo $USER
guest
$ cd /home/guru/C
$ ./getName
NM is SMITH
$ cat myconfig.txt
cat: myconfig.txt: Permission denied

On applying the SUID permission, the other user when trying to execute the script, gets the same permission as the owner on the myconfig.txt file and hence the script is able to read the file.

Python - How to write a list to a file?

guru@theunixschool.com (Guru Prasad) — Tue, 13 Aug 2013 10:36:00 +0530

How to write an entire list of elements into a file? Let us see in this article how to write a list l1 to a file 'f1.txt':

1. Using write method:

#!/usr/bin/python

l1=['hi','hello','welcome']

f=open('f1.txt','w')
for ele in l1:
    f.write(ele+'\n')

f.close()

The list is iterated, and during every iteration, the write method writes a line to a file along with the newline character.

2. Using string join method:

#!/usr/bin/python

l1=['hi','hello','welcome']

f=open('f1.txt','w')
s1='\n'.join(l1)
f.write(s1)
f.close()

Using join method, the entire list can be joined using a delimiter and formed into a string. Now, using the write method, the string can be written to the file. In this, only one write method is needed for the entire list.

3. Using string join along with with open syntax:

#!/usr/bin/python

l1=['hi','hello','welcome']

with open('f1.txt','w') as f:
  f.write('\n'.join(l1))

The with open syntax automatically closes the file at the end of the block, hence no need of explicit call to the close method. Instead of using a variable to contain the joined string, it is directly given to the write method.

4. Using the writelines method:

#!/usr/bin/python

l1=['hi','hello','welcome']

f=open('f1.txt','w')
l1=map(lambda x:x+'\n', l1)
f.writelines(l1)
f.close()

Python provides a method, writelines, which is very useful to write lists to a file. write method takes a string as argument, writelines takes a list. writelines method will write all the elements of the list to a file. Since it writes to the file as is, before invoking the writelines method, the list elements should be appended with newline characters, so that the list elements will come in individual lines. This is achieved using the map function which will add a newline to every list element.

Python - How to find the location of a module?

guru@theunixschool.com (Guru Prasad) — Tue, 30 Jul 2013 11:55:00 +0530

When a module is imported into a program, how to find out the path from where the module is getting picked?

In the Unix world, when a command say ls is executed, the operating system searches in a list of directories for the command executable and once found, executes the command. The list of directories is maintained in the PATH variable.
Similarly, in Python, whenever a module is imported, say "import collections', python also searches in a list of directories to find the module file "collections.py", and once found, includes the module. The list of directories to be searched is maintained in a list, sys.path, from the sys module. Hence, this can be accessed using the sys.path.

Let us write a script, findmod.py, to find the path of the module 'collections' :

#!/usr/bin/python

import sys,os
from collections import Counter

file_name='collections.py'

for folder in sys.path:
    fname=os.path.join(folder,file_name)
    if os.path.isfile(fname):
       print fname

On running the above script, we get:

$ ./findmod.py
/usr/lib/python2.7/collections.py

sys.path contains the list of directories from where the modules is picked up. Since the collections module is present in one of these directories, the file is to be checked for existence in every directory. Once found, it is printed.

os.path.join -> This joins all the arguments given with the path separator( If Unix flavor, the path separator will be / , if Windows it will be \ ). The advantage of using join rather than normal string conatenation is portability. join will use the appropriate path separator depending on the operating system.

os.path.isfile -> This function checks whether a file exists or not.

Perl : Foreach loop variable is an alias

guru@theunixschool.com (Guru Prasad) — Fri, 19 Jul 2013 10:16:00 +0530

In Perl, when an array is traversed using the foreach loop, the variable holding the element of the array during the iteration is actually an alias to the actual element itself. Due to this, any change done to this variable automatically updates the array itself.
Let us see some examples where every element of an array has to be added by 10:

1. Array elements get updated when updating $_:

#!/usr/bin/perl
use warnings ;
use strict ;
$\="\n";

my @arr=(5..8);
print "@arr";
foreach (@arr){
        $_+=10;
}
print "@arr";

This snippet when run outputs:

5 6 7 8
15 16 17 18

As seen above, just by updating the $_, the array elements get updated automatically.

2. Array elements get updated even when using the holding variable:

#!/usr/bin/perl
use warnings ;
use strict ;
$\="\n";

my @arr=(5..8);
print "@arr";
foreach my $x (@arr){
        $x+=10;
}
print "@arr";

Instead of using the special variable $_, this example uses a lexical variable $x to hold the values in iteration.
The above snippet outputs:

5 6 7 8
15 16 17 18

3. List of scalar values also gets updated:

#!/usr/bin/perl
use warnings ;
use strict ;
$\="\n";

my ($x,$y,$z)=(20,30,40);

foreach my $a ($x,$y,$z){
        $a+=10;
}
print "$x $y $z";

The sample snippet outputs:

20 30 40
30 40 50

Even when a set of scalars is given, they too get udpated the same way. The holding variable $a is just an alias to these scalar variables.

Linux For You - A magazine for Open Source techies

guru@theunixschool.com (Guru Prasad) — Wed, 17 Jul 2013 09:19:00 +0530

Linux For You(LFY) or the OpenSource For You is the magazine which you should be having if you are a techy, especially for the ones working in Unix or Linux flavors. I accidentally got my hands on the magazine once, and made it a point to subscribe to it immediately.

My 5 reasons why it is a must in your shelf:

1. Knowledge on Linux World: Linux is one such technology which has no limits. You keep learning, Linux keeps providing something new for you to satisfy your hunger. However, its not possible to learn everything by ourself through our experiments. It takes lot of time to learn anything by oneself. This magazine is a place wherein you get lot of knowledge, good knowledge, in a short span of time. The topics covered can be of any programming language, scripting language, system administration stuff, apps programming stuff, review of any particular software, etc. Last but not the least, the articles always come with lots of explanation.

2. All important Open Source : Most of the softwares we use today are open source. I use Ubuntu for my desktop, and the rest of the software goes without saying. The fact is, the majority of the softwares coming up these days are open source, and the OpenSource storm has already taken the tech world in a different direction. This will lead to a day wherein each and every electronic item we use in our life will have something of open source associated to it. Hence, it is very important for us to be in the complete know-how of things. The articles on Open Source in the magazine could range from servers, networking, security, tools, etc.

3. Hot technologies in the market: Cloud computing, Big data, Hadoop are some of the trending technologies in the market. You like it or not, it cannot be avoided. Very soon, these terminologies are going to find a mention in our day to day life. You will always find dedicated articles on the latest market offerings in LFY. Getting exposed to new technologies will help us grow in this competitive market. Also, a lot of coverage on the Android apps programming is provided which is a hot-cake today.

4. Utilities : Every developer or admin works on some utilities or the other. And the developer/admin is only aware about those utilities which he or she has come across. Come to LFY, you will see articles on quite a lot of utilities. Though we may not need to know too much about it, having a basic understanding of important ones will always keep us in good stead. Also, with every edition you get a CD of one of the Linux distros enclosed.

5. Career options: LFY has a section devoted for career options. This section takes up a hot career option in the market and talks about in detail, about the scope of it, the advantages of the technology, the different training institutes providing training on the technology and of course,the salary range for freshers who want to get into this segment. This section is very helpful for every techy, especially for those who are trying to shift to a new domain.

P.S : I subscribed to this magazine around 6 months back, and I can surely say I gained quite a bit of knowledge of the OpenSource market and its offerings. You always feel you are in sync with the OpenSource world which is more important in today's competitive market.

Perl : Remove duplicate elements from arrays

guru@theunixschool.com (Guru Prasad) — Tue, 14 May 2013 10:48:00 +0530

How to remove duplicate element from arrays in Perl? Let us see in this article how can duplicates be removed in different ways?

1. Copying distinct elements to new array using grep function:

my @arr=qw(bob alice alice chris bob);
my @arr1;
foreach my $x (@arr){
        push @arr1, $x if !grep{$_ eq $x}@arr1;
}
print "@arr1";

A loop is run on the array elements. For every element, it is checked to find out whether the element is already present in the new array. If present, it is skipped, else added to the new array.

2. Same way using regex:

my @arr=qw(bob alice alice chris bob);
my @arr1;
foreach my $x (@arr){
        push @arr1, $x if !grep{/^$x$/}@arr1;
}
print "@arr1";

This method is same as the above except that instead of string comparison, the array element is checked using the regular expression.

3. Using hash:

my @arr=qw(bob alice alice chris bob);
my %h1;
foreach my $x (@arr){
        $h1{$x}=1;
}
@arr=keys%h1;
print "@arr";

Every element is stored in a hash with key as the array element and value as 1. Since a hash cannot have duplicate keys, after populating the hash, the keys from the hash will consist of distinct array elements.

4.Using hash and map function:

my @arr=qw(bob alice alice chris bob);
my %h1=map{$_=>1}@arr;
@arr=keys%h1;
print "@arr";

Same as last example, instead of running a loop on the array elements, the hash is populated using the map function where the array element is the key and the value is 1. Map function returns a key-value pair for every array element where the key is the array element and the value is 1.

5. Using hash in array context for keys:

my @arr=qw(bob alice alice chris bob);
my %h1;
@h1{@arr}=(1..@arr);
@arr=keys%h1;
print "@arr";

The hash is populated little differently here. Hash can be populated either for a single key-value or for multiple key value pairs. @h1{@arr}=(1..@arr) translates to @h1{"bob","alice","alice","chris","bob"}=(1,2,3,4,5)

Perl - 6 examples to fetch date / time without Linux date command

guru@theunixschool.com (Guru Prasad) — Wed, 10 Apr 2013 16:31:00 +0530

How to fetch date and time related stuff within in Perl?
Sometimes, Perl developers resort to using of Unix system date command to fetch a date or time within Perl script. This is not needed at all since Perl itself has a set of built-ins using which dates and epoch can be retrieved easily without using the system date command. In this article, let us see how to simulate the date related commands in Perl.

1. Getting the date command output:
Linux command:

$ date

To get the date command like output in Perl:

my $x=localtime;
print $x;

localtime is a Perl built-in, which when called without any arguments returns the current date and time. The above program will return an output like this:

Wed Apr 10 15:54:10 2013

2. Extract only the hour,minute and seconds from the date command:
Linux:

$ date '+%H%M%S'

Perl:

my ($s,$m,$h)=localtime;
print "$h$m$s";

localtime built-in depends on the context in which it is being used:
In scalar context, returns the date and time like the Unix date command.
In list context, returns a list of values : seconds,minutes,hour,day,month(0-11),year,and so on. In the above example, only the seconds, minutes and hours are stored.

3. Get the Unix(epoch) time:
Linux:

$ date '+%s'

Perl:

my $tm=time;
print $tm;

time is a perl built-in which gives the Unix Time (epoch).

4. Get yesterday's date:
Linux:

$ date -d "yesterday" '+%y%m%d'

Perl:

use POSIX qw(strftime);

my $x=strftime("%y%m%d",localtime(time-86400));
print $x;

2 things to note:
a. localtime built-in can take a Unix time as an argument and return the date/time for the specific Unix time. In other words, $x=localtime is same as $x=localtime(time) . By subtracting, time with 86400(24*60*60), we get the Unix time of yesterday.
b. strftime is a function which takes a localtime as argument, and formats the result as a string. The format specifiers are similar to the system date command.

5. Get Unix time(epoch) of yesterday's date:
Linux:

$ date -d "yesterday" '+%s'

Perl:

my $tm=mktime(localtime(time-86400));
print $tm;

mktime is another Perl built-in which can calculate the Unix time for a given localtime. This above snippet calculates the Unix time for the last day.
6. Get epoch of a specific date/time:
Linux:

$ date -d "08 May 2012 11AM 30minutes" '+%s'

Perl:

my $tm=mktime(0,30,11,8,4,112);
print $tm;

Since mktime takes the input of the format of localtime command's output, by providing the appropriate date and time as arguments to the mktime, the Unix time can be calculated for any pre-defined date and time.

Perl - How to find the sum of digits in a number?

guru@theunixschool.com (Guru Prasad) — Tue, 26 Mar 2013 16:10:00 +0530

How to find the sum of all the digits in a number or a string? OR How to find the sum of all elements in array? Let us see the different ways how it can be done in Perl?

1. Divide and accumulate:

my $x=3456;
my $sum=0;
while ($x) {
        $sum+=$x%10;
        $x/=10;
}
print $sum;

The digits are extracted by taking the modulus of the number with 10, and then dividing the number by 10. This is repeated till the original number becomes zero at the end of which the sum is retrieved.

2. Using regular expression:

my $x=3456;
my @arr=$x=~/./g;
my $sum=0;
$sum+=$_ foreach(@arr);
print $sum;

The number is broken into individual digits using the regular expression and extracted into an array. The array elements are looped using the foreach loop and the sum is calculated.

3. Using eval and join:

my $x=3456;
my @arr=$x=~/./g;
print eval join "+",@arr;

The number is separated into digits as in the last example. Using join, all the digits are joined using "+" and then the eval function evaluates the string.

4. Using CPAN module List::Util:

use List::Util qw(sum);

my $x=3456;
my @arr=$x=~/./g;
print sum @arr;

The CPAN module List::Util contains a subroutine sum which takes an array as input and calculates the returns sum of all elements in the array.

How to remove leading zeros in a string in Linux?

guru@theunixschool.com (Guru Prasad) — Thu, 14 Mar 2013 16:13:00 +0530

How to remove leading zeros in a variable or a string?

Let us consider a variable "x" which has the below value :

$ x=0010
$ echo $x
0010

1. Using typeset command of ksh:

$ typeset -LZ x
$ x=0010
$ echo $x
10

The variable x is declared using the typeset command. The -Z option of typeset will strip the leading zeros present when used along with -L option.

2. Using sed command:

$ echo $x | sed 's/^0*//'
10

The expression '^0*' will search for a sequence of 0's in the beginning and delete them.

3. Using awk :

$ echo $x | awk '{sub(/^0*/,"");}1'
10

Using the sub function of awk, explanation same as sed solution.

4. awk with printf:

$ echo $x| awk '{printf "%d\n",$0;}'
10

Using printf, use the integer format specifier %d, the leading zeros get stripped off automatically.

5. awk using int:

$ echo $x | awk '{$0=int($0)}1'
10

The int function converts a expression into a integer.

6. Perl using regex:

$ echo $x | perl -pe 's/^0*//;'
10

7. Perl with printf:

$ echo $x | perl -ne 'printf "%d\n",$_;'
10

8. Perl with int:

$ echo $x | perl -pe '$_=int;'
10

Perl - What is DATA ?

guru@theunixschool.com (Guru Prasad) — Tue, 26 Feb 2013 16:03:00 +0530

What is __DATA__ ? Why is it used?
Generally, Perl reads data from a file which in the Perl program can be accessed using open and readline functions. However, for users who just want to test something for an urgent requirement, it is pretty boring to put the test data in a file and access it using the open and readline functions. The solution for this : __DATA__.

Let us consider a sample text file:

$ cat f1.txt
AIX,1
Solaris,2
Unix,3
Linux,4

1. The common way of reading or processing a file in Perl:

#!/usr/bin/perl
use warnings;
use strict;

open my $fh, '<', 'f1.txt' or die 'Cannot open f1.txt';

while(<$fh>){
        print;
}
close ($fh);

open function opens the file and associates the file with the filehandler $fh. This file handler with diamond(<> )operator reads a line from the file. Using while loop, the entire file is read and printed till the end of the file is reached. close function closes the file associated with the file handler. This method is little cumbersome in a way that every time to read a file, one needs to have open and close commands.

2. Using __DATA__ : Input file content in the source file itself:
Let us see how to use this:

#!/usr/bin/perl

use warnings;
use strict;

while(<DATA>){
        print;
}

__DATA__
AIX,1
Solaris,2
Unix,3
Linux,4

__DATA__ :
Any input data to be provided to the Perl program is put below the line __DATA__ . This __DATA__ line indicates that the content following is input file content. Perl automatically attaches the DATA file handler to access this data for the programmer which can then be read with the diamond operator(<>) . This section should be put only at the very end of the program.
Note: Perl allows to use __END__ as well in place of __DATA__. However, the file handler associated will be DATA itself.

Reading entire file into an array:

#!/usr/bin/perl

use warnings;
use strict;

my @arr=<DATA>;
print @arr;

__DATA__
AIX,1
Solaris,2
Unix,3
Linux,4

Just by assigning the DATA handler to an array (for that matter any file handler), the entire file will be read into the array. By loading the entire file into an array, any particular record of a file can be accessed from the array.

What is the time command in Linux for?

guru@theunixschool.com (Guru Prasad) — Tue, 19 Feb 2013 16:53:00 +0530

time command in Unix is used to find the time taken by a particular command to execute. This is very useful when we want to time a particular command or a particular script to know the amount of time it consumes. Not only the elapsed time, the time command also tells the amount of time spent in the user and kernel spaces as well.

$ time

real    0m0.000s
user    0m0.000s
sys     0m0.000s

time command output shows 3 different components:
real - The total time elapsed during the command which includes the time taken for the process.
user - The time taken for the process only in the user space. This does not include kernel calls.
sys - The time taken for the process in system space (for kernel related calls )

1. Using time for a command:

$ time ls
f1.c  f2.c

real    0m0.01s
user    0m0.00s
sys     0m0.01s

This indicates the ls command has taken a total of 0.01 second to finish, which is the time elapsed.

2. Using time command for a simple script:
Let us write a script, try.sh, which prints a welcome message:

$ cat try.sh
#!/bin/bash

echo "welcome"

Running the time command:

$ time ./try.sh
welcome

real    0m0.03s
user    0m0.01s
sys     0m0.02s

This script has taken a total of 0.03 seconds to execute.

3. Timing the sleep in the script:
sleep command's duration do not go against user or sys, it is always against the real output.

$ cat try.sh
#!/bin/bash

sleep 3
echo "welcome"

Running the time command:

$ time ./try.sh
welcome

real    0m3.04s
user    0m0.03s
sys     0m0.01s

As seen above, the sleep duration goes entirely in the real, not in the user or sys.

Capturing time output:
time command output is re-directed to standard error terminal, and not standard output like other Unix commands. Hence, time command output cannot be re-directed to a file in a normal way.

1. To retrieve only the real time of the time output:

$ time ./try.sh | grep real
welcome

real    0m1.05s
user    0m0.02s
sys     0m0.04s

As seen, grep does not help since the time command output is not in the STDOUT.

$ (time ./try.sh) 2>&1 | grep real
real    0m1.05s

By re-directing the standard error to standard output, the time command output can now be passed as standard input to other commands.

2. To re-direct the time command output to a file:

$ (time ls ) 2>file

sed - 10 examples to replace / delete / print lines of CSV file

guru@theunixschool.com (Guru Prasad) — Tue, 12 Feb 2013 16:25:00 +0530

How to use sed to work with a CSV file? Or How to work with any file in which fields are separated by a delimiter?

Let us consider a sample CSV file with the following content:

cat file
Solaris,25,11
Ubuntu,31,2
Fedora,21,3
LinuxMint,45,4
RedHat,12,5

1. To remove the 1st field or column :

$ sed 's/[^,]*,//' file
25,11
31,2
21,3
45,4
12,5

This regular expression searches for a sequence of non-comma([^,]*) characters and deletes them which results in the 1st field getting removed.

2. To print only the last field, OR remove all fields except the last field:

$ sed 's/.*,//' file
11
2
3
4
5

This regex removes everything till the last comma(.*,) which results in deleting all the fields except the last field.

3. To print only the 1st field:

$ sed 's/,.*//' file
Solaris
Ubuntu
Fedora
LinuxMint
RedHat

This regex(,.*) removes the characters starting from the 1st comma till the end resulting in deleting all the fields except the last field.

4. To delete the 2nd field:

$ sed 's/,[^,]*,/,/' file
Solaris,11
Ubuntu,2
Fedora,3
LinuxMint,4
RedHat,5

The regex (,[^,]*,) searches for a comma and sequence of characters followed by a comma which results in matching the 2nd column, and replaces this pattern matched with just a comma, ultimately ending in deleting the 2nd column.
Note: To delete the fields in the middle gets more tougher in sed since every field has to be matched literally.

5. To print only the 2nd field:

$ sed 's/[^,]*,\([^,]*\).*/\1/' file
25
31
21
45
12

The regex matches the first field, second field and the rest, however groups the 2nd field alone. The whole line is now replaced with the 2nd field(\1), hence only the 2nd field gets displayed.

6. Print only lines in which the last column is a single digit number:

$ sed -n '/.*,[0-9]$/p' file
Ubuntu,31,2
Fedora,21,3
LinuxMint,45,4
RedHat,12,5

The regex (,[0-9]$) checks for a single digit in the last field and the p command prints the line which matches this condition.

7. To number all lines in the file:

$ sed = file | sed 'N;s/\n/ /'
1 Solaris,25,11
2 Ubuntu,31,2
3 Fedora,21,3
4 LinuxMint,45,4
5 RedHat,12,5

This is simulation of cat -n command. awk does it easily using the special variable NR. The '=' command of sed gives the line number of every line followed by the line itself. The sed output is piped to another sed command to join every 2 lines.

8. Replace the last field by 99 if the 1st field is 'Ubuntu':

$ sed 's/\(Ubuntu\)\(,.*,\).*/\1\299/' file
Solaris,25,11
Ubuntu,31,99
Fedora,21,3
LinuxMint,45,4
RedHat,12,5

This regex matches 'Ubuntu' and till the end except the last column and groups each of them as well. In the replacement part, the 1st and 2nd group along with the new number 99 is substituted.

9. Delete the 2nd field if the 1st field is 'RedHat':

$ sed 's/\(RedHat,\)[^,]*\(.*\)/\1\2/' file
Solaris,25,11
Ubuntu,31,2
Fedora,21,3
LinuxMint,45,4
RedHat,,5

The 1st field 'RedHat', the 2nd field and the remaining fields are grouped, and the replacement is done with only 1st and the last group , resuting in getting the 2nd field deleted.

10. To insert a new column at the end(last column) :

$ sed 's/.*/&,A/' file
Solaris,25,11,A
Ubuntu,31,2,A
Fedora,21,3,A
LinuxMint,45,4,A
RedHat,12,5,A

The regex (.*) matches the entire line and replacing it with the line itself (&) and the new field.

11. To insert a new column in the beginning(1st column):

$ sed 's/.*/A,&/' file
A,Solaris,25,11
A,Ubuntu,31,2
A,Fedora,21,3
A,LinuxMint,45,4
A,RedHat,12,5

Same as last example, just the line matched is followed by the new column.

Note: sed is generally not preferred on files which has fields separated by a delimiter because it is very difficult to access fields in sed unlike awk or Perl where splitting fields is a breeze.

How to delete every nth line in a file in Linux?

guru@theunixschool.com (Guru Prasad) — Wed, 6 Feb 2013 16:37:00 +0530

How to delete or remove every nth line in a file? The requirement is to remove every 3rd line in the file.

Let us consider a file with the below content.

$ cat file
AIX
Solaris
Unix
Linux
HPUX

1. awk solution :

$ awk 'NR%3' file
AIX
Solaris
Linux
HPUX

NR%3 will be true for any line number which is not multiple of 3, and hence the line numbers which are multiple's of 3 does not get printed.
Note: NR%3 is same as NR%3!=0

2. Perl:

$ perl -lne 'print if $.%3 ;' file
AIX
Solaris
Linux
HPUX

Same logic as awk solution. $. in perl contains the line number of the file.

3. sed:

$ sed 'n;n;d;' file
AIX
Solaris
Linux
HPUX

n commands prints the current line and reads the next line. Hence 2 consecutive n's result in 2 lines getting printed with the 3rd line in the pattern space. d command deletes the line(3rd line) which is present in the pattern space. And this continues till the end of the file.

4. Bash Shell script:

$ x=0
$ while read line
> do
>   ((x++))
>   [ $x -eq 3 ] && { x=0; continue; }
>   echo $line
> done < file
AIX
Solaris
Linux
HPUX

A simple logic of incrementing a variable by 1 after every line is processed. When the count becomes 3, the particular line is not printed and the counter is reset.

The UNIX School

Linux Shell - What is IFS?

Python - How does an iterator work

bash - 5 examples to do arithmetic operations

Python - How to read a file?

bash - 10 examples to find / replace in a string

Python - How to get the NSE daily data feed?

Python - Get nth prime number

Perl - Connect to Oracle database and SELECT

grep vs awk - Part 2

How to use arrays in ksh?

Perl : Find files using File::Find

sed - 20 examples to remove / delete characters from a file

How to use SUID for shell scripts in Linux?

Python - How to write a list to a file?

Python - How to find the location of a module?

Perl : Foreach loop variable is an alias

Linux For You - A magazine for Open Source techies

Perl : Remove duplicate elements from arrays

Perl - 6 examples to fetch date / time without Linux date command

Perl - How to find the sum of digits in a number?

How to remove leading zeros in a string in Linux?

Perl - What is __DATA__ ?

What is the time command in Linux for?

sed - 10 examples to replace / delete / print lines of CSV file

How to delete every nth line in a file in Linux?

Perl - What is DATA ?