Unix Shell Scripts

Wishing Thanks to My Readers.

2011-06-07T06:00:00.001-07:00

Hi reader’s,

I’m very much happy to see that lot of people from across the globe visited my blog and visitor’s count has crossed 3000 + with in 40 days of starting the blog. I would request my readers to start posting your valuable comments and requests so that I can make the blog more informative & interactive . For viewers information who visit my Informatica blog, to visit my SQL or Data warehousing blogs too as I spent a lot of time posting the information, which is very useful for the job seekers or Data warehousing professionals in your day today activities at work or can also be used as the quick reference guide before taking up any interview.

I’m planning to publish a book on informatica soon which will cover all major concepts of the data warehousing in detail, including the SQL concepts used by an ETL consultants, Basic Unix Commands and Introduction to the Scheduler tools like Autosys, Tivoli and off course Informatica with simple examples.

I would highly appreciate if few people come forward and post the interview questions or the scenarios which they have encountered in there day to day work for discussion at the discussion board.

Regular Expressions

2011-04-23T03:03:00.001-07:00

What Are Regular Expressions?

A regular expression is a pattern template you define that a Linux utility Uses to filter text. A Linux utility (such as the sed editor or the gawk program)matches the regular expression pattern against data as that data flows Into the utility. If the data matches the pattern, it’s accepted for processing.

If the data doesn’t match the pattern, it’s rejected. The regular expression pattern makes use of wildcard characters to represent one or more characters in the data stream.

Types of regular expressions:

There are two popular regular expression engines:

The POSIX Basic Regular Expression (BRE) engine
The POSIX Extended Regular Expression (ERE) engine

Defining BRE Patterns:

The most basic BRE pattern is matching text characters in a data stream.

Eg 1: Plain text

$ echo "This is a test" | sed -n ’/test/p’

This is a test.

$ echo "This is a test" | sed -n ’/trial/p’

$ echo "This is a test" | gawk ’/test/{print $0}’

This is a test.

$ echo "This is a test" | gawk ’/trial/{print $0}’

Eg 2: Special characters

The special characters recognized by regular expressions are:

.*[]^${}\+?|()

For example, if you want to search for a dollar sign in your text, just precede it with a backslash character:

$ cat data2

The cost is $4.00

$ sed -n ’/\$/p’ data2

The cost is $4.00

Eg 3: Looking for the ending

The dollar sign ($) special character defines the end anchor.

$ echo "This is a good book" | sed -n ’/book$/p’

This is a good book

$ echo "This book is good" | sed -n ’/book$/p’

Eg 4: Using ranges

You can use a range of characters within a character class by using the dash symbol.

Now you can simplify the zip code example by specifying a range of digits:

$ sed -n ’/^[0-9][0-9][0-9][0-9][0-9]$/p’ data8

60633

46201

45902

Extended Regular Expressions:

The POSIX ERE patterns include a few additional symbols that are used by some Linux applications and utilities. The gawk program recognizes the ERE patterns, but the sed editor doesn’t.

Eg 1: The question mark

The question mark indicates that the preceding character can appear zero or one time, but that’s all. It doesn’t match repeating occurrences of the character:

$ echo "bt" | gawk ’/be?t/{print $0}’

$ echo "bet" | gawk ’/be?t/{print $0}’

Bet

$ echo "beet" | gawk ’/be?t/{print $0}’

$ echo "beeet" | gawk ’/be?t/{print $0}’

Eg 2: The plus sign

The plus sign indicates that the preceding character can appear one ormore times, but must be present at least once. The pattern doesn’t match if the character is not present:

$ echo "beeet" | gawk ’/be+t/{print $0}’

beeet

$ echo "beet" | gawk ’/be+t/{print $0}’

beet

$ echo "bet" | gawk ’/be+t/{print $0}’

bet

$ echo "bt" | gawk ’/be+t/{print $0}’

Eg 3: The pipe symbol

The pipe symbol allows to you to specify two or more patterns that the regular expression engine uses in a logical OR formula when examining the data stream. If any of the patterns match the data stream text, the text passes. If none of the patterns match, the data stream text fails.

The format for using the pipe symbol is:

expr1|expr2|...

Here’s an example of this:

$ echo "The cat is asleep" | gawk ’/cat|dog/{print $0}’

The cat is asleep

$ echo "The dog is asleep" | gawk ’/cat|dog/{print $0}’

The dog is asleep

$ echo "The sheep is asleep" | gawk ’/cat|dog/{print $0}’

Eg 4: Grouping expressions

When you group a regular expression pattern, the group is treated like a standard character. You can apply a special character to the group just as you would to a regular character.

For example:

$ echo "Sat" | gawk ’/Sat(urday)?/{print $0}’

Sat

$ echo "Saturday" | gawk ’/Sat(urday)?/{print $0}’

Saturday

Automated FTP File Transfer

2011-04-22T22:02:00.001-07:00

Automated FTP File Transfer :

You can use a here document to script an FTP file transfer. The basic idea is shown here.

ftp -i -v -n wilma <<END_FTP
user randy mypassword
binary
lcd /scripts/download
cd /scripts
get auto_ftp_xfer.ksh
bye
END_FTP

Process a File line by line

2011-04-22T21:49:00.001-07:00

Process a File line by line :

There are Numerous methods to achieve the same one such method is discussed here :

Method 1:

Let’s start with the most common method that I see, which is catting a file and piping the file output to a while read loop. On each loop iteration a single line of text is read into a variable named LINE. This continuous loop will run until all of the lines in the file have been processed one at a time.

The pipe is the key to the popularity of this method. It is intuitively obvious that the output from the previous command in the pipe is used as input to the next command in the pipe. As an example, if I execute the df command to list file system statistics and it scrolls across the screen out of view, I can use a pipe to send the output to the more command, as in the following command:

df | more

When the df command is executed, the pipe stores the output in a temporary system file. Then this temporary system file is used as input to the more command, allowing me to view the df command output one page/line at a time. Our use of piping output to a while loop works the same way; the output of the cat command is used as input to the while loop and is read into the LINE variable on each loop iteration. Look at the
complete function in Listing 2.2.

function while_read_LINE
{
cat $FILENAME | while read LINE
do
echo “$LINE”
:
done
}

Each of these test loops is created as a function so that we can time each method using the shell script. You could also use () C-type function definition if you wanted, as shown in Listing 2.3.

while_read_LINE ()
{
cat $FILENAME | while read LINE
do
echo “$LINE”
:
done
}

Whether you use the function or () technique, you get the same result. I tend to use the function method more often so that when someone edits the script they will know the block of code is a function. For beginners, the word “function” helps understanding the whole shell script a lot. The $FILENAME variable is set in the main body of the shell script. Within the while loop notice that I added the no-op (:) after the echo statement. A no-op (:) does nothing, but it always has a 0, zero, return code. I use the no-op only as a placeholder so that you can cut the function code out and paste it in one of your scripts. If you should remove the echo statement and leave the no-op, the while loop will not fail; however, the loop will not do anything either.

Shell Scripts for beginners

2011-04-10T05:26:00.001-07:00

Sample 1 :Interactive Shell scripts

#SS1

#Usage : SS1

# An Interactive Shell Scripts

echo What is Your name\?

read name

echo Hello $name. Happy Programming.

Instructions : Type the same in an Notepad and save the file with the extension (.sh)

./SS1

SS1:execute permission Denied (By default every file gets created with the permission 666 i.e., rw-rw-rw-).

the next thing that we got to do is

$umask

0022 ( with this we can read and write the file but we can execute it)

$ chmod 744 SS1(with this option we get the read,write,execute option open up for our file).

Sample 2 : Arithmetic in Shell Scripts

#SS2

#Usage : SS2

#Example of Arithmetic Operations

a=20 b=10

echo ‘expr $a + $b’

echo ‘expr $a - $b’

echo ‘exp $a * $b’

echo ‘expr $a /$b’

on execution of this shell scripts we get the following output :

200

Sample 3 : if – then –else – fi statement

#SS3

#usage : SS3

Enter the Source and target file names

read source target

if cp $ source $ target

then

echo file copied successfully

else

echo failed to copy the file

for further Study on the Unix Shell Scripting one can contact us.

Sample Unix shell Scripts

2011-04-10T04:56:00.001-07:00

#ksh: JobNum.sh

#Author: Prasad Degela

#Date: 03/13/2006

#Reviewed by By:

#Date:

#Project: Appeals & Grivances Tracking

####################################################################################

#This script will call .env file to get common variables.

# SET PARAMETERS - Part 1

# These parameters are provided for all interfaces.

# $1 = environment (DEV)

####################################################################################

#set -x

####################################################################################

. /etldata/aagt/common/scripts/aagt.env

inboxdir=$PROJECTDIR/qnxt/inbox

srcfiledir=$PROJECTDIR/qnxt/estage

errorfiledir=$PROJECTDIR/qnxt/error

tempfiledir=$PROJECTDIR/nice/tstage

tempfiledirqnxt=$PROJECTDIR/qnxt/tstage

ERROR_EXIT_STATUS=99

SUCCESS_EXIT_STATUS=0

echo "Control Moved to the Inbox Directory = "$inboxdir

cd $inboxdir

fileconv=`echo "PDP*.txt"`

echo "fileconv = "$fileconv

files=`ls -tr $fileconv`

if [[ $? -ne 0 ]]

then

echo "No Source File in the InBox "$inboxdir

exit $SUCCESS_EXIT_STATUS

echo "List Of Files in the Inbox Directory "$inboxdir

echo "{ $files }"

for pfilename in $files

cd $inboxdir

cp $pfilename $srcfiledir

gzip $pfilename

cp $pfilename.gz $ARCHFILEDIR

rm -f $pfilename.gz

rm -f TRAIL*.txt

echo "Source File Name Is "$pfilename

pDSjobname="sjAG01"

pINVOKE_ID=$pfilename

pJobStatusFile=`echo $pfilename".stat"`

pJobLogFile=`echo $pfilename".log"`

####################################################################################

# RESET AND RUN job sequencer #

####################################################################################

$DSBINDIR/dsjob -jobinfo $DSPROJECT $pDSjobname.$pINVOKE_ID > $LOGFILEDIR/$pJobStatusFile

JOBSTATUS=`grep "Job Status" $LOGFILEDIR/$pJobStatusFile | cut -f2 -d '('|cut -f1 -d ')'`

echo "$pDSjobname Job status code is "$JOBSTATUS

if [[ $JOBSTATUS = 3 ]]

then

echo "Job Status = "$JOBSTATUS

$DSBINDIR/dsjob -run -mode RESET -wait -jobstatus $DSPROJECT $pDSjobname.$pINVOKE_ID

echo "************************* Reset the Job Sequencer *******************"

echo AGDB: $AGDB

echo AGDBUserID: $AGDBUserID

echo ORADB: $ORADB

echo ORADBUSRID: $ORADBUSRID

echo NICEDB: $NICEDB

echo NICEDBUserID: $NICEDBUserID

echo SrcFile: $pfilename

echo SrcFileDir: $SRCFILEDIR

echo TempFileDirectory: $TEMPFILEDIR

echo TempFileDirectoryqnxt:$TEMPFILEDIRQNXT

echo ErrorFileDirectory: $ERRORFILEDIR

echo ScriptFileDirectory: $SCRIPTFILEDIR

echo "calling $Jobname Sequence"

$DSBINDIR/dsjob -run \

-param AGDB=$AGDB \

-param AGDBUserID=$AGDBUserID \

-param AGDBPswd=$AGDBPswd \

-param ORADB=$ORADB \

-param ORADBUSRID=$ORADBUSRID \

-param ORADBPswd=$ORADBPswd \

-param NICEDB=$NICEDB \

-param NICEDBUserID=$NICEDBUserID \

-param NICEDBPswd=$NICEDBPswd \

-param SrcFile=$pfilename \

-param SrcFileDir=$SRCFILEDIR \

-param TempFileDirectory=$TEMPFILEDIR \

-param TempFileDirectoryqnxt=$TEMPFILEDIRQNXT \

-param ErrorFileDirectory=$ERRORFILEDIR \

-param ScriptFileDirectory=$SCRIPTFILEDIR \

-wait -jobstatus $DSPROJECT $pDSjobname.$pINVOKE_ID

$DSBINDIR/dsjob -jobinfo $DSPROJECT $pDSjobname.$pINVOKE_ID > $LOGFILEDIR/$pJobStatusFile

$DSBINDIR/dsjob -logsum $DSPROJECT $pDSjobname.$INVOKE_ID > $LOGFILEDIR/$pJobLogFile

JOBSTATUS=`grep "Job Status" $LOGFILEDIR/$pJobStatusFile | cut -f2 -d '('|cut -f1 -d ')'`

echo "$pDSjobname Job status code is "$JOBSTATUS

if [[ $JOBSTATUS = 1 ]]

then

echo "Job Status = "$JOBSTATUS

echo "Removing File from $srcfiledir on successful processing of file $pfilename"

cd $srcfiledir

rm -f $pfilename

else

exit $ERROR_EXIT_STATUS

done

echo 99 END OF THE PROG

exit $SUCCESS_EXIT_STATUS

#####################################################################################################

Standards for ETL UNIX Shell Scripts for use with PowerCenter 7.1.3

2011-04-10T04:50:00.001-07:00

Standards for ETL UNIX Shell Scripts for use with PowerCenter 7.1.3

Scripting Standards (for PowerCenter version 7.1.3):

Scripting standards include the use of a UNIX shell script, which the scheduling tool uses to start the PowerCenter job, and a separate file which contains the username and password for the user called in the script. The following is a template that should be used to create scripts for new scheduled jobs. Following is the script and explanation. This script has been provided as an example and is named etl_unix_shell_script.sh.

# Program: etl_unix_shell_script.sh

# Author: Kevin Gillenwater

# Date: 7/24/2003

# Purpose: Sample UNIX shell script to load environment variables

# needed to run PowerMart jobs, pass username and password variables

# and start the job using the pmcmd command line.

# $1 = Project Id Parameter (ie. ud, hr, dss, siq, etc.)

# Example usage: etl_unix_shell_script.sh dss

# NOTE: Enter the Project ID parameter that is designated in the

# directory structure for your team

# (ie. dss would be used for the DWDAS team as the

# directory is /usr/local/autopca/dss/)

#-----------------------------------------------------------------

# Call the script to set up the Informatica Environment Variables:

#-----------------------------------------------------------------

. /usr/local/bin/set_pm_var.sh

#-----------------------------------------------------------------

# Read ETL configuration parameters from a separate file:

#-----------------------------------------------------------------

ETL_CONFIG_FILE=$JOBBASE/$1/remote_accts/test_etl.config

ETL_USER=`grep ETL_USER $ETL_CONFIG_FILE | awk -F: '{print $2}'`

ETL_PWD=`grep ETL_PWD $ETL_CONFIG_FILE | awk -F: '{print $2}'`

#-----------------------------------------------------------------

# Start the job

#-----------------------------------------------------------------

$PM_HOME/pmcmd startworkflow -u $ETL_USER -p $ETL_PWD -s $MACHINE:4001 -f DWDAS_

LOAD_dssqa –wait s_m_CENTER_INSTITUTE

#-----------------------------------------------------------------

# Trap the return code

#-----------------------------------------------------------------

rc=$?
if [[ $rc -ne 0 ]] then
exit $rc
fi

Notes Regarding the Script/Standards:

1. The beginning of each script should call and execute the script set_pm_var.sh to set up the PowerCenter variables used in the session (. /usr/local/bin/set_pm_var.sh). The set_pm_var.sh, located on each machine (Leisure for Production and Glance for Development), will provide ease of maintenance if changes need to be made to PowerCenter variables, and will provide one source for the scripts on a machine. The following is the code in the script. You will not need to do anything with this script, it is for informational purposes only:
# Program: set_pm_var.sh

# Author: Kevin Gillenwater

# Date: 7/3/2003

# Purpose: UNIX script which sets the variables for running PowerMart 6.2

# when called from a shell script (ie. script run by Autosys).

#-------------------------------------------

#Set up the Informatica Variables

#-------------------------------------------

export MACHINE=`hostname`

export SHLIB_PATH=/usr/local/pmserver/informatica/pm/infoserver

export PM_HOME=/usr/local/pmserver/informatica/pm/infoserver

#---------------------------------------------------------------------

# Set the environment variables needed for scheduling jobs.

# The value of JOBBASE differs based on the account. For AUTOSYS

# and AUTODBA, the variable should evaluate to /usr/local/autopca.

# For all other accounts, it should evaluate to their $HOME variable.

#---------------------------------------------------------------------

case $HOME in

/home/autopca/autopca) JOBBASE=/usr/local/autopca ;;

/home/autopca/autodba) JOBBASE=/usr/local/autopca ;;

*) JOBBASE=$HOME ;;

esac

export JOBBASE

2. The second section of etl_unix_shell_script.sh sets up ETL parameters for username and password usage by PowerCenter when running the workflow. The username and password are no longer stored as part of the main shell script. In the script example, the filename test_etl_config contains the username and password to be used when running the workflow. Following is the contents of test_etl_config:

ETL_USER:etlguy

ETL_PWD:ou812

This file (your password file) must be located in your team directory under the remote_accts folder (i.e. /usr/local/autopca/dss/remote_accts/). The folder permissions on the Production machine will only permit PCA’s access to this folder. The permissions should be 6-4-0 (rw,r,-) on this file.

3. The third section of etl_unix_shell_script.sh contains the command to run the workflow in PowerCenter. Please follow the script exactly. The only thing that will need to be changed in this section is the folder name and the workflow within the folder that is being executed

(i.e. ... DWDAS_LOAD_dssqa wflw_m_CENTER_INSTITUTE).

4. The final section of etl_unix_shell_script.sh contains code to trap the return code of the script that indicated success or failure..
*8/27/03 – Added ‘-wait’ parameter to the pmcmd command line to start the scheduled job.
*10/6/04 - Updated to generalize the scheduling tool used to run the shell scripts as UC4 has been chosen to replace Autosys as the scheduling tool used to run PowerMart workflows.

Pattern Matching

2011-03-25T07:15:00.000-07:00

Command	Options	Description
*		Matches any number of character including NONE. * does not match all files beginning with dot <.>.
	ls -l chap*	Matches all the files which starts with chap.
	ls -x chap*	Matches all the files which starts with chap and prints in multi column way.

?		Matches single character.
	ls -l chap?	Matches all the files with only 5 character name and should start with chap.

[ ijk ]		Matches single character either i or j or k.
[ !ijk ]		Matches single character that is not i or j or k
[ x - z ]		Matches single character that is not within the ASCII range of character x and z.
[ ! x - z ]		Matches single character that is not within the ASCII range of character x and z

	ls -l chap0[1 - 4]	Range specification is also available.
	ls -l [ a – z A-Z]	Matches all file names beginning with alphabet irrespective of case.
	ls -l chap[!0 - 9]	Matches all file names beginning with alphabet chap and not ending with any number.
	ls *.ist	Print all the files which end with ist extensions.
	cp chap?? abc	Copy all files starts with chap to abc directory.
	cmp chap[12]	Compares file chap1 and chap2.
	mv * ../bin	Moves all the files to bin directory.
	cat chap[!0 - 9]	Concatenates all the files beginning with chap and not ending with number.
	ls -l *	* does not match all files beginning with dot <.>.
	ls -l .???*	The above problem can be solved with specifying first few character using meta character <?> explicitly.

Escaping Backslash ( \ )		Playing with file names which uses meta character in their file name.
	ls - l chap*	Print all files whose names start with chap but not the one whose name is chap*.
	ls - l chap\*	The above problem can be solved by escaping the special character.

Pipe \|		To pass the standard output of one command as the standard input to another.
	who \| wc -l	Output of who command <three users> passed as input to wc which count the number of lines present.

Tee	who \| tee users list	Tee saves the output of who command in user list as well as display it also.

Shell Variable		Shell variables are initialized to null value < by default > so it returns null. All words starts with $ are considered as variable unless single quoted or escaped.
	a=ab, b=cd, z=$a$b. echo $z	shell concatenate two variable.
	echo '$10' eg: $10	All words starts with $ are considered as variable unless single quoted or escaped.
	echo "$10" eg: 0	Shell is looking for $1 variable which is undefined so passes null for this. $1 is part of positional parameter.

Control Structures

2011-03-25T07:08:00.000-07:00

Pattern Searching

2011-03-24T10:58:00.001-07:00

Commands	Options	Description
/	/Unix	Forward search of Unix keyword in file
?	?Unix	Backward search of Unix keyword in file
n		Repeat the last search
.		Repeat the previous command

:1, $s/<search string>/<Replace String >/g		Pattern search and replacement.
	1,$	Represent all lines in the file.
	g	Stands for globally
:3,10s/gagan/deep/g		search between lines 3 and 10
: .s/gagan/deep/g		Only the current line.
: $s/gagan/deep/g		Only the last line.
: $s/gagan/deep/gc	c	Ask for confirmation for replacement

grep
	-c	counting occurrence.
	-n	display line number for record.
	-v	skip records that contain directory.
	-l	display files containing record.
	-i	ignore case.
	[PQR]	match any single character.
	[c1-c2]	match char with ASCII range
	[^PQR]	match single character which is not PQR.
	^<pat>	beginning with pattern.
	<pat>$	ending with pattern.
	{a-Z A-Z 0-9}	match any single character.
	ls –l \|grep “^d”	Prints only directories.

egrep
	egrep '[aA]g+[ar][ar]wal' test1.txt	match one or more occurrence matches ag & agg.
	egrep '[aA]gg?[ar][ar]wal' test1.txt	match zero or one occurrence.
	egrep prashant\|director test1.txt	match eap1 or eap2 (finds prashant or line with director)
	egrep (das\|sen)gupta test1.txt	match exp x1x3 or x2x3 (like dasgupta & sengupta)
	egrep -f <pattern_file_name> test1.txt	Huge list of pattern search can passed in the form of file name .pattern stored in file eg(prashant\|admin\|director)
	+	Matches one or more occurrence of previous character.
	?	Matches zero or one occurrence of previous character.

Fgrep		fgrep and egrep accepts multiple pattern both form command line and a file but unlike grep and egrep does not accept regular expression.
	fgrep -f pattern file emp file	Faster than grep and egrep family

Zipping Files

2011-03-24T09:06:00.001-07:00

Command	Options	Description
1. gzip	gzip etl_code eg:etl_code.gz
	gunzip etl_code.gz eg: etl_code
	zip filesql eg:* file.zip
	unzip file.zip eg:file*sql

2. tar		create backup of files recursively.
	-cvf eg :tar -cvf /home/gagan/sqlbackup ./*.sql
	- x eg:tar-xcvf /home/gagan/sqlbackup ./*.sql	files are restored using -x option.

3. Cut	-c <column start and no> eg : cut -c -5,6-12 test file	column (by specifying position). C stands for column cut.
	-f <field start and end no> eg:cut -f 1,5 test file	field (default deliminator tab). F stands for field cut.
	'-d -f <field start and end no> eg: cut -d "\|" -f 1,5 test file \| new file	Cut the field b/w 1 and 5 and piped the output to new file.

4. sort	sort test file	By default the sorting starts with first character of each line. Priority1.space, tabs 2. numerals 3.uppercase letters 4. Lower case letters.
	-t eg : sort -t "\|" +2 test file	Sorting starts from 3rd field skipping 2nd field, overring the default.delimiter to distinguish b/w start and end of field.
	-r eg : sort -t \\| -r +2 test file	Reverse sort starting with 3rd field.
	eg : sort -t \\| +2r test file	The above command can be written in another way.
	-o eg : sort -o abc.txt abc_sort.txt	save sorted data in file.
	eg : sort -t "\|" +1 -2 +3 <>	Sorting based on different field as in case of order by clause.Sorting starts with 2nd field then with 4th field, -2 indicate to stop the sorting after 2nd field and resume it with 3rd field.
	-n eg : sort -n <>	Numeric sort.
	-u eg : sort -u <>	Unique sort.

5.paste	-d eg:paste -d "\|" <> <>	deliminator.

6.tr	eg : tr '\|\' '~-' <test.txt	translate all \| with ~ and \ with - .
	eg : tr '[a-z]' '[A-Z] <test.txt	translate to upper case.
	-d eg : tr -d '\|' test.txt	delete all occurrence of \| .

7.uniq		Unique require sorted file as input.
	-u eg : cut -d "\|" -f3 <>\|sort\|uniq -u	remove duplicate.
	-d eg: cut -d "\|" -f3 <>\|sort\|uniq -d	select only dup records.
	-c eg : cut -d "\|" -f3\|sort\|uniq -c	duplicate count.

8.Changing time stamp		touch mon date hrs mins <file>.
ls -lt	time of last modification	touch -m 01290430 <file>.
ls -lu	time of last access	touch -a 01290415 <file>.

9.Change Date	date 09181754

10.wall	wall -g dba "hello"	to selectively send msg to dba group.

11.shutdown	shutdown -g2	power down after 2 mins.
	shutdown -y -g0	immediate shutdown.
	shutdown -y -g0 -i6	shutdown and reboot (init level 6).
	shutdown 17:30	shutdown at 17:30.
	shutdown -r now	shutdown immediate and reboot.

12. du		Disk usage.
	du /home/expimp/create_db	tree output for each directory inside.
	du -s //home/expimp/create_db	summary.

13.find	find <loc> <option> <pattern> <action>	find in root dir abc in emp.lst file
	'-mtime = mod time eg: find . -mtime -2 -print	find file modi in less then 2 days.
	'atime = access time eg: find . -atime +365 -print	find the file not accessed in last 1 year.
	! –newer eg: find / -name "*.pl" ! -newer last_backup -print	file modi before last_backup.
	-size. eg: find . -size +2048 -print	files greater then x blocks.
	-a (and) -o (OR) *eg: find . $ -name ".sh" -o -name ".lst" $ -print*	double quotes necessary.
	-exec eg: find . -atime +180 -ok rm -f {} \;	remove the files which are not modi for last 20 days
	-ok eg: find . -atime +180 -ok rm -f {} \;	before removing prompt for confirmation.
	xargs eg: find . -mtime +20 \| xargs rm -f	remove all file rm will be executed only once.
	xargs -n -p –t eg:find . -mtime +20 \| xargs -n20 -p -t rm -f	remove at max 20 files in batch and in interactive mode.
	-type eg: find / -name log -type f -print	f for file and d for directory.
	-prune eg: find . -name *.log -prune exe –print.	don't descend exe directory.

Basic Commands Part2

2011-03-24T08:49:00.001-07:00

Command

Options

Description

25.Head

head newfile

head -5 newfile

head -1 newfile | wc -c

vi 'ls -t | head -1'

head -c 512 newfile

head -c 1b newfile

head -c 2m newfile

Display top content of file

Display top 10 lines of file when used without argument

Display top 5 lines of file<newfile> when used without argument

To count the number of character present in first line.

To open the last modified file for editing.

To pick special no of character from file.

Print first 512 byte character

Print first 2 blocks (1 MB each)

26.Tail

tail -5

tail +10

tail -f 6_load.log

Display end of file

Display last 5 lines of file.

start dispplaying 10 line onward till the end.

Print the gworth of file

27.$$

Prints the Process id of current shell

28.ps

ps -f

ps -f -u a_cmdb

ps -a

ps -e

Prints the Processes associated with current user.

Prints the Processes associated with current user with their hierarchy.f stands for full.

Prints all the processes associated with a_cmdb user.

Prints all the users processes.

Prints system process.

prints all the ksh process running in the system by first searching for all the process.

29.Background job

ps -f

ps -f -u a_cmdb

ps -a

ps -e

Running job in background

Run the search job in background by printing its process id.Shell become the parent of all background process.

This command print all the jobs runnung in background with their sequence no of exceution.

Running fg will bring most recently started background process <LAST IN FIRST OUT>

First job will come into foreground.

30.nohup

nohup ps -eaf | grep 'ksh' &

Shell is parent of all background jobs and it dies when user logs out and ultimately all the child process also dies.To avoid dying of shell even when the user logs out run the background process using nohup command.

System process init having PID as 1 is parent of all SHELL processes and when the user login this PID become the PPID of SHELL prrocess. Now after running the process in nohup mode kernel has reassigned the PPID of ps process with PID of system process which will not die even after SHELL dies through logging out.

31.Kill

kill 520

kill 640

kill 1

kill $!

kill 0

To kill the command.

Kill the process with 520 process id.

Kill the parent process id which inturn kill all the child process.

Killing system process init is not possible. A Process having process id as 1 is parent of all SHELL processes.

No need to remember Process id of last background procees its in $!

Kill all process in the system except login shell process.

$$ stores the PID of current login shell

32.Nice

nice who | wc - l &

nice -n 10 who | wc -l &

Running the command with low priority

Nice value range from 1 to 39, higher the nice value lower the priority.default nice value is 20.

nice value becomes 30.

33.At

at 2:10 load.ksh

To Run the job at specied time of the day.

Indicate the load.ksh will execute today by 2:10

34.Batch

batch load.ksh

batch command executes the process whenever the CPU is free.

35.Cron

cron executes process at regular intervals it keeps checking the control file at /user/spool/cron/crontabs for jobs.

36.Finger

finger

finger a_cmdb

Produces list of all logged users when run without any argument. The 2nd and last field of output is taken fron passwd file from /etc directory.

Prints details about a_cmdb user.

37.Set

set 10 20 30

-x

set the posional parameter.

This will set $1, $2, $3 , $* and $# posional parameter

When this statement is used at start of ksh script it echoes each statement at the terminal

38.Shift

shift

shift 2

shift command transfer the contents of positional parameter to next lower number.

shift one position.

shift 2 places

39.Chown

chown jaspreet testfile

chown -R jaspreet testfile

To change the owner of file can only be done by owner or sys admin.

change the owner from gagan to jaspreet for testfile. Now gagan cannot change the ownership of test file what he can do is just create the similar copy of file and then he can become the owner of file.

Recursively changes the owner all files under current directory.

40.Chgrp

chgrp GTI test file

chgrp -R GTI test file

To change the group of file can only be done by owner or sys admin.

group changed to GTI from IB as user is still the same and he can again change the group of file to IB.

Recursively changes group all files under current directory.

41.Touch

touch 03161430 test file

touch -m 04161430 test file

touch -a 05161430 test file

Changing the time stamps of file

touch without any argument will change both modified time and access time.

Changes only modification time.

Changes only access time.

42.Linking in

ln 6_load.ksh 7_load.ksh

rm 6_load.ksh

Linking two file, doing this both the file will have same inode number and changes in one will reflect in another also.

ls -li 6_load.ksh 7_load.ksh will give us same inode number.

Drop the link between two files.

43.df

df -t /home/oracle

Prints the amount of free space available on the disc.

Prints the free and total space under oracle file system.

44.du

du -s

du -s /home/*

Prints the disk usage of specific directory tree.

By default du prints disk usage for each sub directory.

Prints summary of whole directory tree.

Prints disk usage for every user

Basic Commands Part1

2011-03-24T08:32:00.001-07:00

Command

Option

Description

1. Uname

uname -r

uname -n

To find the version of unix os

To find the machine name

2. Man

man grep

man -e grep

man -k copy

Print the user manual for any command.

To find the user manual for grep command.

One line definition of grep usage.

Helpful when we donot know which command will solve the purpose.e.g :- gives the different option available for copying.

3.Cal

cal 2009

cal dec

Print the calendar year or month

Print the calendar 2009 year

Print calender for current december month

4. Date

date

date +%m

date +%h

date +'%h%m'

date +%d

date +%y

H,M and S

Tue Mar 09 07:10:10 IST 2009

March

Mar-03

5.Who

who

who am I

who -H

who -u

6.Passwd

To change the password

7.Lock

lock -45

lock the terminal for 45 minutes

8.Bc

To use the calculator

9.ls

ls -x

ls -F

ls -a

ls -r

ls -x abc

ls -R

ls -ld

ls -il testfile

ls -lt

ls -ltr

ls -lut

To Print the contents of current directory.

Prints Multi column output of files with their names.

To Print only directoris and executable files.< * > indicate all files containing executable code and </> indicate the directories.

Print all files including hidden files. <a> stands for all.

Reverse the sort order of all the files with their names.

Print the content of abc directory.

Print all files and sub-directories in directory TREE.

Print all the directories.

Prints the inode number of file.

Prints all the file with their last modification time keeping latest modified at top.

Prints all the file with their last modification time in reverse order keeping last modified at bottom.

prints last access time of files.

10.Cat

cat testfile

cat > testfile

Print the content of any file.

Create a new file with testfile name.

11.CP

cp -i testfile abc

cp -r abc newabc

To copy file and directories.

copying testfile to abc directory interactively.

copying entire directiry structure abc to newabc recursively.

12.rm

rm -f newabc

rm -r newabc

All the options of cp command is applicable for rm command.

Removing directory newabc forcefuly.

Recursive deletion will not remove the write protected files.

13.lp

Used for Printing the files

14.Split

split testfile

split -72 testfile

split testfile newfile

To split the big files in different small files of 1000 lines each<default>.

Split the files into small files.

change the default split line size to 72.

Renaming the testfile to new name as <newfile>.

15. Cmp
comm
Diff

These three command are used to find the difference in two files

16.wc

wc -w testfile

wc -c testfile

wc -l testfile

wc testfile

Print the numbers of words, lines and character in the file

17.Path

PATH=$PATH:/home/
gagan:/home/deep

Set the path variable for SHELL to use these directory path to locate the executables commands.

18.stty

stty -a

stty intr \^c

stty eof \^a

Print current setting of terminal

To delete the character during backspacing

To change the interrupt key to Ctrl c instead of 'DELETE' default

To change the termination control of input during file creation using cat command from Ctrl d to Ctrl a.

19.Alias

alias

alias l ='ls -ltr'

alias showdir ='cd $1;ls -l'

unalias

Print all the alias set in the system.

To create short hand names of command.

Passing positional parameter ETL to showdir alias will take us to that directory.

To redefine and unset the alias.

20.History

history -5

history 10 15

HISTSIZE 1200

To find the previous command that has been used.

Print the last 5 command that has been used.

Every command has event number. Display event number between 10 an d15.

To change the defalult setting of command history saving feature.

21. r
!!

r-2

r 20

Repeat previous command in Korn

Repeat previous command in Bash

Repeat previuos to prevoius command

Repeat the command with 20 event number.

22.Tilde(~)

Changes control to home directory

Changes absolute path home directory of user a_cmdb, the value of home directory can be seen by $HOME environment variable

23.cd -

To switch between current working directory and most recently used directory.

24.chmod

chmod u+rwx newfile

chmod g+x newfile

chmod 757 newfile

chmod 457 newfile

chmod -R a+x abc

chmod -R 001 .

Changing the permission of file for user <u>, group <g>, others <o> or all <a> with + as assign , - as remove and = as absolute permission.r read<4>, w write<2> and x execute<1>.

Recursively changes the permission to execute for all directory and sub directory in abc directory for all users.

. Indicate the current directory.

Why do we Use Shell Scripting

2011-03-24T07:11:00.001-07:00

We use Shell Scripts to obtain the following purposes.

1. Customizing your work environment.

2. Automating Your Daily Task.

3.Automating Repetitive Task.

4.Executing Important Procedures like shutting down the system,formatting a disk,Creating a file system on it,mounting the file system,letting the users use the floppy and finally un mounting the disk.

5.Performing the same operation on many files.

When Not to use Shell Scripting :

Is too complex,such as writing an entire billing system.
Requires a high degree of efficiency
Requires a variety of Software tools.

Usage of Shell Scripts with in Informatica:

1.To run an batch via pmcmd.

2. To have header and footer in case we are goanna write to an flat file

3.To run the command task.

4. To update parameter file with session start time and end time

Introduction to Shell Scripting

2011-03-24T06:57:00.001-07:00

A shell script is a script written for the shell, or command line interpreter, of an operating system. It is often considered a simple domain-specific programming language. Typical operations performed by shell scripts include file manipulation, program execution, and printing text.

Types of shell scripts : Basically we have three types of the shell scripts [Most Prominent one’s ,widely used in industry].

1. Bourne Shell :

The Bourne shell, or sh, was the default Unix shell of Unix Version 7, and replaced the Thompson shell, whose executable file had the same name, sh. It was developed by Stephen Bourne, of AT&T Bell Laboratories, and was released in 1977 in the Version 7 Unix release distributed to colleges and universities. It remains a popular default shell for Unix accounts. The binary program of the Bourne shell or a compatible program is located at /bin/sh on most Unix systems, and is still the default shell for the root superuser on many current Unix implementations.

Features of the Bourne shell include:

Scripts can be invoked as commands by using their filename
May be used interactively or non-interactively
Allow both synchronous and asynchronous execution of commands
supports input and output redirection and pipelines
provides a set of built-in commands
provides flow control constructs, quotation facilities, and functions.
typeless variables
provides local and global variable scope
scripts do not require compilation before execution
does not have a goto facility, so code restructuring may be necessary
Command substitution using back quotes: `command`.
Here documents using << to embed a block of input text within a script.
"for ~ do ~ done" loops, in particular the use of $* to loop over arguments.
"case ~ in ~ esac" selection mechanism, primarily intended to assist argument parsing.
sh provided support for environment variables using keyword parameters and exportable variables.
It contains strong provisions for controlling input and output and in its expression matching facilities.

2. C Shell :

The C shell (csh or the improved version, tcsh, on most machines) is a Unix shell that was created by Bill Joy while a graduate student at University of California, Berkeley in the late 1970s.

The C shell is a command processor that's typically run in a text window, allowing the user to type commands which cause actions. The C shell can also read commands from a file, called a script. Like all Unix shells, it supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration. What differentiated the C shell, especially in the 1980s, were its interactive features and overall style. Its new features made it easier and faster to use. The overall style of the language looked more like C and was seen as more readable.

Features of C Shell :

Some of the features of the C shell are listed here:

Customizable environment.
Abbreviate commands. (Aliases.)
History. (Remembers commands typed before.)
Job control. (Run programs in the background or foreground.)
Shell scripting. (One can write programs using the shell.)
Keyboard shortcuts.

3. Korn Shell :

The Korn shell (ksh) is a Unix shell which was developed by David Korn (AT&T Bell Laboratories) in the early 1980s and announced at Toronto USENIX on July 14 1983

ksh is backwards-compatible with the Bourne shell and includes many features of the C shell as well, such as a command history, which was inspired by the requests of Bell Labs users.

The Korn shell's major new features include:

Command-line editing , allowing you to use vi or emacs -style editing commands on your command lines.
Integrated programming features : the functionality of several external UNIX commands, including test , expr , getopt , and echo , has been integrated into the shell itself, enabling common programming tasks to be done more cleanly and without creating extra processes.
Control structures , especially the select construct, which enables easy menu generation.
Debugging primitives that make it possible to write tools that help programmers debug their shell code.
Regular expressions , well known to users of UNIX utilities like grep and awk , have been added to the standard set of filename wildcards and to the shell variable facility.
Advanced I/O features , including the ability to do two-way communication with concurrent processes (coroutines ).
New options and variables that give you more ways to customize your environment.
Increased speed of shell code execution.
Security features that help protect against "Trojan horses" and other types of break-in schemes.