...................Linux

Ansil4Linux Blog Moving to new site https://linxlabs.com/

2017-12-10T12:39:00.000-08:00

I've been studying new topics for some time and decided to start a new website

Site is now live and can be accessed at https://linxlabs.com/

Many articles were already posted on "Golang"

Suscribe and follow me on below places

YouTube : https://www.youtube.com/channel/UCCA7cWxxqOC95VR8JPEFn_w

FB : https://www.facebook.com/OpenLinuxLabs

Google+ : https://plus.google.com/communities/109444869997457310738

LVM thin provisioning - file system usage and pool usage dosn't match

2017-07-23T06:27:00.000-07:00

When I was demonstrating LVM thin provisioning to new batch of campus hires ; they pointed out an important mismatch between thin pool usage and actual file system usage.

I thought , it would be worthy to give it a try to find the cause.

So here we go; I created a thin pool with size of 100MB and thin volume of 1GB

Thin volume formatted with ext4 and mounted over /test_fs

Everything was going smooth as per the plan until we did a tar backup of /usr over /test_fs.

Here we go ;

* Created vg001 - size 12GB

[root@ol7-san ~]# vgdisplay vg001

--- Volume group ---

VG Name vg001

System ID

Format lvm2

Metadata Areas 1

Metadata Sequence No 12

VG Access read/write

VG Status resizable

MAX LV 0

Cur LV 0

Open LV 0

Max PV 0

Cur PV 1

Act PV 1

VG Size 12.00 GiB

PE Size 4.00 MiB

Total PE 3071

Alloc PE / Size 0 / 0

Free PE / Size 3071 / 12.00 GiB

VG UUID kroGDG-Rs8L-1c0e-WBjJ-g0OG-4u8k-T4yaJl

[root@ol7-san ~]#

* Created a thin pool with size of 100MB

[root@ol7-san ~]# lvcreate -L 100M -T vg001/mythinpool
Logical volume "mythinpool" created.
[root@ol7-san ~]#

* Created a thin volume of size 1GB

[root@ol7-san ~]# lvcreate -V1G -T vg001/mythinpool -n thinvolume
Logical volume "thinvolume" created.
[root@ol7-san ~]#

* Formatted thinvolume with ext4

[root@ol7-san ~]# mkfs.ext4 /dev/vg001/thinvolume
mke2fs 1.42.9 (28-Dec-2013)
..(snipped)...
Writing superblocks and filesystem accounting information: done

[root@ol7-san ~]#

* Mounted the file system over directory /test_fs

[root@ol7-san ~]# mount /dev/vg001/thinvolume /test_thin
[root@ol7-san ~]# df -h /test_thin/
Filesystem                    Size Used Avail Use% Mounted on
/dev/mapper/vg001-thinvolume 976M 2.6M 907M   1% /test_thin
[root@ol7-san ~]#

* Everything is good and lets do a tar backup

[root@ol7-san test_thin]# tar -cf usr_bkp.tar /usr
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets

* Lets check the FS usage

[root@ol7-san ~]# df -h /test_thin/
Filesystem                    Size Used Avail Use% Mounted on
/dev/mapper/vg001-thinvolume 976M 312M 597M 35% /test_thin
[root@ol7-san ~]#

* Lets check the thinpool usage as well

[root@ol7-san test_thin]# lvs
LV         VG    Attr         LSize   Pool       Origin Data% Meta% Move Log Cpy%Sync Convert
mythinpool vg001 twi-aotz--   100.00m                   100.00 1.95
thinvolume vg001 Vwi-aotz--   1.00g   mythinpool        9.77
[root@ol7-san test_thin]#

Pool is 100% utilized and it clearly shows its size of 100MB

SUPERSIZE..!!! :) File system usage shows 312MB , but our thin pool is only 100MB.

So where is the remaining 212MB stored ???

* ) I pressed Ctrl+C on tar as its hung and tried to do a sync

[root@ol7-san test_thin]# sync

It hung..!!! Gotcha.. the contents were stored in cache in memory and not really written to disk.

So that solves the mystery of file system usage and pool usage mismatch.

You may extend the pool after breaking tar to get the sync commit all dirty page to disk and return to terminal.

One more catch - removing file won't release thin pool usage ; you need to execute fstrim over the file system to get the space back to the pool

2016-08-12T04:13:00.002-07:00

SFTP auto login without keys / without password

sftp command doesn't have option to specify password for auto login.

Recently faced this issue while writing a script on SUSE Linux for transferring system performance report to a central location, which is also a SuSE system.

We can accomplish this with "lftp" or "expect" command, but SLES doesn't install those binaries default.

I've to live with what I got, so let’s try how we can accomplish this with sftp command.

The idea is

1.Set "SSH_ASKPASS" environment variable and force "sftp" command to read password from there.

2."sftp" will always ask password if its executing from a terminal, so we need to daemonize sftp command with "setsid" command

Below code snippets will give some idea how we get it to the solution

pathnames.h

/* Default path to ASKPASS program */

#define _PATH_SSH_ASKPASS_DEFAULT "/usr/X11R6/bin/ssh-askpass"

ssh.h

/*
* Environment variable for overwriting the default location of askpass
*/
#define SSH_ASKPASS_ENV "SSH_ASKPASS"

readpass.c

...
...

if (use_askpass && getenv("DISPLAY")) {
if (getenv(SSH_ASKPASS_ENV))
askpass = getenv(SSH_ASKPASS_ENV);
...
...

execlp(askpass, askpass, msg, (char *) 0);
...
...

So here we have all parts in place and we need to assemble it.

Now let’s start the real work

1.User name is "testvsftp" and password is "pass123"
2.echo "echo pass123" >/tmp/pass.sh
3.chmod 755 /tmp/pass.sh
4.export SSH_ASKPASS=/tmp/pass.sh
5.export DISPLAY=":0" # As mentioned in readpass.c code , askpass = getenv(SSH_ASKPASS_ENV); will execute only if we have "DISPLAY" variable set.
6.setsid sftp -o StrictHostKeyChecking=no testvsftp@hostname << EOF
pwd
ls
EOF

That’s it..!!!..Now you know how to automate sftp login in scripts

Note:- “StrictHostKeyChecking=no” option in sftp command is to accept keys from SFTP server automatically , otherwise if you are accessing the server first time, it may wait and ask for accepting the keys , eventually your script won’t continue from there.

Linux OS backup and Restoration using ReaR – Relax & Recover

2012-03-18T12:50:00.000-07:00

ReaR – Relax and Recover is an Opensource tool for backing up your OS.

The Physical to Virtual and Virtual to Physical migration of OS is also possible with ReaR

You can also integrate your favorite backup tools like IBM TSM or HP Data Protector with ReaR to backup the OS to External Storage

Here is the setup I used

OS :openSuSE 11.4

ReaR package: rear-1.10.0-23.1.noarch

Pre-requisites 1:

RPMs

mingetty

binutils

iputils

tar

gzip

ethtool

syslinux

iproute2

lsb

genisoimage

nfs-client

rpcbind

rpmlib(PayloadFilesHavePrefix) <= 4.0-1

rpmlib(CompressedFileNames) <= 3.0.4-1

/bin/bash

rpmlib(PayloadIsLzma) <= 4.4.6-1

Pre-requisite 2:

NFS server ( I used RHEL5 )with a share /rear/backup and the share must have write permission

/etc/exports

/rear/backup *(rw,no_root_squash)

service nfs start

showmount –e

----------------------------------------------

If yor NFS server is already running, then

Add “/rear/backup *(rw,no_root_squash)” to /etc/exports

exportfs –a

showmount –e

The main configuration file is /etc/rear/local.conf

Make sure below entries are present on file

OUTPUT=ISO

BACKUP=NETFS

NETFS_URL=nfs://ip.of.nfs.server/rear/backup

Then execute rear to create a backup with bootable ISO which can be used for booting incase of system crash and using that file we can restore OS from NFS partition

You can also configure a PXE boot server to recover the OS without making ISO to DVD for booting

mysql01:~ # rear mkbackup

Relax & Recover Version 1.10.0 / 2011-02-20

WARNING ! The NETFS backup method should not be your primary

backup solution, because it does not fulfill the requirements

of a professional backup solution !

The preparation phase OK

Physical devices that will be recovered: /dev/sda

Creating root FS layout OK

WARNING: To login as root via ssh you need to setup an authorized_keys file in /root/.ssh

Copy files and directories OK

Copy program files & libraries OK

Copy kernel modules OK

Checking udev OK

Create initramfs \

It will take some time to backup all partitions ,depends on the data inside every partition

After completion of backup ,the files will be stored in NFS share.

The files will be inside a directory with the name same as host name.In my case the host name is mysql01

So I got a directory named mysql01 in /rear/backup

The contents inside the directory is listed below

total 1430088

-rw-r--r-- 1 root root 1418761646 Mar 18 14:11 backup.tar.gz

-rw-r--r-- 1 root root 8329424 Mar 18 16:59 backup.txt

-rw-r--r-- 1 root root 202 Mar 18 14:11 README

-rw-r--r-- 1 root root 35321 Mar 18 14:11 rear-2012-03-18T14:11:38+0530.log

-rw-r--r-- 1 root root 35821568 Mar 18 14:11 rear-mysql01.iso

-rw-r--r-- 1 root root 368 Mar 18 14:11 VERSION

The rear-mysql01.iso need to written to a CD or DVD to boot the system after a crash

Suppose my server mysql01 is crashed because of File system corruption ,then we will follow the below steps

1.Write the rear-mysql01.iso to a DVD/CD

2.Boot from the CD/DVD which is now created

3.on boot: prompt enter rear and press Enter

boot:rear àpress enter

4.Login with user name root (It will not ask for password)

5.We will get a prompt

6.Set up network using IP command

# ip addr add 192.168.56.11/24 dev eth0

# ip link set eth0 up

If you need to add a route ,use “ip route” command (man ip for more details)

7.Now check whether the NFS server is reachable using PING

8.If the server is accessible then execure rear recover

# rear recover

The command will mount the NFS share and it will create all partitions as previous and restores all data

9.Once the restoration completed ,remove the DVD/CD and reboot

10.You will get your system in perfect condition

If you are going to restore the OS backup of one server to another,

1. Boot from DVD/CD

2. Login as root (Will not ask for password)

3.Set up networking

4.rear recover

3.Mount /dev/system/root /mnt

4.cat /mnt/etc/fstab and check the disk ids and change it to present disks id (you can use VI to edit file)

/dev/disk/by-id/ata-XXXXX ace79de-c82e0d0d-part1 /boot ext4 acl,user_xattr 1 2

5.chroot /mnt

6.grub

GNU GRUB version 0.97 (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported. For the first word, TAB

lists possible command completions. Anywhere else TAB lists the possible

completions of a device/filename.]

grub> root (hd0,0)

Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)

Checking if "/boot/grub/stage1" exists... no

Checking if "/grub/stage1" exists... yes

Checking if "/grub/stage2" exists... yes

Checking if "/grub/e2fs_stage1_5" exists... yes

Running "embed /grub/e2fs_stage1_5 (hd0)"... 15 sectors are embedded.

succeeded

Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub

.conf"... succeeded

Done.

7.Remove the DVD/CD and reboot it

8.You got the OS clone in different server

Happy backup Day ... ;)

HMC vtmenu exit

2011-10-07T12:27:00.001-07:00

You can exit the cosole of an LPAR ,that has been taken using HMC vtmenu ,by typing
"~." (press dot key after tild without qoutes )

If you had taken HMC vtmenu using SSH ,your ssh connection to HMC will alse loses.

Read /etc/passwd using TFTP venerability - How to resolve in AIX

2011-09-23T06:43:00.000-07:00

1.vi /etc/tftpaccess.ctl

allow:/tftpboot

Save the file

You are done..!!!

Now the clients can only access /tftpboot using TFTP

FTP Banner in AIX

2011-08-29T14:44:00.000-07:00

1.vi /etc/ftpaccess.ctl

herald: /path/to/banner/file ---->Enter your welcome message in this file

motd: on

2.refresh -s inted

You are done

DRBD on RHEL6 with YUM group

2011-08-19T12:29:00.000-07:00

Building DRBD for 2.6.32-71.el6.x86_64 kernel

1.mkdir -p /root/rpmbuild/SOURCES/

2.wget http://oss.linbit.com/drbd/8.3/drbd-8.3.10.tar.gz

3.cp drbd-8.3.10.tar.gz /root/rpmbuild/SOURCES/

4.tar -xvf drbd-8.3.10.tar.gz

5.cd drbd-8.3.10

6../configure --with-km --with-distro=redhat --enable-spec

7.rpmbuild -bb drbd.spec

8.rpmbuild -bb drbd-km.spec

9.cd /root/rpmbuild/RPMS/x86_64

10.mkdir /YUM/DiskReplicator

11.cp drbd-* /YUM/DiskReplicator/

12.cd /YUM/DiskReplicator/

13.createrepo . ---> The "." is important

14.vi /etc/yum.repos.d/rhel6-1.repo

[DiskReplicator]

name=rhel6-1 LoadBalancer repo

baseurl=ftp://192.168.8.2/DiskReplicator

enabled=1

gpgcheck=0

15.cd /tmp/ -->To avoid rpm name clash

16.yum install yum-utils

17.yum-groups-manager -n "Disk Replicator" --id=drbd --description="DRBD, packages from ANSIL" --save=comps.xml --mandatory drbd.x86_64 --mandatory drbd-bash-completion.x86_64 --mandatory drbd-heartbeat.x86_64 --mandatory drbd-km-2.6.32_71.el6.x86_64.x86_64 --mandatory drbd-pacemaker.x86_64 --mandatory drbd-udev.x86_64 --mandatory drbd-utils.x86_64 --mandatory drbd-xen.x86_64

18.createrepo -g comps.xml /YUM/DiskReplicator/

19.yum grouplist "Disk Replicator"

20.yum groupinstall "Disk Replicator"

How to Configure YUM in RHEL6

2011-08-15T13:39:00.000-07:00

Easy way to create your own repo
1. Put your RHEL6 media on DVD drive
2. Create a partition with size 6GB and mount it as /yum
3. Download the below script to /yum
https://sites.google.com/site/ansilh/rhel6_repo.sh
4. Give execute permission to the downloaded script
5. Please open the file and try to understand it. If you found all safe ,then go to next step
6. Execute it...You will get an RHEL repo accessible through ftp

***Please feel free to comment your experience ***

=======================================

Manual Steps (*Different approach from above script .The repo file not look like usual one)

YouTube Link : http://www.youtube.com/watch?v=scBtTTwLze0

1.create a directory to store packages .(Better you create a partition with 5GB size and mount it as /YUM)
mkdir /YUM
2.mount RHEL6 DVD
mount /dev/dvd /mnt
3.go to the mounted directory
cd /mnt/Packages/
4.copy all contents of DVD to /YUM (Note the "." in command )
cp . /YUM/
5.Install createrepo package with dependancies
rpm -ivh deltarpm-**.el6.x86_64.rpm python-deltarpm-**.el6.x86_64.rpm createrepo-**.rpm
6.install FTP server
rpm -ivh vsftpd-2.2.2-6.el6.x86_64.rpm
7.Create repository using
createrepo -g /YUM/HighAvailability/repodata/****-comps-rhel6-HighAvailability.xml –o \
/YUM/HighAvailability/ /YUM/Packages/

createrepo -g /YUM/LoadBalancer/repodata/*****-comps-rhel6-LoadBalancer.xml –o \
/YUM/LoadBalancer/ /YUM/Packages/

createrepo -g /YUM/ResilientStorage/repodata/****-comps-rhel6-ResilientStorage.xml –o \
/YUM/ResilientStorage/ /YUM/Packages/

createrepo -g /YUM/ScalableFileSystem/repodata/****-comps-rhel6-ScalableFileSystem.xml –o \
/YUM/ScalableFileSystem/ /YUM/Packages/

createrepo -g /YUM/Server/repodata/****-comps-rhel6-Server.xml -o /YUM/Server/ /YUM/Packages/
8.Create a repo file in /etc/yum.repos.d/

The Server id contains the main path to the packages.(bold in [Server] baseurl )

vi /etc/yum.repos.d/rhel6-1.repo
[Server]
name=rhel6-1 Server repo
baseurl=ftp://192.168.8.2/Server
ftp://192.168.8.2/Packages
enabled=1
gpgcheck=0
[HighAvailability]
name=rhel6-1 HighAvailability repo
baseurl=ftp://192.168.8.2/HighAvailability
enabled=1
gpgcheck=0
[ScalableFileSystem]
name=rhel6-1 ScalableFileSystem repo
baseurl=ftp://192.168.8.2/ScalableFileSystem
enabled=1
gpgcheck=0
[ResilientStorage]
name=rhel6-1 ResilientStorage repo
baseurl=ftp://192.168.8.2/ResilientStorage
enabled=1
gpgcheck=0
[LoadBalancer]
name=rhel6-1 LoadBalancer repo
baseurl=ftp://192.168.8.2/LoadBalancer
enabled=1
gpgcheck=0
9.Edit ftp server configuration file to make /YUM as anonymous root
vi /etc/vsftpd/vsftpd.conf
anon_root=/YUM
10.Restart ftp service
service vsftpd restart
11.Add a rule in iptables to accept ftp connections in /etc/sysconfig/iptables
vi /etc/sysconfig/iptables
-A INPUT -p tcp -s 192.168.8.0/24 --dport 21 -i eth0 -j ACCEPT

12.Add ip_conntrack_ftp module in /etc/sysconfig/iptables-config which is needs to be added for FTP support[ip_nat_ftp]

vi /etc/sysconfig/iptables-config
IPTABLES_MODULES="ip_conntrack_ftp"
service iptables restart
13.Install the following packages[both clients and server need these packages] to transfer files
You need to install it from DVD
Follow step 2 and 3 and then
rpm -ivh libedit-2.11-4.20080712cvs.1.el6.x86_64 openssh-clients-5.3p1-20.el6.x86_64 ftp-0.17-51.1.el6.x86_64.rpm
14.transfer yum configuraion files to clients

scp /etc/yum.repos.d/rhel6-1.repo *.*.*.*:/etc/yum.repos.d/
15.Check yum configuratoin

yum grouplist will give you these ouputs

Installed Groups:
E-mail server
FTP server
Available Groups:
Additional Development
Afrikaans Support

...

Enjoy RHEL6 Package Management

I've made a video for easy understanding

http://www.youtube.com/watch?v=scBtTTwLze0

MySQL backup script

2011-05-06T20:58:00.000-07:00

This script is made using dialogues
So without dialogue package it will not work

You can start a mysql instance in full customized way

The script is mainly focused on MySQL compiled from source code

The script is using a config file to get necessary information to store values like

path to mysql daemon ,myql dump,mysql safe ,user name and password to start mysql instance

You can download the script from the following link

The advantage is ,You can take the entire database as well as a particular table of a schema using this script.

https://sites.google.com/site/ansilh/mysql-backup-script/mybackup.sh

Screen saver using script / CLI screen saver

2011-04-25T04:32:00.000-07:00

Can you imagine we can make animations using shell script..?

Yes ...Here is one I made screen.sh
A floating box in CLI with my name inside

You can execute the script in any row - column resolution
You can re-size the window if you are accessing system using PuTTy(little bit disturbance if you are resizing from large to small window but it will stabilize after few seconds)

The script is released under GNU public license

Download it from here

https://sites.google.com/site/ansilh/screen.sh

Try and post your comments

Terminal input output logger / Teminal recording

2011-04-20T06:49:00.000-07:00

Logging terminal input and output
------------------------------------------

After a lot of googling I found a tool that can log all input and output of terminal in Unix system

The tool is rootsh and its not only used for logging but also used for root login for normal users using sudo.

So that the command entered by the user will be logged in a file as well as on syslog

We can make rootsh as a shell , So that all the activity by that particular user will be logged

The tool is released under GNU Open source license

You can download the tool from

http://sourceforge.net/projects/rootsh/

I made a modified copy of mine on
https://sites.google.com/site/ansilh/rootsh

The tool will log username,commands and its output in a file -- But no client IP
The command can be started as a shell -But it will take the default shell

I made some modification on this and it's working perfectly on my Linux machine
The link here

https://sites.google.com/site/ansilh/rootsh/rootsh-1.5.3-final.tar.gz

The extra modules I added are here
username.c --->This will challenge the user to enter actual name of user and
purpose of login
https://sites.google.com/site/ansilh/rootsh/username.c

readconf.c ----->This will read configuration file /etc/rootsh.conf
format of the file will be

:
e.g
ansil:/bin/bash

do not add space on end of the line (need some code changes)
https://sites.google.com/site/ansilh/rootsh/readconf.c

Keep an eye on here ..more to come like remote logging with TLS security on a separate file for a separate action

Shell Scripting - part 3

2011-04-20T06:40:00.000-07:00

Fun with Shell scripting
-----------------------------

tput cup command
-----------------
This command is used to set properties of terminal

tput cup 0 0
This command will set courser position to top left corner of terminal

e.g:-
[root@localhost]#vi file6.sh
tput cup 0 0
echo "Ansil"

Save and execute script
The output "Ansil" will be displayed on top left corner of the terminal

e.g:-
Lets make this example more interesting

#!/bin/bash
clear
tput cup 12 35
echo "Ansil"
read

Save the file and execute the script
The out put will display "Ansil" on almost center of the screen,
Surprisingly you will not get your terminal back until you press enter OR you have to kill/terminate the script(we will discuss this later)

The read command is used to read values from terminal(by default standard input)

here the read command read values till an occurrence of ENTER key ,but the value will not be stored

e.g:-
Make this example some more interesting

#!/bin/bash
clear
tput cup 12 35
echo -e "\033[31mAnsil\033[0m"
read

The echo line having some extra characters ..!!

More on echo command
---------------------------
echo -e --->what is -e
:- enable interpretation of backslash escapes
echo -e "\033[31m ------>what is \033[31m
:-Start printing charecter from here in red
:-\033[ is a backslash escape character
:- in 31 3-forground 1-for red
echo -e"\033[31mAnsil" ----->It will print Ansil in red color ,Then why

echo -e "\033[31mAnsil\033[0m"
:-The \033[0m will make the terminal to normal state
:- i,e from \033[0m the characters will be printed normally
:- If you did't typed this on your script ,after execution also the character printed will be in red

Colors and attributes using echo
---------------------------------
Foreground
-----------
\033[30m - BLACK
\033[31m - RED
\033[32m - GREEN
\033[33m - YELLOW
\033[34m - BLUE
\033[35m - MAGENTA
\033[36m - CYAN
\033[37m - WHITE

background
-----------
\033[41m - BLACK
\033[41m - RED
\033[42m - GREEN
\033[43m - YELLOW
\033[44m - BLUE
\033[45m - MAGENTA
\033[46m - CYAN
\033[47m - WHITE
More
------------
\033[4m - UNDERLINE
\033[24m - STOP UNDERLINE
\033[5m - BLINK
\033[7m - INVERT FORGROUND AND BACKGROUNG
\033[1m - BOLD

e.g:-
print Ansil in bold with color green and background red in bottom right corner of terminal

#!/bin/bash
clear
tput cup 23 00
echo -e "\033[41m\033[32m\033[1mAnsil\033[0m"

we will split the echo line
echo -e ---->You know this
\033[41m --->Make background red
\033[32m --->Make foreground green
\033[1m ---->Make foreground bold
Ansil -------->The word to be printed
\033[0m ---->Make default printing colors

Shell Scripting - part 2

2011-04-14T09:04:00.000-07:00

Command Line Arguments
-------------------------------
Here we will discuss some more about command line arguments( values passed along with script like flags in commands)
and how they can be accessed in script

1)Name of the program
The name of the program is stored in variable '0'
So the value can be accessed as ${0}
e.g:- vi file1.sh
---------------------------------------------------------------------
#!/bin/bash
echo ${0}
---------------------------------------------------------------------
The output of the program will be like
./file.sh
2)All arguments passed along with script
All values passed to the script(command line arguments) will be stored in variable '@'
So the value can be accessed as ${@}
e.g:- vi file2.sh
---------------------------------------------------------------------
#!/bin/bash
echo ${0}
---------------------------------------------------------------------
If you execute the script with out any arguments,there will be no output
So to verify the script ,execute with some arguments like

root@localhost]#./file2.sh a b c

The output of the program will be like
a b c
3) Number of command line arguments
Number of arguments passed to the program will be stored in variable '#'
So the value can be accessed as ${#}
e.g:- vi file3.sh
---------------------------------------------------------------------
#!/bin/bash
echo ${#}
---------------------------------------------------------------------
If you execute the script with out any arguments,there will be no output
So to verify the script ,execute with some arguments like

root@localhost]#./file2.sh a b c

The output of the program will be like
3
4) PID of shell executing the script
variable that will store PID of shell executing will be on $ or _
So the value can be accessed as ${$} or ${_}
The PID of shell executing will depend on how you execute the program

The script can be executed in two way
a) on current shell
e.g:-
root@localhost]#. file3.sh
5388
b) on a separate shell
root@localhost]#./file3.sh
5344

On e.g (a) the PID displayed will be the PID of current shell
You can verify it using 'ps' command
root@localhost]#ps
PID TTY TIME CMD
5388 pts/0 00:00:00 bash
5464 pts/0 00:00:00 ps
5) Command line arguments
So finally we are on the actual arguments
The values that are passed to the script for processing

The values start with 1

e.g:- vi file4.sh
---------------------------------------------------------------------
#!/bin/bash
echo ${1}
echo ${2}
echo ${3}
---------------------------------------------------------------------
Execute the program with some arguments

root@localhost]#./file4.sh a b c
a b c

Thats for Now we will continue with more on coming posts

Shell Scripting - part 1

2011-04-12T22:50:00.000-07:00

The echo command
------------------
The easiest command in shell script
used for display text

e.g:-
echo "Hello world"

Variables in script
--------------------
variables are used for storing values
It may be pe-defined or the value will be assigned on the fly (On execution of script)

The variable manipulation in scripting
VAR=value

${VAR} for retrieving the value assigned to VAR

a)Pre-defined
e.g:-
VAR=10

b)On the fly
e.g:-
VAR=`expr ${i} + 1`

In example (a) the variabe VAR is assigned a value of 10
The value of VAR will be 10 through out the execution life of script(unless that variable is not alterd by another assignment)

In example (b)the variable VAR assigned a value that depend on the value of 'i' (another variable)
We will split the example to understand two command

expr ${i} + 1
and
VAR=

i) evaluat the expression ${i} + 1
ii) Assign the result to VAR

Using the echo command we can display the value assigned to a variable

e.g:- Write a script that having a variable VAR and assign a value 10 to it
Display the value of VAR in terminal
---------------------------------------------------------------------

#!/bin/bash
#
# Dispaly the value of a variable

VAR=10
echo ${VAR}
---------------------------------------------------------------------

Basic Shell Scripting-step by step

2011-04-11T04:09:00.000-07:00

Basic Shell Scripting-step by step

If you are not a beginner go to summery page

What is a script?

Group of commands in one file for a specific or a number of actions to achieve a result

1. Select your editor

You must have a text editor to write shell programs

Select an editor in which you are comfortable

Most common editors are Vi and Emacs

2. Structure of a script

a)First line

“#!/path/to/shell”

b) Comments

Purpose of script, author, created date, modifications, bug fix notes etc

c) Body of script

Commands to be executed

This line will decide which shell to be used for executing the commands in script

If the line is not present, system will use default shell

E.g.:- 1

------------------------------------

#!/bin/bash

echo “Hello world”

E.g. Explained

------------------------------------

The above script will execute

echo “Hello world”

in bash shell

3. Execute a script

If you logged in as root, the home directory will be /root/

We will create a file in /root using vi editor

[root@localhost]# vi hello.sh

The files have an extension of “.sh” so that others can easily identify the file as script

-------------------------------------------------------------------------------------------------------------------------------

Note: - The “file” is a command used to identify, what kind of file it is.

Suppose we have file with name “abcd” How can I say it is a shell script or an executable binary or else another type..?

Here we will use “file” command

Eg:-

[root@localhost]# file abcd

ASCII Long line text

-------------------------------------------------------------------------------------------------------------------------------

We are in vi now (I’m going ahead with the assumption that you have basic knowledge of VI editor)

Type the text mentioned in e.g.: 1

#!/bin/bash

echo “Hello world”

Save the file

Now we have created a” bash shell script” named as “file.sh”

To execute a file that need execute permission

So give execute permission to “file.sh” using chmod command

[root@localhost]# chmod 755 file.sh

Now we can execute the script as

[root@localhost#]./file.sh

Hello world

[root@localhost#]

Summary

1. Create a script that will print “Hello world”

[root@localhost#]

[root@localhost#]vi file.sh

#!/bin/bash

echo “Hello world”

[root@localhost#]chmod 755 file.sh

[root@localhost#]./file.sh

Hello world

[root@localhost#]

Tomcat Application Vertical Cluster

2010-12-19T05:30:00.000-08:00

Application Cluster

Packages Needed

1. 1. jdk-1_5_0_19-linux-i586.bin

(http://www.oracle.com/technetwork/java/javase/downloads/index-jdk5-jsp-142662.html)

2. 2.. apache-tomcat-5.5.27.tar.gz

(http://archive.apache.org/dist/tomcat/tomcat-5/v5.5.27/bin/apache-tomcat-5.5.27.tar.gz)

3. 3. httpd-2.2.3-11.el5_1.3 (RHEL DVD)

4. 4. httpd-devel-2.2.3-11.el5_1.3.i386.rpm (RHEL DVD)

5. 5.tomcat-connectors-1.2.28-src.tar.gz (http://archive.apache.org/dist/tomcat/tomcat-connectors/jk/source/jk-1.2.28/tomcat-connectors-1.2.28-src.tar.gz)

1. Install JDK

1.Copy jdk file to /tmp

2.[root@localhost]# chmod 755 jdk-1_5_0_19-linux-i586.bin

3.[root@localhost]#./ jdk-1_5_0_19-linux-i586.bin

Accept license agreement by typing “yes”

A new directory will be created on /tmp directory with name jdk1.5.0_19

Copy the jdk1.5.0_19 directory to /usr/local

4. .[root@localhost]# cp -rv jdk1.5.0_19 /usr/local

Create a symlink of /usr/local/jdk1.5.0_19 to /usr/loca/java

5.[root@localhost]#ln –s /usr/local/jdk1.5.0_19 /usr/loca/java

Open /etc/profile add these lines

5.[root@localhost]#vi /etc/profile

PATH=/usr/local/java/bin:$PATH

JAVA_HOME=/usr/local/java

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME

Log out the session and login again

Type echo $JAVA_HOME ,will show /usr/local/java

2. Install Tomcat

Create a system user with name “tomcat”

1.[root@localhost]#useradd –r tomcat

Create four directories /app1,/app2 ,/app3 and /applog

2.[root@localhost]#mkdir /app1 /app2 /app3 /applog

Change ownership of those directories to “tomcat”

3.[root@localhost]#chown tomcat:tomcat /app1 /app2 /app3 /applog -R

We will install three instances of tomcat for load balanced cluster, for that we need three directories /app1,/app2 and /app3

All log files related to application goes to /applog directory

Unzip and untar the package

4.[root@localhost]#tar -xzvf apache-tomcat-5.5.27.tar.gz

apache-tomcat-5.5.27 directory will be created after executing the command

Copy all the contents from apache-tomcat-5.5.27 to application directories

5.[root@localhost]#cd apache-tomcat-5.5.27

6.[root@localhost]#cp –rv * /app1

7.[root@localhost]#cp –rv * /app2

8.[root@localhost]#cp –rv * /app3

*.Change tomcat default log location from /app1/logs/ to /applog/app1

Create three directories inside /applog

9.[root@localhost]#cd /applog

10.[root@localhost]#mkdir app1 app2 app3

11.[root@localhost]#chown –R tomcat.tomcat /applog

12.Edit /app1/bin/catalina.sh

Change "$CATALINABASE"/logs/catalina.out path to /applog/app1/catalina.out

Do the same for remaining two instances

*.Disable all logging for application to increase tomcat performence

Rename /app1/conf/logging.properties to /app1/conf/logging.properties.org

[root@localhost]#mv /app1/conf/logging.properties /app1/conf/logging.properties.org

Do the same for remaining two instances

3. Install Apache

Install packages for apache

[root@localhost]#rpm -ivh apr-devel-1.2.7-11.i386.rpm

[root@localhost]#rpm -ivh apr-util-devel-1.2.7-7.el5.i386.rpm

[root@localhost]#rpm -ivh httpd-2.2.3-11.el5_1.3.i386.rpm

[root@localhost]#rpm -ivh httpd-devel-2.2.3-11.el5_1.3.i386.rpm

4. Install Tomcat JK connector

we have to give apache extension binary file path before compiling JK connector

[root@localhost]#which apxs

The command will give path of apxs

/usr/sbin/apxs

[root@localhost]#tar -zxvf tomcat-connectors-1.2.28-src.tar.gz

[root@localhost]#cd tomcat-connectors-1.2.28-src/native

[root@localhost]#./configure --with-apxs=/usr/sbin/apxs

[root@localhost]#make

After make the mod_jk.so will be created under apache-2.0 directory

Copy mod_jk.so module to /etc/httpd/modules directory

[root@localhost]#cd apache-2.0/

[root@localhost]#cp mod_jk.so /etc/httpd/modules/

4. Install Application and Configure Cluster

Copy application directories to /app1/webapps , /app2/webapps and /app3/webapps

For example ADMINAPP and USERAPP

[root@localhost]# cp –rv ADMINAPP USERAPP /app1/webapps

[root@localhost]#cp –rv ADMINLIC LICUSER /app2/webapps

[root@localhost]#cp –rv ADMINLIC LICUSER /app3/webapps

Copy necessary jar files for connection pooling to tomcat instances

[root@localhost]#cp commons-pool-1.4.jar mysql-connector-java-3.0.11-stable-bin.jar commons-dbcp-1.2.2.jar /app2/common/lib

Edit configuration files for tomcat

[root@localhost]#vi /app1/conf/server.xml

1. Comment HTTP connector tag

Do the same for remaining two instances

2. Change tomcat shutdown port

vi /app1/conf/server.xml

vi /app2/conf/server.xml

vi /app3/conf/server.xml

3.Add JVM route to Catalina engine

vi /app1/conf/server.xml

vi /app2/conf/server.xml

vi /app3/conf/server.xml

4.Add Context and Resources to tomcat

vi /app1/conf/server.xml

Add the lines indicated bold in Host tag

Do the same for remaining two instances

5. Enable Session replication and Clustering

vi /app1/conf/server.xml

Uncomment Cluster tag

<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"

managerClassName="org.apache.catalina.cluster.session.DeltaManager"

expireSessionsOnShutdown="false"

useDirtyFlag="true"

notifyListenersOnReplication="true">

……….striped…….

</Cluster>

Do the same for remaining two instances

6.Change Multicast port for Cluster listener

Edit Receiver tag in Cluster and change tcpListenPort

vi /app1/conf/server.xml

<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4001"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

vi /app2/conf/server.xml

<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4002"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

vi /app3/conf/server.xml

<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="auto"

tcpListenPort="4003"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

7. Change AJP 13 Protocol port for tomcat

vi /app1/conf/server.xml

<Connector port="8009"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

vi /app2/conf/server.xml

<Connector port="8109"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

vi /app3/conf/server.xml

<Connector port="8209"

enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

7. Add cluster parameter to Application’s web.xml

/app1/webapps/USERAPP/WEB-INF/web.xml

/app2/webapps/USERAPP/WEB-INF/web.xml

/app3/webapps/USERAPP/WEB-INF/web.xml

/app1/webapps/ADMINAPP/WEB-INF/web.xml

/app2/webapps/ADMINAPP/WEB-INF/web.xml

/app3/webapps/ADMINAPP/WEB-INF/web.xml

add at the end of the file.

Install mysql jdbc driver

cp /app1/common/lib/mysql-connector-java-3.0.11-stable-bin.jar /usr/local/java/jre/lib/ext/

Create a worker file for tomcat instances

vi workers.properties

workers.tomcat_home=/app1

workers.java_home=/usr/local/java

ps=/

worker.list=loadbalancer,jkstatus

worker.app1.port=8009

worker.app1.host=127.0.0.1

worker.app1.type=ajp13

worker.app1.lbfactor=1

worker.app1.socket_timeout=1800

worker.app1.socket_keepalive=1

worker.app2.port=8109

worker.app2.host=127.0.0.1

worker.app2.type=ajp13

worker.app2.lbfactor=1

worker.app2.socket_timeout=1800

worker.app2.socket_keepalive=1

worker.app3.port=8209

worker.app3.host=127.0.0.1

worker.app3.type=ajp13

worker.app3.lbfactor=1

worker.app3.socket_timeout=1800

worker.app3.socket_keepalive=1

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=app1,app2,app3

worker.jkstatus.type=status

Copy the worker file to /etc/httpd/conf.d/

cp workers.properties /etc/httpd/conf.d/

Configure Apache

vi /etc/httpd/conf/httpd.conf

In Main Server configuration add these lines

LoadModule jk_module modules/mod_jk.so

JkWorkersFile "/etc/httpd/conf.d/workers.properties"

JkLogFile "/etc/httpd/logs/mod_jk.log"

JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

JkRequestLogFormat "%w %V %T"

JkLogLevel info

JkShmFile logs/jk.shm

Create a virtual Host listening on all interfaces in 80 port

ServerAdmin root@lms.icaps.com

#DocumentRoot /var/www/html

ServerName lms.icaps.com

ErrorLog logs/lms.icaps.com-error_log

CustomLog logs/lms.icaps.com-access_log common

JkMount jkstatus

Order deny,allow

Deny from all

Allow from 127.0.0.1

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

Order deny,allow

Deny from all

JkMount / loadbalancer

JkMount /* loadbalancer

JkMount /ICMSUSER loadbalancer

JkMount /USERAPP/* loadbalancer

JkMount /ADMINAPP loadbalancer

JkMount /ADMINAPP/* loadbalancer

Create a Script for starting Application Cluster with name “tomcat”

#!/bin/sh

############################################

# Startup script for Tomcat Servlet Engine #

# Author : Ansil Hameed #

# Last Modified : 28/10/2010 #

############################################

# User under which tomcat will run

TOMCAT_USER=tomcat

APP1_DIR="/app1"

APP2_DIR="/app2"

APP3_DIR="/app3"

RETVAL=0

. /etc/init.d/functions

# start, debug, stop, status and flush-log functions

flush(){

LOG_DATE=`date +%d-%m-%y`

cp /applog/app1/catalina.out /applog/app1/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

>/applog/app1/catalina.out

cp /applog/app2/catalina.out /applog/app2/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

>/applog/app2/catalina.out

cp /applog/app3/catalina.out /applog/app3/catalina.out-${LOG_DATE} 2>>/applog/tomcat-flush.log

if [ $? -ne 0 ]

then

echo "Log flush for App1 failed .Check Log file for details"

>/applog/app3/catalina.out

}

start() {

SHUTDOWN_PORT=`netstat -vatn|grep LISTEN|grep 8005|wc -l`

if [ $SHUTDOWN_PORT -ne 0 ]; then

echo -e "\033[31mAPP1 already started\033[0m"

else

echo -e "\033[31m....... \033[32mStarting APPLICATON CLUSTER\033[31m ......\033[0m"

echo -e "\033[32mStarting APP1...\033[0m"

echo ""

chown -R $TOMCAT_USER:$TOMCAT_USER ${APP1_DIR}/*

How to Configure Squid with Mysql DB authentication

2010-02-23T02:30:00.001-08:00

Download squid source from http://www.squid-cache.org/Versions/v3/3.0/squid-3.0.STABLE24.tar.gz
Copy the downloaded file to /tmp
Tar –zxvf squid-3.0.STABLE24.tar.gz
Cd squid-3.0.STABLE24
./configure --enable-basic-auth-helpers=DB
make
make install
Download Current Mysql5.1 Source RPM from http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-5.1.44-1.glibc23.src.rpm/from/ftp://mirror.anl.gov/pub/mysql/
Copy MySQL-5.1.44-1.glibc23.src.rpm /tmp
Rpm –ivh MySQL-5.1.44-1.glibc23.src.rpm
Cd /usr/src/redhat/SPEC
Rpmbuid –bb mysql-5.1.44.spec
After compilation the RPM files will be in /usr/src/redhat/`uname –m`/RPMS
Cd /usr/src/redhat/`uname –m`/RPMS
Rpm –ivh MySQL-server-5.1.44-1.glibc23.ia64.rpm
The service will start automatically after installation
Here we are using Default Mysql database engine MyISAM (Non transactional Engine)
In a large application Database we will use Engine INNODB (Transactional engine).It has many configurable options to get mysql to work efficiently with maximum performance. The default database directory will be /var/lib/mysql.
Rpm –ivh MySQL-client-5.1.44-1.glibc23.ia64.rpm
The mysql service can be stop/start by running #service mysql stop/start
Login to Mysql as root
#mysql –u root
You must have to set a password for mysql root user
Mysql>set password=PASSWORD('your password');
Mysql>/q
Test your authentication
#mysql -u root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
#mysql –u root –p
Enter your password, then you will get mysql prompt.
The we have to create one Database and table for squid as follows

mysql>create database squid;

mysql>grant select on squid.* to squiduser@localhost identified by 'squid';

mysql>use squid;

mysql> CREATE TABLE `passwd` (

          `user` varchar(32) NOT NULL default '',

          `password` varchar(35) NOT NULL default '',

          `enabled` tinyint(1) NOT NULL default '1',

          `fullname` varchar(60) default NULL,

          `comment` varchar(60) default NULL,

          PRIMARY KEY  (`user`)

);

mysql>; insert into passwd values('testuser','test',1,'Test User','for testing purpose');
mysql>\q

#/usr/local/squid/libexec/squid_db_auth –-user  squiduser -–password squid -–plaintext –-persist

testuser test

OK

Enter Test user name and password ,separated with space and press enter ,if it shown OK ,your authentication will work .If you got "ERR unknown login", you missed something
Now your database is ready and we are going to configure squid
Go to /usr/local/squid/etc/
Cp squid.conf squid.conf.org
Search for auth_param and add these lines
auth_param basic program /usr/local/squid/libexec/squid_db_auth --user squiduser --password squid --plaintext --persist
auth_param basic children 5
auth_param basic realm Web-Proxy
auth_param basic credentialsttl 1 minute
auth_param basic casesensitive off

Now you have to create an ACL and Rule to authenticate
Here we will use two separate user ACL groups

A) Master users with all allowed sites as MSTUSR

B) Normal users with minimum access NRMUSR

Two ACL for IP based authentication
a) Allowed IPs as ALLOWIP
b) Denied IPs DENYIP
Two ACL for Allowed and Denied sites
a)Allowed sites as ALLOWSITE
b)Denied sites as BLOCKSITES
The ACLs in squid.conf as follows

acl localnet src 10.0.0.0/8 #Your

acl localnet src 172.16.0.0/12 #Possible

acl localnet src 192.168.0.0/16 #Internal Network

acl ALLOWIP src "/usr/ansil/proxy/policy/allowip"

acl DENYIP src "/usr/ansil/proxy/policy/denyip"

acl ACLAUTH proxy_auth REQUIRED

acl MSTUSR proxy_auth "/usr/ansil/proxy/policy/mstusr"

acl ALLOWSITE url_regex "/usr/ansil/proxy/policy/allowsite"

acl NRMUSR proxy_auth "/usr/ansil/proxy/policy/normaluser"

acl BLOCKSITES url_regex -i "/usr/ansil/proxy/policy/blocksites"

Now we have to assign rules for according to our ACLs

http_access deny BLOCKSITES

http_access allow ACLAUTH MSTUSR

http_access allow ALLOWIP

http_access deny DENYIP

http_access allow ACLAUTH ALLOWSITE ICMSUSER

Now save the configuration file.
Restart squid service ,for that you need to create one script for squid
Create a script as follows and put it on /etc/ini.d/ with name as squid
#chmod 755 /etc/init.d/squid
chkconfig --add squid

#!/bin/bash

PATH=/usr/bin:/sbin:/bin:/usr/sbin

export PATH

. /etc/rc.d/init.d/functions

. /etc/sysconfig/network

if [ -f /etc/sysconfig/squid ]; then

. /etc/sysconfig/squid

SQUID_OPTS=${SQUID_OPTS:-"-D"}

SQUID_PIDFILE_TIMEOUT=${SQUID_PIDFILE_TIMEOUT:-20}

SQUID_SHUTDOWN_TIMEOUT=${SQUID_SHUTDOWN_TIMEOUT:-100}

[ -f /usr/sbin/squid ] && SQUID=squid

if [ "$1" == "status" ]; then

[ -z "$SQUID" ] && exit 4

else

[ -z "$SQUID" ] && exit 1

prog="$SQUID"

CACHE_SWAP=`sed -e 's/#.*//g' /usr/local/squid/etc/squid.conf | \

grep cache_dir | awk '{ print $3 }'`

[ -z "$CACHE_SWAP" ] && CACHE_SWAP=/var/spool/squid

RETVAL=0

probe() {

# Check that networking is up.

[ ${NETWORKING} = "no" ] && exit 1

# check if the squid conf file is present

[ -f /usr/local/squid/etc/squid.conf ] || exit 6

}

start() {

probe

$SQUID -k parse

RETVAL=$?

if [ $RETVAL -ne 0 ]; then

echo -n $"Starting $prog: "

echo_failure

echo

return 1

for adir in $CACHE_SWAP; do

if [ ! -d $adir/00 ]; then

echo -n "init_cache_dir $adir... "

$SQUID -z -F -D >> /var/log/squid/squid.out 2>&1

done

echo -n $"Starting $prog: "

$SQUID $SQUID_OPTS >> /var/log/squid/squid.out 2>&1

RETVAL=$?

if [ $RETVAL -eq 0 ]; then

timeout=0;

while : ; do

[ ! -f /var/run/squid.pid ] || break

if [ $timeout -ge $SQUID_PIDFILE_TIMEOUT ]; then

RETVAL=1

break

sleep 1 && echo -n "."

timeout=$((timeout+1))

done

[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$SQUID

[ $RETVAL -eq 0 ] && echo_success

[ $RETVAL -ne 0 ] && echo_failure

echo

return $RETVAL

}

stop() {

echo -n $"Stopping $prog: "

$SQUID -k check >> /var/log/squid/squid.out 2>&1

RETVAL=$?

if [ $RETVAL -eq 0 ] ; then

$SQUID -k shutdown &

rm -f /var/lock/subsys/$SQUID

timeout=0

while : ; do

[ -f /var/run/squid.pid ] || break

if [ $timeout -ge $SQUID_SHUTDOWN_TIMEOUT ]; then

echo

return 1

sleep 2 && echo -n "."

timeout=$((timeout+2))

done

echo_success

echo

else

echo_failure

echo

return $RETVAL

}

reload() {

$SQUID $SQUID_OPTS -k reconfigure

}

restart() {

stop

start

}

condrestart() {

[ -e /var/lock/subsys/squid ] && restart || :

}

rhstatus() {

status $SQUID && $SQUID -k check

}

case "$1" in

start)

start

;;

stop)

stop

;;

reload)

reload

;;

restart)

restart

;;

condrestart)

condrestart

;;

status)

rhstatus

;;

probe)

probe

return 0

;;

echo $"Usage: $0 {start|stop|status|reload|restart|condrestart|probe}"

exit 2

esac

exit $

2010-02-01T05:36:00.001-08:00

Apache Cluster Using Redhat Cluster Suit

1	DNS	10.10.40.211	Dns.ansil.com
2	YUM/NIM	10.10.40.211
3	Xen Dom0	10.10.40.60	Xen-base.ansil.com
4	Xen DomU1(node1)	10.10.40.61	Xen1.ansi.com
5	Xen DomU2(node2)	10.10.40.62	Xen2.ansil.com
6	Cluster Service IP	10.10.40.63
7	Cluster management UI	10.10.40.60

10.10.40.211

DNS Configuration (BIND9)

Forward Zone Definition

www A 10.10.40.63

xen1 A 10.10.40.61

xen2 A 10.10.40.62

dns A 10.10.40.211

Reverse Zone Definition

63 PTR www.ansil.com

61 PTR xen1.ansil.com

62 PTR xen2.ansil.com

211 PTR dns.ansil.com

YUM

You need one 4 GB Partition mounted on /install

Copy All the contents from RHEL 5 DVD to /install directory

Take one copy of the following files to /tmp from /install/Server, Cluster, ClusterStorage and VT

comp-rhel5-server-core.xml

comp-rhel5-cluster.xml

comp-rhel5-cluster-st.xml

comp-rhel5-vt.xml

Remove repodata directory from Server, Cluster, ClusterStorage and VT in /install directory

Install creteaterepo .x.x.x.rpm first

#rpm –ivh creteaterepo .x.x.x.rpm

#cretaerepo –g /tmp/comp-rhel5-server-core.xml /install/Server

#cretaerepo –g /tmp/comp-rhel5-cluster.xml /install/ Cluster

#cretaerepo –g /tmp/comp-rhel5-cluster-st.xml /install/ ClusterStorage

#cretaerepo –g /tmp/comp-rhel5-vt.xml /install/ VT

Now your repository is ready

Create your client repodata files in /etc/yum.repos.d/

Server.repo, Cluster.repo, ClusterStorage.repo, VT.repo

========================================================

[Server] # Name of Repo
Name=Ansil’s repo # Information of repo
Baseurl=ftp://10.10.40.211/Server/ # Path to Packages
enabled=1 # Enable repository access

========================================================

The above contents will be in Server.repo

Likewise you have to create remaining repodata files

NIM(Network Installation Manager)

To install Linux through network you need one of the services like FTP/HTTP/NFS.

Here we are installing through FTP, for that we have to install vsftpd.x.x.rpm

#rpm –ivh vsftpd.x.x.rpm

Open /etc/vsftpd/vsftpd.conf

#vi /etc/vsftpd/vsftpd.conf

Add anon_root=/install to the end of the file

#service vsftpd start

#chkconfig –level 35 vsftpd on

Now our YUM and NIM are ready

Install RHEL 5 on a High end server with virtualization. That system will be your Dom0.

After installation edit your /etc/resolve.conf to point your name server

And also you need to ensure /etc/hosts entry.

You must have one partition with 25 GB space where you have to create your DomUs

Xen DomU1 & Xen DomU2.

Create your DomUs follow these steps

# virt-install

It will ask you

1.Name of the virtual machine(will referred by xen manager)

2.Amount of RAM in MB(eg:-265)

3.Disk path for installation(must specify with a file name like /xen-path/node1)

4.Enable graphical support (no)

5.Install location (ftp://10.10.40.60)

After providing the information the xen will start gust Os installation. Once it started installation it will be same as the normal redhat installation. Select with default options (most cases)

If all information you provided are valid you will get a window like this and Linux installer anaconda will be loaded

Select your Language

Select IPv4 Support and Manual configuration (You can’t use DHCP in Production environment)

Provide IP address, Subnet mask ,Default gateway, and Name server

The next section will load files needed for running anaconda from NIM server

Anaconda starts installation in text mode

Welcome message

Enter Installation number

Ask your permission to erase data before starting installation.

Select installation media

Once again it will ask confirmation before erasing all data from selected drives

You need to modify your partition layout for a production system,but here we will go with default layout.So select No

Once agian you can modify the network informations(It will show the values given at the time of creation)

Go with defaul values.

Enter a host name as pre-planned

Select Time zone.

Provide root password

Starting installation

Go with default packages

The complete information of installation will be in /root/install.log

Formating file systems

Starting package installation

You can follow the above steps for creating second DomU exept IP address and Host name

Now we will install Redhat Cluster Suite on Base Xen system,be sure your gust Operatting systems are down

#yum install “Clustering”

After executing this command you will be asked for a confirmation,type y.

It will install Redhat cluster suite in base system.

After installation start start ricci service

# service ricci start

Create one file called cluster.conf in /etc/cluster/ with the following contents

Now our base system is ready for Xen clustering.

# luci_admin init

Here you can set password for admin user here.

Start Luci service for management interface

#service luci start

You can now access cluster management in https://10.10.40.60:8084

Go to add an existing Cluster

Here you can give one of the node in cluster, and it will be xen-base.Now our base cluster configuration is ready.

Start two services cman & rmanager

# service cman start

# service rgmanager start

Now You can see your base cluster in Luci window in Cluster List

After starting these services we can setup our xen guest Operating system.

Start guest node1.

#xm create node1

Now node1 will start and you can access the console using

# xm console node1

Once the node is up transfer the repo files from Base system’s /etc/yum.rerpos.d/

#scp /et/yum.repos.d/*.repo xen1.ansil.com:/etc/yum.repos.d/

Accept the ssh key and type guest’s password the files will be transferred to xen1.ansil.com

Start the second node and do all the same done for node1.

Now we can install cluster suit on both systems using YUM.

# yum install “Clustering”

Also install apache on both nodes

#yum install http*

Now Cluster suit and web service are ready.

Now we have to create IP based virtual host on both nodes with cluster ip.

The configuration in /etc/httpd/httpd.conf will be like this

ServerAdmin root@xen1.ansil.com

ServerName www.ansil.com

NameVirtualHost 10.10.40.63:80

ServerAdmin root@xen1.ansil.com

DocumentRoot /var/www/html

ServerName xen1.ansil.com

ErrorLog logs/xen1.ansil.com-error_log

CustomLog logs/xen1.ansil.com-access_log common

Transfer this file to node2 also and change xen1 field to xen2

Now our Cluster service is ready.

Here the Game starts……………………

Now we have to start ricci service on both nodes

# Service ricci start

Go to Luci Cluster management -Cluster Create a new Cluster.

Cluster name – webclust

Node or host name – xen1.ansil.com

Node or host name – xen2.ansil.com

Also enter password for both nodes and click submit.

Now it will go through 4 stages

Download packages—rebooting—creating cluster—joining cluster

After completion of these stages you must have to enable cman and rgmanager in auto start

#chkconfig –level 35 cman on

#chkconfig –level 35 rgmanager on

And ensure ricci service is started

# service ricci status

Now you can see your cluster in Cluster list on Luci Web UI

Hurray…………………Our Operating system cluster is up………….

Now we have to add our web service-Apache in Cluster…..

Here the second part of Game starts….

Add a cluster ip as a resource to our cluster

Go to Cluster ->Configure -> Select Webclust->resources->Add a resource->Select a resource type ->IP address

Enter the cluster ip here and be sure there will be tick mark on “Monitor this link”

Now we have to create our cluster service

Go to Cluster àConfigure à Select WebclustàServicesàAdd a Service

Service name ---webhttp

Click “Add a service to this resource”

Select our pre defined resource from “Existing global resource “

And click on submit …now it will configure and add cluster service to our Cluster…

Here the third part of Game ….Fencing….

Go to Cluster ->Configure -> Select Webclust->Shared Fence device->Add a fence device->Select virtual machine fencing

Name-- xvm

Click on “Add this shared fence device”

Our fencing mechanism is ready.

Now we have to add this fencing mechanism to our cluster nodes

Cluster Configure Select Webclust->Nodes->xen1.ansil.com->Main fencing methodà

Add a fence to this level

Select xvm(Virtual machine fencing) from drop down list

Click on Update main fence properties.

Do the same for xen2.ansil.com

…………………………………………………………….………………………………………………………………………………..

Now we have to test our cluster Setup….so check your cluster status.

Create one index.html in /var/www/html on both nodes

#clustat

Here you can see the cluster is in Quorate state

Members in cluster

Status and resource manager

Service name - Owner of service - and state.

Now open your browser and type http://10.10.40.63

Yes My Website is running on cluster IP……!!!!!!!

Now we have to test our failover

So check in which node currently active

Ok…xen2.ansil.com is active..

First we will stop web service on xen2.ansil.com…

I’ve stopped apache on xen2 and cluster state shows “recoverable”…!!!!

And owner changed to xen1.ansil.com…yes our service shifted to xen1.ansil.com….

But after a moment the service came back to xen2.ansil.com

Cluster Suit will restore the service if it is recoverable….!!!!!

Now we are going to shutdown xen2.ansil.com….here we will see actual failover….

The above picture will give the full picture to you…analyze it and note down the changes…and check your website..

Yes xen2.ansil.com is off line and what about our website…!!!? Let’s check…

Check your Cluster ip

# ip addr list

Yes Still Working… “ON xen1.ansil.com”

Thanks

Ansil Hameed

ansil4linux.blogspot.com

Note:- # dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4k count=1

2009-11-10T05:33:00.000-08:00

CHROOT-BIND

Before starting
This tutorial is written "only for a beginner,by a beginner".NOT FOR PRODUCTION

Why we are using CHROOT-BIND

The idea behind running BIND in a chroot jail is to limit the amount of access any malicious individual could gain by hacking BIND.
It is for the same reason that we run BIND as a non-root user.

CHROOT-BIND configuration
========================================
/var/named/chroot/---will be the root ("/") directory

ie;/etc/named.conf will be /var/named/chroot/etc/named.conf
/var/named/ will be /var/named/chroot/var/named/

KEEP THIS IN MIND EVERY TIME....
we will not refer original location ie; /var/named/chroot/etc/named.conf
Will refer as /etc/named.conf

=========================================
/etc/named.conf
=========================================
options
{
directory "/var/named";
};

controls {
inet 127.0.0.1 allow { localhost;127.0.0.1;} keys { rndckey; };
};

acl "safe-subnet" { 10.10.40.0/24; };
view "internal" {

match-clients { localnets; localhost; safe-subnet; };
match-destinations { localnets; localhost; safe-subnet; };
recursion yes;

zone "." IN {
type hint;
file "named.ca";
};

zone "localhost" IN {
type master;
file "localhost.zone";
allow-update { none; };
};

zone "0.0.127.in-addr.arpa" IN {
type master;
file "named.local";
allow-update { none; };
};
include "/var/named/zones/internal/internal_zones.conf";
};

include "/etc/rndc.key";
============================================
1.The first line "options" contains only one directive pointing to
where to store default zone related files

2.The "controls" section is used to administer named deamon using "rndc"

3.The "acl" is used to define control list (Just like in squid proxy)

4.The "view" section is used to define internal and external connection
We have only internal view so that internal netwok can query the name server

5.The three zone definitions are needed for a production system
a)The first zone contains all root domains (13 Servers)
b)Forward zone file for local host
c)Revers file for local host

6.The first include directive is pointing to our internal zone(internal view).

7.The last include directive is used for administer named deamon using rndc
The file contains an algorithm and an md5-key for encrypting data while using rndc
from remote server

===============================================
Our Zone file definitions
===============================================
/var/named/zones/internal/internal_zones.conf
===============================================
zone "40.10.10.in-addr.arpa" IN {
type master;
file "/var/named/zones/internal/10-10-40.zone";
allow-update { none; };
};

zone "ansil.com" IN {
type master;
file "/var/named/zones/internal/ansil.com.zone";
allow-update { none; };
};
==============================================
/var/named/zones/internal/10-10-40.zone
==============================================
$TTL 3D
@ IN SOA ns1.ansil.com. root.ansil.com. (
200911101 ; serial number
8H ; refresh, seconds
2H ; retry, seconds
4W ; expire, seconds
1D ) ; minimum, seconds

NS www ; Nameserver Address

211 PTR www.ansil.com.
=============================================
/var/named/zones/internal/ansil.com.zone
=============================================
$TTL 3D
@ IN SOA ns1.ansil.com. root.ansil.com. (
200911101 ; serial#
3600 ; refresh, seconds
3600 ; retry, seconds
3600 ; expire, seconds
3600 ) ; minimum, seconds

NS ns1.ansil.com ; Inet Address of nameserver
ansil.com. MX 10 mail ; Primary Mail Exchanger

localhost A 127.0.0.1
www A 10.10.40.211
mail A 10.10.40.212
ns1 A 10.10.40.213
=============================================
/etc/rndc.key
=============================================
You can create your own rndc key file using
# rndc-confgen >rndc.kye
The file will shown like

key "rndckey" {
algorithm hmac-md5;
secret "YKPl5gxHe1d2J6kyjDGZFg==";
};

options {
default-key "rndckey";
default-server 127.0.0.1;
default-port 953;
};

We will not use options here;because we already mentioned
the same in "controls" section.So just keep "rndckey" entry
At last the file /etc/rndc.key will be like this
============================================
key "rndckey" {
algorithm hmac-md5;
secret "YKPl5gxHe1d2J6kyjDGZFg==";
};
============================================
At last you have to copy some files from/usr/share/doc/bind-9.3.3/sample/var/named
to /var/named/

1.named.root
2.named.local
3.localhost.zone

Rename named.root to named.ca

open /etc/resolve.conf and add
nameserver 127.0.0.1
===========================================

Check your configuration using "host" command
1.Forward Lookup
# host www.ansil.com
www.ansil.com has address 10.10.40.211
2.Reverse lookup
# host 10.10.40.211
211.40.10.10.in-addr.arpa domain name pointer www.ansil.com.

The Reverse internal zone file contains only one pointer www.ansil.com to 10.10.40.211
The Forward internal zone file contains many entries like
1.Mail exchanger
2.Name server
3.A record will give the IP of a domain

Clustering Apche Using RHCS(Redhat Cluster Suit)

2009-11-03T05:51:00.000-08:00

How to Install Redhat Clustering Suite

For a cluster setup you need minimum two machines

1. Node1

2. Node2

In this setup we will install cluster management software on another machine called Manager

The machine names and IPs are

1. node1.ansil.com --- 10.10.40.212

2. node2.ansil.com --- 10.10.40.213

3. manager.ansil.com --- 10.10.40.211

Be sure that you can access these machines through their names(FQDN).For this use a dns server/flat dns(/etc/hosts)

Management Server

1. Configure YUM

2. Install Cluster management software Luci

3. Initialise Luci cluster manager

4. Set password for Luci admin login

5. Start Luci service

1. Configure YUM

First you have to setup a yum repository in management server. You can find the procedure for creating yum is in earlier post.

2. Install Cluster management software Luci

# yum install luci*

3. Initialise Luci cluster manager

# luci_admin init

4. Set password for Luci admin login

Enter password for ‘Admin’ user

Enter password:

Reenter password:

5. Start Luci service

# service luci start

After starting service you can access Luci cluster management using

https://10.10.40.211:8084

More on here...

Enter “admin” as user name and password that you entered while initializing Luci

How to Configure YUM

2009-10-23T06:26:00.001-07:00

Most of the beginners in Linux faces a big problem when they are deploying a package i.e.; dependency failure. To resolve this issue redhat introduced a technology called YUM.
YUM is a repository system contains information about packages and files inside every RPMs
Don’t worry about redhat repository .Create your own Yum repository in a local system and enjoy
Here we go

1. You must have all CDs used for Redhat installation
2. 3 GB partition formatted in ext3
3. Ftp service
4. Create repo package

Create a directory in / called yum

# mkdir /yum

Create one partition using fdisk with 3 gb size
After creating partition reboot the system or use partprob command to rebuild partition table

Format partition

# mkfs.ext3 /dev/sdX

Mount the newly created partition to /yum

# mount /dev/sdX /yum

Add the same to fstab for mounting at system startup

Mount your 1st CD to /mnt

# mount /dev/cdrom /mnt

Copy Server, Cluster, ClusterStorage, VT directories to /yum

# cp –ai /mnt/Server /yum
# cp –ai /mnt/Cluster /yum
# cp –ai /mnt/ClusterStorage /yum
# cp –ai /mnt/VT /yum

Unmount 1st CD and mount 2nd CD to /mnt

Umount /dev/cdrom
Insert 2nd CD

# mount /dev/cdrom /mnt

Copy all the contents from /mnt/Server directory to /yum/ Server
Copy all the contents from /mnt/Cluster directory to yum/Cluster
Copy all the contents from /mnt/ClusterStorage to /yum/ ClusterStorage
Copy all the contents from /mnt/VT directory to /yum/VT

# cp –ai /mnt/Server /* /yum/Server/
# cp –ai /mnt/Cluster /* /yum/ Cluster /
# cp –ai /mnt/ClusterStorage /* /yum/ ClusterStorage /
# cp –ai /mnt/VT /* /yum/ VT /

Copy /yum/Server/repodata/comp-rhel5-server-core.xml to /tmp
Copy /yum/Cluster/repodata/comp-rhel5-cluster.xml to /tmp
Copy /yum/ClusterStorage/repodata/comp-rhel5-cluster-st.xml to /tmp
Copy /yum/VT/repodata/comp-rhel5-vt.xml to /tmp

# cp /yum/Server/repodata/comp-rhel5-server-core.xml /tmp
# cp /yum/Cluster/repodata/comp-rhel5-cluster.xml /tmp
# cp /yum/ClusterStorage/repodata/comp-rhel5-cluster-st.xml /tmp
# cp /yum/VT/repodata/comp-rhel5-vt.xml /tmp

Remove /yum/Server/repodata directory
Remove /yum/ Cluster /repodata directory
Remove /yum/ ClusterStorage /repodata directory
Remove /yum/ VT /repodata directory

# rm –fr /yum/Server/repodata
# rm –fr /yum/ Cluster /repodata
# rm –fr /yum/ ClusterStorage /repodata
# rm –fr /yum/ VT /repodata

Create your group repository using createrepo

#createrepo -g /tmp/ comp-rhel5-server-core.xml /yum/Server/
#createrepo –g /tmp/ comp-rhel5-cluster.xml /yum/ Cluster
#createrepo –g /tmp/c omp-rhel5-cluster-st.xml/yum/ ClusterStorage
#createrepo –g /tmp/ comp-rhel5-vt.xml /yum/ VT

Create repodata file in /etc/yum.repos.d/
1. Copy rhel-debuginfo.repo as Server.repo to the same directory
2. Open Server.repo
3. And change the portions highlighted
-----------------------------------------------------------
[Server]
Name=Ansil’s repo
Baseurl=ftp://your ip/Server/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
-----------------------------------------------------------

Copy Server.repo as Cluster.repo, ClusterStorage.repo, and VT.repo to the same directory (/etc/yum.repos.d/)

Change filed in “[ ]” for each file and put Cluster, ClusterStorage.repo, and VT for Cluster.repo, ClusterStorage.repo, and VT.repo respectively

Now your YUM repository is ready

Also you have to configure Ftp service for accessing this repository from outside

Open /etc/vsftpd/vsftpd.conf

1. Check line
anonymous_enable=YES
is uncommented

2. Add
anon_root=/yum
at the end of file

Restart Ftp service
# service vsftpd restart
# chkconfig –level 35 vsftpd on

ALL… DONE…!!!!!

Go to Application->Add/Remove Software