FTP prerequisite

We need to make sure that the ftp server “vsftpd” is installed on every server in our cluster. You can check if it is installed by typing the following command ;

root@gandalf:~# rpm -q vsftpd
vsftpd-2.0.5-16.el5_5.1
root@gandalf:~#

If is not installed, we need to run the following command to instal it on the servers where it’s not installed ;

root@bilbo:~# yum install vsftpd

We must make sure the vsftpd is not started and doesn’t start upon reboot. To do so use the following commands on all servers;

root@bilbo:~# service vsftpd stop
Shutting down vsftpd:                                      [FAILED]
root@bilbo:~# chkconfig vsftpd off

Script to stop/start/status our FTP service

Now we need to create a script for each of our services (ftp and web) that the cluster software will use to stop and start the appropriate service and add it to our cluster configuration. We’ll put these scripts if our /cadmin GFS filesystem, so it’s accessible by our 3 servers. We will start by creating the script for the ftp service. The script used by the Red Hat Cluster Suite,  receive one parameter when called by the cluster software. These parameter can be “stop”, “start” and “status”.  You can download a copy of the script and the vsftpd configuration file if you want. but remember that if you want to use them as is, you must put them in the /cadmin filesystem. The “srv_ftp.sh” script will go in a subdirectory name “/cadmin/srv” and the configuration file “srv_ftp.conf” must go in “/cadmin/cfg” directory. But nothing beat an example, let’s built the one for our FTP service.

#! /bin/bash
# ---------------------------------------------------------------------------------
# Script to stop/start and give a status of ftp service in the cluster.
# This script is build to receive 3 parameters.
#    - start :  Executed by cluster to start the application(s) or service(s)
#    - stop  :  Executed by cluster to stop  the application(s) or service(s)
#    - status:  Executed by cluster every 30 seconds to check service status.
# ---------------------------------------------------------------------------------
# Author    : Jacques Duplessis - April 2011
# ---------------------------------------------------------------------------------
#set -x
CDIR="/cadmin"              ; export CDIR       # Root directory for Services
CSVC="$CDIR/srv"            ; export CSVC       # Service Scripts Directory
CCFG="$CDIR/cfg"            ; export CCFG       # Service Config. Directory
INST="srv_ftp"              ; export INST       # Service Instance Name
LOG="$CDIR/log/${INST}.log" ; export LOG        # Service Log file name
HOSTNAME=`hostname -a`      ; export HOSTNAME   # HostName
VSFTPD="/usr/sbin/vsftpd"   ; export VSFTPD     # Service Program name
FCFG="${CCFG}/${INST}.conf" ; export FCFG       # Service Config. file name
RC=0                        ; export RC         # Service Return Code
DASH="---------------------"; export DASH       # Dash Line

# Where the Action Start
# ---------------------------------------------------------------------------------
case "$1" in
  start)  echo -e "\n${DASH}" >> $LOG 2>&1
          echo -e "Starting service $INST on $HOSTNAME at `date`" >> $LOG 2>&1
          echo -e "${VSFTPD} ${FCFG}" >> $LOG 2>&1
          ${VSFTPD} ${FCFG} >> $LOG 2>&1
          RC=$?
          FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
          echo "Service $INST started on $HOSTNAME - PID=${FPID} RC=$RC">> $LOG
          echo "${DASH}" >> $LOG 2>&1
          ;;
  stop )  echo -e "\n${DASH}" >> $LOG 2>&1
          echo -e "Stopping Service $INST on $HOSTNAME at `date` " >> $LOG
          ps -ef | grep ${FCFG}| grep -v grep >> $LOG 2>&1
          FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
          echo -e "Killing PID ${FPID}" >> $LOG 2>&1
          kill $FPID  >> $LOG 2>&1
          echo -e "Service $INST is stopped ..." >> $LOG 2>&1
          RC=0
          echo "${DASH}" >> $LOG 2>&1
          ;;
  status) COUNT=`ps -ef | grep ${FCFG}| grep -v grep | wc -l`
          FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
          echo -n "`date` Service $INST ($COUNT) on $HOSTNAME">> $LOG 2>&1
          if [ $COUNT -gt 0 ]
             then echo " - PID=${FPID} - OK"  >> $LOG 2>&1
                  RC=0
             else echo " - NOT RUNNING" >> $LOG 2>&1
                  ps -ef | grep -i ${FCFG} | grep -v grep  >> $LOG 2>&1
                  RC=1
          fi
          ;;
esac
exit $RC

This script is placed in the directory “/cadmin/svc” and name “svc_ftp.sh”. Now, let’s add this script to our ftp service, run the “system-config-cluster” command to start the cluster configuration GUI.

Add ftp script to our ftp cluster service

Now, let’s add this script to our ftp service, run the “system-config-cluster” command.

root@gandalf:~# system-config-cluster &

Click on “Resources” on the left side and then on  the  “Create a Resource”‘ button at the bottom right of the screen. This will allow us insert our ftp service script into the cluster configuration.

Select “Script” from the “Resource Type” list and then enter the name of our ressource “srv_ftp”  and then specify the name of the script the service will use, with it’s full path. Here, like I said we decided to place it in our “/cadmin” GFS filesystem so it is seen by every node in the cluster.

Now we need to edit our “srv_ftp” service to add the resource we just created.

Select the “srv_ftp” service at the bottom left of the screen and then press the “Edit Service Properties” button.

Click on the “Add a Shared Resource to this service” button. This will bring up the screen below, where we select the “srv_ftp” script that we want to add the our service.

After adding our script to the resource, press the “Close” button.

We are now ready to push our new configuration to the member of our cluster, press the “Send to Cluster” button to do so.

Web site prerequisite

Make sure that the “httpd” and the “php” package is installed on every server in our cluster. You can check if it is installed by typing the following command ;

root@gandalf # rpm -q httpd php
httpd-2.2.3-45.el5
php-5.1.6-27.el5_5.3
root@gandalf #

#

If is not installed, we need to run the following command to instal them on the servers where it’s not installed ;

root@bilbo:~# yum install httpd php

We must make sure the “httpd” is not started and doesn’t start upon reboot. To do so use the following commands on all servers;

root@bilbo:~# service httpd stop
Shutting down httpd:                                      [FAILED]
root@bilbo:~# chkconfig httpd off

Script to stop/start/status our Web service

We have simplify the configuration of our web site to the minimum. This was done intentionnaly, we wanted to demonstrate the cluster functionnaly and not the “httpd” possibilites. But our web site will be functionnal and redundant. As with the ftp script, the function of our web server script is very similar. You can download this script and the httpd configuration file if you want, but remember that if you want to use them as is, you must put them in the /cadmin filesystem. The “srv_www.sh” script will go in a subdirectory name “/cadmin/srv” and the configuration file “srv_www.conf” must go in “/cadmin/cfg” directory.

#! /bin/bash
# ---------------------------------------------------------------------------------
# Script to stop/start and give a status of our web service in the cluster.
# This script is build to receive 3 parameters.
#    - start :  Executed by cluster to start the application(s) or service(s)
#    - stop  :  Executed by cluster to stop  the application(s) or service(s)
#    - status:  Executed by cluster every 30 seconds to check service status.
# ---------------------------------------------------------------------------------
# Author    : Jacques Duplessis - April 2011
# ---------------------------------------------------------------------------------
#set -x
CDIR="/cadmin"              ; export CDIR       # Root directory for Services
CSVC="$CDIR/srv"            ; export CSVC       # Service Scripts Directory
CCFG="$CDIR/cfg"            ; export CCFG       # Service Config. Directory
INST="srv_www"              ; export INST       # Service Instance Name
LOG="$CDIR/log/${INST}.log" ; export LOG        # Service Log file name
HOSTNAME=`hostname -a`      ; export HOSTNAME   # HostName
HTTPD="/usr/sbin/httpd"     ; export HTTPD      # Service Program name
HCFG="${CCFG}/${INST}.conf" ; export HCFG       # Service Config. file name
RC=0                        ; export RC         # Service Return Code
DASH="---------------------"; export DASH       # Dash Line

# Where the Action Start
# ---------------------------------------------------------------------------------
case "$1" in
 start)  echo -e "\n${DASH}" >> $LOG 2>&1
         echo -e "Starting service $INST on $HOSTNAME at `date`" >> $LOG 2>&1
         echo -e "${HTTPD} ${HCFG}" >> $LOG 2>&1
         ${HTTPD} -f ${HCFG} >> $LOG 2>&1
         RC=$?
         HPID=`cat ${CCFG}/${INST}.pid`
         echo "Service $INST started on $HOSTNAME - PID=${HPID} RC=$RC">> $LOG
         echo "${DASH}" >> $LOG 2>&1       
         ;;
 stop )  echo -e "\n${DASH}" >> $LOG 2>&1
         echo -e "Stopping Service $INST on $HOSTNAME at `date` " >> $LOG
         HPID=`cat ${CCFG}/${INST}.pid`
         echo -e "Killing PID ${HPID}" >> $LOG 2>&1
         kill $HPID  > /dev/null 2>&1
         echo -e "Service $INST is stopped ..." >> $LOG 2>&1
         RC=0
         echo "${DASH}" >> $LOG 2>&1       
         ;;
 status) COUNT=`ps -ef | grep ${HCFG}| grep -v grep | wc -l`
         HPID=`cat ${CCFG}/${INST}.pid`
         echo -n "`date` Service $INST ($COUNT) on $HOSTNAME">> $LOG 2>&1
         if [ $COUNT -gt 0 ]
            then echo " - PID=${HPID} - OK"  >> $LOG 2>&1
                 RC=0
            else echo " - NOT RUNNING" >> $LOG 2>&1
                 ps -ef | grep -i ${HCFG} | grep -v grep  >> $LOG 2>&1
                 RC=1
         fi
         ;;
esac
exit $RC

Updating our cluster Configuration

To add our web service, please follow the same sequence as we did when we inserted our ftp service into the cluster configuration. You only need to replace “srv_ftp.sh” by “srv_www.sh” and the script path will be the same, we have decide to place our scripts into the directory “/cadmin/srv”. Once we have push the new configuration to all servers in the cluster, we should now have a working cluster.The web site define in the configuration have its “Root Directory” set to “/cadmin/www/html” it contains only one file that will display the name of the it is running on. The will help us testing our cluster configuration.

I you wish to use the cluster configuration, scripts and configuration files we have used in this series of articles, I would encourage you to download the “cadmin.tar” file. The file is the actual content of the “/cadmin” directory used throught out this article. To use it, download the “cadmin.tar” file then copy it to your “/cadmin” directory and enter the command “tar -xvf ./cadmin.tar”. This will explode the tar file and then you will have the working envirionnment I used in this article.

Testing our ftp service

So here we are (finally you would say … hum me too), we have now a fully working cluster. So if we issue the “clustat” command this is what we should see.

root@gollum:/# clustat
Cluster Status for our_cluster @ Sat Apr 16 11:37:25 2011
Member Status: Quorate

 Member Name                        ID   Status
 ------ ----                       ---- ------
 hbbilbo.maison.ca                    1 Online, rgmanager
 hbgandalf.maison.ca                  2 Online, rgmanager
 hbgollum.maison.ca                   3 Online, Local, rgmanager

 Service Name                     Owner (Last)            State        
 ------- ----                     ----- ------            -----        
 service:srv_ftp                  hbgollum.maison.ca      started      
 service:srv_www                  hbbilbo.maison.ca       started      
root@gollum:/#

From the information above, we can see that all our cluster member status are online and that the resource manager is running on all of them. The resource manager is important, it is responsable for moving service around when needed. Our service “srv_ftp” is started (running) on the “hbgollum” server and “srv_www” is running on the “hbbilbo” like we decided at the beginning (remember ?

root@gollum:/# ip addr show | grep 192
    inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.204/24 scope global secondary eth0
root@gollum:/#
root@gollum:/# ps -ef | grep vsftpd | grep -v grep
root      7858     1  0 10:05 ?        00:00:00 /usr/sbin/vsftpd /cadmin/cfg/srv_ftp.conf
root@gollum:/#

The command “ip addr show | grep 192” is confirming that the virtual IP is defined on “hbgollum” server and if we check if the ftp process is also running, we can see that it is. So let’s try to do an FTP to our virtual IP that we have name “ftp.maison.ca” (192.168.1.204). We will try it from the “gandalf” server and we see that it is working.

root@gandalf:/# ftp ftp.maison.ca
Connected to ftp.maison.ca.
220 ftp.maison.ca
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (ftp.maison.ca:root):

Now let’s move the ftp service from “hbgollum” to “hbbilbo”, to see if the ftp service will continue to work. To move the service we will use the “clusvcadm” command, we need to specify the service name we need to relocate (-r) and the server (-m for machine) we wish to move it. You can issue the “clusvccmd” command on any of the server within our cluster. So enter the following command to move our service to “hbbilbo” ;

root@gandalf:/# clusvcadm -r srv_ftp -m hbbilbo
'hbbilbo' not in membership list
Closest match: 'hbbilbo.maison.ca'
Trying to relocate service:srv_ftp to hbbilbo.maison.ca...Success
service:srv_ftp is now running on hbbilbo.maison.ca
root@gandalf:/#

Notice that just after pressing [Enter], we got “‘hbbilbo’ nit in membership list”, this is because we did not mention the domain name ‘maison.ca’, but it managed to assume that we were refering to “hbbilbo.maison.ca”. So our command succeeded, so let’s see if everything went like it should have.

Fisrt, let’s execute the clustat command to see in the “srv_ftp” service is now running on ‘hbbilbo’.

root@gandalf:/# clustat 
Cluster Status for our_cluster @ Sat Apr 16 12:10:01 2011
Member Status: Quorate

 Member Name                      ID   Status
 ------ ----                      ---- ------
 hbbilbo.maison.ca                    1 Online, rgmanager
 hbgandalf.maison.ca                  2 Online, Local, rgmanager
 hbgollum.maison.ca                   3 Online, rgmanager

 Service Name            Owner (Last)            State
 ------- ----            ----- ------            -----
 service:srv_ftp         hbbilbo.maison.ca       started      
 service:srv_www         hbbilbo.maison.ca       started
root@gandalf:/#

We can see that our ftp service is now running on “hbbilbo”, let’s see if it reallly is. If we check if the 192.168.1.204 (ftp.maison,ca) is now defined on “hbbilbo’ we can see that it is. The FTP process is also running now on the server.

root@bilbo:~# ip addr show | grep 192
    inet 192.168.1.111/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.211/24 scope global secondary eth0
    inet 192.168.1.204/24 scope global secondary eth0
root@bilbo:~# ps -ef | grep vsftpd | grep -v grep
root      8616     1  0 12:05 ?        00:00:00 /usr/sbin/vsftpd /cadmin/cfg/srv_ftp.conf
root@bilbo:~#

But what happen on ‘hbgollum’, the IP 192.168.1.204 should not be there anymore and the FTP process should not be running anymore. So that’s what happen the IP is gone and the ftp process is no longer running. So far so good, the move to ‘hbbilbo’ server have worked.

root@gollum:/etc/profile.d# ip addr show | grep 192
    inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0
root@gollum:/etc/profile.d# ps -ef | grep vsftpd | grep -v grep
root@gollum:/etc/profile.d#

The last test if to try to do an ftp to ftp.maison.ca and see if it respond.

root@gandalf:/# ftp ftp.maison.ca
Connected to ftp.maison.ca.
220 ftp.maison.ca
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (ftp.maison.ca:root):

Great evrything worked !! Let’s move the ftp process back to ‘hbgollum’ before testing our web site. Open another terminal window and enter the “clustat -i 2” command and see watch the status change from ‘started/stopping/starting/started’ while the move is going on. Check your /var/log/message and familiar yourself with the line recorded when the move happen.

root@gandalf:/# clusvcadm -r srv_ftp -m hbgollum
'hbgollum' not in membership list
Closest match: 'hbgollum.maison.ca'
Trying to relocate service:srv_ftp to hbgollum.maison.ca...Success
service:srv_ftp is now running on hbgollum.maison.ca
root@gandalf:/#

One of the test we should make, is to unplug the network cable or poweroff “hbgollum” and see if the service move to the next server in the failover domain (It will). So we have now completed and tested our ftp service. It as been a long road but it woth it, no ?

Testing our web service

You know now, how to move service from server to another, Let’s do the same test with our web server service. The web site is actually just one simple page. It just display the name of the server that it is running on, this simplify our testing.

If we issue the “clustat” command we should have the following picture ;

# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:15:37 2011
Member Status: Quorate

Member Name                      ID   Status
------ ----                      ---- ------
hbbilbo.maison.ca                    1 Online, rgmanager
hbgandalf.maison.ca                  2 Online, Local, rgmanager
hbgollum.maison.ca                   3 Online, rgmanager

Service Name            Owner (Last)            State
------- ----            ----- ------            -----
service:srv_ftp         hbgollum.maison.ca      started
service:srv_www         hbbilbo.maison.ca       started

Let’s see if it is working, open your browser type this URL “http://www.maison.ca” , you should have a the following ;

Now, let’s move the wev site to “gandalf” server, type the following command ;

root@gollum:/cadmin/cfg# clusvcadm -r srv_www -m hbgandalf
'hbgandalf' not in membership list
Closest match: 'hbgandalf.maison.ca'
Trying to relocate service:srv_www to hbgandalf.maison.ca...Success
service:srv_www is now running on hbgandalf.maison.ca
root@gollum:/cadmin/cfg#
root@gollum:/cadmin/cfg# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:27:14 2011
Member Status: Quorate

 Member Name                         ID   Status
 ------ ----                         ---- ------
 hbbilbo.maison.ca                       1 Online, Local, rgmanager
 hbgandalf.maison.ca                     2 Online, rgmanager
 hbgollum.maison.ca                      3 Online, rgmanager

 Service Name               Owner (Last)               State
 ------- ----               ----- ------               -----
 service:srv_ftp            hbgollum.maison.ca         started
 service:srv_www            hbgandalf.maison.ca        started 

We can see that the web site is now running on “gandalf” server.

Disabling and Enabling Services

There may come a time, when you need to stop a service completely. We will demonstrate how to acheive that, first let’s display the status of our cluster

root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:39:48 2011
Member Status: Quorate

 Member Name                                  ID   Status
 ------ ----                                  ---- ------
 hbbilbo.maison.ca                                1 Online, Local, rgmanager
 hbgandalf.maison.ca                              2 Online, rgmanager
 hbgollum.maison.ca                               3 Online, rgmanager

 Service Name                        Owner (Last)                        State
 ------- ----                        ----- ------                        -----
 service:srv_ftp                     hbgollum.maison.ca                  started
 service:srv_www                     hbgandalf.maison.ca                 started  

We are going to disable the “srv_www” service, to do so enter the following command ;

root@bilbo:~# clusvcadm -d srv_www 
Local machine disabling service:srv_www...Success

The clustat command show us that the service is now disable.

root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:40:04 2011
Member Status: Quorate

 Member Name                                  ID   Status
 ------ ----                                  ---- ------
 hbbilbo.maison.ca                                1 Online, Local, rgmanager
 hbgandalf.maison.ca                              2 Online, rgmanager
 hbgollum.maison.ca                               3 Online, rgmanager

 Service Name                        Owner (Last)                        State
 ------- ----                        ----- ------                        -----
 service:srv_ftp                     hbgollum.maison.ca                  started
 service:srv_www                     (hbgandalf.maison.ca)               disabled

We will now enable the service, but this time we will enable it on another server than “hbgandalf”. This command enable the “srv_www” service on the server “hbbilbo”.

root@bilbo:~# clusvcadm -e srv_www -m hbbilbo
'hbbilbo' not in membership list
Closest match: 'hbbilbo.maison.ca'
Member hbbilbo.maison.ca trying to enable service:srv_www...Success
service:srv_www is now running on hbbilbo.maison.ca
root@bilbo:~#

We can see the it is now running on “hbbilbo”.

root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:47:36 2011
Member Status: Quorate

 Member Name                                  ID   Status
 ------ ----                                  ---- ------
 hbbilbo.maison.ca                                1 Online, Local, rgmanager
 hbgandalf.maison.ca                              2 Online, rgmanager
 hbgollum.maison.ca                               3 Online, rgmanager

 Service Name                        Owner (Last)                        State
 ------- ----                        ----- ------                        -----
 service:srv_ftp                     hbgollum.maison.ca                  started
 service:srv_www                     hbbilbo.maison.ca                   started  
root@bilbo:~#

This conclude our implementation of a small cluster. It was intended just to show every one how the Red Hat Cluster Suite actually work and to give a brief overview how it work. We will now move on to other interesting topic. Don’t know what it will be, but I can assure that it should fit into one article, So I hope you appreciate it and hope to see you soon.

Part 1 – Creating a Linux ReadHat/CentOS cluster

Part 2 – Creating a Linux ReadHat/CentOS cluster

Part 3 – Creating a Linux ReadHat/CentOS cluster

Part 4 – Creating a Linux ReadHat/CentOS cluster

Part 5 – Creating a Linux ReadHat/CentOS cluster

Adding a SAN disk to our servers

The Linux operating system is installed on the internal disks for each of our server. We will now add a SAN disk that will be visible be each of our server. I assume here that your SAN and your Brocade switch are configure accordingly. Explaining how to set up the SAN and the Brocade switch is not in the scope of this article. But I think that you get the idea that the new disk must be visible by every node in our cluster. In the example below we already have a SAN disk (sda) with one partition (sda1) on it. Adding a disk to the server, can be done (live) without any interruption of service, if you follow the steps below. I would suggest you practice on a test server, to become familiar with the procedure.

Before we add a disk, let’s see what are the visible disks on the system, by looking at the /proc/partitions file. We can see that we already have a disk (sda) with one partition on it. So the new disk that we’re going to add, should be seen as “sdb”.

root@gollum~# grep sd /proc/partitions
8     0  104857920 sda
8     1  104856223 sda1

Let’s rescan the SCSI bus by typing the command below. This command must be run on each of the server within the cluster. Here, we have only one HBA (Host Base Adapter)  card connected to the SAN on each server. If you have a second HBA, you need to run the same command for the second HBA, but replace the “host0” by “host1”.

root@gollum~#  echo “- – -” > /sys/class/scsi_host/host0/scan
root@gandalf~#  echo “- – -” > /sys/class/scsi_host/host0/scan
root@bilbo~#     echo “- – -” > /sys/class/scsi_host/host0/scan

Let’s see if we have some new disk(s) that were detected (sdb) (check each servers)

root@gollum~# grep sd /proc/partitions
8     0  104857920 sda
8     1  104856223 sda1
8    16   15728640 sdb

Let’s create a LVM partition on our new disk (sdb) by running the “fdisk” command..

# fdisk /dev/sdb

Command (m for help): p

Disk /dev/sdh: 107.3 GB, 107374510080 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot      Start         End      Blocks   Id  System

Command (m for help): n (n=new partition)
Command action
e   extended
p   primary partition (1-4) p (p=primary partition)
Partition number (1-4): 1 (first partition =1)
First cylinder (1-13054, default 1):  1 (Start at the beginning of disk)
Last cylinder or +size or +sizeM or +sizeK (1-13054, default 13054): 13054 (End of the Disk)
Command (m for help): p (p=Print partition information)

Disk /dev/sdh: 107.3 GB, 107374510080 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       13054   104856223+  83  Linux

Command (m for help): t (t=type of partition)
Selected partition 1 (Change type of partition 1)
Hex code (type L to list codes): 8e (8e=LVM partition – Type L to list partition code)
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): p (p=print partition information)

Disk /dev/sdh: 107.3 GB, 107374510080 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       13054   104856223+  8e  Linux LVM

Command (m for help): w (w=write partition to disk)

The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

If we look again at our /proc/partition, we should see our new disk and partition are being seen by this server.

root@gollum~# grep sd /proc/partitions
8 0 104857920 sda
8 1 104856223 sda1
8 16 15728640 sdb
8 16 15727658 sdb1

Now we need to make sure that the new disk and partition are seen by every servers within the cluster. We now need to go on every servers in the cluster and run the command “partprobe” (Partition probe). After running the command, you should check like we did on “gollum”, if all the disks and partitions are seen by each servers.

root@gollum~# partprobe
root@gandalf~# partprobe
root@bilbo~# partprobe

Creating our physical volume

Now that we know that the disk is seen by every node, let’s create the physical volume on one of the server and then check on all others servers in the cluster, the physical should be seen on all the servers. The command to create a physical volume is “pvcreate”, so what we are really doing here, is creating a physical volume of the partition (/dev/sdb1) we created earlier.

# pvcreate /dev/sdb1
Physical volume “/dev/sdb1” successfully created

Let’s run a pvscan on every node, to validate that every node can actually see the new disk.

# pvscan
PV /dev/sda1           VG     datavg      lvm2 [100.00 GB / 22.02 GB free]
PV /dev/sdb1                                         lvm2 [100.00 GB]

Create our clustered volume group

We will now create a new volume group named “sharevg” and we will assign the physical volume “/dev/sdb1” as part of that group. If we ever ran out of disk space within “sharevg”, we could add another physical volume to the volume group and continue to work without any service disruption. This is a real advantage when working in production environment.

# vgcreate  sharevg /dev/sdb1
Clustered volume group “sharevg” successfully created

Display the “sharevg” volume group properties

We can display the volume group properties by issueing the “vgdisplay” command. We can see that our volume group is “Clustered”, so it is cluster aware. This will allow later on,  to create “LV” (Logical volume/Filesystem) on one server and have the cluster software automatically advise the cluster member that a new logical volume (filesystem) is available.

root@gollum~# vgdisplay sharevg
 --- Volume group ---
 VG Name               sharevg
 System ID            
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  25
 VG Access             read/write
 VG Status             resizable
 Clustered             yes
 Shared                no
 MAX LV                0
 Cur LV                0
 Open LV               0
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               100.00 GB
 PE Size               4.00 MB
 Total PE              24999
 Alloc PE / Size       1 / 2.10 MB
 Free  PE / Size       24980 / 98.00 GB
 VG UUID               V8Ag76-vdW2-NAk4-JjOo-or3l-GuPz-x5LEKP

Create a logical volume of 1024MB named “cadmin” of the “sharevg” volume group

We will create a logical volume named “cadminlv” (Cluster Admin), in our sharevg volume group. The command below is asking to create a logical volume of 1024MB, name “cadminlv” in the volume group “sharevg”.  This command can be done one server and the logical volume will be seen by every member of the cluster.

#/usr/sbin/lvcreate -L1024M -n cadminlv sharevg
Logical volume “cadminlv” created

The “lvs” command allow you to display a list of all your logical volumes. Since this is currently the only one on the volume group “sharevg”, we filter the list (with the grep command) to only display the logical volume on “sharevg” volume group.Let’s check if it seen by all nodes.

root@gandalf~# lvs | grep sharevg
cadminlv sharevg -wi-a- 1024.00M

root@gollum~# lvs | grep sharevg
cadminlv sharevg -wi-a- 1024.00M

root@bilbo~# lvs | grep sharevg
cadminlv sharevg -wi-a- 1024.00M

Creating the /cadmin GFS Filesystem

Finally, we are going to create a GFS filesystem within that logical volume “cadminlv” we’ve just created. But first we need to create our GFS filesystem mount point on every nodes. We need to do that, because we want this filesystem to be mounted on every node and be available for the 3 nodes.

root@gandalf~# mkdir /cadmin

root@bilbo~#    mkdir /cadmin

root@gollum~# mkdir /cadmin

We have choosen to have the GFS filesystem “/cadmin” to be mounted on all servers at all time. We could have include it as part of our service, so it would be mounted only when a service is started. But we found out, that the process of unmounting and mounting a couple of GFS take time and this time adds up to the time it take to move a service from one server to another.  In production we have 5 servers in a cluster for more that 2 years now,  we have around 30 GFS mounted at all time on all the five servers and we had very little problem. The only thing you have to be careful about is the number of journals that you assign to each GFS. One journal is needed for each concurrent mount in the cluster, in our case we have at least 5 journals for each of our GFS filesystem (more on that below).Create the GFS on the LVM cadmin created previously. This need to be done only on one node, the creation is done once and all the nodes are made aware of the new GFS by the cluster daemon.

The command we use to create a GFS filesystem is “gfs_mkfs”.  We need to use a couple of options and I will explain them all.

First, the “-O”  prevents  “gfs_mkfs”  from  asking  for confirmation before creating the filesystem.

The option “-p lock_dlm“, indicate the name of the  locking  protocol to use.  The  locking  protocol should  be  “lock_dlm” for a clustered file system.

The “-t our_cluster:cadminlv” It’s the cluster-name, followed by “:” and the logical volume name.  The cluster name must match the one you have defined in your cluster configuration file (in our case “our_cluster”),  only members  of  this cluster are permitted to use this file system.  The filesystem name (cadminlv)  is a unique file system name used to distinguish this GFS file system from others created (1 to 16  characters).

The “-j 4” is the number of journals for gfs_mkfs to create.  You need at least one  journal  per machine that will mount the filesystem. This number should have been 3, but I always add one more, in case a add a member in the cluster. This number is important, if I had put 3 and I added a node within the cluster and I wanted the 4 nodes to mount simultaneously this filesystem, I would need to make sure that the filesystem have 4 journals, because the GFS wound not mount. You can always add a journal to an existing GFS filesystem with the “gfs_jadd” command. Each journal reserve 128 MB in the filesystem, so you need take into consideration. Let look at our example, we want all our nodes to mount the “/cadmin” GFS filesystem, we created an logical volume of 1024M and on it we created a GFS, we reserved 4 journals (4*128=512MB) , so will have only around 500 MB available for data out of the 1024MB we allocated to our logical volume.

The last parameter “/dev/sharevg/cadmlv” is the name of the logical volume we created previously.

# /sbin/gfs_mkfs -O -p lock_dlm -t our_cluster:cadmlv -j 4 /dev/sharevg/cadmlv
Device:                    /dev/sharevg/cadmlv
Blocksize:                 4096
Filesystem Size:           98260
Journals:                  4
Resource Groups:           8
Locking Protocol:          lock_dlm
Lock Table:                our_cluster:cadminlv
Syncing…
All Done

We are now able to mount our GFS filesystem on all the servers, by using the command below on all the servers,

# mount -t gfs /dev/sharevg/cadminlv  /cadmin

We want that filesystem to be mounted every time a server boot, so don’t forget to add your filesystem to /etc/fstab, so it will mount after the next reboot and don’t forget to change the owner and protection of the filesystem.

# echo “/dev/sharevg/cadminlv /cadmin  gfs defaults 0 0″ >> /etc/fstab

The filesystem should be available on all our nodes.

# df -h /cadmin
Filesystem                                                  Size     Used    Avail    Use%  Mounted on
/dev/mapper/sharevg-cadminlv  510 M 1.5M   508M   1%      /cadmin

So we’ve just created our first GFS filesystem and it is mounted on all our nodes in the cluster.

Defining the fencing devices

The fencing device is use by the cluster software to power-off the node when it is considered in problem. We need to define a fencing device for all the nodes that we defined in the previous article.

In our cluster, the fencing device used is the HP ILO interface. Select “the “HP ILO Device” from the pull down menu and enter the device information needed to connect to it. If you would like to see what are the fencing device supported by Red Hat cluster, you can consult the fencing FAQ on this page.

You could use manual fence for testing purpose but it is not supported in a production environment, since manual intervention is required to fence a server.

• We choose to prefix the fencing device name with a lowercase “f_”, followed by the name assigned to its IP in /etc/hosts.
• The login name to access the device.
• The password used to authenticate to the device.
• The host name as defined is our host file used to connect to the device.

Repeat the operation for each server in the cluster.

Defining the fence level

Next, we need to associate a fencing device with each server in our cluster. Let’s begin by clicking on the node name “hbbilbo.maison.ca”, then press on the button name (“Manage Fencing For This Node”) at the bottom right of the screen. You will be presented with a similar screen than the one below.

Now click on the “Add a New Fence Level” button, this will create a new fence level named “Fence_Level-1”.

Next, select “Fence-Level-1” on the left of the screen and then click on the button “Add a New Fence to this Level” . A little pop-up will appear, allowing you to assign the  fencing device to the host name. In our case we select the fencing device name “f_bilbo”. So we just created an association between a fencing device and a node.

Click on the “OK”  button to close the screen above and then press the “Close”  button.

Back on this screen, press the “Close” button again to end the definition of our fence level.

We need to repeat the operation for each node in our cluster.

Once a fence level is created for all nodes, you can proceed with the next step.

Define Failover Domain

A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. The name of our failover domain, will always begin with “fd_” (lowercase), this is the standard that I choose.

So let’s begin defining our failover domain, by clicking on “Failover Domains” on the right side of the GUI and pressing the “Create a Failover Domain” button.

Now enter the name of the failover domain, in this case for “bilbo” server will be “fd_bilbo” to stick to our standard.  Press the “OK” button to proceed.

The failover domain configuration, list all the servers that will be part of the specified failover domain. We choose, that each member of the cluster will be listed in our failover domain.

So select each server one at a time from the selection list “Available Cluster Nodes” , until they have all been selected.  When all the servers are selected, the selection list will display “No Cluster Nodes Available”.

We choose to restrict the failover domain to the list of servers we have included. This means all of them for this cluster, but they can be situation were you could have six nodes in a cluster and we want one service to run on only three of the servers because they have more cpu and memory then the three others .

We also want the service to be prioritized, check the “Prioritized List” checkbox.

You may then highlight the nodes listed in the “Member Node” box and click the “Adjust Priority”  buttons to select the order in which the failover should occur. The node with the highest priority (priority 1) will be the initial node for running the service. If that node fails, the service will be relocated to the node with the next higher priority (priority 2) and so forth.

So for the failover domain  “fd_bilbo”  the “hbbilbo”  will be prioritize (1), if the server is power-off then the service will be move to “hbgandalf” (2) our passive node and if it unavailable then service will move to the server “hbgollum” (3).

The preferred server for the failover domain “fd_gollum” will be “hbgollum” (1), if it is not available then our passive server “hbgandalf”  (2) will take over. And in the eventuality that the two servers are not available then “hbbilbo” (3) will take over.

For the failover domain “fd_gandalf” we have choosen to give “hbgandalf”  priority one, “hbgollum” priority two and three to “hbbilbo” .

Defining cluster resources

Cluster resource can be IP addresses, scripts, NFS and/or GFS filesystem mount point, database that are needed to run a service. In this case our services will be a FTP and a web site.

If you remember inl our cluster network diagram, we’ll be running 2 services in our cluster. The first service will be a FTP server name “ftp.maison.ca” running at IP 192.168.1.204 and a web service “www.maison.ca” running at IP 192.168.2.211.

So let’s define an “IP Resource” for our ftp server at 192.168.1.204. Click on “Resources” on the left part of the screen and then click on the button “Create a Resource”  at the lower right of the screen. From the drop down list, select the “IP Address” resource type.

Enter the IP of our ftp server “192.168.1.204” and make sure that the “Monitor Link” check box is selected. That means that if for a reason or another this IP is no longer responding, then this will trigger the move of our ftp service to the next server that we have defined in our failover domain for that node.

Repeat the process for our web site at IP 192.168.1.211.

Defining our cluster services

We have decided that we would be running two services in our cluster, one for the ftp and one for the web site.

Now is the time to define these services, we will begin by creating our ftp service. Click on “Services” on the bottom left of the screen and then press the button called “Create a Service” to begin the process.

First, we need to assign a name to our service, we choose to prefix our service by “srv_” , this will be our standard. We choose a short name, because they are nicer when they are displayed by the “clustat” command that we will see later on. So enter the service name “srv_ftp” and click on the “OK” button.

Next you will see a screen similar to this one.

First we need to assign a failover domain to our ftp service. As planned on our cluster network diagram we were going to run the ftp service on the gollum server, so from the failover domain drop down list we select the failover domain “fd_gollum” for our ftp service.

Next we need to make our FTP server IP (192.168.1.204)  part of the service. To do so click on the button “Add a Shared Resource to this service” and select the IP used for our ftp service.

After pressing the button above,  you should see a screen similar to this one.

Click on the IP that our ftp server will use (192.168.2.204) and press the “OK” button.

One last thing before finishing the definition of our ftp service, make sure you select “Relocate” from the “Recovery Policy”. This allow our service to be move automatically to another server if the FTP service become inaccessible.

We can now press the “Close” button and repeat the operation to create of our web service.

Follow the name procedure as the ftp service to create the web service.

The name that we decided to call the service for the web site is “srv_www” .

The web site service “srv_www” will be part of the “fb_bilbo” failover domain, so it will start on the “bilbo” server.

If the web service become unavailable, the service will be relocated to another server within the cluster. The server to where the web service will be move, is be based on the priority order we have put in the failover domain. So the IP will automatically be move when the service is relocated to another server within the cluster. We will demonstrate that later on in this article.

Propagate cluster configuration

We should have now a working cluster, it does not have the final configuration but we will be able to bring it up and move services from one node to another. The missing part of our cluster in order to finalize it, is the creation of the GFS filesystem and our scripts that will bring up and down our ftp/web site. We will look at that in the next article.

But for now let’s push this configuration to our cluster, by pressing the “Sent to Cluster” button. This will copy our new configuration file “/etc/cluster/cluster.conf” to every member of the cluster and activate it.

We will need to confirm our intention, by pressing the “Yes” button to the pop up below.

I had some problem pushing the initial configuration to other member of the cluster, on my first attempt. If this happen, you may have to copy it manually. The cluster configuration file is stored in /etc/cluster and is named cluster.conf.  If you have ran the “system-config-cluster” GUI on “bilbo” server, then issue the following command on “bilbo” to copy the configuration file to the other servers ;

scp /etc/cluster/cluster.conf  gandalf:/etc/cluster

scp /etc/cluster/cluster.conf  gollum:/etc/cluster

If the copy is done manually, you will have to restart the cluster services on each node or reboot all the nodes. Once you have a working cluster, you will not have to manually copy the cluster configuration file, pressing the button “Send to Cluster” will be enough. You can also the command “ccs_tools” to propagate the cluster configuration change (describe below).

Manual adjustment to cluster configuration file

I made the following changes to the cluster configuration file to prevent the reboot of the server after it is fenced (For the HP ILO). When a server is fenced, the service is transferred to another node and the server that was hosting that service is power off (fenced). If we do not make the following change, the server that was power off will automatically reboot and perhaps then same error condition may occur (network, switch, FC problem), it would power off / power on, / power off / power on …. We want to eliminate that.

Every time you update the cluster configuration file manually you need to increment the configuration version number, so it will trigger an update on the others servers .

root@gandalf:~# vi /etc/cluster/cluster..conf

Increment the version number

Since we are changing the configuration file, need to increment the version number.

Before change

<cluster alias=”our_cluster” config_version=”74” name=”our_cluster”>

After the change

<cluster alias=”our_cluster” config_version=”75” name=”our_cluster”>

Changes to prevent rebooting of the node after a power-off

We need to do this modification for each of the “clusternode name” section. This will prevent a restart of the node after it has been fenced.

Before the change

<clusternode name=”hbbilbo.maison.ca” nodeid=”1″ votes=”1″>
<fence>
<method name=”1″>
<device name=”f_bilbo”/>
</method>
</fence>
</clusternode>

After the change

<clusternode name=”hbbilbo.maison.ca” nodeid=”1″ votes=”1″>
<fence>
<method name=”1″>
<device action=”off” name=”f_bilbo”/>
</method>
</fence>
</clusternode>

Increase resource manager verbosity

I include here something optional, if you are having some problem with your cluster, increasing verbosity of the resource manager daemon “rgmanager” may help you.  Changing the <rm> line will add more debugging information in the /var/log/rgmanager file (syslogd need to be restarted).

Before the change

</fencedevices>
<rm>
<failoverdomains>

After the change

</fencedevices>
<rm log_facility=”local4″ log_level=”7″>
<failoverdomains>

Whenever you update the configuration file manually you can use the “ccs_tool” command to propagate the new cluster configuration file to all cluster members. Don’t forget to update the version number.

Distribute new cluster config file

Checking cluster functionality

We can check the status of your cluster in two different ways. The simplest one is to use the “clustat” command (see output below).  If you want a continuous display, you can add “-i 2” to the “clustat” command, to have the output refreshed every two seconds (Press CTRL-C to stop the display).

clustat command output

This is the normal output you should have. All our members are “Online”  with the resource manager running on them.

The name of each members of the cluster, their “Node ID” and their status are displayed.

Our two services “srv_ftp” and “srv_www”  are running (started) on the selected server.

system-config-cluster (cluster mgmt tab)

The second way to get the cluster status, is to run the cluster GUI (system-config-cluster) and then click on the “Cluster Management” tab.

In the upper part of the screen, we can see the current member of our cluster and their “Node ID” .

Below, we can visualise the status of each services define within the cluster.

We can also see that our service “srv_ftp”, is running (started) on “hbgollum” and that the service “srv_www” is running on “hbbilbo.maison.ca” .

With the “Cluster Management” tab , we can disable (stop), enable (start) or restart each of these services. More on this later.

Let’s check if our services IP are alive.

The IP address of the ftp service that we have define is “192.168.1.204” and from the “clustat” command above we know that it is running on the server “gollum”. So let’s logon on that server and check if that IP is active. Be aware that the output of the “ifconfig -a”  does not include our FTP IP, we need to use the “ip” command to see it.  From the output below, we can see that server IP “192.168.1.104”  is active on “eth0” and that our ftp server IP  “192.168.1.204” is also active on the same interface. So our cluster software is doing it’s job, we have the “ftp.maison.ca” IP defined on the interface “eth0”. You will also notice that on “eth1”, we have our heartbeat IP “10.10.10.104” is active.

root@gollum:~# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:da:68:df:9b brd ff:ff:ff:ff:ff:ff
inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.204/24 scope global secondary eth0
inet6 fe80::250:daff:fe68:df9b/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:02:a5:b1:a0:d4 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.104/24 brd 10.10.10.255 scope global eth1
inet6 fe80::202:a5ff:feb1:a0d4/64 scope link
valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
root@gollum:~#

On the server “bilbo” we have our web IP (192.168.1.211) active on the interface “eth0” along with the server IP (192.168.1.111). Our heartbeat IP is defined on the interface “eth1” .

root@bilbo:~# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:01:02:75:80:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.111/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.211/24 scope global secondary eth0
inet6 fe80::201:2ff:fe75:8058/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:8b:f4:c5:59 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.111/24 brd 10.10.10.255 scope global eth1
inet6 fe80::250:8bff:fef4:c559/64 scope link
valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
root@bilbo:~#

I think we should stop the article here, we have now a working cluster that we will exploit in our next article.

In our next article, we will create our services scripts and our GFS filesystem so that our server can share the same filesystem. This feature will give us the ability to share information between server via a common filesystem. So when the service has to moved from one server to another, the service can start using the same data it was using on the previous server.

I don’t know if the next article will be the last one of this series of article on the cluster, but we will see.

Stay tune for part 4 of this series of article on creating a Linux Red Hat cluster.

Part 1 – Creating a Linux ReadHat/CentOS cluster

Part 2 – Creating a Linux ReadHat/CentOS cluster

Part 3 – Creating a Linux ReadHat/CentOS cluster

Part 4 – Creating a Linux ReadHat/CentOS cluster

Part 5 – Creating a Linux ReadHat/CentOS cluster

Here is the second article on how to to build a Red Hat/CentOS cluster. The “Part One of this article” was posted a couple of month ago, I’m sorry for the delay but I have been a bit busy at work and at home.  I can promise that Part 3 will not take a couple of month to post. I want to terminate this series on the cluster and post some new ideas that I have.  In this article, we continue our journey on how-to build our cluster and stay tune for Part 3 coming soon.

Installing the cluster software

If you are using Red Hat, you need to register your server in order to download new software or update. You also need to subscribe to the “Clustering” and “Cluster Storage” channel in order to install these groups of software. With CentOS, this is not needed since we can download these groups of software without registration. Let’s install the clustering software, by typing this command;

# yum groupinstall Clustering

Since we will be using GFS filesystem, we will need the “Cluster Storage” software group.

# yum groupinstall “Cluster Storage”

I also found out that this package is also needed by the cluster software, so let install it.

# yum install perl-Crypt-SSLeay

If you are using a 32 bits kernel and you have more than 4 GB of memory on it, you need to install the PAE kernel and GFS modules. This will ensure that you are using all the memory available on the server.

# yum install kmod-gnbd-PAE kmod-gfs-PAE kernel-PAE

Finally, let’s make sure that you have that the servers have the latest OS update.

# yum –y update

Setting the locking type for GFS filesystem

To use the GFS (Global File System) with the cluster you need to activate the GFS locking in the /etc/lvm/lvm.conf file. We need to change the “locking_type” variable from 0 to 3, to inform LVM that we will be dealing iwth GFS volume group and GFS filesystem. This command needs to be run on all the servers.

# grep -i locking_type /etc/lvm/lvm.conf
locking_type = 0
# lvmconf --enable-cluster
# grep -i locking_type /etc/lvm/lvm.conf
locking_type = 3

Making sure SELinux and firewall are disable

We do not want to deal with SELinux and the firewall in our cluster, so we will disable them. From the gnome desktop run the following the following command ;

# system-config-securitylevel

Disabling SELinux need a reboot

Disable Firewall

Activating cluster service

Let’s make sure that the cluster service are started each time the server is started.

Activate cluster services

We now we can start the cluster services manually or reboot the server.

Start cluster services

Starting cluster configuration

There are two tools you can use to configure and maintain your cluster. The first one is a web interface called “Conga” that require the installation of a agent on each node (ricci) and a centralize configuration center named (lucy).  If you want to use the web interface (Seem to be working a lot better now),  it is advisable to install the configuration center on a Linux server outside of the cluster. This was fairly new when I first created my first cluster and we decided to used the second interface name “Cluster Configuration”.  This tool can be started from any node within the cluster and does not require to install any additional software. To start the cluster configuration GUI, type the following command using the “root” account ;

Start cluster GUI from the command line

new_cluster_config_warning

The first time you run the cluster configuration GUI, a warning message may display.  It is just informing us that the cluster configuration file “/etc/cluster/cluster.conf” was not found. Click on the “Create New Configuration” button and let’s move on.

Cluster general setting

• Next we need to enter the name of our cluster, we have choose to name it “our_cluster”.  The name of a cluster cannot be change. The only way to change the name of the cluster is to create a new one, so choose it wisely.
• We use the recommend Lock Manager (Dynamic Lock Manager), the GULM is depreciated.
• The multicast address we used for the heartbeat will be “239.1.1.1”.
• The usage of the “Quorum Disk” is outside the scope of this article. But basically, you define a small disk on the SAN that is shared among the nodes in the cluster and node status are regularly written to that disk. If a node haven’t updated its status for a period of time, it will be considered down and the cluster will then fence that node. If you are interested to use the “Quorum Disk” there is an article here that describe how to set it up.

Cluster Properties

Cluster Properties

Now let’s check some of the default setting given to our cluster. Click on “Cluster” on the left hand side of the screen and then click on the “Edit Cluster Properties”.

The Configuration Version value is by default to 1 and automatically incremented each time you modify your cluster configuration.

The Post-Join Delay parameter is the number of seconds the fence daemon (fenced) waits before fencing a node after the node joins the fence domain. The Post-Join Delay default value is 3. A typical setting for Post-Join Delay is between 20 and 30 seconds, but can vary according to cluster and network performance.

The Post-Fail Delay parameter is the number of seconds the fence daemon (fenced) waits before fencing a node (a member of the fence domain) after the node has failed. The Post-Fail Delay default value is 0. Its value may be varied to suit cluster and network performance. CHANGE THE “POST FAIL DELAY” to 30 Seconds.

Adding our nodes to cluster

Insert our first node "bilbo"

To add node into the cluster let’s select “Cluster Nodes” on the upper left side of the screen and click on the “Add a Cluster Node” button. You will be then presented the “Node Properties” screen.  Enter the node name, the “Quorum Votes” and the name of the interface used for the “Multicast” (Heartbeat), in our case it is eth0. The avoid problem, I always used the first interface for the network cluster heartbeat.

• Enter the name of the host name used for the heartbeat, for our first node it will be “hbbilbo.maison.ca. Remember that this name MUST be defined in our hosts file and in your DNS (if you have one).

• Quorum is a voting algorithm used by the cluster manager. We say a cluster has ‘quorum’ if a majority of nodes are alive, communicating, and agree on the active cluster members. So in a thirteen-node cluster, quorum is only reached if seven or more nodes are communicating. If the seventh node dies, the cluster loses quorum and can no longer function. For our cluster we will leave the Quorum Votes to the default value of 1.

If we would have a two-node clusters, we would need to make a special exception to the quorum rules. There is a special setting “two_node” in the /etc/cluster.conf file that looks like this:  <cman expected_votes=”1″ two_node=”1″/>

Repeat operation for every node you want to include in the cluster

Insert "gandalf" node in our cluster

Insert "gollum" node in our cluster

This article was getting quite long, so I decided to stop here, and continue in a another article.

We are almost there stay tune, the third one will be posted soon.

Part 1 – Creating a Linux ReadHat/CentOS cluster

Part 2 – Creating a Linux ReadHat/CentOS cluster

Part 3 – Creating a Linux ReadHat/CentOS cluster

Part 4 – Creating a Linux ReadHat/CentOS cluster

Part 5 – Creating a Linux ReadHat/CentOS cluster

The cluster hardware

Our cluster will have 3 HP servers, they will all have 4GB of memory, 36GB of mirrored internal disks, one Qlogic fiber card connected to our SAN system, 2 network cards and for our fencing device we will use the on-board HP ILO (Integrated Light Out). This is my setup, yours does not need to be the same.  You do not need to use HP server to build a cluster, you do not need to have mirrored disks (although recommended) and you do not need to have to use a SAN infrastructure either (NFS share can also be used). One thing I would recommend is a fencing device, on the HP server there is a network port on the back of each server called “ILO”.  This will allow the cluster software to power on, power off or restart a server remotely. Red Hat cluster package allow you to use a lot of similar fencing devices. This part of the cluster is important because this will prevent at one point some nodes in the cluster to write to a non-shareable file-system and create data corruption. If you do not have a fencing device, you can always use the manual fencing method it works, but it not supported.

The Setup

Our cluster will be contains 3 servers, we will have 2 actives servers and one passive. For our example, we will have one server running an HTTP service, the second server will be an FTP server. The third server will be use as a fail-over server, if the first or second server have network, SAN, or hardware problem, the service it is running will move the third server.

Although not required, having a passive server offer some advantage, first it make sure that if one server have problem the passive server will be able to handle the load of any of your server. If we did  not have that passive server we would need to make sure that either of the server would be capable of handling the load of the two servers on one server.

Clustering environment offer some other advantage when time come to do hardware update on a server. Let say we need to add memory to the first server, we could move the HTTP service from one server to the passive node, add memory to the first server and then move back to the service on the original node when ready. Another advantage of having a passive server is that you can update the OS of your node one by one without affecting the service (if reboot is necessary).

So the name of our node will be “bilbo” the will host the http service, “gollum” the will host the FTP service and “gandalf” will be our passive node.

Click on image to view it enlarge.

As you can see above, each server use 3 network cards.

The first NIC (eth0), is use to offer the service HTTP and FTP to the users and it is the host main network card.

The second network card (eth1) is used by cluster software for his heartbeat. In a corporate environment, this network should be isolated from the rest of your network. There will be a lot of  broadcast on it and the response time is important.

The ILO network will be used by the cluster software to remotely poweron/poweroff server in our cluster, when a problem arise.

So out /etc/hosts file look like this :

#
127.0.0.1       localhost.localdomain   localhost
#
# Host Real IP
192.168.1.103   gandalf.maison.ca       gandalf
192.168.1.104   gollum.maison.ca        gollum
192.168.1.111   bilbo.maison.ca         bilbo
#
# Service Virtual IP
192.168.1.204   ftp.maison.ca           ftp
192.168.1.211   www.maison.ca           www
#
# HeartBeat IP Address
10.10.10.103    hbgandalf.maison.ca     hbgandalf
10.10.10.104    hbgollum.maison.ca      hbgollum
10.10.10.111    hbbilbo.maison.ca       hbbilbo
#
# HP ILO IP Address
192.168.1.3     ilo_gandalf.maison.ca   ilo_gandalf
192.168.1.4     ilo_gollum.maison.ca    ilo_gollum
192.168.1.11    ilo_bilbo.maison.ca     ilo_bilbo
#

We should define all the ip address that our cluster need to function properly in your /etc/hosts. Of course they should also be define in your DNS, but in case your DNS does not respond, the /etc/hosts will assure you that the cluster will continue to work properly. We will use the domain name “maison.ca¨ through out these articles.

In our second article, we will be installing the cluster software and begin the configuration of the cluster.

Part 1 – Creating a Linux ReadHat/CentOS cluster

Part 2 – Creating a Linux ReadHat/CentOS cluster

Part 3 – Creating a Linux ReadHat/CentOS cluster

Part 4 – Creating a Linux ReadHat/CentOS cluster

Part 5 – Creating a Linux ReadHat/CentOS cluster