RIMServ

How To Backup LDOM Configurations

noreply@blogger.com (Abhijeet Medhekar) — Tue, 22 Feb 2011 20:03:00 +0000

A system administrator must backup the LDOM configuration before rebooting the box. This is for those situations wherein later you feel - "If only I had taken the backup of the configuration files ... bla bla!" Now, there are number of commands that should help you. I have a condensed form, in the name of the script - backupldom.ksh

#!/bin/ksh

# This script creates XML backup files of all configured domains
# and saves an SP config
# export Variables
HOST=`hostname`
SCR=`basename $0`
LOG=/tmp/$SCR.`date +%Y+%m.log`
PATH=/opt/SUNWldm/bin:/bin:/usr/sbin/:/sbin
BDIR=/var/adm/doc
ID=`/usr/ucb/whoami`

usage(){
echo "You must be root to run this script"
}

export_config(){
if [[ ! -d /var/adm/doc ]];then
mkdir -p /var/adm/doc
fi

echo "Moving current XML files in $BDIR to 'filename'.backup"
for XML in `ls $BDIR/*.xml`
do
mv $XML $XML.backup
done
echo ""

for LDOM in `ldm ls |grep -v ^NAME| awk '{print $1}'`
do
if [[ $LDOM = primary ]];
then
echo "Backing up LDOM $LDOM to $BDIR as $LDOM-$HOST.xml"
ldm ls-constraints -x $LDOM > $BDIR/$LDOM-$HOST.xml
else
echo "Backing up LDOM $LDOM to $BDIR as $LDOM.xml"
ldm ls-constraints -x $LDOM > $BDIR/$LDOM.xml
fi
done

echo "Saving extended information 'ldm ls -e -p' to ldm-extended.out"
mv $BDIR/ldm-extended.out $BDIR/ldm-extended.out.backup
ldm ls -e -p > $BDIR/ldm-extended.out

echo ""
echo "-------------------------"
echo "Backup of LDOM's complete"
echo "If this is a cluster environment, ensure you scp the guest LDOM"
echo "XML files to the adoptive nodes"
echo "-------------------------"
echo ""
}

save_sp_config(){
echo ""
echo "We will now backup the LDOM Configuration to the Service Processor"
echo ""
echo "Current list of the saved Service Processor configs:"
ldm ls-spconfig
echo ""
echo "Is there an existing configuration you want to over-write?[Y|N]"
read overwrite
# if [ $overwrite = "Y" -o $overwrite = "y" ];then
if [[ $overwrite = "Y" || $overwrite = "y" ]];then
echo "Enter the name of config you want to overwrite:"
read overwrite_config
echo "Removing/adding the config...."
ldm rm-spconfig $overwrite_config
ldm add-spconfig $overwrite_config
echo ""
echo "The current SP configurations are:"
ldm ls-spconfig
echo ""
else
echo "OK, we will add a new config, enter the name of the config:[prod]"
read new_config
if [[ -z $new_config ]];then
export new_config=prod
fi
echo "Adding new config $new_config to the SP"
ldm add-spconfig $new_config
echo ""
echo "The current SP configs are:"
ldm ls-spconfig
echo ""
fi
}

# MAIN
if [[ $ID != root ]];then
usage && exit 1
fi
export_config
sleep 5
save_sp_config

Thanks to Kevin for this useful script. Quite handy when you wish to back reference your configuration or have a quick recovery.

How To Recover From A Hung Guest LDOM State

noreply@blogger.com (Abhijeet Medhekar) — Mon, 21 Feb 2011 17:05:00 +0000

Logical Domains or LDOMs was a step in the right direction toward achieving hardware partitioning and in turn virtualization. However, it is not uncommon to find the guest LDOMs in hung state which end up having dependency with the Control Domain. Take for example the following situation -

# ldm list
NAME       STATE      FLAGS   CONS    VCPU MEMORY   UTIL UPTIME
primary    active     -n-cv- SP      8     4G       3.7% 443d 8h
ldom011    active     -n---- 5008    16    16G      100% 115d 11h
ldom012    active     -n---- 5001    16    16G      8.4% 40d 17m
ldom013    active     -n---- 5002    16    16G       33% 39d 23h 41m
ldom014    active     -n---- 5000    8     4G       0.9% 492d 9h
ldom015    active     -t---- 5003    24    24G      0.0% 0s
ldom016    active     -n---- 5004    8     8G       100% 151d 13h
ldom017    active     -n---- 5005    8     8G       100% 153d 7h
ldom018    active     -n---- 5006    8     8G       100% 157d 7h
ldom019    active     -n---- 5007    8     8G       100% 153d 8h

Most of the LDOMs herein are not reachable even via console -

# telnet localhost 5003
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldom015" in group "ldom015" ....
Press ~? for control options ..

telnet> send brk

telnet> q
Connection to localhost closed.

You will observe the guest LDOMs are out of administrative control. Sending Break signal doesn't work either. And attempts to stop / unbind the guest domain result in the following unsuccessful attempts -

root@controldomain
/root
# ldm stop -f ldom015
LDom ldom015 stop notification failed

root@controldomain
/root
# ldm unbind ldom015
LDom ldom015 cannot be unbound because it is active

Under such circumstances and I speak with vendor recommendation that you are left with the only option to reboot the control domain. Magically, after the reboot all the servers come up without any complications. This may be happening very regularly in your environment and the best way out to resolve your problem permanently is to upgrade the LDOM software version from 1.1 to 1.2. For this is a known bug documented with Oracle knowledge database. Checkout Upgrading To Logical Domains 1.2 Software

ErrorCode: Command not found

noreply@blogger.com (Abhijeet Medhekar) — Fri, 04 Feb 2011 13:05:00 +0000

Category*:	UNIX
Error Message*:	Command not found
Elaborate*:	Hi There, First of all, I really like your site. I am an aspiring UNIX system administrator and for some reason I am not able to execute few very obvious commands. I am not sure if I am confusing between different versions of UNIX like AIX, Solaris or Linux command specifics. But whenever I execute these commands it simply gives an error - "Command not found". Can you please help me with the same?

What To Do When Veritas Explorer Hangs

noreply@blogger.com (Abhijeet Medhekar) — Thu, 03 Feb 2011 14:24:00 +0000

Veritas Explorer is the executable package that collects all the relevant information a symantec engineer needs to know the current status of the system and do necessary troubleshooting. The VRTSexplorer is usually executed from the location -

# /opt/VRTSspt/VRTSexplorer/VRTSexplorer

However, ideally you raise a call with the vendor only when you need to do break-fix. So it is very common to find that the data collection tool VRTSexplorer in a hung state. What can be done now?

The solution is simple!

Identify the rock bottom process in the process tree that is hanging a give it a "pill of death" by "KILL -9" signal.

Here are the step by step execution details, for your execution and reference -

1. Identify the VRTSexplorer process id. The second column herein the output identifies the process id.

# ps -aef | egrep -i VRTSexplorer | egrep -v egrep

Lets say the process id is 1043

2. Then to identify the rock bottom process, use the command -

# ptree 1043

It gives you the hierarchy of child processes with process id running under process id 1043.

3. Identify the last process id within this hierarchy (say it is 1056) and as said before - give it a "pill of death"

# kill -9 1056

Bingo! And you will find that the VRTSexplorer will breath life again!

How To Implement Sun Trunking And Achieve Fast Data Transfer Rate

noreply@blogger.com (Abhijeet Medhekar) — Mon, 31 Jan 2011 15:15:00 +0000

The sun trunking software 1.3 helps to aggregate ethernet adapters in solaris to provide higher aggregated data transfer rates. I am describing here how exactly one can implement it on their servers. It is indeed an amazing concept giving you extra ordinary speed for data transfer within your infrastructure.

Pre-Requisite: SUN Gigaswift Ethernet Adapter drivers should be there ( ce drivers are present in my environment). Download the Sun Trunking Software.

1. Install the SUN Trunking Software through the automated script ( ./install ) provided in SUN Zipped Trunking Package - Sun_Trunking_1[1].3_s10_2.zip ( This will install the necessary patches for a successful installation ) .

2. Make sure that the Link mode is set to run Full Duplex. Also the settings need to be changed at the Switch Level to support Trunking .

At the LINK mode :

# ndd -set /dev/ce instance ( instance number )
# ndd -set /dev/ce adv_100fdx_cap 1
# ndd -set /dev/ce adv_100hdx_cap 0
# ndd -set /dev/ce adv_autoneg_cap 0

3. Unplumb the Interface before configuring Sun Trunking .

4. Setup the Trunk and Plumb the interface. ( Note : We need to figure out the policy here )

# nettr -setup 10 device=ce members=0,1 policy=1

5 . Put the entries into /etc/opt/SUNWconn/bin/nettr.sh file and check the configuration after a successful implementation through nettr –conf command . This is to preserve the settings across reboot.

# vi /etc/opt/SUNWconn/bin/nettr.sh
nettr -setup 10 device=ce members=0,1 policy=1

6. Plumb the interface

# ifconfig ce0 inet [ IP address ][ Netmask ] [ Broadcast ] plumb up

7. Create the /etc/hostname.driver# (where # corresponds to the instance number used as the trunk head i.e. – ce0 ) file and add the entry into /etc/hosts file.

8. Finally, add routes as applicable to your network.

# route add net 20.254.52.112/28 20.254.52.110

Note1: Only one interface, i.e. ce0 will be plumbed. The other interface that is trunked will remain unplumbed but rest assured that it will be working beneath and providing high aggregated speed.

Note2: Use DLADM SHOW-DEV to observe the effects!!

How To Find Quickly The Status Of All The Servers In A Data Center

noreply@blogger.com (Abhijeet Medhekar) — Sun, 30 Jan 2011 03:47:00 +0000

Complete data center shutdown is an yearly planned event in most organizations. It gives the organization a nice idea about the disaster recovery procedures it has in place and an opportunity to follow up with the tech teams for re-mediating delinquencies. Power outages, DR testing, SAN offline events, power supply replacement, microcode upgrade (if you don't have multipathing), etc are some of the other events that require outages on a large number of servers. The servers usually come online once the power is restored. To ensure the same one must set the OBP parameter -

ok> setenv auto-boot? true
# eeprom auto-boot?=true

When the servers are coming online, a system administrator is in a great time crunch. It is very plausible that many servers may not come online smoothly. It becomes very difficult for sys admins to figure out where should they be channelizing their energies and how to find the defects in first place when you have 5000+ servers booting up simultaneously with great management pressure. Well, here is a small PING script to your rescue. This must be executed from a command server that has access routes to all the servers in the Data Center.

# cat findstatus.sh

# Purpose:      To identify servers online/offline
# Usage:        During mega DR, SAN Migration, Power Down Events
# Requirements: Use the "list" file to mention the server names/line

# Clean And Initialize Temporary Files

cat /dev/null > down
cat /dev/null > up

# Run Main Program Logic

for i in `cat list`
do
        echo ""
        echo "Checking For $i"
        ping $i 2
        if [ $? -ne 0 ]
        then
                echo $i >> down
        else
                echo $i >> up
        fi
done

echo ""
echo "###################################"

# Calculate Numbers

ntotal=`cat list | wc -l`
nup=`cat up | wc -l`
ndown=`cat down | wc -l`

# Reporting Section

echo "Total Servers Checked:\t$ntotal"
echo "###################################"
echo "Total Servers Up:\t$nup"
cat up
echo "###################################"
echo "Total Servers Down:\t$ndown"
cat down
echo "###################################"

Sample LIST file requried for the script execution is as follows and it contains the FQDN of all the servers of whom you wish to check the status -

# cat list
linux_server_web_1
linux_server_db_2
solaris_server_application_1
solaris_server_web_2
hp-ux_server_db_1
aix_server_db_1

Here is how you can use the script and the relevant sample output -

# ./findstatus.sh

Checking For linux_server_web_1
linux_server_web_1 is alive

Checking For aix_server_db_1
aix_server_db_1 is alive

...

...

###################################
Total Servers Checked:        6
###################################
Total Servers Up:             6
linux_server_web_1
linux_server_db_2
solaris_server_application_1
solaris_server_web_2
hp-ux_server_db_1
aix_server_db_1
###################################
Total Servers Down:            0
###################################

Note: If the servers are down, such a script will usually run very slowly. The reason that makes the script faster in execution is the command usage -

ping $i 2

which sets a default npackets to be sent across as only 2. This makes the execution very fast and is good enough a try- to figure out the server statuses. Anyhow, this is only the initial step of the troubleshooting!

How To Increase A Cluster File System

noreply@blogger.com (Abhijeet Medhekar) — Wed, 26 Jan 2011 18:57:00 +0000

It's all too simple with VXRESIZE when you have a VXFS. However, the question here is how to increase a CFS, i.e. a Cluster File System. The previously discussed concept of Master-Slave Node and Primary for Storage Foundation Cluster File System comes into picture herein. If you would like to brush up the concepts there, check out - How To Identify Master-Slave And Primary-Secondary Cluster Nodes

To begin with learn the process and make it a thumb rule -

Grow the volume from the CVM Master, and then grow the File System from any SF CFS node. Please note that the CVM Master and the SF CFS node can be two different nodes.

Figure out the primary of the cluster file system by -

# fsclustadm -v showprimary mount_point

Figure out the master CVM node in the cluster by-

# vxdctl -c mode

To increase the size of the file system, execute the following on CVM master -

# vxassist –g shared_disk_group growto volume_name newlength

And then on SF CFS node, execute-

# fsadm –F vxfs –b newsize –r device_name mount_point

That is it! The results will be replicated to all the cluster nodes. It may require a "VXDCTL ENABLE" on other nodes, but should be fair and square. Enjoy the new space!

How To Ensure CRS Has Stopped Completely

noreply@blogger.com (Abhijeet Medhekar) — Tue, 25 Jan 2011 17:55:00 +0000

Symantec recommends to place the CRS cluster as an application resource with start, stop, monitor and clean scripts- at the top of the dependency matrix within a veritas cluster. It is advisable to bring the cluster down before the server is being shut down. Also, you must ensure that you bring down the clustered environment always with the command -

# shutdown -y -i5 -g0

This is a safe way to bring all the services down systematically. It may be time consuming, but you become more confident about a smooth start up sequence when the server boots up. REBOOT is a dangerous way to do it which simply kills the running processes to ensure that the server has been brought down.

Now, coming to the main point - how can you ensure that the CRS has stopped completely? GABCONFIG is the command that will show you the ports status. This is after you have executed -

# hastop -all

Just follow the sequence below and ensure you have a smooth CRS shutdown.

Solaris-Box2 # /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Error while stopping resources. Possible cause: CRSD is down.
Shutdown has begun. The daemons should exit soon.

Solaris-Box2 # gabconfig -a
GAB Port Memberships
===========================================
Port a gen   444e08 membership 01
Port b gen   444e0c membership 01
Port d gen   444e07 membership 01
Port f gen   444e17 membership ;1
Port f gen   444e17    visible 0
Port o gen   444e0b membership 01
Port q gen   444e15 membership ;1
Port q gen   444e15    visible 0

Solaris-Box2 # pkill vxfsckd

Solaris-Box2 # gabconfig -a
GAB Port Memberships
===========================================
Port a gen   444e08 membership 01
Port b gen   444e0c membership 01
Port d gen   444e07 membership 01
Port f gen   444e17 membership ;1
Port f gen   444e17    visible 0
Port o gen   444e0b membership 01
Port q gen   444e15 membership ;1
Port q gen   444e15    visible 0

Solaris-Box2 # fsclustadm cfsdeinit

Solaris-Box2 # gabconfig -a
GAB Port Memberships
===========================================
Port a gen   444e08 membership 01
Port b gen   444e0c membership 01
Port d gen   444e07 membership 01
Port o gen   444e0b membership 01
Port q gen   444e15 membership ;1
Port q gen   444e15    visible 0

Solaris-Box2 # pkill qlogckd

Solaris-Box2 # gabconfig -a
GAB Port Memberships
===========================================
Port a gen   444e08 membership 01
Port b gen   444e0c membership 01
Port d gen   444e07 membership 01
Port o gen   444e0b membership 01
Port q gen   444e15 membership ;1
Port q gen   444e15    visible 0

Solaris-Box2 # qlogclustadm deinit

Solaris-Box2 # gabconfig -a
GAB Port Memberships
===========================================
Port a gen   444e08 membership 01
Port b gen   444e0c membership 01
Port d gen   444e07 membership 01
Port o gen   444e0b membership 01

Phew! Finally, CRS stopped completely!

ErrorCode: VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible

noreply@blogger.com (Abhijeet Medhekar) — Mon, 24 Jan 2011 18:40:00 +0000

Category*:	Veritas Volume Manager
Error Message*:	VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible
Elaborate*:	I just installed Veritas Storage Foundation 5.0 flawlessly. It was without a doubt a clean install. There were no warning messages and errors. But once the installation has completed, I am unable to run any of the veritas commands. I don't think I have missed any step, but then it may be a small thing that has just slipped my mind. Can you please help me resolve it without logging a symantec case?

How To Start HACMP Cluster In AIX

noreply@blogger.com (Abhijeet Medhekar) — Mon, 24 Jan 2011 12:05:00 +0000

This is all about a simple situation when you have a perfectly configured HACMP cluster. No troubleshooting, not a single break fix- just plain old cluster start up sequence - and that is only what I intend to elaborate herein. The advantage of AIX is that things are very much menu based and the key to that menu is - smitty!! All selected options within the menu are marked by "<---"

# smitty hacmp
~~~~~~~~~~~~~~~~~~~~~~~~~
Initialization and Standard Configuration
Extended Configuration
System Management (C-SPOC) <---
Problem Determination Tools
~~~~~~~~~~~~~~~~~~~~~~~~~
HACMP Communication Interface Management <---
HACMP Resource Group and Application Management
HACMP Log Viewing and Management
HACMP File Collection Management
HACMP Security and Users Management
HACMP Logical Volume Management
HACMP Concurrent Logical Volume Management
HACMP Physical Volume Management
Configure GPFS
Open a SMIT Session on a Node
~~~~~~~~~~~~~~~~~~~~~~~~~
Manage HACMP Services
Start Cluster Services <---
Stop Cluster Services
Show Cluster Services
~~~~~~~~~~~~~~~~~~~~~~~~~
* Start now, on system restart or both [now+]
Start Cluster Services on these nodes[AIX_Box_1]+
* Manage Resource Groups [Automatically+]
BROADCAST message at startup? [true+]
Startup Cluster Information Daemon? [true+]
Ignore verification errors?[ false+]
Automatically correct errors found during [Interactively+]
cluster start? <---
~~~~~~~~~~~~~~~~~~~~~~~~~

And then you may see loads of warning messages, but it is alright as long as the messages on the console are not errors. Shortly, you should be seeing your HACMP cluster up and running!

How To Identify Master-Slave And Primary-Secondary Cluster Nodes

noreply@blogger.com (Abhijeet Medhekar) — Sun, 23 Jan 2011 13:45:00 +0000

The concept of Master-Slave and Primary-Secondary comes into picture when the environment is Active-Active with parallel clusters. It is implicit that the clustered environment will have clustered file systems that are mounted parallel on multiple cluster nodes. It is very easy to confuse among the terminology and usage. So here is a brief explanation for all future use -

# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: UNIX-Node-1
=> Shows master and slave cluster nodes

# fsclustadm -v showprimary /cfs/mount/point
UNIX-Node-1
=> Shows primary and implicitly secondary for a given cfs filesystem

The Master-Slave concept can be used when you wish to fire veritas cluster commands. Note that they must be fired from the Master cluster node.

The Primary-Secondary concept can be used when you wish to fire veritas configuration commands that are specific to the cluster file system in question. Like increasing the cluster file system size, decreasing the same, etc. This must be fired from Primary node for the given cluster file system.

Now can you tell me "How To Change Master-Slave And Primary-Secondary Cluster Nodes"??

ErrorCode: No directory!

noreply@blogger.com (Abhijeet Medhekar) — Sat, 22 Jan 2011 17:27:00 +0000

Category*:	UNIX
Error Message*:	No directory!
Elaborate*:	I get this message every time I try to login in one of our production Solaris box. I am sure that my user id and directory is created within the server.

FSCK To Your Rescue Against File System Inconsistencies

noreply@blogger.com (Abhijeet Medhekar) — Sat, 22 Jan 2011 15:00:00 +0000

It is very common to have inconsistent data in your old file systems. This usually surfaces when you are trying to mount a file system over a mount point. Take this instance for example-

# mount -F vxfs /dev/vx/dsk/diskgroup_1/volume_1 /mount_point_1
UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/diskgroup_1/volume_1 is corrupted. needs checking

Remember, FSCK is your best friend for not only such file system checks, but also for on the fly corrections.

# fsck -F vxfs /dev/vx/rdsk/diskgroup_1/volume_1
log replay in progress
replay complete - marking super-block as CLEAN

FSCK is usually run on the raw disk and that implies "rdsk" in the volume path. Please note that it is important to specify the type of file system with the -F option which can be ufs, vxfs, etc.

So for UFS file system the command becomes-

# fsck -F ufs /dev/md/rdsk/d30

Before, correcting any block level, file level, directory level or inode level consistency, the utility generally asks for a confirmation. And if there are lots of such inconsistencies, it usually becomes frustrating for a system administrator to give the same input of "y" again and again. Here is a way out which says to go "y" by default!!

# fsck -ofy /dev/rdsk/c1t0d0s0

Essential Integrity Check During Solaris Services Startup

noreply@blogger.com (Abhijeet Medhekar) — Fri, 21 Jan 2011 15:27:00 +0000

SVC.CONFIGD is the repository daemon responsible for maintaining /etc/svc/repository.db. The repository.db must come clean during this integrity check otherwise it is a "no go" for usual boot sequence to run level 3. The repository may get corrupted due to various hardware issues, software bugs, disk write failures, etc. A corrupted /etc/svc/repository.db displays the following message on the console -

svc.configd: smf(5) database integrity check of:

/etc/svc/repository.db

failed. The database might be damaged or a media error might have
prevented it from being verified. Additional information useful to
your service provider is in:

/etc/svc/volatile/db_errors

The system will not be able to boot until you have restored a working
database. svc.startd(1M) will provide a sulogin(1M) prompt for recovery
purposes. The command:

/lib/svc/bin/restore_repository

can be run to restore a backup version of your repository. See
http://sun.com/msg/SMF-8000-MY for more information.

Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
svc.configd exited with status 102 (database initialization failure)

Since, the file plays such an important role and is precisely responsible for starting the overall system services Solaris system always keeps a default copy of the repository.
Login to the system in single user mode -

Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Off course to resolve the problem you just need to follow the instructions and execute the script -

/lib/svc/bin/restore_repository

Difference Between Command Output Of WHOAMI And WHO AM I

noreply@blogger.com (Abhijeet Medhekar) — Fri, 21 Jan 2011 13:29:00 +0000

One of the basic level 1 interview question to a solaris administrator is - what is the difference between the commands "# who am i" and "# whoami". I won't go into explaining the difference but would love to show it practically. Here are the difference in output-

solserv $ who am i
testuser pts/1 Jul 9 12:55 (mgmtserver.in.client.com.au)
solserv $ whoami
testuser
solserv $ sudo su -
Password:
Last login: Fri Jul 9 2010 12:55:43 from pts/1
solserv $ who am i
testuser pts/1 Jul 9 12:55 (mgmtserver.in.client.com.au)
solserv $ whoami
root

Hopefully, it is crystal clear

Data Corruption When You Loose SCSI 3 Persistent Reservation

noreply@blogger.com (Abhijeet Medhekar) — Thu, 20 Jan 2011 13:44:00 +0000

Troubleshooting is the name of the game. The simple error below can malfunction clustered production environments - big time. And I wish to share the details of the same -

Apr 02 11:48:28 Solaris emcp: [ID 801593 kern.notice] Error: MpxRegister: PRO service action 46 failed, 0x5/0x24/0x0

EMC powerpath records the error under the following circumstances-

"Sun Cluster Error: MpRegister: PRO service action 6 failed, 0x5/0x24/0x0"
ID: emc144905
Usage: 5
Date Created: 09/24/2006
Last Modified: 06/26/2009
STATUS: Approved
Audience: Customer

Knowledgebase Solution

Environment: EMC SW: PowerPath 4.1
Environment: EMC Product: Symmetrix DMX Family
Environment: Application SW: Sun Cluster 3.1
Environment: OS: Solaris
Problem: Error: MpRegister: PRO service action 6 failed, 0x5/0x24/0x0
Problem: 591F errors logging on the Symmetrix, streaming on each FA attached to the host.
Root Cause: In this particular instance, the PER bit setting of the devices in question had been removed by the customer, and the host could not set the SCSI reservation.
Fix: Set the SCSI3 Persistent Reservation bit. The procedure is documented in primus solution emc92950.
Notes: This error is reported when there is a problem with the Persistent Reserve Out command (PRO). This indicates a SCSI reservation issue.

Please note that this cannot be omitted. VCS highly banks upon SCSI3 Persistent Disks for IO Fencing which is the basis for protection from data corruption in a clustered environment. If you loose this characteristic you are potentially inviting data corruption of your clustered environment. The same can be viewed from Veritas Cluster Logs in the form of below warning messages-

Apr 02 12:48:28 Solaris vxfs: [ID 702911 kern.warning] WARNING: msgcnt 1 mesg 038: V-2-38: vx_dataioerr - /dev/vx/dsk/ora__dg/u900 file system file data write error in dev/block 0/41473032

And that will bring the VCS resources like application, DB and UNIX filesystems crashing down!

How To Know Whether A Port Is Open Or Closed From The Client Side

noreply@blogger.com (Abhijeet Medhekar) — Fri, 14 Jan 2011 16:34:00 +0000

In an infrastructure production environment, almost always a UNIX server lies behind a firewall. The firewall generally ensures which authorized services should be provided and manages the same via opening / closing the ports for a particular service. To know a mapping of the services with the ports you can use the command -

# cat /etc/services

But a system administrator is always put on the hook to identify whether a given port is open or closed? This can only be checked from the client end and here is the trick for you -

# telnet master_server 13782
Trying xxx.xxx.xxx.xxx...
Connected to master_server.linux.com
Escape character is '^]' .

Connection to master_server.linux.com closed by foreign host.

You can also use where the service is associated with the port in the /etc/service file

# telnet master_server bpcd
Trying xxx.xxx.xxx.xxx...

Note that the connection is immediate and won't take more than 5 seconds at most. So if you get a connected response consider that the connection is through and the port is open. If you do not get one get your firewall administrator on the hook!

How To Ensure A Given Application User Never Logs Into The UNIX System

noreply@blogger.com (Abhijeet Medhekar) — Thu, 13 Jan 2011 16:16:00 +0000

Let's accept it there is a never ending war among UNIX system administrators, network engineers, DBAs, and application guys. Let's say there is a interfering application user who should only be bothered about his application and application login but has developed sizable interest in how UNIX works at the backend of his/her application. You are a busy system administrator and you don't like so much interference - not only for the sake of it but also from the security point of view. You also cannot delete this user for the application usage is linked with the UNIX user created. But you don't want to let him/her login to the UNIX box. What do you do?

There are some who will suggest you use /bin/noshell or /bin/false. However, I have witnessed that this does not work for all operating environment. The best and the smartest way to do it is as follows-

# su - application-user
# echo "exit" >> .profile
# exit

It's funny but I really won't like to explain that!! If you are system administrator you know how frustrating you can make them feel. And if you are reading this and not - I just hope you are not an application user ;-)

How To Find The Network ID for /ETC/NETMASK

noreply@blogger.com (Abhijeet Medhekar) — Wed, 12 Jan 2011 07:37:00 +0000

There are 3 files that help in persistently configuring a network IP in a Solaris operating system, namely-

1. /etc/hosts
2. /etc/netmask
3. /etc/hostname.<interface>

When you get in touch with network architects they shall generally tell you about the IP and netmask to be configured for a new server to be built. To have any network conversions to hexadecimal format use - Netmask Chart. However, there is a field in /ETC/NETMASK known as Network ID. This can be calculated the harder way using all those crosses and 0 and 1 or you can use the simple way of using Online Subnet Calculator

And the question still remains, where is the Network ID? Well, the Subnet ID that comes out here after you have chosen the IP address and netmak, is nothing but your required Network ID. Use the same in the /ETC/NETMASK and you should be fine with your configuration.

However, please note that /ETC/NETMASK can have multiple fields if there are multiple networks connected to your given system. How do you configure networks in such environments?

How To Send Mail From A UNIX OS Command Line

noreply@blogger.com (Abhijeet Medhekar) — Tue, 11 Jan 2011 15:02:00 +0000

I would really love to keep this smart and simple. You can use a simple command and if everything is configured right; you should be able send across a mail using this simple command and its options. The command is MAILX.

The command options are more important and must be brushed first -

-s Mail Subject
-r From Address
-a Mail Attachment

The command syntax is as follows -

# mailx -s "Subject" -r "FromAddress" -a "AttachmentFilePath" "ToAddress"

Hello!! You must be wondering!! Where is the mail body? Well the beauty lies in the details for here is the command usage-

# mailx -s "How To Send Mail From A UNIX OS From Command Line" admin@rimserv.com
This post contains all the details with options to help you send a simple mail using the command MAILX. Hope you succeed in your first attempt. Unless things aren't configured on your client or if proper routes aren't there - Admin@RIMServ should be able to receive this mail.
Best Regards,
Abhijeet
CTRL + D
#

That's it!!

How To Temporarily Allocate IP To A Local Zone From A Global Zone

noreply@blogger.com (Abhijeet Medhekar) — Mon, 10 Jan 2011 11:15:00 +0000

One way of allocating IP to a local zone is via ZONECFG. It is a permanent way of doing it and every time your local zone restarts the IP shall be persistent. However, there are many situations in development and UAT environments that requires a temporary IP to be allocated to the local zone on the fly and without a reboot.

The secret lies in using the same old IFCONFIG command with a smart keyword ZONE. Execute it from the global zone and see how smoothly it gets a temporary IP allocated to your local zone.

# ifconfig ce0:2 plumb 192.168.1.3 netmask + broadcast + up
# ifconfig ce0:2 zone localzone1

Execute the following to see the same NIC ce0:2 with the keyword 'zone localzone1" allocated-

# ifconfig -a

This can also be very useful to ensure network routes are properly configured or testing and ensuring how a new application IP will function without actually restarting the application or renewing the existing IP.

How To Import A Missing Disk In Veritas Storage Foundation

noreply@blogger.com (Abhijeet Medhekar) — Sun, 09 Jan 2011 05:35:00 +0000

I had a recent veritas explorer which clearly showed me a disk by the name c9t4d9s2. The disk was there but now when I execute VXDISK LIST, it seems that the disk has simply vanished in thin air. Although the disk group (orabackup_dr_dg) was running perfectly fine in the last healthcheck reports. A brief investigation of the /var/adm/message logs show-

Sep 20 15:10:06 Solaris-Box1 vxvm:vxconfigd: [ID 935587 daemon.warning] V-5-1-8222 slave: missing disk 1182007918.201.Solaris-Box1
Sep 20 15:10:06 Solaris-Box1 vxvm:vxconfigd: [ID 918256 daemon.warning] V-5-1-7830 cannot find disk 1182007918.201.Solaris-Box1

So lets start the investigation. I checked the status from the other cluster node where the disk group is supposed to be shared and mapped-

Solaris-Box2 # vxdisk -s list | grep -i 1182007918.201.Solaris-Box1
diskid: 1182007918.201.Solaris-Box1

Great! So the disk was last imported on Solaris-Box1.

As expected, I couldn't import the disk group. Without the missing disk it was bound to fail. This may occur for various reasons - like write errors, failing disk overlooked and failing flag turned off, recovery after existing split brain, whatever may be the reason - the current status is that we have a missing disk.

Since it is a missing disk, I do not have any abilities to import the disk via veritas disk media name or disk access name. All veritas disk commands simply revert with non-existent device. However, VXDG has got one such option to import a disk from the disk ID name you have. My previous records show me the Disk ID and so do the error logs.

Solaris-Box2 # vxdisk -s list | grep -i 1182007918.201.Solaris-Box1
diskid: 1182007918.201.Solaris-Box1

And The Missing Disk -
Disk:   c9t4d9s2
type:   auto
flags: online ready private autoconfig shared autoimport imported
diskid: 1182007918.201.Solaris-Box1
dgname: orabackup_dr_dg
dgid:   1183404627.90.Solaris-Box1
clusterid: Linux-RPS-dr-uat-Cluster
info:   format=cdsdisk,privoffset=256,pubslice=2,privslice=2

The command syntax is-

# vxdg [ -Cfst ] [ -n newname] [ -o clearreserve] [ -o groupreserve=fencekey] [ -o selectcp=diskid] [ -o noreonline] import diskgroup

So my command becomes-

# vxdg -Cf -o selectcp=1182007918.201.Solaris-Box1 import orabackup_dr_dg

Bingo! Disk Group import was a success!!

When Veritas Cluster GAB - Global Membership Atomic Broadcast - Timeouts

noreply@blogger.com (Abhijeet Medhekar) — Sat, 08 Jan 2011 11:55:00 +0000

Veritas Cluster is all about monitoring various cluster resources to ensure that they are ONLINE and if they aren't - take up corrective measure as defined in the cluster configuration. The primary way of identifying whether a resource is ONLINE or OFFLINE is via monitoring scripts and when the scripts timeout, you know that corrective measures like failover have to be kick started from the cluster end.

However, what will happen if your HAD becomes unresponsive.

Apr 8 08:28:11 SolarisUNIX Had[6270]: [ID 702911 daemon.alert] VCS WARNING V-16-1-51047 HAD Self Check: Excessive delay in the HAD heartbeat to GAB (10 seconds)
Apr 8 08:28:15 SolarisUNIX Had[6270]: [ID 702911 daemon.alert] VCS WARNING V-16-1-53034 HAD Signal SIGABRT received

Within a VCS cluster node, GAB and HAD processes regularly heartbeats. By default after (VCS_GAB_TIMEOUT) 15000 milliseconds, if fails to heartbeat, GAB would declare HAD is unresponsive and tries to kill the HAD process. When HAD stops sending heartbeats to GAB, GAB tries to kill the HAD process five times, after which GAB declares that HAD is dead on the affected node and by default triggers intentional PANIC, when the HAD process becomes unresponsive even to kill. If the HAD process can be killed, hashadow daemon would restart the HAD process and perform new cluster membership.

The most common cause of the error is due to high system load that affects the ability of HAD to heartbeat with GAB. The action taken by GAB to kill HAD process succeeds depending upon the severity of system performance affected.It is important to review the system performance data for any possible performance bottleneck in system resource to identify the possible cause.

SSH Comes With Few Internal Security Checks

noreply@blogger.com (Abhijeet Medhekar) — Fri, 07 Jan 2011 12:11:00 +0000

I have discussed before during one my brainstorming sessions about How To Implement Passwordless SSH. However, this never occurred to me that SSH before establishing a session thoroughly checks for permissions. There is an inbuilt security check that ensures that the encrypted key is kept safe and to achieve the same the absolute path of the location where the SSH key is placed is checked out.

Let's say you store your public keys in /var/tmp/unixuser/.ssh/authroized_keys where /var/tmp/unixuser is the home directory of the user - unixuser. Do you think you will be able to establish a clean SSH session. Note the permissions of the directory /var/tmp.

# ls -ld /var/tmp

It shall revert you with 1777 or drwxrwxrwt, with sticky bit set. This should be the same for all standard UNIX based servers. These permissions, i.e. with sticky bit imply that users who do not own the file won't be allowed to delete the file. But is it safe? SSH clearly tells you - it's not!! For anyone can copy your encrypted key in this fashion and use it for their own nefarious deeds. Here is how my attempts were rightly stopped while doing SSH from another server -

sshd: Authentication refused:
bad ownership or modes for directory /tmp

So do figure out a safe place for keeping your SSH keys and home directories must not have any parent directory with permissions 1777 for SSH to work the way you want!

How To Know Your Primary And Secondary SRDF Status

noreply@blogger.com (Abhijeet Medhekar) — Thu, 06 Jan 2011 12:40:00 +0000

SRDF stands for Symmetrix Remote Data Facility. It is a product that achieves similar function as executed by Symantec Product Veritas Volume Replicator (VVR). SRDF belongs to the family of EMC products and is used for replicating data bit by bit on the raw storage arrays from primary production site to secondary disaster recovery site, and in times of need vice versa.

I believe, Veritas Storage Foundation, Veritas Volume Replicator and Veritas Global Cluster much better integrates in comparison to SRDF. SRDF puts an added step and complexity of making the primary and secondary sites ReadWrite / WriteDisabled, as the case may be applicable. Also, this is an added manual step and cannot be managed via Global Cluster.

It is very much required that a system administrator check the status of primary and secondary sites before bringing the cluster online on either of the sites for this missing step could very well lead to data corruption. To check whether your primary (R1) or secondary (R2) site is in WD (WriteDisabled) or RW (ReadWrite) state use the following set of commands-

# /opt/emc/SYMCLI/V6.5.0/bin/symrdf -g MODE_SRDF query

Device Group (DG) Name             : MODE_SRDF
DG's Type                          : RDF2
DG's Symmetrix ID                  : 000190102489    (Microcode Version: 5773)
Remote Symmetrix ID                : 000190106930    (Microcode Version: 5773)
RDF (RA) Group Number              : 30 (1D)

      Target (R2) View                 Source (R1) View     MODES
--------------------------------    ------------------------ ----- ------------
            ST                  LI      ST
Standard      A                   N       A
Logical       T R1 Inv   R2 Inv K       T R1 Inv   R2 Inv       RDF Pair
Device Dev   E Tracks   Tracks S Dev   E Tracks   Tracks MDA   STATE
-------------------------------- -- ------------------------ ----- ------------

DEV001 0D11 WD       0        0 RW 070B RW       0        0 A..   Consistent

Total          -------- --------           -------- --------
Track             0        0                  0        0
MB              0.0      0.0                0.0      0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
D(omino)           : X = Enabled, . = Disabled
A(daptive Copy)    : D = Disk Mode, W = WP Mode, . = ACp off

And only bring the cluster online at the site where you find the state is RW. Ensure that you have kept the cluster Service Groups in AutoDisabled state where you have the site in WD state.