UNIX my first love...

Apache-Subversion deployment - Legacy Solaris 8, different experience!

noreply@blogger.com (Nilesh Joshi) — Thu, 13 Mar 2014 10:34:00 +0000

I got a requirement wherein customer requested to have Apache Subversion setup to be configured on quite old legacy version, Solaris 8 server; hence getting binaries for SVN on Solaris 8 is almost impossible hence I eventually decided to compile and install Subversion & all it's dependent components using source code.

Installing Subversion needs to fulfil some dependencies, below are the required dependent packages to install SVN successfully.

= Expat 2.x
= Apache 2.x with WebDAV support
= APR (Apache Portable Runtime)
= APR-UTIL
= NEON
= Compilation of SQLite-amalgamation C source code for SQLite which is a requirement for Subversion configure script and compilation purpose
= Compilation of zlib

This write-up is quite basic and gives enough insights on what compile string works fine on Solaris 8 and one might save lots of time.. trust me it's a pain to get Subversion compliled on Solaris 8 along with Apache and it's all dependent components.

Compile Expat
=============

# cd expat-2.0.1
#./configure && make && make install

# cd.. 

Compliling Apache 2.x
=====================

#./configure --prefix=/usr/local/apache2 --enable-so \

--enable-mods-shared=most --enable-ssl=static \

--with-ssl=/usr/local/ssl --enable-dav --enable-dav-fs
# make && make install

# cd.. 

Starting Apache Webserver
-------------------------

#/usr/local/apache2/bin/apachectl start

root@XXXX # ps -eaf | grep httpd | grep -v grep
root 24948    1 2 21:35:49 ?        0:02 /usr/local/apache2/bin/httpd -k start
svn 24952 24948 0 21:35:51 ?        0:00 /usr/local/apache2/bin/httpd -k start
svn 24949 24948 0 21:35:51 ?        0:00 /usr/local/apache2/bin/httpd -k start
svn 24950 24948 0 21:35:51 ?        0:00 /usr/local/apache2/bin/httpd -k start
svn 24951 24948 0 21:35:51 ?        0:00 /usr/local/apache2/bin/httpd -k start
svn 24953 24948 0 21:35:51 ?        0:00 /usr/local/apache2/bin/httpd -k start

Apache processes are onwed by user svn, you may use any user for this purpose.

Compile APR
===========

# cd subversion-1.6.6
# cd apr
# ./configure --prefix=/usr/local/apr
# make
# make install
# cd ..

Compile APR-UTIL
================

# cd apr-util
# ./configure --prefix=/usr/local/apr --with-apr=/usr/local/apr
# make
# make install
# cd ..

Compile NEON
============

# cd neon
# ./configure --prefix=/usr/local/neon --with-ssl --with-libs=/usr/local/ssl --enable-shared
# make
# make install
# cd ..

Compile Subversion
==============

# ./configure --prefix=/usr/local/subversion \
--with-apxs=/usr/local/apache2/bin/apxs --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr --with-neon=/usr/local/neon
# make
# make install

If everything success subversion will add 2 modules into your apache modules and added 2 module line into your httpd.conf like below -

# more /usr/local/apache2/conf/httpd.conf
....
LoadModule dav_svn_module /usr/lib/apache2/mod_dav_svn.so
LoadModule authz_svn_module /usr/lib/apache2/mod_authz_svn.so 
...

Restart Your Apache Services
----------------------------------------

# /usr/local/apache2/bin/apachectl stop
# /usr/local/apache2/bin/apachectl start

SVN repository configuration via WebDAV
-------------------------------------------------------

Under HTTP configuration file "/usr/local/apache2/conf/httpd.conf" put below configuration -

---------------------------------------------------------------------

Add users for basic authentication
==========================

# /usr/local/apache2/bin/htpasswd /usr/local/svn/repos/authz njoshi01
New password:
Re-type new password:
Adding password for user njoshi01

The above command can be used for adding user and modifying the password. Now to access the SVN repository user needs to authenticate and then access the repository contents.

--------------------------------------------------

To make sure the httpd service comes online automatically after server reboot will have to create startup script. Let's create a startup script -

-rwxr--r-- 6 root sys 734 Feb 13 10:24 /etc/init.d/apache

#!/sbin/sh

APACHE_HOME=/usr/local/apache2
CONF_FILE=/usr/local/apache2/conf/httpd.conf
RUNDIR=/var/run
PIDFILE=${RUNDIR}/httpd.pid

if [ ! -f ${CONF_FILE} ]; then
        exit 0
fi

if [ ! -d ${RUNDIR} ]; then
        /usr/bin/mkdir -p -m 755 ${RUNDIR}
fi

case "$1" in
start)
        /bin/rm -f ${PIDFILE}
        cmdtext="starting"
        ;;
restart)
        cmdtext="restarting"
        ;;
stop)
        cmdtext="stopping"
        ;;
*)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

echo "httpd $cmdtext."

/bin/sh -c "${APACHE_HOME}/bin/apachectl $1" 2>&1
status=$?

if [ $status != 0 ]; then
        echo "exit status $status"
        exit 1
fi
exit 0

PS: For Solaris 10 you have to create SMF service. Creating SMF is not described in this write-up.

--------------------------------------------------

How developers access the SVN repository?
=================================

Either via Eclipse or Web based Subversion portal.

I hope above notes will stand helpful..

Online data from storage luns in ZPOOL

noreply@blogger.com (Nilesh Joshi) — Wed, 01 Jan 2014 19:42:00 +0000

Wishing you & your family very Happy New Year, may this year brings you all the success, good health & wealth.

Today will be writing about online storage (LUNs) migration for ZFS zpool. I'll be replacing the LUNs from old array to new array.

The best thing about the procedure is that during migration all the data which was present in ZFS file systems in ZPOOL nsrpool is still accessible without any issues, no performance lag. No impact to production system and no down time!!!

Each disk in zfs pool has the same size.

One can do this storage migration in two ways online.

1. One can use attach/detach options, zpool attach -f [pool] [device] [new-device] and then once re-silver done zpool detach [pool] [old-device]
2. Or simply one can simply do device replace, zpool replace [pool] [device] [new-device]

I’m going to use the 2nd option because I think this is the easiest way to achieve this LUN migration.

Newly allocated disks are as below,

/dev/rdsk/c6t6006016070312700CEB537F88C48E311d0s2 - 60G format
/dev/rdsk/c6t6006016070312700FA0468009348E311d0s2 - 60G format
/dev/rdsk/c6t600601606B312700FBA40FCCC0BCE211d0s2 - 60G format
/dev/rdsk/c6t600601606B3127002E623AB38B48E311d0s2 - 60G format
/dev/rdsk/c6t600601606B3127000283C9778948E311d0s2 - 60G format
/dev/rdsk/c6t6006016070312700008A87119148E311d0s2 - 60G format

Have formatted the above disks palcing everything on slice 0.

root@XXXXXX:/root# zpool status nsrpool
  pool: nsrpool
 state: ONLINE
 scan: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        nsrpool                                  ONLINE       0     0     0
          c6t60060160E6C31D00625335009BF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00C8B0F7B898F1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D000C0C7E699AF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00AB24671C98F1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D0060312EA39BF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00CCF4057397F1DB11d0  ONLINE       0     0     0

errors: No known data errors

Let's start the migration -

root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00625335009BF1DB11d0 c6t6006016070312700CEB537F88C48E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00C8B0F7B898F1DB11d0 c6t6006016070312700FA0468009348E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D000C0C7E699AF1DB11d0 c6t600601606B312700FBA40FCCC0BCE211d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00AB24671C98F1DB11d0 c6t600601606B3127002E623AB38B48E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D0060312EA39BF1DB11d0 c6t600601606B3127000283C9778948E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00CCF4057397F1DB11d0 c6t6006016070312700008A87119148E311d0

Now that resilvering started, will have to wait.

root@XXXXXX:/root# zpool status nsrpool
pool: nsrpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Dec 4 13:04:21 2013
    44.4G scanned out of 220G at 75.1M/s, 0h39m to go
    44.4G scanned out of 220G at 75.1M/s, 0h39m to go
    44.4G resilvered, 20.24% done
config:

        NAME                                         STATE     READ WRITE CKSUM
        nsrpool                                    ONLINE       0     0     0
          replacing-0                              ONLINE       0     0     0
            c6t60060160E6C31D00625335009BF1DB11d0 ONLINE       0     0     0
            c6t6006016070312700CEB537F88C48E311d0 ONLINE       0     0     0 (resilvering)
          replacing-1                              ONLINE       0     0     0
            c6t60060160E6C31D00C8B0F7B898F1DB11d0 ONLINE       0     0     0
            c6t6006016070312700FA0468009348E311d0 ONLINE       0     0     0 (resilvering)
          replacing-2                              ONLINE       0     0     0
            c6t60060160E6C31D000C0C7E699AF1DB11d0 ONLINE       0     0     0
            c6t600601606B312700FBA40FCCC0BCE211d0 ONLINE       0     0     0 (resilvering)
          replacing-3                              ONLINE       0     0     0
            c6t60060160E6C31D00AB24671C98F1DB11d0 ONLINE       0     0     0
            c6t600601606B3127002E623AB38B48E311d0 ONLINE       0     0     0 (resilvering)
          replacing-4                              ONLINE       0     0     0
            c6t60060160E6C31D0060312EA39BF1DB11d0 ONLINE       0     0     0
            c6t600601606B3127000283C9778948E311d0 ONLINE       0     0     0 (resilvering)
          replacing-5                              ONLINE       0     0     0
            c6t60060160E6C31D00CCF4057397F1DB11d0 ONLINE       0     0     0
            c6t6006016070312700008A87119148E311d0 ONLINE       0     0     0 (resilvering)

errors: No known data errors

After a long time, resilvering done and now we see zpool with new disks - so our zpool online migration is done.

root@XXXXXX:/root# zpool status nsrpool
errors: No known data errors
pool: nsrpool
state: ONLINE
scan: resilvered 219G in 2h55m with 0 errors on Wed Dec 4 16:00:07 2013
config:

        NAME                                       STATE     READ WRITE CKSUM
        nsrpool                                  ONLINE       0     0     0
          c6t6006016070312700CEB537F88C48E311d0 ONLINE       0     0     0
          c6t6006016070312700FA0468009348E311d0 ONLINE       0     0     0
          c6t600601606B312700FBA40FCCC0BCE211d0 ONLINE       0     0     0
          c6t600601606B3127002E623AB38B48E311d0 ONLINE       0     0     0
          c6t600601606B3127000283C9778948E311d0 ONLINE       0     0     0
          c6t6006016070312700008A87119148E311d0 ONLINE       0     0     0

It was easy! Hope this helps...

Migrating disks within coordinator disk group from one array to another.

noreply@blogger.com (Nilesh Joshi) — Sun, 10 Nov 2013 08:22:00 +0000

Last week I was working on storage migration tasks in which I envisioned to migrate disks within coordinator disk group from old array to newly deployed array.

To me, it was first time doing this task so I decided to note down the steps and after successfully completing the task decided to share with my all friends out there!

There can be several methods to do this task however for me below steps worked very well but if anyone has any better set of instructions doing this then request to share those.

Let's start then -

1. If VCS is running, shut it down locally:

# hastop -all -force

2. Stop I/O fencing on all nodes. This removes any registration keys on the disks.

# /etc/init.d/vxfen stop   (on all 3 cluster nodes)

3. Import the coordinator disk group. The file /etc/vxfendg includes the name of the disk group (for example, vxfencoorddg) that contains the coordinator disks, so use the command -

# vxdg -tfC import 'cat /etc/vxfendg'
                                       OR
# vxdg -tfC import vxfencoorddg

Where:

-t specifies that the disk group is imported only until the system restarts.
-f specifies that the import is to be done forcibly, which is necessary if one or more disks is not accessible.
-C specifies that any import blocks are removed.

4. Turn off the coordinator attribute value for the coordinator disk group.

# vxdg -g vxfencoorddg set coordinator=off

5. Remove old disks and add new disks

    First remove N-1 disks from fence disk group

       # vxdg -g vxfencoorddg rmdisk vxfencoorddg01
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg02
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg03
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg04
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg05
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg06

   Add disk for fence disk group

       # vxdisk list | egrep 'apevmx13_139|apevmx13_140|apevmx13_143|apevmx13_145|apevmx14_139|apevmx14_140|apevmx14_141'
    apevmx13_139 auto:cdsdisk    -            -            online
    apevmx13_140 auto:cdsdisk    -            -            online
    apevmx13_143 auto:cdsdisk    -            -            online
    apevmx13_145 auto:cdsdisk    -            -            online
    apevmx14_139 auto:cdsdisk    -            -            online
    apevmx14_140 auto:cdsdisk    -            -            online
    apevmx14_141 auto:cdsdisk    -            -            online

    # vxdg -g vxfencoorddg adddisk vxfencoorddg01=apevmx13_139
    # vxdg -g vxfencoorddg adddisk vxfencoorddg02=apevmx13_140
    # vxdg -g vxfencoorddg adddisk vxfencoorddg03=apevmx13_143
    # vxdg -g vxfencoorddg adddisk vxfencoorddg04=apevmx13_145
    # vxdg -g vxfencoorddg adddisk vxfencoorddg05=apevmx14_139
    # vxdg -g vxfencoorddg adddisk vxfencoorddg06=apevmx14_140

    Remove the remaining one disk from enclosure apedmx06

    # vxdg -g vxfencoorddg rmdisk vxfencoorddg07

    Add the 7th disk from enclosure apevmx14

    # vxdg -g vxfencoorddg adddisk vxfencoorddg07=apevmx14_141

6. Set the coordinator attribute value as "on" for the coordinator disk group.

# vxdg -g vxfencoorddg set coordinator=on

7. Run disk scan on all nodes

# vxdisk scandisks   (Run on all cluster nodes)

8. Check if fencing disks are visible on all nodes

# vxdisk -o alldgs list | grep fen

9. After replacing disks in a coordinator disk group, deport the disk group:

# vxdg deport 'cat /etc/vxfendg'
                                 OR
# vxdg deport vxfencoorddg

10. Verify if the fencing diskgroup is deported

# vxdisk -o alldgs list | grep fen

11. On each node in the cluster, start the I/O fencing driver:

# /etc/init.d/vxfen start (on all 3 cluster nodes)

12. hastart on all cluster nodes.

# hastart (on all 3 cluster nodes)

That's it, these 12 steps takes you through migrating disks within coordinator disk group from one array to another.

HTH someone!

Adding UFS, ZFS, VxVM FS, Raw FS, LOFS to Non-Global Zone - Some useful examples

noreply@blogger.com (Nilesh Joshi) — Fri, 01 Nov 2013 19:13:00 +0000

In day to day administration we deal with these tasks like adding a raw device to zone, delegating ZFS Datasets to a Non-Global Zone, Adding FS/Volume etc.

In this post I'll be only talking about diffrent types of filesystem operations associated to zones.

Before we start I would like reiterate - Zones are cool and dynamic !!!

So let's start with -

Adding UFS filesystem to Non-Global Zone
____________________________________

global # zonecfg -z zone1
zonecfg:zone1> add fs
zonecfg:zone1:fs> set dir=/u01
zonecfg:zone1:fs> set special=/dev/md/dsk/d100
zonecfg:zone1:fs> set raw=/dev/md/rdsk/d100
zonecfg:zone1:fs> set type=ufs
zonecfg:zone1:fs> add options [nodevices,logging]
zonecfg:zone1:fs> end
zonecfg:zone1> verify
zonecfg:zone1> commit
zonecfg:zone1> exit

Adding ZFS filesystem/dataset/Volume to Non-Global Zone
__________________________________________________

Points to ponder before associating ZFS datasets with zones -

Can add a ZFS file system or a clone to a non-global zone with or without delegating administrative control.
Can add a ZFS volume as a device to non-global zones.
Cannot associate ZFS snapshots with zones
A ZFS file system that is added to a non-global zone must have its mountpoint property set to legacy. If the filesystem is created in the global zone and added to the local zone via zonecfg, it may be assigned to more than one zone unless the mountpoint is set to legacy.

global # zonecfg -z zone1
zonecfg:zone1> add fs
zonecfg:zone1:fs> set type=zfs
zonecfg:zone1:fs> set special=dpool/oradata-u01
zonecfg:zone1:fs> set dir=/u01
zonecfg:zone1:fs> end
zonecfg:zone1> verify
zonecfg:zone1> commit

Adding ZFS filesystem via lofs filesystem
__________________________________________

In a order to use lofs, actual zfs filesystem should be mounted in global zone.

global # zonecfg -z zone1
zonecfg:zone1> add fs
zonecfg:zone1:fs> set special=dpool/oradata-u01
zonecfg:zone1:fs> set dir=/u01
zonecfg:zone1:fs> set type=lofs
zonecfg:zone1:fs> end
zonecfg:zone1> verify
zonecfg:zone1> commit

global # mkdir -p /zoneroot/zone1/root/u01 
global # mount -F lofs /rpool/oradata-u01 /zoneroot/zone1/root/u01 

global # zlogin zone1 df -h /u01
Filesystem             size   used  avail capacity  Mounted on
/oradata-u01             3G    21K   3G     1%      /u01

Delegating Datasets to a Non-Global Zone
_________________________________________

global # zonecfg -z zone1
zonecfg:zone1> add dataset
zonecfg:zone1:dataset> set name=dpool/oradata-u01
zonecfg:zone1:dataset> set alias=oradata-pool
zonecfg:zone1:dataset> end

Within the zone1 zone, this file system is not accessible as dpool/oradata-u01, but as a virtual pool named oradata-pool. The zone administrator is able to set properties on the dataset, as well as create children. It allows the zone administrator to take snapshots, create clones, and otherwise control the entire namespace below the added dataset.

Adding ZFS Volumes to a Non-Global Zone
________________________________________

global # zonecfg -z zone1
zonecfg:zone1> add device
zonecfg:zone1:device> set match=/dev/zvol/dsk/dpool/oradata/u01
zonecfg:zone1:device> end

Adding VxVM filesystem to Non-Global Zone
___________________________________________
global # zonecfg -z zone1
zonecfg:zone11> add fs
zonecfg:zone1:fs> set type=vxfs
zonecfg:zone1:fs> set special=/dev/vx/dsk/oradg/u01
zonecfg:zone1:fs> set raw=/dev/vx/rdsk/oradg/u01
zonecfg:zone1:fs> set dir=/u01
zonecfg:zone1:fs> end
zonecfg:zone1> commit
zonecfg:zone1> verify
zonecfg:zone1> exit

Create & Add UFS filesystem on VxVM volume
___________________________________________

global # vxassist -g zone1_dg make home-ora1-zone1 1g
global # mkfs -F ufs /dev/vx/rdsk/zone1_dg/home-ora1-zone1 2097152

NOTE: 2097152 is sector size.

global # mount -F ufs /dev/vx/dsk/zone1_dg/home-ora1-zone1 /zones/zone1/root/home/oradata/ora1

=========================================================================

Adding the filesystem to Non-Global Zone
____________________________________
global # zonecfg -z zone1
zonecfg:zone1> add fs
zonecfg:zone1:fs> set type=ufs
zonecfg:zone1:fs> set special=/dev/vx/dsk/zone1_dg/home-ora1-zone1
zonecfg:zone1:fs> set raw=/dev/vx/rdsk/zone1_dg/home-ora1-zone1
zonecfg:zone1:fs> set dir=/zones/zone1/root/home/oradata/ora1
zonecfg:zone1:fs> end
zonecfg:zone1> verify
zonecfg:zone1> commit
zonecfg:zone1> exit

global # zlogin zone1 df -k | ora1
/home/oradata/ora1 986095 1041 886445 1% /home/oradata/ora1

Adding raw device to Non-Global Zone
______________________________________

global # zonecfg -z zone1
zonecfg:zone1> add device
zonecfg:zone1:device> set match=/dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0
zonecfg:zone1:device> end
zonecfg:zone1>exit

Ideally we need to reboot non-global zone in order to see added raw device however there is a hack available to do it dynamically. See - Dynamically-adding-raw-device-to-Non-global-zone

Well, this is it. Hope this helps to our community friends for their day to day work!

BTW, In India it's a festive season, Diwali celebration time!!!!
So all my friends - Wish you & you’re your family a very happy, prosperous & safe Diwali.

Enjoy !!!!

Split & Migrate VERITAS Sub-disks from one array to another.

noreply@blogger.com (Nilesh Joshi) — Tue, 22 Oct 2013 14:38:00 +0000

Hi All, I’m still breathing and alive! Yes again, it’s been a long time that I haven’t posted any new investigational post since past few months. Anyways, it’s just that I was bit busy and got mangled in my daily routine.

Today I'll be writing about subject "Sub Disks". Split and migrate sub disks from one array to another.

I'm having a situation wherein I've to migrate the sub disk of size 45Gb from its existing array to another new array. From new array I got storage disks assigned each of size 17Gb and the sub disk which needs to be migrated is of size 45Gb. The limitation here is that the disk size is inelastic to 17Gb by Storage for new array, now with non-contiguous space we can't join the sub disks as a one sub disk so ONLY option available in here to split the existing 45Gb sub disks into three sub disks of equal sizes and then move the split subdisks to the newly created sub disks from the disks coming from new array.

So let's do it.

The first step would be to initializing the new disks and take those disks into VERITAS control and then add them to disk group. well, after doing so start splitting the sub disk -

root:XXXXXXXXXXX:/root # vxsd -g GAPRMANdg -s 33554432 split EMC0_4-05 EMC0_4-06 EMC0_4-07

BEFORE:

sd EMC0_4-05    db_GAPRMAN-01 EMC0_4 48234496 94371840 52428800 UNIX177_4 ENA

AFTER:

sd EMC0_4-06    db_GAPRMAN-01 EMC0_4 48234496 33554432 52428800 UNIX177_4 ENA
sd EMC0_4-07    db_GAPRMAN-01 EMC0_4 81788928 60817408 85983232 UNIX177_4 ENA

Now we will split sub disk EMC0_4-07 into another 2 sub disk each of size 14.5Gb.

root:XXXXXXXXXXX:/root # vxsd -g GAPRMANdg -s 30408704 split EMC0_4-07 EMC0_4-08 EMC0_4-09

BEFORE:

sd EMC0_4-06    db_GAPRMAN-01 EMC0_4 48234496 33554432 52428800 UNIX177_4 ENA
sd EMC0_4-07    db_GAPRMAN-01 EMC0_4 81788928 60817408 85983232 UNIX177_4 ENA

AFTER:

sd EMC0_4-06    db_GAPRMAN-01 EMC0_4 48234496 33554432 52428800 UNIX177_4 ENA
sd EMC0_4-08    db_GAPRMAN-01 EMC0_4 81788928 30408704 85983232 UNIX177_4 ENA
sd EMC0_4-09    db_GAPRMAN-01 EMC0_4 112197632 30408704 116391936 UNIX177_4 ENA

Create sub disks from newly allocated disks -

root:XXXXXXXXXXX:/root # vxmake -g GAPRMANdg sd EMC2_24-01 EMC2_24,0,33554432
root:XXXXXXXXXXX:/root # vxmake -g GAPRMANdg sd EMC2_25-01 EMC2_25,0,30408704
root:XXXXXXXXXXX:/root # vxmake -g GAPRMANdg sd EMC2_26-01 EMC2_26,0,30408704

Now move data off the old sub disk to new sub disk -

root:XXXXXXXXXXX:/root # vxsd -g GAPRMANdg -o rm mv EMC0_4-06 EMC2_24-01
root:XXXXXXXXXXX:/root # vxsd -g GAPRMANdg -o rm mv EMC0_4-08 EMC2_25-01
root:XXXXXXXXXXX:/root # vxsd -g GAPRMANdg -o rm mv EMC0_4-09 EMC2_26-01

So now the migration has been completed.

BEFORE:

sd EMC0_4-06    db_GAPRMAN-01 EMC0_4 48234496 33554432 52428800 UNIX177_4 ENA
sd EMC0_4-08    db_GAPRMAN-01 EMC0_4 81788928 30408704 85983232 UNIX177_4 ENA
sd EMC0_4-09    db_GAPRMAN-01 EMC0_4 112197632 30408704 116391936 UNIX177_4 ENA

AFTER:

sd EMC2_24-01   db_GAPRMAN-01 EMC2_24 0        33554432 52428800 UNIX168_23 ENA
sd EMC2_25-01   db_GAPRMAN-01 EMC2_25 0        30408704 85983232 UNIX168_24 ENA
sd EMC2_26-01   db_GAPRMAN-01 EMC2_26 0        30408704 116391936 UNIX168_25 ENA

Final task would be to free up the old disk from disk group and taking it out of VERITAS control.

root:XXXXXXXXXXX:/root # vxdg -g GAPRMANdg rmdisk EMC0_4
root:XXXXXXXXXXX:/root # vxdiskunsetup -C UNIX177_4
root:XXXXXXXXXXX:/root # vxdisk rm UNIX177_4

Isn't it such a flexible feature from VERITAS. Hope this helps someone, some day!

Increase Cluster File System - CVM Cluster

noreply@blogger.com (Nilesh Joshi) — Mon, 20 May 2013 15:11:00 +0000

In this post I would like to discuss and demonstrate about increasing the Cluster File System (CFS) in CVM environment.

- The requirement is to increase the filesystem by 1TB.

As a best practice, in a CVM/CFS environment, volume should be grown on CVM master and file system should be grown on a CFS Primary. Please note that the CVM Master and the SF CFS node can be two different nodes.

Just off the topic, to know how to grow volume on CVM master & then grow the filesystem on CFS Primary -

To increase the size of the file system, execute the following on CVM master -

# vxassist –g shared_disk_group growto volume_name newlength

And then on CFS node, execute-

# fsadm –F vxfs –b newsize –r device_name mount_point

On other hand, if the system is both CVM master and CFS primary, then "vxresize" command could be executed on the system without any issues.

The above mentioned statement is true below VERITAS version 3.5 but from and above VERITAS version 3.5 vxresize can be run on any nodes within the cluster provided that attribute named 'HacliUserLevel' is set to value "COMMANDROOT". The default value of this attribute is 'NONE' which prevents user to run vxresize command from all the nodes in the cluster.

In my case it is set to "COMMANDROOT"

# /opt/VRTSvcs/bin/haclus -display | grep Hacli
HacliUserLevel COMMANDROOT

But if the value of attribute 'HacliUserLevel' is set to "NONE" then below is method can be used to change the value.

To change the value to COMMANDROOT, run:

# /opt/VRTSvcs/bin/haconf -makerw
# /opt/VRTSvcs/bin/haclus -modify HacliUserLevel COMMANDROOT
# /opt/VRTSvcs/bin/haconf -dump -makero

This change allows vxresize to call hacli command, which then allows any command to be run on any system within the cluster.

Okay, back to our requirement -

So first let's find out the CFS primary node -

# fsclustadm -v showprimary /u01-zz/oradata
XXXXX

Now let's find out the master CVM node in the cluster by -

# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: XXXXX

Now that I need to increase the filesystem by 1TB hence I'm checking if disk group has enough space in it.

# vxassist -g dde1ZZGO0 maxsize
Maximum volume size: 2136743936 (1043332Mb)

Well, we have enough space available under disk group dde1ZZGO0.

Let increase the FS by 1TB now -

# vxresize -b -F vxfs -g dde1ZZGO0 ZZGO0_v0 +1000g

BEFORE:

# df -kh /u01-zz/oradata
Filesystem             size   used avail capacity Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0
                       5.0T   5.0T    17G   100%    /u01-zz/oradata

AFTER:

# df -kh /u01-zz/oradata
Filesystem             size   used avail capacity Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0
                       6.0T   5.0T   954G    85%    /u01-zz/oradata

Good learning about attribute HacliUserLevel, isn't it?

Change mount point under VCS.

noreply@blogger.com (Nilesh Joshi) — Sun, 24 Mar 2013 18:26:00 +0000

It's my B'day today!!! On this occasion I can’t miss an opportunity to write a blog entry as I think, being a tech savvy guy I should write something which would be the nice gift from me to myself..

On this special day, today I would like to share a blog entry with you all. So let's start...

Past week I was requested to change mount point of /opt/vdf to /data along with applicable alteration in VCS configuration to make sure VCS functionality should work as expected as before for this volume & mount resource.

The mount point needs to be renamed to /data is as follows –

# df -kh /opt/vdf/
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
400G 165M 375G 1% /opt/vdf

Hence in order to do so, I’ll have to –

Unmount the /opt/vdf filesystem and re-mount it on /data as mount point. Before we do so, check if any data resides under filesystem and if any application active processes running within the filesystem which needs to be renamed. In case if any application is running and using the filesystem which needs renamed then at first place you need to stop the application and verify if no application process is hooked with the filesystem & If data resides on filesystem then simply create a temporary filesystem with temporary mountpoint and then copy over the data from original filesystem needs to be renamed. I trust this is simple one and any experienced SA can certainly do it.

Now in order to remount the filesystem with new mountpoint, first you need to modify your VCS configuration a bit so that you can un-mount the filesystem successfully.

Obliviously, the first basic thing to verify that VCS cluster configuration is currently read-only mode or write mode. To verify that you can use below command.

# haclus -display | grep -i 'readonly'
ReadOnly 1

Where,

0 = write mode
1 = read only mode

Well, currently VCS configuration is in Read-only mode so let’s make it write mode.

# haconf -makerw

# haclus -display | grep -i 'readonly'
ReadOnly 0

Good, now we have VCS configuration in write mode so we can make appropriate changes to the configuration and save them.

Now change the appropriate attributes in VCS configuration for filesystem/mount point resource that we are supposed to rename. This may vary, in my case I’m not changing the volume name and changing just a mountpoint name so I’m modifying only “MountPoint” attribute for resource “optvdf_mnt”

# hares -modify optvdf_mnt MountPoint "/data"

To verify.

# hares -display optvdf_mnt | grep -i MountPoint
optvdf_mnt ArgListValues adevdf01s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt ArgListValues adevdf02s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt MountPoint global /data

At this point we are good to un-mount the filesystem with mountpoint named /opt/vdf.

# umount /opt/vdf
UX:vxfs umount: ERROR: V-3-26388: file system /opt/vdf has been mount locked

Error!!! Whenever I get any error I feel very happy as every error teaches something new & especially new errors which I never came across!

The Solaris native OS command "umount" may through below error -

Well this error occurs if the mount point is locked by VCS. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control. The purpose of the VERITAS File System (VxFS) Mount Lock is to prevent accidental unmounting of a VxFS file system. The feature is enabled for the VCS Mount resource by default. This feature can be disabled by setting the Mount resource attribute VxFSMountLock to 0.

# hagrp -resources vdfapp_sg
vdfapp_dg
vdfappIP
optcdvdf_mnt
optvdf_mnt
optcdvdf_vol
optvdf_vol

# hares -display vdfapp_dg | grep -i UmountVolumes
vdfapp_dg ArgListValues adevdf01s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg ArgListValues adevdf02s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg UmountVolumes global 0

To get rid of this error use the VxFS umount command for Solaris to manually unmount the file system.

# /opt/VRTS/bin/umount -o mntunlock=VCS /opt/vdf

To mount filesystem/volume on new mountpoint use –

# mount -F vxfs -o mntlock=VCS /dev/vx/dsk/vdfdg/opt_vdf_vol /data

# df -kh /data
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
400G 165M 375G 1% /data

Make sure to save and make cluster configuration read-only.

# haconf -dump -makero

That's it! Done, isn't this easy! :)

Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS)

noreply@blogger.com (Nilesh Joshi) — Fri, 22 Mar 2013 19:41:00 +0000

Hi there! busy.. busy.. busy... stuck with routine work.. lots of work! :)

Anyways, today I finally decided to write a blog entry on a bit unusual task I did recently - Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS). In past have done several migrations, like UFS to ZFS, UFS/SDS to VxVM, VxVM to ZFS however this time I been contacted for some different and backward migration which is VxVM to SDS. When SDS is getting obsolete but on other hand still there are some requirement comes, in which demanding for such things… anyways, good to perform anything which you feels interesting and uncommon. Then let’s do it…

Point To Ponder - Make sure you have full backup of the system on which you're going to operate.

Let's first see which disks are part of bootdg.

Disk_0 auto rootdisk rootdg online c3t0d0s2
Disk_5 auto rootmirror rootdg online c0t0d0s2

Which all volumes needs to be converted to the SVM -

# df -kh | grep bootdg
/dev/vx/dsk/bootdg/rootvol 5.9G 4.3G 1.5G 75% /
/dev/vx/dsk/bootdg/var 5.9G 4.7G 1.1G 81% /var
/dev/vx/dsk/bootdg/opt 5.9G 1.8G 4.0G 32% /opt

Take the pre-veritas vfstab copy.

root@XXXXXX# cat /etc/vfstab.prevm

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no nologging
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /var ufs 1 no nologging
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /opt ufs 2 yes nologging
/dev/dsk/c0t2d0s0 /dev/rdsk/c0t2d0s0 /var/crash ufs 2 yes nologging
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes -

#/dev/dsk/c3t0d0s0 is currently mounted on /.
#/dev/dsk/c3t0d0s1 is currently used by swap.
#/dev/dsk/c3t0d0s5 is currently mounted on /opt.
#/dev/dsk/c3t0d0s6 is currently mounted on /var.

Let's first unencapsulate the rootdisk using the vxunroot.

Detach all the plexes associated with the 'rootmirror' disk if applicable & Verify the rootmirror plexes have been detached.

# vxprint -qhtg rootdg -s | grep -i rootmirror | awk '{print $3}' > /var/tmp/subs.plex && cat /var/tmp/subs.plex
rootvol-02
swapvol-02
opt-02
var-02

# for x in `cat /var/tmp/subs.plex`

> do
> vxplex -g rootdg dis $x
> vxprint -qhtg rootdg -p $x
> done
pl rootvol-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 12584484 0 Disk_5 ENA
pl swapvol-02 - DISABLED - 31458321 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 12584484 31458321 0 Disk_5 ENA
pl opt-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-03 opt-02 rootmirror 44042805 12584484 0 Disk_5 ENA
pl var-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-04 var-02 rootmirror 56627289 12584484 0 Disk_5 ENA

# /etc/vx/bin/vxunroot

VxVM vxunroot NOTICE V-5-2-1564
This operation will convert the following file systems from
   volumes to regular partitions:
opt rootvol swapvol var

   VxVM vxunroot INFO V-5-2-2011
Replacing volumes in root disk to partitions will require a system
reboot. If you choose to continue with this operation, system
configuration will be updated to discontinue use of the volume
  manager for your root and swap devices.

Do you wish to do this now [y,n,q,?] (default: y) y
VxVM vxunroot INFO V-5-2-287 Restoring kernel configuration...
VxVM vxunroot INFO V-5-2-78
A shutdown is now required to install the new kernel.
You can choose to shutdown now, or you can shutdown later, at your
   convenience.

Do you wish to shutdown now [y,n,q,?] (default: n) n

VxVM vxunroot INFO V-5-2-258
Please shutdown before you perform any additional volume manager
   or disk reconfiguration. To shutdown your system cd to / and type

          shutdown -g0 -y -i6

# sync;sync;sync;shutdown -g0 -y -i6

Well, after 2-3 reboots server came back online and now I've UFS filesystems for OS volumes.

# df -kh / /var /opt
Filesystem size used avail capacity Mounted on
/dev/dsk/c3t0d0s0 5.9G 4.3G 1.5G 74% /
/dev/dsk/c3t0d0s6 5.9G 4.7G 1.1G 81% /var
/dev/dsk/c3t0d0s5 5.9G 1.8G 4.0G 32% /opt

Just to be sure, once just try booting from mirror disk too.

Now let's create a partition slice for metadb.

# format c3t0d0

selecting c3t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c3t0d0s0 is currently mounted on /. Please see umount(1M).
/dev/dsk/c3t0d0s1 is currently used by swap. Please see swap(1M).
/dev/dsk/c3t0d0s5 is currently mounted on /opt. Please see umount(1M).
/dev/dsk/c3t0d0s6 is currently mounted on /var. Please see umount(1M).

FORMAT MENU:
                disk - select a disk
                type - select (define) a disk type
                partition - select (define) a partition table
                current - describe the current disk
                format - format and analyze the disk
                repair - repair a defective sector
                label - write label to the disk
                analyze - surface analysis
                defect - defect list management
                backup - search for backup labels
                verify - read and display labels
                save - save new disk/partition definitions
                inquiry - show vendor, product and revision
                volname - set 8-character volume name
                ![cmd] - execute [cmd], then return
                quit
format> p

PARTITION MENU:
        0 - change `0' partition
        1 - change `1' partition
        2 - change `2' partition
        3 - change `3' partition
        4 - change `4' partition
        5 - change `5' partition
        6 - change `6' partition
        7 - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name - name the current table
        print - display the current table
        label - write partition map and label to the disk
        ![cmd] - execute [cmd], then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 0 0 (0/0/0) 0

partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 23960
Enter partition size[0b, 0c, 23960e, 0.00mb, 0.00gb]: 128mb
partition> p
Current partition table (unnamed):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 23960 - 24050 128.37MB (91/0/0) 262899

partition> l
Ready to label disk, continue? yes

Good enough, now let's create metadb.

# metadb -a -f -c 3 c3t0d0s7

# metadb -i
       flags first blk block count
         a u     16      8192     /dev/dsk/c3t0d0s7
         a u     8208    8192     /dev/dsk/c3t0d0s7
         a u     16400   8192     /dev/dsk/c3t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Set SDS on root file system.

# metainit -f d11 1 1 c3t0d0s0
d11: Concat/Stripe is setup

# metainit d10 -m d11
d10: Mirror is setup
# metaroot d10

After executing metaroot, check the changes in /etc/vfstab and /etc/system.

#live-upgrade: updated boot environment

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap -no nologging
/dev/dsk/c3t0d0s1 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/dsk/c3t0d0s6 /dev/rdsk/c3t0d0s6 /var ufs 1 no nologging,nosuid
/dev/dsk/c3t0d0s5 /dev/rdsk/c3t0d0s5 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,10,blk
* End MDD root info (do not edit)

Good enough.

# metastat -ac
d10           m 6.0GB d11
   d11        s 6.0GB c3t0d0s0

# metainit -f d31 1 1 c3t0d0s6
d31: Concat/Stripe is setup
# metainit d30 -m d31
d30: Mirror is setup

# metainit -f d51 1 1 c3t0d0s5
d51: Concat/Stripe is setup
# metainit d50 -m d51
d50: Mirror is setup

# metainit -f d1 1 1 c3t0d0s1
d1: Concat/Stripe is setup
# metainit d0 -m d1
d0: Mirror is setup

# metastat -ac
d0      m 15GB d1
   d1   s 15GB c3t0d0s1
d50     m 6.0GB d51
   d51  s 6.0GB c3t0d0s5
d30     m 6.0GB d31
   d31  s 6.0GB c3t0d0s6
d10     m 6.0GB d11
   d11  s 6.0GB c3t0d0s0

So in above,

d0 - swap
d10 - /
d30 - /var
d50 - /opt

Now it's time to change the vfstab to change slices into metadb devices.

# vi /etc/vfstab

# cat /etc/vfstab
#live-upgrade: updated boot environment
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap - no nologging
/dev/md/dsk/d0 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/md/dsk/d30 /dev/md/rdsk/d30 /var ufs 1 no nologging,nosuid
/dev/md/dsk/d50 /dev/md/rdsk/d50 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

Reboot the box once.

Now it's time to add mirror disk to existing metadevices.

At this stage the mirror disk is part of rootdg and for rootdg it's the last disk in the disk group hence we need to destroy rootdg.

# vxdg destroy rootdg

Unsetup the disk so it will be out of VERITAS control.

# vxdiskunsetup -C Disk_5

Cool, now I'm all set to create/copy partition table on mirror disk as exisitng disk part of metadevices.

# prtvtoc /dev/rdsk/c3t0d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Fine, Set redundant database on mirror disk.

# metadb -a -f -c 3 c0t0d0s7

# metadb -i

        flags    first blk    block count
     a m p luo   16           8192         /dev/dsk/c3t0d0s7
     a   p luo   8208         8192         /dev/dsk/c3t0d0s7
     a   p luo   16400        8192         /dev/dsk/c3t0d0s7
     a      u    16           8192         /dev/dsk/c0t0d0s7
     a      u    8208         8192         /dev/dsk/c0t0d0s7
     a      u    16400        8192         /dev/dsk/c0t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Create metadevices on mirror.

# metainit -f d12 1 1 c0t0d0s0
d12: Concat/Stripe is setup

# metainit -f d2 1 1 c0t0d0s1
d2: Concat/Stripe is setup

# metainit -f d52 1 1 c0t0d0s5
d52: Concat/Stripe is setup

# metainit -f d32 1 1 c0t0d0s6
d32: Concat/Stripe is setup

Attach metadevices.

# metattach d10 d12
d10: submirror d12 is attached

# metattach d30 d32
d30: submirror d32 is attached

# metattach d50 d52
d50: submirror d52 is attached

# metattach d0 d2
d0: submirror d2 is attached

# metastat -ac
d0              m 15GB d1 d2 (resync-76%)
     d1         s 15GB c3t0d0s1
d2              s 15GB c0t0d0s1
     d50        m 6.0GB d51 d52
d51             s 6.0GB c3t0d0s5
     d52        s 6.0GB c0t0d0s5
d30             m 6.0GB d31 d32
     d31        s 6.0GB c3t0d0s6
d32             s 6.0GB c0t0d0s6
     d10        m 6.0GB d11 d12
d11             s 6.0GB c3t0d0s0
     d12        s 6.0GB c0t0d0s0

Install boot block on both disks.

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c3t0d0s0

Well, that completes VxVM to SDS migration. I'm sure hardly anyone need to do such backward migration but just in case then this method will certainly help you... :)

Have a good weekend!

UNABLE TO MIRROR VxVM VOLUME

noreply@blogger.com (Nilesh Joshi) — Tue, 25 Dec 2012 20:31:00 +0000

First of all... Merry Christmas, everybody!

Yes, again after a long time... past few weeks were very hyperactive and quite attentive but finally, yes… finally one more unsleeping late night… I decided to take some timeout to post this interesting post for all my sysadmin friends!

Few days back I got a request to mirror an existing volume. Easy enough right? - Yes, it is! but then I revealed into an interesting problem causing me to unable mirror the existing volume.

Let's see what was the issue and what logic I used to get rid of the issue.

Volume DXarchive90d was not mirrored for some unknown reason and so I've been requested to mirror it.

root:XXXXXXXX:/root # vxprint -htg asmm7 DXarchive90d

V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 418426880 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA

In order to mirror it I decided to create the new sub disks, new plex and attach the plex to make mirror for volume DXarchive90d. When I tried so, I got an interesting error -

1) I created another plex of same size and tried to attach but get below message.

- Create sub-disk

Syntax - vxmake [-g diskgroup] sd subdisk diskname,offset,length

root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_21m-01 asmm7_21m,0,167270400
root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_22m-01 asmm7_22m,0,167270400
root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_23m-01 asmm7_23m,0,83886080

- Create Plex of same size

Syntax - vxmake [-g diskgroup] plex plex sd=subdisk1[,subdisk2,...]

root:XXXXXXXX:/root # vxmake -g asmm7 plex DXarchive90d-02 sd=asmm7_21m-01,asmm7_22m-01,asmm7_23m-01

- Attach the plex

root:XXXXXXXX:/root # vxplex -g asmm7 att DXarchive90d DXarchive90d-02
VxVM vxplex ERROR V-5-1-4035 Plex DXarchive90d-02 would be a sparse plex of Volume DXarchive90d

Tried with "force" option.

root:XXXXXXXX:/root # vxplex -g asmm7 -o force att DXarchive90d DXarchive90d-02
VxVM vxplex ERROR V-5-1-1204 Volume DXarchive90d has no complete plexes, cannot attach

2) Tried another way,

root:XXXXXXXX:/root # vxassist -v -b -g asmm7 mirror DXarchive90d layout=mirror-concat alloc="asmm7_21m asmm7_22m asmm7_23m"
VxVM vxassist ERROR V-5-1-437 Cannot allocate space to mirror 419430400 block volume

Later, carefully looking at things - Ahh.... See that -

Volume length: 419430400
Plex length: 418426880

The mirror attach operations are failing due to the addressable size of storage in the volume being larger than that of the plex assigned to it. i.e the volume has no complete plex that will store the entire volume data.

So what is the way out to such situation?

Well, I got the link where it says -

http://www.symantec.com/business/support/index?page=content&id=TECH67644

- Remove the volume
- Then attached the plex to the volume.

Well, unfortunately this solution was not for me! So, I figured out another way -

So, there is 1003520 sector diffrence between Volume length and Plex length. So how about associating subdisk of size 1003520 sectors.... just wild idea - let's see how it goes...

root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_10-04 asmm7_10,0,1003520
root:XXXXXXXX:/root # vxsd -g asmm7 assoc DXarchive90d-01 asmm7_10-04

BEFORE:

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 418426880 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA

AFTER:

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA
sd asmm7_10-04 DXarchive90d-01 asmm7_10 0 1003520 418426880 usp006_16 ENA

Good enough - the root cause of volume mirror not happening was mismatch between volume lenth and plex lenth so all what I did is matched those sizes by associating subdisk of the size of diffrence to existing plex.

Now I think remove the plex which I created for mirroring...

root:XXXXXXXX:/root # vxedit -g asmm7 -rf rm DXarchive90d-02

Execute -

root:XXXXXXXX:/root # vxassist -v -b -g asmm7 mirror DXarchive90d layout=mirror-concat alloc="asmm7_21m asmm7_22m asmm7_23m"

SUCCESS!!!! Mirroring done OK.

See the mirroring progress...

root:XXXXXXXX:/root # vxtask list

TASKID PTID TYPE/STATE PCT PROGRESS
559 ATCOPY/R 01.10% 0/419430400/4608000 PLXATT DXarchive90d DXarchive90d-02 asmm7

root:XXXXXXXX:/root # vxtask -l list
Task: 559 RUNNING
Type: ATCOPY
Operation: PLXATT Vol DXarchive90d Plex DXarchive90d-02 Dg asmm7
Started: Thu Dec XX 12:07:12 2012
Throttle: 0
Progress: 1.23% 5156864 of 419430400 Blocks
Work time: 43 seconds (57:34 remaining)

root:XXXXXXXX:/root # vxprint -htg asmm7 DXarchive90d

V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA
sd asmm7_10-04 DXarchive90d-01 asmm7_10 0 1003520 418426880 usp006_16 ENA
pl DXarchive90d-02 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_21m-01 DXarchive90d-02 asmm7_21m 0 167270400 0 usp009_20 ENA
sd asmm7_22m-01 DXarchive90d-02 asmm7_22m 0 167270400 167270400 usp009_21 ENA
sd asmm7_23m-01 DXarchive90d-02 asmm7_23m 0 84889600 334540800 usp009_22 ENA

Well, now in good shape... Hope this helps someone who gets such a interesting issue...

Weird issue with VERITAS after replacing the disk

noreply@blogger.com (Nilesh Joshi) — Sun, 21 Oct 2012 19:07:00 +0000

When performing a disk replacement in VxVM 4.1 and VxVM 5.0, the disk being replaced does not show up in the output of a "vxdisk list" command. Instead a "NONAMEs2" entry is seen.

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
VxVM vxdisk ERROR V-5-1-2272 NONAMEs2 not a valid dmpnode disk access name.
NONAMEs2 auto - - online
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
c1t0d0s2 auto rootdisk rootdg online c1t0d0s2
c1t1d0s2 auto - - online c1t1d0s2
c1t2d0s2 auto data01 data online c1t2d0s2
c1t5d0s2 auto data02 data online c1t5d0s2
c1t8d0s2 auto data03 data online c1t8d0s2
c1t9d0s2 auto data04 data online c1t9d0s2
c1t10d0s2 auto - - online c1t10d0s2
c1t11d0s2 auto - - online c1t11d0s2
c1t12d0s2 auto - - online c1t12d0s2
c1t13d0s2 auto - - online c1t13d0s2
- - rootmirror rootdg removed was:c1t3d0s2

To get rid of this, solution is to re-create device mapping, restart vxconfigd, initialize the device, add the disk back to the diskgroup.

# vxdisk rm NONAMEs2 (Remove the device entry in Volume Manager)
# cd /dev/vx/rdmp (Delete all files pertaining to NONAME*)
# cd /dev/vx/dmp (Delete all files pertaining to NONAME*)
# mv /etc/vx/disk.info /etc/vx/disk.info.old
# mv /etc/vx/array.info /etc/vx/array.info.old
# devfsadm -Cv
# vxddladm stop eventsource
# vxconfigd -k (This will recreate the device mapping and the disk.info file)

If the above solution fails, Rerun all commands up to and including: "mv /etc/vx/disk.info /etc/vx/disk.info.old" then run a reconfigure reboot.

It worked!!!! disk c1t3d0s2 is back....

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
c1t0d0s2 auto rootdisk rootdg online c1t0d0s2
c1t1d0s2 auto - - online c1t1d0s2
c1t2d0s2 auto data01 data online c1t2d0s2
c1t3d0s2 auto - - online c1t3d0s2
c1t5d0s2 auto data02 data online c1t5d0s2
c1t8d0s2 auto data03 data online c1t8d0s2
c1t9d0s2 auto data04 data online c1t9d0s2
c1t10d0s2 auto - - online c1t10d0s2
c1t11d0s2 auto - - online c1t11d0s2
c1t12d0s2 auto - - online c1t12d0s2
c1t13d0s2 auto - - online c1t13d0s2
- - rootmirror rootdg removed was:c1t3d0s2

Now as the disk is back in proper format perform action of -

Initialize the device,
Add the disk to the disk group
Recover plex.

root@XXXXX# vxdisksetup -i c1t3d0 format=sliced
root@XXXXX# vxdg -g rootdg -k adddisk rootmirror=c1t3d0

root@XXXXX# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdisk rootdg online
c1t1d0s2 auto:none - - online invalid
c1t2d0s2 auto:sliced data01 data online
c1t3d0s2 auto:sliced rootmirror rootdg online
c1t5d0s2 auto:sliced data02 data online
c1t8d0s2 auto:sliced data03 data online failing
c1t9d0s2 auto:sliced data04 data online
c1t10d0s2 auto:none - - online invalid
c1t11d0s2 auto:none - - online invalid
c1t12d0s2 auto:none - - online invalid
c1t13d0s2 auto:none - - online invalid

root@XXXXX# vxrecover -bg rootdg

Check if recovery is in progress (vxtask list)

root@XXXXX# vxtask -l list
Task: 436 RUNNING
Type: PARENT
Operation: VXRECOVER
Started: Fri Oct 19 18:38:17 2012
Progress: 20.00% (1 of 5 jobs, 1 active)
Task: 441 RUNNING (Parent: 436)
Type: ATCOPY
Operation: PLXATT Vol opt Plex opt-02 Dg rootdg
Started: Fri Oct 19 18:40:57 2012
Throttle: 0
Progress: 30.16% 6326272 of 20972736 Blocks
Work time: 1 minute, 54 seconds (04:23 remaining)

root@XXXXX# vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1027335905.1025.XXXXX
dm rootdisk c1t0d0s2 auto 10175 143339136 -
dm rootmirror c1t3d0s2 auto 9919 143328960 -

v home - ENABLED ACTIVE 8405376 ROUND - gen
pl home-01 home ENABLED ACTIVE 8405376 CONCAT - RW
sd rootdisk-03 home-01 rootdisk 75566975 8405376 0 c1t0d0 ENA
pl home-02 home ENABLED ACTIVE 8405376 CONCAT - RW
sd rootmirror-05 home-02 rootmirror 75516096 8405376 0 c1t3d0 ENA

v opt - ENABLED ACTIVE 20972736 ROUND - gen
pl opt-01 opt ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdisk-04 opt-01 rootdisk 54594239 20972736 0 c1t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 20972736 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 54543360 20972736 0 c1t3d0 ENA

v rootvol - ENABLED ACTIVE 4202688 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 4202688 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 20351 4202688 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 4202688 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4202688 0 c1t3d0 ENA

v swapvol - ENABLED ACTIVE 33560448 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 33560448 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 4233215 33560448 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 33560448 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 4202688 33560448 0 c1t3d0 ENA

v var - ENABLED ACTIVE 16780224 ROUND - gen
pl var-01 var ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-05 var-01 rootdisk 37803839 16780224 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 16780224 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 37763136 16780224 0 c1t3d0 ENA

Hope this helps someone who facing this issue..... Have a good night.

Resolving Duplicate disk/device entries in “vxdisk list” or vxdisksetup.

noreply@blogger.com (Nilesh Joshi) — Mon, 17 Sep 2012 19:46:00 +0000

One fine morning I had a undertaking to replace the disk which was part of VxVM. Easy enough – just another routine stuff so my words were - “Ahh it’s simple, it’s just a disk replacement! will finish this off quickly, then go get a cup of coffee over some relax time” – Nope, I was wrong & wasn’t lucky enough to find myself relaxing in office garden over a hot coffee. Anyways, learnt and realized the fact that - There’s no such thing as ‘risk free.’ Everything you do or don’t do has an inherent risk !!!

Anyways, enough with the story – let’s come to the real one. I replaced the disk and inserted new one but after doing so, I started seeing duplicate entries for the replaced disk in the “vxdisk list” output. As per Symantec notes to get rid of this issue we should perform the reconfiguration reboot. The server I was working on was unluckily Solaris 5.8 with vintage VERITAS version 3.5

The problem can be seen when running vxdisk list command:

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

Or the problem can be seen when running vxdisksetup command:

root@XXXXX# vxdisksetup -i c1t2d0
vxdisksetup: c1t2d0: Duplicate DA records encountered for this device.
                     Refer to the troubleshooting guide to clear them

Right above in vxdisk list output you can see disk c1t2d0 has duplicate entries.

Now let’s see how to get rid of this not often (at least in my case) seen issue.

First of all, remove c1t2d0s2 entries from VxVM control & run it for all duplicate entries. There can be 2 or more than 2 duplicate entries for the same disk.

In my case, there were two entries –

root@XXXXXXX# vxdisk rm c1t2d0s2
root@XXXXXXX# vxdisk rm c1t2d0s2

Remove the disk c1t2d0s2 using luxadm. Remove device c1t2d0s2 using ” luxadm remove_device ” command.

root@XXXXXXX# luxadm remove_device /dev/rdsk/c1t2d0s2

WARNING!!! Please ensure that no filesystems are mounted on these device(s).
All data on these devices should have been backed up.

The list of devices which will be removed is:
1: Box Name:     "FCloop" slot 2
Node WWN:        20000004cfa1b23c
Device Type:Disk device
Device Paths:
             /dev/rdsk/c1t2d0s2

Please verify the above list of devices and
then enter 'c' or to Continue or 'q' to Quit. [Default: c]:
stopping: Drive in "FCloop" slot 2....Done
offlining: Drive in "FCloop" slot 2....Done

Hit after removing the device(s).

Drive in Box Name "FCloop" slot 2
Notice: Device has not been removed from the enclosure.
It has been removed from the loop and is ready to be
removed from the enclosure, and the LED is blinking.

Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
c1t2d0s0
c1t2d0s1
c1t2d0s2
c1t2d0s3
c1t2d0s4
c1t2d0s5
c1t2d0s6
c1t2d0s7

Execute “devfsadm” command to clean-up unlinked device references followed by vxdctl enable command.

root@XXXXXXX# devfsadm -Cv

devfsadm[6117]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600

root@XXXXXXX# vxdctl enable

At this point of time we have removed all dev_t associated with the device. BTW, Within the kernel, the dev_t type is used to hold device numbers—both the major and minor parts.

Okay, now we have one entry less.

root@XXXXXXX# vxdisk -e list

DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

Again remove *ALL* duplicate c1t0d0s2 entries from VxVM control.

root@XXXXXXX# vxdisk rm c1t2d0s2

Well, no more entry for disk c1t2d0s2.

root@XXXXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

To remove all possible stale dev_t will offline all the paths to the disk.

root@XXXXXXX# luxadm -e offline /dev/dsk/c1t2d0s2

Clean up the things using devfsadm command –

root@XXXXXXX# devfsadm -Cv
devfsadm[6369]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:a
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s0
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:b
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s1
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:c
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s2
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:d
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s3
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:e
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s4
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:f
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s5
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:g
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s6
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:h
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s7

root@XXXXXXX# vxdctl enable

Now it’s time to insert the disk.

root@XXXXXXX# luxadm insert_device

Once the disk have been inserted or replaced, use "vxdctl enable" and "vxdiskadm" option 5 after syncing with the remaining mirror.

root@XXXXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced rootmirror rootdg online c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2

Here we go! Done with replacing & re-mirroring the VxVM disk in Solaris 8 with legacy VERITAS version 3.5, that too without rebooting the box. Actually, ultimate solution for this situation is a reconfiguration reboot(s), and however standard reboots have been found to occasionally solution for such condition but rebooting the host is most of the time not a practical substitute for mission-critical servers and applications hosted on it.

Just in case if you face this issue on servers with Solaris 10 with version prior or equal to VERITAS Storage Foundation 5.0 MP3 then following methods will surely will be a big help.

Solution 1:
===========

With Storage Foundation 5.0 MP3, and above the following commands can be tried first without restarting vxconfigd:

# rm /etc/vx/disk.info
# rm /dev/vx/dmp/*
# rm /dev/vx/rdmp/*
# vxddladm -c assign names

Solution 2:
============

With any version prior to 5.0 MP3 where the disk.info file exists, vxconfigd must be restarted in order to recreate the DMP nodes, and the disk.info file.

IMPORTANT: If this is part of a VCS cluster, freeze all service groups before running these commands.

# rm /etc/vx/disk.info
# rm /dev/vx/dmp/*
# rm /dev/vx/rdmp/*
# vxconfigd -k

The special thing about this episode is that I able to remove duplicate disk device entries on Solaris 8 with VERITAS 3.5 without reboot. It worked very well in my case and I’m hopeful that you will find this procedure helpful.

Slice up ASM Disks on Solaris Host.

noreply@blogger.com (Nilesh Joshi) — Sun, 26 Aug 2012 16:59:00 +0000

Yes, yes… I know really a long time…. Past few days were quite demanding and quite occupied with routine work. Anyways, today I finally decided to take some time out and write up something about how to slice up ASM disks on Solaris host.

I’ll start from some basics and get some background about -

What is ASM?

It's - Automatic Storage Management.

ASM provides filesystem and volume manager capabilities built into the Oracle database kernel. With this capability, ASM simplifies storage management tasks, such as creating/laying out databases and disk space management. Since ASM allows disk management to be done using familiar create/alter/drop SQL statements, DBAs do not need to learn a new skill set or make crucial decisions on provisioning.

ASM is a management tool specifically built to simplify the job of the DBA. It provides a simple storage management interface across all server and storage platforms. ASM provides the DBA flexibility to manage a dynamic database environment with increased efficiency. This feature is a key component of Grid Computing and Database Storage Consolidation. The following are some key benefits of ASM:

I/O is spread evenly across all available disk drives to prevent hot spots and maximize performance.
ASM eliminates the need for over provisioning and maximizes storage resource utilization facilitating database consolidation.
Inherent large file support.
Performs automatic online redistribution after the incremental addition or removal of storage capacity.
Maintains redundant copies of data to provide high availability, or leverage 3rd party RAID functionality.
Supports Oracle Database 10g as well as Oracle Real Application Clusters (RAC).
Capable of leveraging 3rd party Multipathing technologies & much more.

Now let's see how to format disk for ASM in Solaris.

NOTE - Currently, in 10g and 11gR1, ASM disks cannot be larger than 2TB, and needs to be a minimum of 4Mb

To prevent Oracle ASM from overwriting the partition table, you cannot use slices that start at cylinder 0 (for example, slice 2).

If you need to feed the disks to ASM, which requires you to start your partitions at cylinder > 0. The disk slices to be used must skip the first cylinder (cylinder 0) to avoid overwriting the disk VTOC (Volume Table of Contents). The VTOC is a special area of disk set aside for aside for storing information about the disk’s controller, geometry and slices. Let me explain this in little details –

It is important to move the starting cylinder for the partition from cylinder 0 to cylinder 1, which can be done in the usual way before labelling the disk. This is because cylinder 0 contains the disk label at block 0. File Systems such as UFS or VxFS etc, which are placed into such partitions will not use cylinder 0, specifically to avoid overwriting the disk label, but other, perhaps less thoughtful, applications using the raw device will overwrite cylinder 0, and therefore delete the disk partition label, making the disk unusable.

SPARC platform uses 8 slices starting from 0 to 7. Below is the common configuration of these eight slices:

slice 0: Holds files and directories that make up the operating system.*
slice 1: Swap, Provides virtual memory, or swap space.
slice 2: Refers to the entire disk, by convention. The size of this slice should not be changed.**
slice 3: /export, Holds alternative versions of the operating system.
slice 4: /export/swap. Provides virtual memory space for client systems. ***
slice 5: /opt. Holds application software added to a system.
slice 6: /usr. Holds operating system commands--also known as executables-- designed to be run by users.
slice 7: /home. Holds files created by users.

Standard practice –

* = Cannot be used as ASM disk. Using this slice causes disk corruption and may render the disk as unusable.
** = Should not be used as ASM Disk, as slice refers to the entire disk (Including partition tables).
*** = Is the recommended slice to be used for ASM disk.

Use the -f option with format to take command input from the file instead of standard input.

It is a best practice on a Solaris system, to create a partition on the disk; such as slice 4 or 6, which skips the first 1Mb into the disk. Skipping 1Mb into the disk is done to skip the OS label/VTOC, as well as to preserve alignment between ASM stripping and storage array internal stripping.

Ownership and permissions –

Take care to set the right permission & ownership for the raw devices. The user oracle must have read and write access to them.

Default owner is root:sys needs to be changed to oracle:dba

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 root sys 118, 64 Feb 16 02:10 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# chown oracle:dba /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 oracle dba 118, 64 Feb 16 03:00 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

Also the device file should have permissions set to 660 (recommended but not mandatory)

NOTE: - If you preparing disk to be added to ASM on RAC nodes then please make sure that you set appropriate permissions and ownership on all the nodes.

root@racnode1:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 16 16:49 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode2:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode3:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode4:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:17 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

Once the disk has been discovered by the ASM instance, it is ready to be added to any DiskGroup.

SQL> select name, path, group_number from v$asm_disk
NAME PATH GROUP_NUMBER
------------------------------ --------------- ------------
          /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001213d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001214d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001215d0s4 0

Disks have various header statuses that reflect its membership state within a DiskGroup. Disks can have the following header statuses, mount statuses & global statuses:

Header status:

    MEMBER:       Disk is member of existing diskgroup.
    CANDIDATE:    No part of diskgroup, may be added.
    PROVISIONED: No part of diskgroup, may be added but administrator must first take additionally platform-specific action.
    INCOMPATIBLE: Version in disk header is not compatible with ASM storage.
    FORMER:       Disk was one part of diskgroup. Cleanly dropped.
    UNKNOWN:      Header has not been read.
    CONFLICT:     Disk not mounted due to a conflict.
    FOREIGN:      Disk contains data created by other products than ASM (e.g. datafile,logfile,OCR).

Mount status:

    CACHED - Normal state for disk which are part of mounted diskgroup.(disk is present in storage system and part of a diskgroup)
    OPENED - Normal state, actively used by instance.(disk is present in storage system and accessed by ASM)
    CLOSED - Disk is present in storage system but not being accessed by ASM.
    CLOSING - ASM is in process of closing this disk.
    IGNORED - Disk is present in storage system but ignored by ASM (disk in system library is the same as in ASM library OR header no longer valid)
    MISSING - ASM metadata indicates: Disk is known in diskgroup, but no disk in storage system is found with name.

Global status:

    NORMAL - Online and operating normally.
    ADDING - Disk is being added and is pending validation by all instances that have the diskgroup mounted.
    DROPPING - Disk has been manually taken offline. Rebalancing will be commence. After this disk is expelled.
    DROPPED - Disk has been fully expelled from the diskgroup.
    HUNG - Disk drop operation hangs because insufficient space to relocate date.
    FORCING - Disk is removed without attempting to offload its data. Date will be recovered from redundant copies(where possible).
    UNKNOWN - Disk state is unknown (typical disk not mounted).

I see three main benifits out of this method, which might be quite helpful in enterprise setups where bulk stuffs needs to be carried out.

i. The disk formatting done via format command file hence this ensures the factor “standardization”. Using the format command file we make sure that all the ASM disks will have an unique VTOC across the environment which in-turns make sure standardization.

ii. Bulk disks can be formatted using simple for loop using this format command file & this will be accomplished in just a matter of one hit. For e.g.

# cat /tmp/ASM_Disks
c2t5006048452A8CCDDd1060
c2t5006048452A8CCDDd1061
c2t5006048452A8CCDDd1062
c2t5006048452A8CCDDd1063
c2t5006048452A8CCDDd1064
c2t5006048452A8CCDDd1065
c2t5006048452A8CCDDd1066
c2t5006048452A8CCDDd1067
c2t5006048452A8CCDDd1068
c2t5006048452A8CCDDd1069
c2t5006048452A8CCDDd1070
c2t5006048452A8CCDDd1071
c2t5006048452A8CCDDd1072

# for disk in `cat /tmp/ASM_Disks`
> do
> /usr/scripts/ASM_format.cmd -f $disk
> done

iii. In case of ASM disk label corruption, as we will use this format command file as a standard therefore unique VTOC will help to avoid chaos about “what was previous VTOC” and re-formatting will be easy and flawless.

Hope this helps!

Plex DISABLED RECOVER/STALE state while volume is ENABLED ACTIVE state.

noreply@blogger.com (Nilesh Joshi) — Fri, 08 Jun 2012 19:49:00 +0000

Recently I had a situation where, plexes went into DISABLED RECOVER state while volumes were in ENABLED ACTIVE state. The reason that all volumes remained ENABLED ACTIVE state because, only one side of mirrored plexes were disabled however other side of mirror was in good state or enabled active. This is quite rarely seen situation (at least in my environment) hence I thought of writing a blog entry about this occurrence.

I believe, due to some temporary IO failure on disk c1t1d0 this had happened and I see some logs in diagnostic messages indicating drive c1t1d0 going offline... see below vxprint output

root:XXXXXXXX:/root # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 29000 1218716511.6.XXXXXXXX

dm rootdisk c1t0d0s2 auto 16384 143328960 -
dm rootmirror c1t1d0s2 auto 20095 143318784 -

v crash - ENABLED ACTIVE 16780224 ROUND - fsgen
pl crash-01 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-03 crash-01 rootdisk 67131071 16780224 0 c1t0d0 ENA
pl crash-02 crash DISABLED RECOVER 16780224 CONCAT - RW
sd rootmirror-02 crash-02 rootmirror 16780224 16780224 0 c1t1d0 ENA

v home - ENABLED ACTIVE 2097152 SELECT - fsgen
pl home-01 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootdisk-09 home-01 rootdisk 102808127 2106432 0 c1t0d0 ENA
pl home-02 home DISABLED RECOVER 2106432 CONCAT - RW
sd rootmirror-04 home-02 rootmirror 35666880 2106432 0 c1t1d0 ENA

v networker - ENABLED ACTIVE 10485760 SELECT - fsgen
pl networker-01 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootdisk-08 networker-01 rootdisk 92316671 10491456 0 c1t0d0 ENA
pl networker-02 networker DISABLED RECOVER 10491456 CONCAT - RW
sd rootmirror-05 networker-02 rootmirror 37773312 10491456 0 c1t1d0 ENA

[... Many lines, skipped for brevity ...]

To recover from such incident I would suggest to try below command first:

# vxrecover -bsE -g diskgroup_name

# vxrecover -bsE -g rootdg

Use vxtask list or vxtask -l list command to check the sync operation status.
Where,

vxrecover - perform volume recovery operations

-b => Performs recovery operations in the background. With this option, vxrecover runs in the background to attach stale plexes and subdisks, and to resyn-chronize mirrored volumes and RAID-5 parity. If this is used with -s, volumes are started before recovery begins in the background.

-E => Starts disabled volumes or plexes even when they are in the EMPTY state. This is useful for start-ing up volumes restored by the vxmake utility when specified along with the -s option.

This would try to recover the plex & sync them again. Just in case if this doesn't works then I would prefer to detach the plex(es) & attach them again.. you can do so via "vxplex det" & "vxplex att".

In my case first option worked like piece of cake.

root:XXXXXXXX:/root # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 29000 1218716511.6.XXXXXXXX

dm rootdisk c1t0d0s2 auto 16384 143328960 -
dm rootmirror c1t1d0s2 auto 20095 143318784 -

v crash - ENABLED ACTIVE 16780224 ROUND - fsgen
pl crash-01 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-03 crash-01 rootdisk 67131071 16780224 0 c1t0d0 ENA
pl crash-02 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootmirror-02 crash-02 rootmirror 16780224 16780224 0 c1t1d0 ENA

v home - ENABLED ACTIVE 2097152 SELECT - fsgen
pl home-01 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootdisk-09 home-01 rootdisk 102808127 2106432 0 c1t0d0 ENA
pl home-02 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootmirror-04 home-02 rootmirror 35666880 2106432 0 c1t1d0 ENA

v networker - ENABLED ACTIVE 10485760 SELECT - fsgen
pl networker-01 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootdisk-08 networker-01 rootdisk 92316671 10491456 0 c1t0d0 ENA
pl networker-02 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootmirror-05 networker-02 rootmirror 37773312 10491456 0 c1t1d0 ENA

[... Many lines, skipped for brevity ...]

Hope this helps someone.

BTW, just for refreshing the basics, let's take a look at Plex states and Condition flags.

EMPTY: This state indicates that you have not yet defined which plex has the good data (CLEAN), and which plex does not have the good data (STALE).

CLEAN: This state is normal and indicates that the plex has a copy of the data that represents the volume. CLEAN also means that the volume is not started and is not currently able to handle I/O (by the administrator's control).

ACTIVE: This state is the same as CLEAN, but the colume is or was currently started, and the colume is or was able to perform I/O.

SNAPDONE: This state is the same as ACTIVE or CLEAN, but is a plex that has been synchronized with the volume as a result of a “vxassist snapstart” operation. After a reboot or a manual start of the volume, a plex in the SNAPDONE state is removed along with its subdisks.

STALE: This state indicates that VxVM has reason to believe that the data in the plex is not synchronized with the data in the CLEAN plexes. This state is usually caused by taking the plex offline or by a disk failure.

SNAPATT: This state indicates that the object is a snapshot that is currently being synchronized but does not yet have a complete copy of the data.

OFFLINE: This state indicates that the administrator has issued the “vxmend off” command on the plex. When the administrator brings the plex back online using the “vxmend on” command, the plex changes to the STALE state.

TEMP: The TEMP state flags (TEMP, TEMPRM, TEMPRMSD) usually indicate that the data was never a copy of the volume’s data, and you should not use these plexes. These temporary states indicate that the plex is currently involved in a synchronization operation with the volume.

NODEVICE: This flag indicates that the disk drive below the plex has failed.

REMOVED: This flag has the same meaning as NODEVICE, but the system admin has requested that the device appear as failed.

IOFAIL: This flag is similar to NODEVICE, but it indicates that an unrecoverable failure occurred on the device, and VxVM has not yet verified whether the disk is actually bad.

Note: I/O to both the public and the private regions must fail to change the state from IOFAIL to NODEVICE.

RECOVER: This flag is set on a plex when two conditions are met:

1) A failed disk has been fixed (by using vxreattach or the vxdiskadm option, “Replace a failed or removed disk”).
2) The plex was in the ACTIVE state prior to the failure.

I'm sure above notes will stand helpful…

Upgrade Solaris 10 Update 3 to Solaris 10 Update 9 + Patch Upgrade + VxVM 4.1 = A Lengthy but Curious Task

noreply@blogger.com (Nilesh Joshi) — Sun, 06 May 2012 22:15:00 +0000

Background –

Recently I’ve been asked to upgrade and patch the system running with Solaris 10 U3 with VxVM 4.1 to Solaris U9 and latest Update 9 kernel patch. OS boot disks are part of VxVM (encapsulated root disk) hence I premeditated to use LiveUpgrade along with Symantec developed & provided LiveUpgrade wrapper scripts namely “vxlustart” & “vxlufinish”. When I actually started performing vxlustart it failed for the reason that someone in past has created few extra volumes on rootdg without un-encapsulating rootdg as vxlustart script was unable to get proper disk slice layout or due to some weird issue and the way partitioning done in past. In addition to this even the slicing was bit confusing at disk level (missing slice on rootmirror disk for one of the OS partition) and I strongly guess that this caused concerns to vxlustart to turn out to be successful. Eventually, I left with only one choice to un-encapsulate rootdg using vxunroot command & then perform the patching along with OS upgrade.

For me this server was like a patient who lost the hope to grow into healthy body ever & it was like a breathing dead body – can only breath (in server language “run”) but can’t do anything useful. Being an UNIX/Solaris system surgeon, I expected to operate on him and get him well-shaped with aesthetic body. Believe me, being an UNIX system surgeon, I love my servers (patients) and yes, of course - I able to successfully offered his life back to him after some never done stuffs!!! and currently server is running like a stunning, most updated DB server ever.

Execution –

Below is comprehensive overview/thought process about steps taken –

- Backed up all added/extra volumes & removed added/extra volumes on rootdg
- Deported all non-rootdg's before executing vxunroot
- Unencapsulated the rootdisk using the vxunroot script
- Rebooted system and got UFS slice based filesystems
- Created alternate boot environment using Live Upgrade and patched/upgraded ABE to Solaris Update 9.
- Booted off from updated and patched ABE, System is now on Update 9 and on latest patch level.
- After patching/upgrading system, added updated disk to rootdg (creation of rootdg with single disk a.k.a encapsulation)
- Created extra volumes while rootdg not mirrored
- Restored data of those extra volumes
- Imported all previously deported data disk group(s) & start, mount volumes in it
- After application owner confirmation, deleted the PBE (Primary Boot Environment), this will free up the disk
- Add disk to rootdg as mirror disk & re-mirror the volumes

Point to ponder – Please take full system backup before acting on below given procedure.

NOTE: Just to refresh the basics before proceeding, according to the fmthard manual page, tag and flag numbers are defined as follows:

tag description
0x00 UNASSIGNED
0x01 BOOT
0x02 ROOT
0x03 SWAP
0x04 USR
0x05 BACKUP
0x06 STAND
0x07 VAR
0x08 HOME

flag description
0x00 MOUNTABLE
0x01 UNMOUNTABLE
0x10 READ-ONLY

Grab as much as information about the server.

- Take a snapshot of VTOC for both disks which are part of rootdg.

root@xxxxxxxx# prtvtoc -s /dev/rdsk/c1t0d0s2
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 31464192 16790400 48254591
1 3 01 0 31464192 31464191
2 5 00 0 286698624 286698623
3 14 01 0 286698624 286698623
4 15 01 286657920 40704 286698623
5 7 00 48254592 16790400 65044991
6 0 00 65044992 16790400 81835391

root@xxxxxxxx# prtvtoc -s /dev/rdsk/c1t2d0s2
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 31504896 16790400 48295295
1 3 01 40704 31464192 31504895
2 5 00 0 286698624 286698623
3 15 01 20352 20352 40703
4 14 01 40704 286657920 286698623

- Take a snapshot of vfstab file.

root@xxxxxxxx# cat /etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/vx/dsk/bootdg/swapvol - - swap - no nologging
/dev/vx/dsk/bootdg/swapvol2 - - swap - no nologging
/dev/vx/dsk/bootdg/rootvol /dev/vx/rdsk/bootdg/rootvol / ufs 1 no nologging
/dev/vx/dsk/bootdg/var /dev/vx/rdsk/bootdg/var /var ufs 1 no nologging,nosuid
/dev/vx/dsk/bootdg/opt /dev/vx/rdsk/bootdg/opt /opt ufs 2 yes nologging
/dev/vx/dsk/bootdg/crash /dev/vx/rdsk/bootdg/crash /var/crash ufs 2 yes nologging
/dev/vx/dsk/bootdg/oracle /dev/vx/rdsk/bootdg/oracle /opt/oracle vxfs 2 yes -
/dev/vx/dsk/bootdg/opt_openv /dev/vx/rdsk/bootdg/opt_openv /opt/openv vxfs 2 yes -
/dev/vx/dsk/bootdg/pa /dev/vx/rdsk/bootdg/pa /opt/pa vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid
/dev/vx/dsk/adbm/oracle /dev/vx/rdsk/adbm/oracle /adbm/oracle vxfs 3 yes -
/dev/vx/dsk/adbm/devel /dev/vx/rdsk/adbm/devel /adbm/devel vxfs 3 yes -
/dev/vx/dsk/adbm/adbm_db /dev/vx/rdsk/adbm/adbm_db /adbm/db vxfs 3 yes -
/dev/vx/dsk/adbm/dumps /dev/vx/rdsk/adbm/dumps /adbm/dumps vxfs 3 yes -

- Take a snapshot of pre-VxVM vfstab file

root@xxxxxxxx# cat /etc/vfstab.prevm
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no nologging
/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /var ufs 1 no nologging
/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /opt ufs 2 yes nologging
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes -

- Take a snapshot of rootdg configuration.

root@xxxxxxxx# vxprint -qht -g rootdg > /var/tmp/rootdg.vxprint
root@xxxxxxxx# cat /var/tmp/rootdg.vxprint
dg rootdg default default 72000 1175781890.8.xxxxxxxx

dm hot01 Disk_4 auto 20095 286657920 SPARE
dm rootdisk Disk_0 auto 40703 286657920 -
dm rootmirror Disk_2 auto 20095 286657920 -

v crash - ENABLED ACTIVE 31457280 SELECT - fsgen
pl crash-01 crash ENABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-05 crash-01 rootdisk 81835391 31464192 0 Disk_0 ENA
pl crash-02 crash ENABLED ACTIVE 31464192 CONCAT - RW
sd rootmirror-05 crash-02 rootmirror 81835392 31464192 0 Disk_2 ENA

v opt - ENABLED ACTIVE 16790400 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-03 opt-01 rootdisk 65044991 16790400 0 Disk_0 ENA
pl opt-02 opt ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 65044992 16790400 0 Disk_2 ENA

v opt_openv - ENABLED ACTIVE 4194304 SELECT - fsgen
pl opt_openv-01 opt_openv ENABLED ACTIVE 4212864 CONCAT - RW
sd rootdisk-07 opt_openv-01 rootdisk 134282495 4212864 0 Disk_0 ENA
pl opt_openv-02 opt_openv ENABLED ACTIVE 4212864 CONCAT - RW
sd rootmirror-07 opt_openv-02 rootmirror 134282496 4212864 0 Disk_2 ENA

v oracle - ENABLED ACTIVE 20971520 SELECT - fsgen
pl oracle-01 oracle ENABLED ACTIVE 20982912 CONCAT - RW
sd rootdisk-06 oracle-01 rootdisk 113299583 20982912 0 Disk_0 ENA
pl oracle-02 oracle ENABLED ACTIVE 20982912 CONCAT - RW
sd rootmirror-06 oracle-02 rootmirror 113299584 20982912 0 Disk_2 ENA

v pa - ENABLED ACTIVE 8388608 SELECT - fsgen
pl pa-01 pa ENABLED ACTIVE 8405376 CONCAT - RW
sd rootdisk-09 pa-01 rootdisk 169226879 8405376 0 Disk_0 ENA
pl pa-02 pa ENABLED ACTIVE 8405376 CONCAT - RW
sd rootmirror-09 pa-02 rootmirror 169226880 8405376 0 Disk_2 ENA

v rootvol - ENABLED ACTIVE 16790400 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 31464191 16790400 0 Disk_0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-02 rootvol-02 rootmirror 31464192 16790400 0 Disk_2 ENA

v swapvol - ENABLED ACTIVE 31464192 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-B0 swapvol-01 rootdisk 286657919 1 0 Disk_0 ENA
sd rootdisk-01 swapvol-01 rootdisk 0 31464191 1 Disk_0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 31464192 CONCAT - RW
sd rootmirror-01 swapvol-02 rootmirror 0 31464192 0 Disk_2 ENA

v swapvol2 - ENABLED ACTIVE 30720000 SELECT - fsgen
pl swapvol2-01 swapvol2 ENABLED ACTIVE 30731520 CONCAT - RW
sd rootdisk-08 swapvol2-01 rootdisk 138495359 30731520 0 Disk_0 ENA
pl swapvol2-02 swapvol2 ENABLED ACTIVE 30731520 CONCAT - RW
sd rootmirror-08 swapvol2-02 rootmirror 138495360 30731520 0 Disk_2 ENA

v var - ENABLED ACTIVE 16790400 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 48254591 16790400 0 Disk_0 ENA
pl var-02 var ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 48254592 16790400 0 Disk_2 ENA

root@xxxxxxxx# vxprint -v -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v crash fsgen ENABLED 31457280 - ACTIVE - -
v opt fsgen ENABLED 16790400 - ACTIVE - -
v opt_openv fsgen ENABLED 4194304 - ACTIVE - -
v oracle fsgen ENABLED 20971520 - ACTIVE - -
v pa fsgen ENABLED 8388608 - ACTIVE - -
v rootvol root ENABLED 16790400 - ACTIVE - -
v swapvol swap ENABLED 31464192 - ACTIVE - -
v swapvol2 fsgen ENABLED 30720000 - ACTIVE - -
v var fsgen ENABLED 16790400 - ACTIVE - -

- Take a snapshot of all disks on system.

root@xxxxxxxx# vxdisk -eo alldgs list > /var/tmp/vxdisk_list.output
root@xxxxxxxx# cat /var/tmp/vxdisk_list.output
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
Disk_0 auto rootdisk rootdg online c1t0d0s2
Disk_2 auto rootmirror rootdg online c1t2d0s2
Disk_4 auto hot01 rootdg online c1t1d0s2
Disk_5 auto - - online c1t3d0s2
clar044_0 auto clar044_0 adbm online c4t5006016939A014F9d4s2
clar044_1 auto clar044_1 adbm online c4t5006016939A014F9d5s2
clar044_2 auto clar044_2 adbm online c4t5006016939A014F9d2s2
clar044_3 auto clar044_3 adbm online c4t5006016939A014F9d7s2
clar044_4 auto clar044_4 adbm online c4t5006016939A014F9d9s2
clar044_5 auto clar044_5 adbm online c4t5006016939A014F9d3s2
clar044_6 auto clar044_6 adbm online c4t5006016939A014F9d12s2
clar044_7 auto clar044_7 adbm online c4t5006016939A014F9d10s2
clar044_8 auto clar044_8 adbm online c4t5006016939A014F9d8s2
clar044_9 auto clar044_9 adbm online c4t5006016939A014F9d6s2
clar044_10 auto clar044_10 adbm online c4t5006016939A014F9d0s2
clar044_11 auto clar044_11 adbm online c4t5006016939A014F9d1s2
clar044_12 auto clar044_12 adbm online c4t5006016939A014F9d11s2
clar045_0 auto clar045_0 adbm online c4t5006016939A01AE9d4s2
clar045_1 auto clar045_1 adbm online c4t5006016939A01AE9d3s2
clar045_2 auto clar045_2 adbm online c4t5006016939A01AE9d7s2
clar045_3 auto clar045_3 adbm online c4t5006016939A01AE9d6s2
clar045_4 auto clar045_4 adbm online c4t5006016939A01AE9d11s2
clar045_5 auto clar045_5 adbm online c4t5006016939A01AE9d0s2
clar045_6 auto clar045_6 adbm online c4t5006016939A01AE9d2s2
clar045_7 auto clar045_7 adbm online c4t5006016939A01AE9d12s2
clar045_8 auto clar045_8 adbm online c4t5006016939A01AE9d8s2
clar045_9 auto clar045_9 adbm online c4t5006016939A01AE9d1s2
clar045_10 auto clar045_10 adbm online c4t5006016939A01AE9d10s2
clar045_11 auto clar045_11 adbm online c4t5006016939A01AE9d9s2
clar045_12 auto clar045_12 adbm online c4t5006016939A01AE9d5s2

- Detach all the plexes associated with the 'rootmirror' disk if applicable & Verify the rootmirror plexes have been detached

root@xxxxxxxx# vxprint -qhtg rootdg -s | grep -i rootmirror | awk '{print $3}' > /var/tmp/subs.plex
root@xxxxxxxx# cat /var/tmp/subs.plex
swapvol-02
rootvol-02
var-02
opt-02
crash-02
oracle-02
opt_openv-02
swapvol2-02
pa-02

root@xxxxxxxx# for x in `cat /var/tmp/subs.plex`
> do
> vxplex -g rootdg dis $x
> vxprint -qhtg rootdg -p $x
> done
pl swapvol-02 - DISABLED - 31464192 CONCAT - RW
sd rootmirror-01 swapvol-02 rootmirror 0 31464192 0 Disk_2 ENA
pl rootvol-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-02 rootvol-02 rootmirror 31464192 16790400 0 Disk_2 ENA
pl var-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 48254592 16790400 0 Disk_2 ENA
pl opt-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 65044992 16790400 0 Disk_2 ENA
pl crash-02 - DISABLED - 31464192 CONCAT - RW
sd rootmirror-05 crash-02 rootmirror 81835392 31464192 0 Disk_2 ENA
pl oracle-02 - DISABLED IOFAIL 20982912 CONCAT - RW
sd rootmirror-06 oracle-02 rootmirror 113299584 20982912 0 Disk_2 ENA
pl opt_openv-02 - DISABLED IOFAIL 4212864 CONCAT - RW
sd rootmirror-07 opt_openv-02 rootmirror 134282496 4212864 0 Disk_2 ENA
pl swapvol2-02 - DISABLED - 30731520 CONCAT - RW
sd rootmirror-08 swapvol2-02 rootmirror 138495360 30731520 0 Disk_2 ENA
pl pa-02 - DISABLED IOFAIL 8405376 CONCAT - RW
sd rootmirror-09 pa-02 rootmirror 169226880 8405376 0 Disk_2 ENA

- Determine a disk with a valid configuration copy

root@xxxxxxxx# vxdg list rootdg > /var/tmp/vxdg.list.rootdg.ouput
root@xxxxxxxx# cat /var/tmp/vxdg.list.rootdg.ouput
Group: rootdg
dgid: 1175781890.8.xxxxxxxx
import-id: 0.1
flags:
version: 120
alignment: 512 (bytes)
ssb: on
detach-policy: global
dg-fail-policy: dgdisable
copies: nconfig=default nlog=default
config: seqno=0.3810 permlen=14803 free=14771 templen=17 loglen=2243
config disk Disk_0 copy 1 len=30015 state=clean online
config disk Disk_2 copy 1 len=14803 state=clean online
config disk Disk_4 copy 1 len=14803 state=clean online
log disk Disk_0 copy 1 len=4547
log disk Disk_2 copy 1 len=2243
log disk Disk_4 copy 1 len=2243

Check location of the private region on a disk with a valid configuration copy as shown above

NOTE: For non-EFI disks of type sliced, VxVM usually configures partition s3 as the private region, s4 as the public region, and s2 as the entire physical disk. An exception is an encapsulated root disk, on which s3 is usually configured as the public region and s4 as the private region.

In our case s3 has private region.

root@xxxxxxxx# prtvtoc -s /dev/rdsk/c1t2d0s2 > /var/tmp/prtvtoc.c0t2d0s2.output
root@xxxxxxxx# cat /var/tmp/prtvtoc.c0t2d0s2.output
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 31504896 16790400 48295295
1 3 01 40704 31464192 31504895
2 5 00 0 286698624 286698623
3 15 01 20352 20352 40703
4 14 01 40704 286657920 286698623

- Record the rootdg disk group configuration via the 'vxprivutil' command

root@xxxxxxxx# /etc/vx/diag.d/vxprivutil dumpconfig /dev/rdsk/c1t2d0s3 > /var/tmp/rootdg.conf

To verify the vxprivutil output use the following command

root@xxxxxxxx# cat /var/tmp/rootdg.conf | vxprint -D - -qht
Disk group: rootdg
dg rootdg default default 72000 1175781890.8.xxxxxxxx

dm hot01 - - - - SPARE
dm rootdisk - - - - -
dm rootmirror - - - - -

pl crash-02 - DISABLED - 31464192 CONCAT - RW
sd rootmirror-05 crash-02 rootmirror 81835392 31464192 0 - DIS

pl opt-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 65044992 16790400 0 - DIS

pl opt_openv-02 - DISABLED IOFAIL 4212864 CONCAT - RW
sd rootmirror-07 opt_openv-02 rootmirror 134282496 4212864 0 - DIS

pl oracle-02 - DISABLED IOFAIL 20982912 CONCAT - RW
sd rootmirror-06 oracle-02 rootmirror 113299584 20982912 0 - DIS

pl pa-02 - DISABLED IOFAIL 8405376 CONCAT - RW
sd rootmirror-09 pa-02 rootmirror 169226880 8405376 0 - DIS

pl rootvol-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-02 rootvol-02 rootmirror 31464192 16790400 0 - DIS

pl swapvol-02 - DISABLED - 31464192 CONCAT - RW
sd rootmirror-01 swapvol-02 rootmirror 0 31464192 0 - DIS

pl swapvol2-02 - DISABLED - 30731520 CONCAT - RW
sd rootmirror-08 swapvol2-02 rootmirror 138495360 30731520 0 - DIS

pl var-02 - DISABLED - 16790400 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 48254592 16790400 0 - DIS

v crash - DISABLED ACTIVE 31457280 SELECT - fsgen
pl crash-01 crash DISABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-05 crash-01 rootdisk 81835391 31464192 0 - DIS

v opt - DISABLED ACTIVE 16790400 ROUND - fsgen
pl opt-01 opt DISABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-03 opt-01 rootdisk 65044991 16790400 0 - DIS

v opt_openv - DISABLED ACTIVE 4194304 SELECT - fsgen
pl opt_openv-01 opt_openv DISABLED ACTIVE 4212864 CONCAT - RW
sd rootdisk-07 opt_openv-01 rootdisk 134282495 4212864 0 - DIS

v oracle - DISABLED ACTIVE 20971520 SELECT - fsgen
pl oracle-01 oracle DISABLED ACTIVE 20982912 CONCAT - RW
sd rootdisk-06 oracle-01 rootdisk 113299583 20982912 0 - DIS

v pa - DISABLED ACTIVE 8388608 SELECT - fsgen
pl pa-01 pa DISABLED ACTIVE 8405376 CONCAT - RW
sd rootdisk-09 pa-01 rootdisk 169226879 8405376 0 - DIS

v rootvol - DISABLED ACTIVE 16790400 ROUND - root
pl rootvol-01 rootvol DISABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 31464191 16790400 0 - DIS

v swapvol - DISABLED ACTIVE 31464192 ROUND - swap
pl swapvol-01 swapvol DISABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-B0 swapvol-01 rootdisk 286657919 1 0 - DIS
sd rootdisk-01 swapvol-01 rootdisk 0 31464191 1 - DIS

v swapvol2 - DISABLED ACTIVE 30720000 SELECT - fsgen
pl swapvol2-01 swapvol2 DISABLED ACTIVE 30731520 CONCAT - RW
sd rootdisk-08 swapvol2-01 rootdisk 138495359 30731520 0 - DIS

v var - DISABLED ACTIVE 16790400 ROUND - fsgen
pl var-01 var DISABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 48254591 16790400 0 - DIS

Well, now we are good to execute vxunroot. We have captured as much as information and we know what was the last good configuration, in case something goes wrong with help of above all information we can fix it.

PS: The 'vxunroot' script will fail if you have other than standard OS volumes (non-system volumes) configured in the rootdg disk group.

root@xxxxxxxx# /etc/vx/bin/vxunroot

   /etc/vx/bin/vxunroot: Error: Disk contains more than 5
   volumes which cannot be unencapsulated. Move some volumes
   to different disk and try again

Now remove the non-system volume with the following command, In this case remove opt_openv, oracle & pa from the rootdg disk group

root@xxxxxxxx# ls -ld /opt/openv
drwxr-xr-x 15 root other 1024 Oct 23 2007 /opt/openv
root@xxxxxxxx# ls -ld /opt/pa
drwxr-xr-x 34 pa dba 2048 Jan 4 2010 /opt/pa
root@xxxxxxxx# ls -ld /opt/oracle/
drwxr-xr-x 6 oracle dba 2048 Feb 9 11:07 /opt/oracle/
root@xxxxxxxx# ls -ld /var/crash/
drwxr-xr-x 4 root root 512 Feb 28 2008 /var/crash/

- Un-mount the filesystems.

root@xxxxxxxx# for vol in /opt/openv /opt/pa /opt/oracle/ /var/crash/
> do
> umount $vol
> done

- Remove volumes.

root@xxxxxxxx# for vol in opt_openv oracle pa crash
> do
> vxedit -g rootdg -rf rm $vol
> done

- Execute "vxunroot"

root@xxxxxxxx# vxunroot
    VxVM vxunroot INFO V-5-2-1562
The following volumes detected on the root disk of your system are not
   derivatives of partitions that were on the pre-encapsulated root disk:

       swapvol2
   VxVM vxunroot NOTICE V-5-2-1560
Please move these volumes and comment out corresponding entries in
/etc/vfstab before you rerun vxunroot.

Ahh.. Error!!! If vxunroot detects volumes other than standard OS volumes then it fails with below error. To get rid of this message let’s remove this extra/added volume.

root@xxxxxxxx# vxedit -g rootdg -rf rm swapvol2
VxVM vxedit ERROR V-5-1-1242 Volume swapvol2 is opened, cannot remove
root@xxxxxxxx# swap -l
swapfile dev swaplo blocks free
/dev/vx/dsk/bootdg/swapvol 270,72002 16 31464176 31401184
/dev/vx/dsk/bootdg/swapvol2 270,72006 16 30719984 30656448
root@xxxxxxxx# swap -d /dev/vx/dsk/bootdg/swapvol2
root@xxxxxxxx# vxedit -g rootdg -rf rm swapvol2

root@xxxxxxxx# vxunroot
    VxVM vxunroot NOTICE V-5-2-1564
This operation will convert the following file systems from
   volumes to regular partitions:

          opt rootvol swapvol var
    VxVM vxunroot INFO V-5-2-2011
Replacing volumes in root disk to partitions will require a system
    reboot. If you choose to continue with this operation, system
    configuration will be updated to discontinue use of the volume
    manager for your root and swap devices.

Do you wish to do this now [y,n,q,?] (default: y) y
   VxVM vxunroot INFO V-5-2-287 Restoring kernel configuration...
   VxVM vxunroot INFO V-5-2-78
A shutdown is now required to install the new kernel.
    You can choose to shutdown now, or you can shutdown later, at your
convenience.

Do you wish to shutdown now [y,n,q,?] (default: n)
    VxVM vxunroot INFO V-5-2-258
Please shutdown before you perform any additional volume manager
    or disk reconfiguration. To shutdown your system cd to / and type

         shutdown -g0 -y -i6

PS: Make sure any data disk group has to be imported before shutdown or in fact before running vxunroot.

Well, after reboot I’m on the normal slice based UFS filesystems –

root@xxxxxxxx# df -kh
Filesystem size used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7.9G 4.5G 3.3G 58% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 41G 1.2M 41G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1
7.9G 4.5G 3.3G 58% /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
7.9G 4.5G 3.3G 58% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/dsk/c1t0d0s5 7.9G 5.4G 2.5G 69% /var
swap 41G 176K 41G 1% /tmp
swap 41G 56K 41G 1% /var/run
/dev/dsk/c1t0d0s6 7.9G 6.5G 1.3G 83% /opt

root@xxxxxxxx# cat /etc/release
Solaris 10 11/06 s10s_u3wos_10 SPARC
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 14 November 2006

Now let’s start with preparing environment for LiveUpgrade.

- Copy VTOC table of primary boot disk to the disk on which you wish to create ABE.

root@xxxxxxxx# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t2d0s2

- Remove LU packages from system and re-install LU packages from U9 media.

root@xxxxxxxx# pkginfo -x| grep "Live Upgrade"
SUNWlucfg Live Upgrade Configuration
SUNWlur Live Upgrade (root)
SUNWluu Live Upgrade (usr)

- Verify if all required patches for LU are installed on system.

root@xxxxxxxx# patchadd -p | grep 121430
Patch: 121430-53 Obsoletes: 121435-04 121437-02 Requires: Incompatibles: Packages: SUNWlucfg SUNWlur SUNWluu
Patch: 121430-57 Obsoletes: 121435-04 121437-02 Requires: Incompatibles: Packages: SUNWlucfg SUNWlur SUNWluu
Patch: 121430-68 Obsoletes: 121435-04 121437-02 Requires: Incompatibles: Packages: SUNWlucfg SUNWlur SUNWluu
Patch: 121430-72 Obsoletes: 121435-04 121437-02 Requires: Incompatibles: Packages: SUNWlucfg SUNWlur SUNWluu

root@xxxxxxxx# cat /etc/vfstab |grep c1t0d0 |awk '{print $1, $3}'
/dev/dsk/c1t0d0s1 -
/dev/dsk/c1t0d0s0 /
/dev/dsk/c1t0d0s5 /var
/dev/dsk/c1t0d0s6 /opt

- Create alternate boot environment.

root@xxxxxxxx# time lucreate -c Sol10u3 -C /dev/dsk/c1t2d0s2 -m -:/dev/dsk/c1t2d0s1:swap -m /:/dev/dsk/c1t2d0s0:ufs -m /var:/dev/dsk/c1t2d0s5:ufs -m /opt:/dev/dsk/c1t2d0s6:ufs -n "Sol10u9"
Determining types of file systems supported
Validating file system requests
Preparing logical storage devices
Preparing physical storage devices
Configuring physical storage devices
Configuring logical storage devices
Analyzing system configuration.
No name for current boot environment.
Current boot environment is named Sol10u3
Creating initial configuration for primary boot environment Sol10u3
The device /dev/dsk/c1t2d0s2 is not a root device for any boot environment; cannot get BE ID.
PBE configuration successful: PBE name Sol10u3 PBE Boot Device /dev/dsk/c1t2d0s2
Comparing source boot environment Sol10u3 file systems with the file system(s) you specified for the new boot environment. Determining which file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
The device /dev/dsk/c1t2d0s0 is not a root device for any boot environment; cannot get BE ID.
Creating configuration for boot environment Sol10u9
Source boot environment is Sol10u3
Creating boot environment Sol10u9
Creating file systems on boot environment Sol10u9
Creating ufs file system for / in zone (global) on /dev/dsk/c1t2d0s0
Creating ufs file system for /opt in zone (global) on /dev/dsk/c1t2d0s6
Creating ufs file system for /var in zone (global) on /dev/dsk/c1t2d0s5
Mounting file systems for boot environment Sol10u9
Calculating required sizes of file systems for boot environment Sol10u9
Populating file systems on boot environment Sol10u9
Checking selection integrity.
Integrity check OK.
Populating contents of mount point /
Populating contents of mount point /opt
Populating contents of mount point /var
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment Sol10u9
Creating compare database for file system /var
Creating compare database for file system /opt
Creating compare database for file system /
Updating compare databases on boot environment Sol10u9.
Making boot environment Sol10u9 bootable.
Population of boot environment Sol10u9 successful.
Creation of boot environment Sol10u9 successful.

real 40m17.793s
user 4m45.033s
sys 4m28.322s

root@xxxxxxxx# lustatus

Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u3 yes yes yes no -
Sol10u9 yes no no yes -

First task is to now patch the system to the latest patch level – 144488-12 (Kernel Bug Fixes post Solaris 10 9/10 (update 9)

All patches are kept under directory /var/logging/patching hence

# cd /var/logging/patching
# luupgrade -n Sol10u9 -s /var/logging/patching -t `cat patch_order`

Once patching done, move forward to OS upgrade.
Let’s now try only dry-run (option –N) to see 'projection' of upgrading new BE to Solaris 10 U9.

root@xxxxxxxx# luupgrade -u -n Sol10u9 -N -s /mnt/
63521 blocks
miniroot filesystem is lofs
Mounting miniroot at /mnt//Solaris_10/Tools/Boot
ERROR: The auto registration file () does not exist or incomplete.
The auto registration file is mandatory for this upgrade.
Use -k (filename) argument along with luupgrade command.

This is known issue. If you try to update to the latest Solaris 10 Update (U9), one new step is now required in order to be able to successfully luupgrade to the desired Update. As mentioned in the Oracle Solaris 10 9/10 Release Notes, a new Auto Registration mechanism has been added to this release to facilitate registering the system using your Oracle support credentials.

Well, if you try the classical luupgrade following incantation, it will fail with the above reported message

So, you now need to set the Auto Registration choice as a mandatory parameter. Here is how it resembles right now:

root@xxxxxxxx# echo "auto_reg=disable" > /tmp/sysidcfg
root@xxxxxxxx# luupgrade -u -n Sol10u9 -N -s /mnt/ -k /tmp/sysidcfg
63521 blocks
miniroot filesystem is lofs
Mounting miniroot at /mnt//Solaris_10/Tools/Boot
#######################################################################
NOTE: To improve products and services, Oracle Solaris communicates
configuration data to Oracle after rebooting.
You can register your version of Oracle Solaris to capture this data
for your use, or the data is sent anonymously.
For information about what configuration data is communicated and how
to control this facility, see the Release Notes or
www.oracle.com/goto/solarisautoreg.
INFORMATION: After activated and booted into new BE Sol10u9,
Auto Registration happens automatically with the following Information
autoreg=disable
#######################################################################
pktool: Invalid verb: genkey
encrypt: cannot open /var/run/autoreg_key
encrypt: invalid key.
Validating the contents of the media /mnt/
The media is a standard Solaris media.
The media contains an operating system upgrade image.
The media contains version 10
Constructing upgrade profile to use.
Locating the operating system upgrade program.
Checking for existence of previously scheduled Live Upgrade requests.
Creating upgrade profile for BE Sol10u9
Performing the operating system upgrade of the BE Sol10u9
Execute Command: /mnt//Solaris_10/Tools/Boot/usr/sbin/install.d/pfinstall -L /a -c /mnt/ /tmp/.liveupgrade.23762.28500/.luupgrade.profile.upgrade
Adding operating system patches to the BE Sol10u9
Execute Command /mnt//Solaris_10/Tools/Boot/usr/sbin/install.d/install_config/patch_finish -R "/a" -c "/mnt/"

This looks okay. Let’s run it in real.

root@xxxxxxxx# time luupgrade -u -n Sol10u9 -s /sol10u9 -k /tmp/sysidcfg
63521 blocks
miniroot filesystem is lofs
Mounting miniroot at /sol10u9/Solaris_10/Tools/Boot
#######################################################################
NOTE: To improve products and services, Oracle Solaris communicates configuration data to Oracle after rebooting.
You can register your version of Oracle Solaris to capture this data for your use, or the data is sent anonymously.
For information about what configuration data is communicated and how
to control this facility, see the Release Notes or
www.oracle.com/goto/solarisautoreg.
INFORMATION: After activated and booted into new BE Sol10u9,
Auto Registration happens automatically with the following Information
autoreg=disable
#######################################################################
pktool: Invalid verb: genkey
encrypt: cannot open /var/run/autoreg_key
encrypt: invalid key.
Validating the contents of the media {/sol10u9}.
The media is a standard Solaris media.
The media contains an operating system upgrade image.
The media contains version 10.
Constructing upgrade profile to use.
Locating the operating system upgrade program.
Checking for existence of previously scheduled Live Upgrade requests.
Creating upgrade profile for BE Sol10u9
Determining packages to install or upgrade for BE Sol10u9.
Performing the operating system upgrade of the BE Sol10u9.
CAUTION: Interrupting this process may leave the boot environment unstable or unbootable.

ERROR: Installation of the packages from this media of the media failed; pfinstall returned these diagnostics:

Processing profile

Loading local environment and services
ERROR: Failure loading local environment
Stat failed: /a///var/sadm/install_data/.clustertoc

INFORMATION: The file /var/sadm/system/logs/upgrade_log on boot environment Sol10u9 contains a log of the upgrade operation.
INFORMATION: Review the files listed above. Remember that all of the files are located on boot environment . Before you activate boot environment , determine if any additional system maintenance is required or if additional media of the software distribution must be installed.
The Solaris upgrade of the boot environment Sol10u9 failed.
Installing failsafe
Failsafe install is complete.

real 0m58.864s
user 0m10.915s
sys 0m23.483s

Ohh.. boy... ERROR!!! Need to find some solution...

To get rid of this issue, I manually copied files /var/sadm/system/admin/.clustertoc & /var/sadm/system/admin/CLUSTER to ABE by mounting it to directory /a/var/sadm/install_data/. Unbelievable!!! Upgrade started working. I’m still not sure why lucreate or in fact pfinstall looking for .clustertoc file at /var/sadm/install_data than its default location /var/sadm/system/admin. The reason might be being u3 is very old and loaded with bugs.

root@xxxxxxxx# time luupgrade -u -n Sol10u9 -s /sol10u9 -k /tmp/sysidcfg
63521 blocks
miniroot filesystem is lofs
Mounting miniroot at /sol10u9/Solaris_10/Tools/Boot
#######################################################################
NOTE: To improve products and services, Oracle Solaris communicates configuration data to Oracle after rebooting.
You can register your version of Oracle Solaris to capture this data for your use, or the data is sent anonymously.
For information about what configuration data is communicated and how
to control this facility, see the Release Notes or
www.oracle.com/goto/solarisautoreg.
INFORMATION: After activated and booted into new BE Sol10u9, Auto Registration happens automatically with the following Information
autoreg=disable
######################################################################
pktool: Invalid verb: genkey

encrypt: cannot open /var/run/autoreg_key
encrypt: invalid key.
Validating the contents of the media {/sol10u9}.
The media is a standard Solaris media.
The media contains an operating system upgrade image.
The media contains version (10).
Constructing upgrade profile to use.
Locating the operating system upgrade program.
Checking for existence of previously scheduled Live Upgrade requests.
Creating upgrade profile for BE {Sol10u9}.
Determining packages to install or upgrade for BE {Sol10u9}.
Performing the operating system upgrade of the BE {Sol10u9}.
CAUTION: Interrupting this process may leave the boot environment unstable or unbootable.
Installation of the packages from this media is complete.
Upgrading Solaris: 100% completed
Updating package information on boot environment {Sol10u9}.
Package information successfully updated on boot environment .
Adding operating system patches to the BE {Sol10u9}.
The operating system patch installation is complete.
INFORMATION: The file (/var/sadm/system/logs/upgrade_log) on boot environment contains a log of the upgrade operation.
INFORMATION: The file (/var/sadm/system/data/upgrade_cleanup) on boot environment {Sol10u9} contains a log of cleanup operations required.
WARNING: (1) packages failed to install properly on boot environment {Sol10u9}.
INFORMATION: The file {/var/sadm/system/data/upgrade_failed_pkgadds} on boot environment {Sol10u9} contains a list of packages that failed to upgrade or install properly.
INFORMATION: Review the files listed above. Remember that all of the files are located on boot environment . Before you activate boot environment {Sol10u9}, determine if any additional system maintenance is required or if additional media of the software distribution must be installed.
The Solaris upgrade of the boot environment {Sol10u9} is partially complete.
Installing failsafe
Failsafe install is complete.

real 20m4.765s
user 6m53.061s
sys 3m33.470s

- Activate the ABE

root@xxxxxxxx# luactivate Sol10u9
A Live Upgrade Sync operation will be performed on startup of boot environment Sol10u9
WARNING: 1 packages failed to install properly on boot environment Sol10u9
INFORMATION: /var/sadm/system/data/upgrade_failed_pkgadds on boot environment Sol10u9 contains a list of packages that failed to upgrade or install properly. Review the file before you reboot the system to determine if any additional system maintenance is required.
WARNING: The following files have changed on both the current boot environment Sol10u3 zone global and the boot environment to be activated :
/var/mail/root-noc
INFORMATION: The files listed above are in conflict between the current boot environment Sol10u3 zone global and the boot environment to be activated Sol10u9 These files will not be automatically synchronized from the current boot environment Sol10u3 when boot environment Sol10u9 is activated.
**********************************************************************
The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the target BE.
**********************************************************************
In case of a failure while booting to the target BE, the following process needs to be followed to fallback to the currently working boot environment:

1. Enter the PROM monitor (ok prompt).

2. Change the boot device back to the original boot environment by typing:
setenv boot-device
/pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e014334f61,0:a

3. Boot to the original boot environment by typing:
boot
**********************************************************************
Modifying boot archive service
Activation of boot environment Sol10u9 successful.

Let’s reboot the server from activated ABE.

root@xxxxxxxx# sync;sync;sync;shutdown -y -i6 -g0

root@xxxxxxxx# cat /etc/release && echo "--------- `uname -a` ----------"
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
--------- SunOS xxxxxxxx 5.10 Generic_144488-12 sun4u sparc SUNW,Sun-Fire-V890 ----------

root@xxxxxxxx# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u3 yes yes no no -
Sol10u9 yes no yes no -

After system came up – I’m on Solaris 10 U9 & latest patch level. Cool. Milestone #1 is finished. Let’s move forward towards Milestone #2.

Milestone #2 talks about – encapsulating disk into rootdg, created added/extra volumes, restore them (if required), delete PBE and mirror second disk to rootdg, import the data disk group.

To encapsulate the boot drive again run the vxdiskadm command selecting option 2.

# vxdiskadm (option 2)

[Many lines here, skipped for brevity]

The Disk_2 disk has been configured for encapsulation.
The first stage of encapsulation has completed successfully. You should now reboot your system at the earliest possible opportunity. The encapsulation will require two or three reboots which will happen automatically after the next reboot. To reboot execute the command:

shutdown -g0 -y -i6

This will update the /etc/vfstab file so that volume devices are used to mount the file systems on this disk device. You will need to update any other references such as backup scripts, databases, or manually created swap devices.

After server reboot, you will find your operating systems volumes as a part of VERITAS Volume Manager.

root@xxxxxxxx# df -kh
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/bootdg/rootvol 7.9G 5.9G 1.9G 76% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 41G 1.6M 41G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1
7.9G 5.9G 1.9G 76% /platform/sun4u-us3/lib/libc_psr.so.1
platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
7.9G 5.9G 1.9G 76% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/vx/dsk/bootdg/var 7.9G 2.6G 5.2G 34% /var
swap 41G 272K 41G 1% /tmp
swap 41G 56K 41G 1% /var/run
swap 41G 0K 41G 0% /dev/vx/dmp
swap 41G 0K 41G 0% /dev/vx/rdmp
/dev/vx/dsk/bootdg/opt 7.9G 6.5G 1.3G 83% /opt

Now create extra volumes which we removed at the time of vxunroot. (opt_openv, oracle, pa, crash & swapvol2)

- Creation of swap volume

root@xxxxxxxx# vxassist -g rootdg make swapvol2 15g
root@xxxxxxxx# swap -a /dev/vx/dsk/rootdg/swapvol2
root@xxxxxxxx# swap -l
swapfile dev swaplo blocks free
/dev/vx/dsk/bootdg/swapvol 270,72002 16 31464176 31464176
/dev/vx/dsk/rootdg/swapvol2 270,72007 16 31457264 31457264

- Create volumes

root@xxxxxxxx# vxassist -g rootdg make opt_openv 2g
root@xxxxxxxx# vxassist -g rootdg make pa 4g
root@xxxxxxxx# vxassist -g rootdg make oracle 10g
root@xxxxxxxx# vxassist -g rootdg make crash 15g

- Create VxFS filesystems

root@xxxxxxxx# for vol in opt_openv pa oracle crash
> do
> mkfs -F vxfs /dev/vx/rdsk/rootdg/$vol
> done
version 6 layout
4194304 sectors, 2097152 blocks of size 1024, log size 16384 blocks
largefiles supported
version 6 layout
8388608 sectors, 4194304 blocks of size 1024, log size 16384 blocks
largefiles supported
version 6 layout
20971520 sectors, 10485760 blocks of size 1024, log size 16384 blocks
largefiles supported
version 6 layout
31457280 sectors, 15728640 blocks of size 1024, log size 16384 blocks
largefiles supported

- Create mountpoints.

root@xxxxxxxx# mkdir -p /opt/openv /opt/pa /opt/oracle

Define entries in vfstab

/dev/vx/dsk/bootdg/crash /dev/vx/rdsk/bootdg/crash /var/crash ufs 2 yes nologging
/dev/vx/dsk/bootdg/oracle /dev/vx/rdsk/bootdg/oracle /opt/oracle vxfs 2 yes -
/dev/vx/dsk/bootdg/opt_openv /dev/vx/rdsk/bootdg/opt_openv /opt/openv vxfs 2 yes -
/dev/vx/dsk/bootdg/pa /dev/vx/rdsk/bootdg/pa /opt/pa vxfs 2 yes -

Mount these volumes with mountall command.

root@xxxxxxxx# df -kh /opt/openv /opt/pa /opt/oracle /var/crash

Filesystem size used avail capacity Mounted on
/dev/vx/dsk/bootdg/opt_openv 2.0G 17M 1.9G 1% /opt/openv
/dev/vx/dsk/bootdg/pa 4.0G 18M 3.7G 1% /opt/pa
/dev/vx/dsk/bootdg/oracle 10G 19M 9.4G 1% /opt/oracle
/dev/vx/dsk/bootdg/crash 15G 20M 14G 1% /var/crash

Well, now we will simply restore the backed up data for these volumes.

Next job is to import the disk group, which we deported at the time of vxunroot.

root@xxxxxxxx# vxdg import adbm

After executing import on DG there might be the case that all volumes belonging to this DG may be in state – “DISABLED CLEAN” so, start all the volumes manually using below command.

root@xxxxxxxx# vxvol -g adbm startall

Now you should be able to see all the volumes as “ENABLED ACTIVE”

Mount the FS belonging to this DG manually using mountall command –

root@xxxxxxxx# df -kh | grep adbm
/dev/vx/dsk/adbm/oracle 32G 75M 30G 1% /adbm/oracle
/dev/vx/dsk/adbm/adbm_db 233G 21G 198G 10% /adbm/db
/dev/vx/dsk/adbm/dumps 291G 216G 70G 76% /adbm/dumps
/dev/vx/dsk/adbm/devel 116G 17G 93G 16% /adbm/devel

Well, this is being 2 weeks now and no complaints so far from customer and he confirmed that everything looks perfect to him. Now we can proceed with post patch and upgrade stuffs.

I found a dirty but quick way to find out current boot disk, so let’s find out what is our boot disk.

root@xxxxxxxx# cat get_bootdisk
BOOTPATH=`prtconf -pv |grep bootpath | tr -d "'" | awk '{print $2}'`
if [ -n "`echo $BOOTPATH | grep "/disk"`" ] ; then
# The bootpath contains "disk," but the /devices block device contains
# either "sd" or "ssd"
BOOTPATH=`echo $BOOTPATH | sed 's/disk@//'`
BOOT_DISK=`ls -l /dev/dsk | sed -e 's/ssd@//' -e 's/sd@//' \ | grep "$BOOTPATH" 2>/dev/null | awk '{print $9}' | sed 's/s[0-7]//'`
else
BOOT_DISK=`ls -l /dev/dsk | grep "$BOOTPATH" 2>/dev/null | awk '{print $9}' | sed 's/s[0-7]//'`
fi

if [ -n "$BOOT_DISK" ] ; then
echo "Your boot disk is ${BOOT_DISK}."
else
echo "Unable to determine logical boot disk."
fi

root@xxxxxxxx# ./get_bootdisk
Your boot disk is c1t2d0.

Now that we don’t have to back out anymore so let’s delete the BE named “Sol10u3”

root@xxxxxxxx# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u3 yes no no yes -
Sol10u9 yes yes yes no -

root@xxxxxxxx# ludelete Sol10u3
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment Sol10u3 deleted.

Well, deleting boot environment Sol10u3 we got disk c1t0d0 free. Let’s add it to rootdg as mirror disk.

root@xxxxxxxx# vxdisksetup -if Disk_0 format=sliced
root@xxxxxxxx# vxdg -g rootdg adddisk rootmirror=Disk_0
# vxmirror -g rootdg rootdisk
! vxassist -g rootdg mirror swapvol
! vxassist -g rootdg mirror rootvol
! vxassist -g rootdg mirror var
! vxassist -g rootdg mirror opt
! vxassist -g rootdg mirror opt_openv
! vxassist -g rootdg mirror pa
! vxassist -g rootdg mirror oracle
! vxassist -g rootdg mirror crash
! vxassist -g rootdg mirror swapvol2
! vxbootsetup -g rootdg

You may check the copy progress using –

root@xxxxxxxx# vxtask -l list
Task: 168 RUNNING
Type: ATCOPY
Operation: PLXATT Vol opt Plex opt-02 Dg rootdg
Started: Fri Apr 27 10:28:04 2012
Throttle: 0
Progress: 39.37% 6610944 of 16790400 Blocks
Work time: 1 minute, 31 seconds (02:20 remaining)

So here is what we achieved!!!

To conclude this task,

- System has been upgraded to Solaris 10 Update 9
- System has been patched to latest patch level i.e. Generic_144488-12
- Completed post patch/upgrade things and system has now mirror volumes in place.

root@xxxxxxxx# cat /etc/release; echo "=========="; uname -a; echo "========="; vxprint -htg rootdg

Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
=========================================================
SunOS xxxxxxxx 5.10 Generic_144488-12 sun4u sparc SUNW,Sun-Fire-V890
=========================================================
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 72000 1175781890.8.xxxxxxxx

dm hot01 Disk_4 auto 20095 286657920 SPARE
dm rootdisk Disk_2 auto 4096 286657920 -
dm rootmirror Disk_0 auto 20095 286657920 -

v crash - ENABLED ACTIVE 31457280 SELECT - fsgen
pl crash-01 crash ENABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-08 crash-01 rootdisk 115436543 31464192 0 Disk_2 ENA
pl crash-02 crash ENABLED ACTIVE 31464192 CONCAT - RW
sd rootmirror-08 crash-02 rootmirror 115436544 31464192 0 Disk_0 ENA

v opt - ENABLED ACTIVE 16790400 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-03 opt-01 rootdisk 65044991 16790400 0 Disk_2 ENA
pl opt-02 opt ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 65044992 16790400 0 Disk_0 ENA

v opt_openv - ENABLED ACTIVE 4194304 SELECT - fsgen
pl opt_openv-01 opt_openv ENABLED ACTIVE 4212864 CONCAT - RW
sd rootdisk-05 opt_openv-01 rootdisk 81835391 4212864 0 Disk_2 ENA
pl opt_openv-02 opt_openv ENABLED ACTIVE 4212864 CONCAT - RW
sd rootmirror-05 opt_openv-02 rootmirror 81835392 4212864 0 Disk_0 ENA

v oracle - ENABLED ACTIVE 20971520 SELECT - fsgen
pl oracle-01 oracle ENABLED ACTIVE 20982912 CONCAT - RW
sd rootdisk-07 oracle-01 rootdisk 94453631 20982912 0 Disk_2 ENA
pl oracle-02 oracle ENABLED ACTIVE 20982912 CONCAT - RW
sd rootmirror-07 oracle-02 rootmirror 94453632 20982912 0 Disk_0 ENA

v pa - ENABLED ACTIVE 8388608 SELECT - fsgen
pl pa-01 pa ENABLED ACTIVE 8405376 CONCAT - RW
sd rootdisk-06 pa-01 rootdisk 86048255 8405376 0 Disk_2 ENA
pl pa-02 pa ENABLED ACTIVE 8405376 CONCAT - RW
sd rootmirror-06 pa-02 rootmirror 86048256 8405376 0 Disk_0 ENA

v rootvol - ENABLED ACTIVE 16790400 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 31464191 16790400 0 Disk_2 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-02 rootvol-02 rootmirror 31464192 16790400 0 Disk_0 ENA

v swapvol - ENABLED ACTIVE 31464192 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-B0 swapvol-01 rootdisk 286657919 1 0 Disk_2 ENA
sd rootdisk-01 swapvol-01 rootdisk 0 31464191 1 Disk_2 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 31464192 CONCAT - RW
sd rootmirror-01 swapvol-02 rootmirror 0 31464192 0 Disk_0 ENA

v swapvol2 - ENABLED ACTIVE 31457280 SELECT - fsgen
pl swapvol2-01 swapvol2 ENABLED ACTIVE 31464192 CONCAT - RW
sd rootdisk-09 swapvol2-01 rootdisk 146900735 31464192 0 Disk_2 ENA
pl swapvol2-02 swapvol2 ENABLED ACTIVE 31464192 CONCAT - RW
sd rootmirror-09 swapvol2-02 rootmirror 146900736 31464192 0 Disk_0 ENA

v var - ENABLED ACTIVE 16790400 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 16790400 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 48254591 16790400 0 Disk_2 ENA
pl var-02 var ENABLED ACTIVE 16790400 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 48254592 16790400 0 Disk_0 ENA

Sorry for publishing so long document however it was needed to record each and every part of the procedure. I hope someone finds this helpful somewhere.

Add VERITAS Volume Manager Disk Group & Filesystem into VERITAS Cluster service group - from the scratch orientation

noreply@blogger.com (Nilesh Joshi) — Sun, 04 Mar 2012 17:21:00 +0000

In this article we will see how to create a new disk group, create 4-way stripe VxVM filesystem & then will see how do we create a new VCS service group and incorporate the newly created Disk Group & VxFS filesystem to the service group.
So let's roll.

Well, what's the requirement? so, here we go!

Create 3 diskgroups namely dapos1dg, dapos2dg, dapos3dg.
Create three VxVM filesystems within above three DiskGroup /twm/dapos1, /twm/dapos2, /twm/dapos3 with 4/128 stripe layout.
Create three VERITAS service groups dapos1_sg, dapos2_sg & dapos3_sg.
Insert/attach/link the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups.

Solution Delivery –

In order to create 4/128 stripe layout volumes/filesystems we need to get storage disks in bunch of 4 hence I ordered for 12x15G LUNs. Each DiskGroup will have 4 disks.

Well, below are the LUNs that we have from storage –

c2t5006048C52A8CCC3d40
c2t5006048C52A8CCC3d41
c2t5006048C52A8CCC3d42
c2t5006048C52A8CCC3d43
c2t5006048C52A8CCC3d44
c2t5006048C52A8CCC3d45
c2t5006048C52A8CCC3d46
c2t5006048C52A8CCC3d47
c2t5006048C52A8CCC3d48
c2t5006048C52A8CCC3d49
c2t5006048C52A8CCC3d50
c2t5006048C52A8CCC3d51

Before formatting the disks verify if someone hasn’t used these disks for any other purpose. (it happens sometimes, some lazy frogs uses unused disks without taking much efforts and without knowing may someone has ordered these LUNs for some purpose)

# for disk in `cat disks`;do vxdisk -eo alldgs list | grep $disk;done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Good. Disks are not being used by any VERITAS DG so let's label the disks with SMI label.

Once done with labelling the disks make sure you run vxdctl enable command & re-scan for new disks devices so that the disks are visible to VxVM.

# vxdctl enable

Now we are all set to get all new disks to VERITAS control. Before taking disks into VxVM control verify if those are visible to VxVM.

# for lun in `cat /tmp/lun-list`; do vxdisk -eo alldgs list | grep -i $lun; done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Now let's initialize the new disks and get them under VERITAS control.

Get the list of new disks visible to VxVM using below command.

# for lun in `cat lun-list`; do vxdisk -e list | grep -i $lun; done | awk '{print $1}' > /tmp/vxvm_disks && cat /tmp/vxvm_disks
EMC0_223
EMC0_224
EMC0_225
EMC0_229
EMC0_226
EMC0_227
EMC0_228
EMC0_231
EMC0_222
EMC0_221
EMC0_220
EMC0_230

Initialize new disks using below command.

# for disk in `cat /tmp/vxvm_disks`;do vxdisksetup -i $disk format=sliced; done

See if all the new disks are online and under VxVM control.

# for lun in `cat vxvm_disks`; do vxdisk list | grep -i $lun; done | sort
EMC0_220 auto:sliced - - online
EMC0_221 auto:sliced - - online
EMC0_222 auto:sliced - - online
EMC0_223 auto:sliced - - online
EMC0_224 auto:sliced - - online
EMC0_225 auto:sliced - - online
EMC0_226 auto:sliced - - online
EMC0_227 auto:sliced - - online
EMC0_228 auto:sliced - - online
EMC0_229 auto:sliced - - online
EMC0_230 auto:sliced - - online
EMC0_231 auto:sliced - - online

Now let's create diskgroups –

NOTE: As these are not the shared disk groups make sure you create them on individual systems where these diskgroups needs to be mounted or else create all three disks groups on one system and then deport –import them on respective systems. Either way!

# vxdg init dapos1dg cds=off s01c0_dapos1dg=EMC0_220 s01c1_dapos1dg=EMC0_221 s01c2_dapos1dg=EMC0_222 s01c3_dapos1dg=EMC0_223
# vxdg init dapos2dg cds=off s01c0_dapos2dg=EMC0_224 s01c1_dapos2dg=EMC0_225 s01c2_dapos2dg=EMC0_226 s01c3_dapos2dg=EMC0_227
# vxdg init dapos3dg cds=off s01c0_dapos3dg=EMC0_228 s01c1_dapos3dg=EMC0_229 s01c2_dapos3dg=EMC0_230 s01c3_dapos3dg=EMC0_231

# vxdg list |egrep 'dapos1dg|dapos2dg|dapos3dg'
dapos1dg enabled 1330089054.668.sys1
dapos2dg enabled 1330089075.670.sys1
dapos3dg enabled 1330089087.672.sys1

Now we will create STRIPE volumes within these newly created three diskgroups.

# vxassist -g dapos1dg make dapos1 3g layout=stripe s01c0_dapos1dg s01c1_dapos1dg s01c2_dapos1dg s01c3_dapos1dg
# vxassist -g dapos2dg make dapos2 3g layout=stripe s01c0_dapos2dg s01c1_dapos2dg s01c2_dapos2dg s01c3_dapos2dg
# vxassist -g dapos3dg make dapos3 3g layout=stripe s01c0_dapos3dg s01c1_dapos3dg s01c2_dapos3dg s01c3_dapos3dg

Well, the volumes are created. Let’s check them out.

# for i in dapos1dg dapos2dg dapos3dg; do vxprint -htg $i | grep -i dapos;done
dg dapos1dg default default 16000 1330089054.668.sys1
dm s01c0_dapos1dg EMC0_220 auto 17023 31420800 -
dm s01c1_dapos1dg EMC0_221 auto 17023 31420800 -
dm s01c2_dapos1dg EMC0_222 auto 17023 31420800 -
dm s01c3_dapos1dg EMC0_223 auto 17023 31420800 -
v dapos1 - ENABLED ACTIVE 6291456 SELECT dapos1-01 fsgen
pl dapos1-01 dapos1 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos1dg-01 dapos1-01 s01c0_dapos1dg 0 1572864 0/0 EMC0_220 ENA
sd s01c1_dapos1dg-01 dapos1-01 s01c1_dapos1dg 0 1572864 1/0 EMC0_221 ENA
sd s01c2_dapos1dg-01 dapos1-01 s01c2_dapos1dg 0 1572864 2/0 EMC0_222 ENA
sd s01c3_dapos1dg-01 dapos1-01 s01c3_dapos1dg 0 1572864 3/0 EMC0_223 ENA
dg dapos2dg default default 68000 1330089075.670.sys1
dm s01c0_dapos2dg EMC0_224 auto 17023 31420800 -
dm s01c1_dapos2dg EMC0_225 auto 17023 31420800 -
dm s01c2_dapos2dg EMC0_226 auto 17023 31420800 -
dm s01c3_dapos2dg EMC0_227 auto 17023 31420800 -
v dapos2 - ENABLED ACTIVE 6291456 SELECT dapos2-01 fsgen
pl dapos2-01 dapos2 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos2dg-01 dapos2-01 s01c0_dapos2dg 0 1572864 0/0 EMC0_224 ENA
sd s01c1_dapos2dg-01 dapos2-01 s01c1_dapos2dg 0 1572864 1/0 EMC0_225 ENA
sd s01c2_dapos2dg-01 dapos2-01 s01c2_dapos2dg 0 1572864 2/0 EMC0_226 ENA
sd s01c3_dapos2dg-01 dapos2-01 s01c3_dapos2dg 0 1572864 3/0 EMC0_227 ENA
dg dapos3dg default default 20000 1330089087.672.sys1
dm s01c0_dapos3dg EMC0_228 auto 17023 31420800 -
dm s01c1_dapos3dg EMC0_229 auto 17023 31420800 -
dm s01c2_dapos3dg EMC0_230 auto 17023 31420800 -
dm s01c3_dapos3dg EMC0_231 auto 17023 31420800 -
v dapos3 - ENABLED ACTIVE 6291456 SELECT dapos3-01 fsgen
pl dapos3-01 dapos3 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos3dg-01 dapos3-01 s01c0_dapos3dg 0 1572864 0/0 EMC0_228 ENA
sd s01c1_dapos3dg-01 dapos3-01 s01c1_dapos3dg 0 1572864 1/0 EMC0_229 ENA
sd s01c2_dapos3dg-01 dapos3-01 s01c2_dapos3dg 0 1572864 2/0 EMC0_230 ENA
sd s01c3_dapos3dg-01 dapos3-01 s01c3_dapos3dg 0 1572864 3/0 EMC0_231 ENA

Next thing is to create a mount point for these volume & create a file system on this volume.

On individual servers…

# for fs in 1 2 3;do mkdir -p /twm/dapos$fs;done && ls –ld /twm/dapos*
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos1
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos2
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos3

Not required to be done on individual servers as the storage disks are shared between cluster nodes.

# mkfs -F vxfs /dev/vx/rdsk/dapos1dg/dapos1
# mkfs -F vxfs /dev/vx/rdsk/dapos2dg/dapos2
# mkfs -F vxfs /dev/vx/rdsk/dapos3dg/dapos3

First time needed only on respective individual servers.

# mount -F vxfs /dev/vx/dsk/dapos1dg/dapos1 /twm/dapos1

Now, furthermore there is requirement to create new VCS service groups & Insert/attach the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups. Let’s on it!

Before doing anything, first thing to check if cluster configuration is read-only or in write mode.

# haclus -display |grep -i 'readonly'
ReadOnly 1

Where,
0 = write mode
1 = read only mode

If the cluster configuration is read-only make it write mode -

# haconf –makerw

# hagrp -add dapos1_sg
# hagrp -modify dapos1_sg SystemList sys1 0 sys3 1 sys4 2 sys2 3
# hagrp -modify dapos1_sg AutoStart 0
# hagrp -modify dapos1_sg AutoStartList sys1
# hagrp -modify dapos1_sg SourceFile "./main.cf"

# hares -add dapos1_dg DiskGroup dapos1_sg
# hares -modify dapos1_dg DiskGroup dapos1dg
# hares -modify dapos1_dg PanicSystemOnDGLoss 0
# hares -modify dapos1_dg StartVolumes 1
# hares -modify dapos1_dg StopVolumes 1
# hares -modify dapos1_dg MonitorReservation 0
# hares -modify dapos1_dg tempUseFence INVALID
# hares -modify dapos1_dg DiskGroupType private
# hares -modify dapos1_dg Enabled 1

# hares -add dapos1_EMCProc EMCProc dapos1_sg
# hares -modify dapos1_EMCProc Enabled 1
# hares -add dapos1_vol Volume dapos1_sg
# hares -modify dapos1_vol Volume dapos1
# hares -modify dapos1_vol DiskGroup dapos1dg
# hares -modify dapos1_vol Enabled 1

# hares -add dapos1_mnt Mount dapos1_sg
# hares -modify dapos1_mnt Critical 0
# hares -modify dapos1_mnt MountPoint "/twm/dapos1"
# hares -modify dapos1_mnt BlockDevice "/dev/vx/dsk/dapos1dg/dapos1"
# hares -modify dapos1_mnt FSType vxfs
# hares -modify dapos1_mnt FsckOpt "%-n"
# hares -modify dapos1_mnt CkptUmount 1
# hares -modify dapos1_mnt SecondLevelMonitor 0
# hares -modify dapos1_mnt SecondLevelTimeout 30
# hares -modify dapos1_mnt Enabled 1

Fine, next part is linking - while linking always remember the syntax -

# hares -link {parent} {child}

Please note in VCS, parent resource depends on child resource, for example in a group where volume depends on DiskGroup resource, DiskGroup will be child & volume would be parent...

# hares -link dapos1_vol dapos1_dg
# hares -link dapos1_mnt dapos1_vol

Upon performing the above configuration you can see the resources added to the service group dapos1_sg.

# hagrp -resources dapos1_sg
dapos1_prod_IPM
dapos1_prod_proxy
dapos1_dg
dapos1_vol
dapos1_mnt

Perform the same procedure for rest of 2 service groups. Once completed check the status of the service groups.

# hastatus -sum | egrep 'dapos1_sg|dapos2_sg|dapos3_sg'
B dapos1_sg sys1 Y N ONLINE
B dapos1_sg sys2 Y N OFFLINE
B dapos1_sg sys3 Y N OFFLINE
B dapos1_sg sys4 Y N OFFLINE
B dapos2_sg sys1 Y N OFFLINE
B dapos2_sg sys2 Y N ONLINE
B dapos2_sg sys3 Y N OFFLINE
B dapos2_sg sys4 Y N OFFLINE
B dapos3_sg sys1 Y N OFFLINE
B dapos3_sg sys2 Y N OFFLINE
B dapos3_sg sys3 Y N ONLINE
B dapos3_sg sys4 Y N OFFLINE

Once completed with the changes do not forget to get the Cluster Server configuration in read-only mode and dump the changes in the main.cf file. You can do the same using below command -

# haconf -dump -makero

Wooh.. we are done with the requirement. It was a good exercise in itself...

HTH.

Dynamically Adding a RAW device to a Solaris zone.

noreply@blogger.com (Nilesh Joshi) — Fri, 10 Feb 2012 20:06:00 +0000

Dynamically Adding a RAW device to a Solaris zone.

By default it is not possible to add raw device to zone without taking a reboot on zone however there is a famous saying => "There is always a way out for those clever enough to find it". So I'm, just kidding, don't take it seriously. :)

I found a little hack to accomplish the objective of adding raw device to zone without rebooting it. Here is a way out -

1) Add the device to the zonecfg

#zonecfg -z barvozone1
zonecfg:barvozone1> add device
zonecfg:barvozone1:device> set match=/dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0
zonecfg:barvozone1:device> end
zonecfg:barvozone1>exit

2) use the mknod command to create the device in the zones dev folder

#ls -l /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0

lrwxrwxrwx   1 root     root          67 Feb 18 15:34 /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0 -> ../../devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

#ls -l /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

crw-r-----   1 root     sys      118, 128 Mar  5 23:55 /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

# cd /barvozone1/zonepath/dev

# mknod c3t60050768018A8023B8000000000000F0d0s0 c 118 128

That's it. The raw device is now visible within zone and now you can start with your stuffs without any downtime. Isn't it cool?

Identify ASM Disks from OS interface

noreply@blogger.com (Nilesh Joshi) — Wed, 08 Feb 2012 07:07:00 +0000

There are several ways to identify ASM disks from the database interface however sometimes or in fact most of the times it might be the case that the System Engineer/Administrator doesn't have access to Oracle (ASM instance) or he may not aware of database operations so in that case how to recognize that a particular disk is being used and under control of ASM.

There are several occasions we need to order SAN disks/LUNs to either expand volume group, zpool or metadevice to increase a filesystem within it. Now consider, there are already few devices/disks available on system and looks free but if you are unsure whether a specific device/disk/LUN is being used by ASM, you can look at the header of the device. The od (octal dump) command can be used. E.g. od --read-bytes=128 --format=c displays the 1st 128 bytes of a device in character format.

For Solaris distributions –

root:aczfild3s:/var/tmp # od -c -N 128 /dev/rdsk/c5t6006016051402800F07C0E23B0CFDF11d0s0
0000000 \0 202 001 001 \0 \0 \0 \0 200 \0 \0 017 A 200 327 272
0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \n 020 \0 \0 \0 017 001 003 V 4 S _ P E R _
0000120 D A T A 0 1 _ 0 0 1 5 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 V 4 S _ P E R _
0000160 D A T A 0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

This is a ASM disk as the ORCLDISK string exists in the header. The ASM disk name is V4S_PER_DATA01_0015.

For Linux distributions –

# od --read-bytes=128 --format=c /dev/dm-39
0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 w 374 211 343
0000020 244 } 030 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \0 \0 020 \n \0 \0 001 003 N C A S _ D I S
0000120 K 1 _ 0 0 0 0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 N C A S _ D I S
0000160 K 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

The ASM disk name is NCAS_DISK1_0000.

For AIX distributions –

AIX is one of the “SENSIBLE” distributions I must say. Why? Check this out –

In Linux or Oracle-Sun Solaris the native volume manager or utilities around it does not identify if the disk is being used by ASM and it simply allow to format/erase the disk and allow to add the device to inline filesystem or volume group however AIX native volume manager is a step ahead & smart - disks being used by Oracle ASM cannot be also used with AIX LVM. The main reason for this is because the Oracle ASM software puts information on the raw disk to identify it as an Oracle disk, which wipes out any existing PVID or VGDA information.

If an attempt is made to use LVM on a disk already in an Oracle ASM disk group the following errors will be seen:

root> extendvg -f apps hdiskpower10
0516-1339 extendvg: Physical volume contains some 3rd party volume group.
0516-1397 extendvg: The physical volume hdiskpower10, will not be added to the volume group.
0516-792 extendvg: Unable to extend volume group.

root> mkvg -y lparvg hdiskpower128
0516-1339 mkvg: Physical volume contains some 3rd party volume group.
0516-1397 mkvg: The physical volume hdiskpower128, will not be added to the volume group.
0516-862 mkvg: Unable to create volume group.

LVM knows to search for the Oracle identifier on these disks, and will fail the operation. You can manually check for the identifier by running the lquerypv command.

# lquerypv -h /dev/hdiskpower7
00000000 00820101 00000000 80000000 F7BE76D7 ..............v.00000010 00000000 00000000 00000000 00000000
................00000020 4F52434C 4449534B 00000000 00000000 ORCLDISK........00000030 00000000 00000000 00000000 00000000 ................00000040 0A100000 00000103 44415441 5F303030 ........DATA_00000000050 30000000 00000000 00000000 00000000 0...............00000060 00000000 00000000 44415441 00000000 ........DATA....00000070 00000000 00000000 00000000 00000000 ................00000080 00000000 00000000 44415441 5F303030 ........DATA_00000000090 30000000 00000000 00000000 00000000 0...............000000A0 00000000 00000000 00000000 00000000 ................000000B0 00000000 00000000 00000000 00000000 ................000000C0 00000000 00000000 01F64610 3B306C00 ..........F.;0l. 000000D0 01F64610 3B9C7C00 02001000 00100000 ..F.;. .........000000E0 0001BC80 0000C800 00000002 00000001 ................000000F0 00000002 00000002 00000000 00000000 ................

Hope that helps.

Oracle-Solaris Patching using Live Upgrade

noreply@blogger.com (Nilesh Joshi) — Mon, 03 Oct 2011 09:34:00 +0000

In today's post I'll be talking about a procedure for patching Solaris server having zones installed on it using Live Upgrade functionality.

The Solaris OS Recommended Patch Cluster provides critical Solaris OS Security, Data Corruption, and System Availability fixes & hence it is advisable to patch your Solaris systems twice in a year (atleast), as per Oracle-Sun's Critical Patch Update release schedule, I prefer to execute patch cycle for my environment in end of April and sometime late October every year.

Oracle-Sun CPUs are released on the Tuesday closest to the 17th of January, April, July, and October –

See - http://www.oracle.com/technetwork/topics/security/alerts-086861.html

In my environment, I use Live Upgrade to patch our Solaris systems. Reason behind using Live Upgrade for patching purpose are -

1. Create a copy of the system environment; that is, a copy of the root (/) file system

2. Live Upgrade has build-in feature for splitting the mirrors of an SVM mirrored root (detach, attach, preserve options on lucreate) hence low overhead to deal with SVM mirror break stuffs separately etc.

3. Less downtime (not more than 15-20 mins) and minimal risk.

4. Better back out option. In case something breaks after patching revert to old BE and be at stage from where started, again that doesn’t take much downtime and safe option.

5. The most appropriate option for those Solaris servers who have zones/containers installed on it.

There might be many more benefits out there, however I find above benefits best fit for my purpose.

So to summarize, all tasks except the reboot can be accomplished on an operational production system; the impact on any running process is minimal. Live Upgrade is a combination of maximizing system availability when applying changes and minimizing risk by offering the ability to reboot to a known working state (your original environment).

Well. let's see how to do it in real life, in my current environment we have many servers which uses Solaris Volume Manager as their primary volume manager to manage the disk and data. So, let's take a look at patching procedure to patch servers who have SVM installed and configured on it along with zones installed on it sitting on ZFS filesystem.

Let us grab the output of metastat to understand metadevice placement -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23 d13
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d100             m   10GB d20 d10
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Alright, my / is on d100 and /var is on d103. Let us create an alternative boot environment out of it.

# lucreate -c Sol10 -n Sol10pu -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d20:detach,attach -m /var:/dev/md/dsk/d3:ufs,mirror –m /var:/dev/md/dsk/d23:detach,attach

Here I'm trying to create a metadevice d0 representing / UFS filesystem having a sub-mirror d20 (sub-mirror d20 first gets detach from d100 and then attach to d0). Same thing applicable for /var filesystem and it's meta device configuration.

In above command I'm creating a new boot environment called Sol10pu using option “-n”, option “-m” Specifies the vfstab information for a new UFS-based BE.

NOTE: The -m option is not supported for BEs based on ZFS file systems.

NOTE: In case you're performing upgrade and patching in one go then point to be ponder - Before upgrading, you must install the Oracle Solaris Live Upgrade packages from the release to which you are upgrading. New capabilities are added to the upgrade tools, so installing the new packages from the target release is important. Example, you need to upgrade from Oracle Solaris 10 update 4 to Oracle Solaris update 8, so you must get the Oracle Solaris Live Upgrade packages from the Oracle Solaris update 8 DVD.

Once above command finishes, you will see you meta device configuration changed as follows -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

d0 and d3 has one sub-mirror and d100 and d100 has one sub-mirror associated.

Also you will be able to see two boot environments on your Solaris system -

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      yes    yes       no     -
Sol10pu                    yes      no     no        yes    -

Fine, so now we have 2 boot environments with us and we are going to patch the alternate BE (sol10pu) using a patching tool called PCA, BTW - I use PCA (Patch Check Advance) tool to apply patches to our Solaris systems. PCA has been setup to download patches via local web proxy to access outside systems.

PCA setup tips can be found at - http://www.par.univie.ac.at/solaris/pca/usage.html

What PCA needs in case setting it up -

- PERL distribution
- At least one server which is internet facing (this server will then act as a proxy to rest of servers)
- Patch cross-reference file called patchdiag.xref (latest one always while patching)
- Valid Oracle support (MOS) user ID and password
- If at all required, some wrapper scripts to PCA

To do so, let us mount the alternate BE on mount point say /a

# lumount Sol10pu /a
/a

Now I'll create a temporary directory to download the missing & required patches,

# mkdir -p /patchman/patches
My next job is to generate patch_order file,

# /bin/pca missingrs -R /a -H --format "%p-%c" > /patchman/patches/patch_order

Where -R stands for Alternative root directory
Where -H stands for Don't display descriptive headers
Where --format stands for Set output format to FORMAT

And go get them, download -

# /bin/pca missingrs -R /a -d -P /patchman/patches/

Where -d stands for download patches
Where -P stands for Patch download directory

Unmount the ABE -

# luumount /a

Now if you populate the /patchman/patches directory then you will see the list of patches in there.

Unzip all those patches -

# for i in `ls *.zip`;do
unzip $i
rm $i
done

Okay, at this stage we are ready upgrade ABE with patches available -

# cd /patchman/patches; luupgrade -n Sol10pu -s /patchman/patches -t `cat patch_order`

NOTE: Reactive patching may occasionally be necessary to address break-and-fix issues so in this case you can use LU with something like -

Apply single patch to ABE -

# luupgrade -n Sol10pu -s /patchman/patches -t

This will update patches on global as well as non-global zones.

Once the patches are installed it will automatically un-mounts the ABE sol10pu mounted on mount point /a.

Now it's time to activate the ABE sol10pu which just been patched using Live Upgrade utility.

# luactivate Sol10pu
A Live Upgrade Sync operation will be performed on startup of boot environment .
**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Enter the PROM monitor (ok prompt).

2. Change the boot device back to the original boot environment by typing:

     setenv boot-device /pci@0/pci@0/pci@2/scsi@0/disk@0,0:a

3. Boot to the original boot environment by typing:

     boot

**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

# init 6
updating /platform/sun4v/boot_archive
SYSTEM GOING DOWN!!!!

NOTE: Live upgrade always uses init 6 or shutdown commands. Halt and reboot commands will create big time bang, be aware!!!

Once system is up it should show new kernel patch version.

Great, it's been a week after patching and application, DB owners are happy with patching stuffs and now we need to perform post patching stuffs upon certain confirmations.

POST PATCHING WORK TO DO - A Week LATER....
=========================================

Now a week later I need to delete the old boot environment and rebuild the metadevices to be in mirror layout.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      no     no        yes    -
Sol10pu                    yes      yes    yes       no     -

# ludelete Sol10
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

So, the meta devices are looking like as below -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Now clear the d100 & d103 mirrors.

# metaclear d100 d103
d100: Mirror is cleared
d103: Mirror is cleared

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1
d13              s   10GB c1t1d0s3
d10              s   10GB c1t1d0s0

Next attach the sub-mirrors d10 & d13 to metadevices d0 and d3 respectively.

# metattach d0 d10
d0: submirror d10 is attached

# metattach d3 d13
d3: submirror d13 is attached

Hence my final meta device placement looks like as follows -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d3               m   10GB d23 d13 (resync-25%)
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d0               m   10GB d20 d10 (resync-45%)
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

That's it. Now your done with patching your Solaris server and zones deployed on it.

Migrating VERITAS Volumes from one storage array to another storage array

noreply@blogger.com (Nilesh Joshi) — Tue, 20 Sep 2011 11:56:00 +0000

Background:

This week I got a task to Migrate Veritas disk groups & their volumes on a specific server from one storage array to new storage array due to old storage array is ruining out of capacity.

To execute this task I split this into 3 phases -

1. Responsibilities with Storage

     a. Target server has to be zoned to new Storage system
     b. Provide LUNs/capacity to server (total capacity required is 3.5 TB 58x60G & 2x15G LUNs)

2. Responsibilities with me

      a. Migrate Veritas disk groups to new storage array
            A) Label all newly added SAN disks (may be using some scripted method) & make sure SAN disks are visible under VxVM
            B) Initialize all those new SAN disks with VxVM
            C) Add the SAN disk to disk group
            D) Mirror the volumes
            E) Verify if sync is completed
            F) Verify from vxprint if you see a new plex added to designated volumes
            G) If all is well then go ahead and detach old plex
            H) Once plex is disassociated from designated volumes then delete the old plex
            I) Verify if data group(s) & their volumes are on new storage array.
     b. Remove disks associated with old storage from Veritas configuration

3. Responsibilities with Storage

     a. Remove disks associated with old storage from server
     b. Take care of redundant paths

Back out -

     a. Have server full backup handy.

Execution -

List the disk groups needs to be migrated to new storage array.

# vxdg list
NAME STATE ID
xxxx_dg enabled,cds 1279726733.18.xxxxx
localswdg enabled,cds 1279726567.16.xxxxx
nass3_dg enabled 1074844579.1535.nassau3

Get the true picture of your VxVM configuration. Save output of this command for future reference

# vxprint -hrt

Now that Storage has attached SAN disks to servers HBA we need to label them and get them to VERITAS control.

List the existing Disks & new disks detected.

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online <<<< All UNIX# disks are from old storage arrary
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto - - online invalid   <<<< All emc_* disks are from new storage arrary
emc_clariion0_1705 auto - - online invalid
emc_clariion0_1706 auto - - online invalid
emc_clariion0_1707 auto - - online invalid
emc_clariion0_1708 auto - - online invalid
emc_clariion0_1709 auto - - online invalid
emc_clariion0_1710 auto - - online invalid
emc_clariion0_1713 auto - - online invalid
emc_clariion0_1800 auto - - online invalid
emc_clariion0_1801 auto - - online invalid
emc_clariion0_1803 auto - - online invalid
emc_clariion0_1804 auto - - online invalid
emc_clariion0_1805 auto - - online invalid
emc_clariion0_1806 auto - - online invalid
emc_clariion0_1807 auto - - online invalid
emc_clariion0_1808 auto - - online invalid
emc_clariion0_1809 auto - - online invalid
emc_clariion0_1810 auto - - online invalid
emc_clariion0_3700 auto - - online invalid

[...]

emc_clariion0_5809 auto - - online invalid
emc_clariion0_5858 auto - - online invalid

Initialize the disks with VxVM and add them to appropriate disk group.

For a single disk -

# vxdisksetup -i emc_clariion0_5858 format=sliced

For multiple disks

# vxdisk list | awk '{print $1}' | grep -i emc > /tmp/EMC_disks
# for d in `cat /tmp/EMC_disks` ; do vxdisksetup -i $d format=sliced; done

Add disks to disk group

#!/bin/sh
#for e.g. vxdg -g nass3_dg adddisk nass3_dg01=emc_clariion0_1704
DG=nass3_dg
DISKS=`vxdisk list | awk '{print $3"="$1}' | grep -i emc`
#Output of command vxdisk list | awk '{print $3"="$1}' | grep -i emc is look like as = nass3_dg01=emc_clariion0_1704
for d in $DISKS;
do vxdg -g $DG adddisk $DISKS;
done

OR you can also use vxdiskadm menu based command to perfrom this activity.

Now you should see something like -

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto:sliced nass3_dg01 nass3_dg online
emc_clariion0_1705 auto:sliced nass3_dg02 nass3_dg online
emc_clariion0_1706 auto:sliced nass3_dg03 nass3_dg online
emc_clariion0_1707 auto:sliced nass3_dg04 nass3_dg online
emc_clariion0_1708 auto:sliced nass3_dg05 nass3_dg online
emc_clariion0_1709 auto:sliced nass3_dg06 nass3_dg online
emc_clariion0_1710 auto:sliced nass3_dg07 nass3_dg online
emc_clariion0_1713 auto:sliced nass3_dg08 nass3_dg online
emc_clariion0_1800 auto:sliced nass3_dg09 nass3_dg online
emc_clariion0_1801 auto:sliced nass3_dg10 nass3_dg online
emc_clariion0_1803 auto:sliced nass3_dg11 nass3_dg online
emc_clariion0_1804 auto:sliced nass3_dg12 nass3_dg online
emc_clariion0_1805 auto:sliced nass3_dg13 nass3_dg online
emc_clariion0_1806 auto:sliced nass3_dg14 nass3_dg online
emc_clariion0_1807 auto:sliced nass3_dg15 nass3_dg online
emc_clariion0_1808 auto:sliced nass3_dg16 nass3_dg online
emc_clariion0_1809 auto:sliced nass3_dg17 nass3_dg online
emc_clariion0_1810 auto:sliced nass3_dg18 nass3_dg online
emc_clariion0_3700 auto:sliced nass3_dg19 nass3_dg online

[...]

emc_clariion0_5809 auto:sliced nass3_dg59 nass3_dg online

Now that we have added new disks to the appropriate datagroups, next task to mirror the volumes.

NOTE: To decide how many disks are require to mirroe a perticular volume is very easy, put those many disks which will meet the current size of volume.

# vxassist -g nass3_dg mirror db_TESTDB_vol alloc=nass3_dg05,nass3_dg02,nass3_dg06,nass3_dg07,nass3_dg08,nass3_dg09,nass3_dg10,nass3_dg11,nass3_dg12,nass3_dg13,nass3_dg14,nass3_dg15,nass3_dg16,nass3_dg17,nass3_dg18,nass3_dg19,nass3_dg20,nass3_dg21,nass3_dg22,nass3_dg23,nass3_dg24,nass3_dg25,nass3_dg26,nass3_dg27,nass3_dg28,nass3_dg29,nass3_dg30,nass3_dg31,nass3_dg32,nass3_dg33,nass3_dg34,nass3_dg35,nass3_dg36,nass3_dg37,nass3_dg38,nass3_dg39,nass3_dg40,nass3_dg41,nass3_dg42,nass3_dg43,nass3_dg44,nass3_dg45,nass3_dg46,nass3_dg47,nass3_dg48,nass3_dg49,nass3_dg50,nass3_dg51,nass3_dg52,nass3_dg53,nass3_dg54,nass3_dg55

Repeat the previous step for rest of the data groups and volumes.

Likewise you can add disks to the remaining volumes and then we have to mirror them.

Check sync progress using -

# vxtask -l list
Task: 5912 RUNNING
Type: ATCOPY
Operation: PLXATT Vol db_TESTDB_vol Plex db_TESTDB_vol-02 Dg nass3_dg
Started: Fri Sep 16 16:05:24 2011
Throttle: 0
Progress: 4.27% 268713984 of 6291456000 Blocks
Work time: 29 minutes, 30 seconds (11:01:11 remaining)

Verify from vxprint , you should see a new plex added to db_TESTDB_vol volume

# vxprint -qthg nass3_dg db_TESTDB_vol

If everything looks good, then detach/disassociate & remove old plex, in short - break the mirror. (Before doing so get application owner consent)

# vxmend -g nass3_dg off db_TESTDB_vol-01
# vxplex -g nass3_dg -o rm dis db_TESTDB_vol-01

Repeat the previous two steps for rest of the plexes.

Well, by this time we can say we are done so verify if datagroup(s) & their volumes are on new storage arrary.

# vxprint -qthg nass3_dg db_TESTDB_vol
v db_TESTDB_vol - ENABLED ACTIVE 6291456000 SELECT db_TESTDB_vol-02 fsgen
pl db_TESTDB_vol-02 db_TESTDB_vol ENABLED ACTIVE 6291456000 STRIPE 3/128 RW
sd nass3_dg05-01 db_TESTDB_vol-02 nass3_dg05 0 125754880 0/0 emc_clariion0_1708 ENA
sd nass3_dg08-01 db_TESTDB_vol-02 nass3_dg08 0 125754880 0/125754880 emc_clariion0_1713 ENA
sd nass3_dg11-01 db_TESTDB_vol-02 nass3_dg11 0 125754880 0/251509760 emc_clariion0_1803 ENA
sd nass3_dg14-01 db_TESTDB_vol-02 nass3_dg14 0 125754880 0/377264640 emc_clariion0_1806 ENA
sd nass3_dg17-01 db_TESTDB_vol-02 nass3_dg17 0 125754880 0/503019520 emc_clariion0_1809 ENA
sd nass3_dg20-01 db_TESTDB_vol-02 nass3_dg20 0 125754880 0/628774400 emc_clariion0_3702 ENA
sd nass3_dg23-01 db_TESTDB_vol-02 nass3_dg23 0 125754880 0/754529280 emc_clariion0_3705 ENA
sd nass3_dg26-01 db_TESTDB_vol-02 nass3_dg26 0 125754880 0/880284160 emc_clariion0_3708 ENA
sd nass3_dg29-01 db_TESTDB_vol-02 nass3_dg29 0 125754880 0/1006039040 emc_clariion0_3800 ENA
sd nass3_dg32-01 db_TESTDB_vol-02 nass3_dg32 0 125754880 0/1131793920 emc_clariion0_3803 ENA

[...]

sd nass3_dg43-01 db_TESTDB_vol-02 nass3_dg43 0 125754880 2/1594132480 emc_clariion0_5704 ENA
sd nass3_dg46-01 db_TESTDB_vol-02 nass3_dg46 0 125754880 2/1719887360 emc_clariion0_5707 ENA
sd nass3_dg50-01 db_TESTDB_vol-02 nass3_dg50 0 125754880 2/1845642240 emc_clariion0_5800
ENA
sd nass3_dg53-01 db_TESTDB_vol-02 nass3_dg53 0 125754880 2/1971397120 emc_clariion0_5803 ENA

Yes, volume has been moved to new storage.

Now we are ready to remove disks associated with old storage from Veritas-configuration.

# vxdg -g nass3_dg rmdisk fls02c4_nass3_dg

Ask storage team to detach/remove the old disks permanently from the server.

This is overall procedure to migrate your data groups & their volumes from one storage to another storage.

Disable Large Segment Offload (LSO) in Solaris 10

noreply@blogger.com (Nilesh Joshi) — Tue, 30 Aug 2011 08:37:00 +0000

In this blog article, I will share my understanding on Large Segment Offload (LSO). I got a task to disable the LSO on few of the servers (include zones).

Let's first understand what is LSO stands for and what is the purpose of using LSO.

As you see above LSO stands for Large Segment Offload.

TCP Offload Engine is an embryonic technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards, hence the no CPU cycle and kernel time will get consumed.

LSO saves valuable CPU cycles by allowing the network protocol stack to handle large segments instead of the traditional model of MSS (TCP Maximum Segment Size) sized segments. In the traditional network stack, the TCP layer segments the outgoing data into the MSS sized segments and passes them down to the driver. This becomes computationally expensive with 10 GigE networking because of the large number of kernel functional calls required for every MSS segment. With LSO, a large segment is passed by TCP to the driver, and the driver or NIC hardware does the job of TCP segmentation (LSO offload the segmentation job on Layer 4 to the NIC driver). An LSO segment may be as large as 64 KByte. The larger the LSO segment, better the CPU efficiency since the network stack has to work with smaller number of segments for the same throughput.

So in simple words, use LSO for better network performance while reducing processor (CPU) utilization.

Segmentation is needed if a full TCP segment does not fit into the Ethernet Maximum Transmission Unit (MTU) size. With LSO, TCP segments do not need to get split in software implementation, this is done on the interface card hardware instead. Being much more effective, this improves the network performance while reducing the workload on the CPUs. LSO is most helpful for 10 Gigabit Ethernet network interfaces and on systems with slow CPU threads or lack of CPU resources.

Solaris LSO can be used if all of the three conditions are met :

1.The TCP/IP stack integrates LSO,

2.The Network Interface Card hardware supports it (for e.g. drivers like e1000g,ixgb,ixgbe etc),
3.The driver for this network card is capable of handling it.

Sadly, in most of the cases LSO seems to be not working that well hence it leads to disable the LSO support. Here is the ways to disable the LSO.

Ways to disable LSO -

Disable LSO by adding the following line in the /kernel/drv/e1000g.conf file (I’m using the e1000g interface/driver hence the file that I'm using is /kernel/drv/e1000g.conf) :

lso_enable=0,0,0,0,0,0,0,0;

After making the changes reboot is required or else if reboot is not possible then you can use ndd utility/command to disable it on a temporary basis and not persist across the reboot.

Using ndd you can disable it as shown below -

# ndd -set /dev/ip ip_lso_outbound 0

Also if you don't want to reboot the server after modifying the file /kernel/drv/e1000g.conf you can simply unplumb all of your e1000g interfaces with ifconfig, do "update_drv e1000g" to reload the .conf file, and then replumb and reconfigure the interfaces with ifconfig however still if I'm going to unplumb the network interfaces then eventually I'll be disturbing the services so reboot is the best option.

I had to disable the LSO as our application folks were experiencing slowness in their web application (response time etc.) It looks like LSO cause unstable connections & hence there are few observations like dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately application folks observed slowness in their web application, NFS stuffs etc.

Installing Perl modules on Solaris 10 - An experience

noreply@blogger.com (Nilesh Joshi) — Fri, 26 Aug 2011 14:35:00 +0000

In recent times I got a task to perform few Perl modules installation. To build the module in question (Compress::Raw::Zlib), I downloaded the module from CPAN (Comprehensive Perl Archive Network), verified that the MD5 checksum was correct, and used the following steps to compile the module:

# perl Makefile.PL
# make
# make install

The ‘make Makefile.PL’ completed successfully, but the make failed with the following errors:

# make
cp lib/Compress/Raw/Zlib.pm blib/lib/Compress/Raw/Zlib.pm
AutoSplitting blib/lib/Compress/Raw/Zlib.pm (blib/lib/auto/Compress/Raw/Zlib)
/usr/bin/perl /usr/perl5/5.8.4/lib/ExtUtils/xsubpp -typemap /usr/perl5/5.8.4/lib/ExtUtils/typemap -typemap typemap Zlib.xs > Zlib.xsc && mv Zlib.xsc Zlib.c
cc -c -I./zlib-src -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO -xO3 -xspace -xildoff -DVERSION=\"2.037\" -DXS_VERSION=\"2.037\" -KPIC "-I/usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE" -DNO_VIZ -DGZIP_OS_CODE=3 -DUSE_PPPORT_H Zlib.c
/usr/ucb/cc: language optional software package not installed
*** Error code 1
make: Fatal error: Command failed for target `Zlib.o'

Above output indicate an error related to compiler & it is clearly showing that it is looking for cc complier however I was trying to build the module using gcc and not the cc (Sun studio) compiler but it seems like by default modules Makefile is referring to Sun Studio compiler and not the gcc complier. I do not have Sun Studio compiler on our systems & as this is being license software I can not even simply install it on any server due to compliance restrictions.

After doing a little bit research on the issue, I found that there is a way to fix this problem. If you want to use gcc to build all Perl modules on a system, you can permanently remove the Sun Studio compiler references by adjusting the “cccdlflags” and “optimize” variables in /usr/perl5/5.8.4/lib/sun4-solaris-64int/Config.pm:

root@XXXX:/usr/perl5/5.8.4/lib/sun4-solaris-64int# egrep '(KPIC|O3)' Config.pm

cccdlflags='-KPIC'
optimize='-xO3 -xspace -xildoff'

After removing the above entries make was successful with gcc complier.

There is yet another method to do the Perl module installation even not performing above changes to Config.pm. You can simply build Perl module using –

- /usr/perl5/bin/perlgcc Makefile.PL
- /usr/sfw/bin/gmake
- /usr/sfw/bin/gmake test
- /usr/sfw/bin/gmake install

What we were discussing in above is how to install Perl modules manually and how to tackle with complier related issues. There is yet another automated method to install Perl module and which is most preferred and quick method to do it.

Simply export the http_proxy environment variable with your proxy server name or IP address and valid port number so that you can directly reach to the internet via proxy.

# http_proxy=myproxy:8080; export http_proxy
# perl -MCPAN -e shell

If this is the first time this script is run it will ask a lot of questions and store the results in Config.pm

Then simply run –

1. Start the CPAN shell script: /usr/perl5/bin/perl -MCPAN -e shell.
2. Install a module, e.g.: install CGI::Session
3. Exit the shell script: exit

To check the already installed modules execute –

# perl -e 'use HTML::Parser;'

If nothing is returned, Perl was able to locate the module. Otherwise, you will see “Can't locate HTML/Parser.pm in @INC”

One more experience to share here. I installed Threads Perl module however user was not able to execute his scripts due to current Perl distribution was not build or complied with “useithreads” configuration option hence he requested to recompile Perl distribution with threads option enable.

Here is a string that I use to compile Perl distribution with threads option.

# sh Configure -Dusethreads -Dprefix=/usr/perl5/5.14.1

With new Perl compilation user was able to execute his scripts with threads module.

Hope this document will help to those who are new to this work.

IBM’s Private Cloud Computing Platform (ISDM)

noreply@blogger.com (Nilesh Joshi) — Sat, 30 Jul 2011 17:41:00 +0000

In last post I've discussed about Cloud Computing at a broad level and as I said would be writing the brief article on IBM private cloud platform. Since past few months I've been involved in evaluating IBM private cloud technology known as ISDM (IBM Integrated Service Delivery Manager) and herein I'm writing about my own understandings and experiences about IBM private cloud PoC that recently deployed in our test environment.

The below document gives very well information about IBM private cloud solution. I hope you will find this document helpful.

A Review of IBM Private Cloud

Cloud computing – Revisited

noreply@blogger.com (Nilesh Joshi) — Sat, 02 Jul 2011 19:46:00 +0000

Well, before I write something about IBM Cloud Computing platform (IBM Integrated Service Delivery Manager) I would like to explain few of my understandings about Cloud Computing & its implementation factors.

I’m sure almost everyone knows about cloud models/types which are namely public cloud, private cloud, hybrid cloud & community cloud also these all clouds have specific service delivery models in particular, IT-as-a-Service, Platform-as-a-service & Software-as-a-Service.

Today, I would like to write an article on implementation factor(s) of cloud computing. Since past 8-9 months have been working on various cloud technology platforms/products and quite a few POC environments to test out & make out which product best suits to an organization specific requirements. There is no ready made cloud product out there which will able meet all of your organization requirements hence you need to take a call whether customizing the solution or altering the organization policies, structure i.e. whatever requirement. Trust me testing the technology is an easy job however to fix on what is the best fit product for an organization is a “not easy” job as this is something which is beyond just a technology & cloud implementers, decision makers and technical people should look at it from business viewpoint too anyways this is not a topic of discussion here.

Let’s start this discussion with traditional IT system management portfolio & service delivery model. In ideal scenario a particular software development project gets a requirement to develop a product and resources required to deliver that product, the project manager or project lead starts identifying the resource requirements and ask IT department to procure necessary systems, OS, software, databases, toolkits or provision necessary systems, OS, software, databases via available IT infrastructures which takes significant amount of time since person who driving the project has to get many things approved by higher management, software compliance team, information security team etc then procurement start which takes several weeks or he raise a request to IT department to provision systems with necessary OS, software, databases which also takes several days and so on. Then only actual application, product development starts, tested & deployed on that infrastructure. We can see several challenges in this model as described below –

Need large Capex - Large investments required to procure the infrastructure required for any product development.

Poor utilization of resources - Application usage is not going to be constant yet the infrastructure is provisioned for peak demand hence the infrastructure remains under-utilized for a major part of the time.

Slow Time-to-Market - This model of procuring and provisioning infrastructure usually requires significant time and reduces the agility of an organization in creating new business solutions.

Now we can clearly see how traditional IT system management portfolio & service delivery model prove expensive in current competitive, difficult economic market.

Also on the other hand, still nothing is accountable as such. For system administrators it’s becomes a challenge to showcase or prove that a particular system or systems are under utilized which is dedicated to a specific functional group and those can be shared among the other groups. In such cases cost grows, complexities increases as number of hardware resources and software resources increases, data center space, cooling cost increases, requires more man power to handle the infrastructure, lack of standardization due to silos & much more to talk about.

Another point of view, Over 70% of IT budgets in a typical Data Centre goes just to "keeping the lights on", especially keeping the “green” lights on! Conversely, only a small portion of each dollar spent on IT today creates a direct business benefit. Since Data Center IT assets become obsolete approximately every 5 years, the vast majority of IT investment is spent on upgrading various pieces of infrastructure and providing redundancy and recover-ability: activities that consume approximately 60 to 80% of IT expenditures without necessarily providing optimal business value or innovation.

To overcome on above challenges to a certain level virtualization & consolidation has played a major role however IT has started seen many challenges with virtualization environment like VM sprawl, costing etc consequently to overcome on those challenges & challenges of overall service delivery methodology Cloud Computing technology evolved.

Cloud Computing is a model of service delivery and access where dynamically scalable and virtualized resources are provided as a service over the Internet.

Cloud Computing offers an alternative approach that profoundly transforms the way in which information and services are consumed and provided and can enable businesses to:

Lower costs by using energy and resources more efficiently

Enhance agility, growth, and profitability

Simplify operations and management

Ensure elastic and trusted collaboration between various groups which results into visibility & standardized, smooth operations.

Faster time to market

On-demand elastic, dynamic infrastructure & lot more.

Cloud computing addresses many of the challenges of IT silos: inefficiencies, high costs, and ongoing support and maintenance concerns, as well as increasing user demand for services.

Evolution towards Cloud

Both "private" and "public" cloud computing is based on qualities such as self-service, pay-as-you-go charge-back, on-demand provisioning, and the appearance of unbounded scalability.

Public cloud has its own benefits and challenges; everybody knows the benefits so we will see what all challenges of public cloud –

Public clouds like Amazon AWS, Microsoft Azure, Google AppEngine offer infrastructure and platforms as services over the internet. In public clouds, resources and costs are shared by users who use them over the internet on pay per use model.

This model appeals especially to startups and small organizations that have not invested in hardware resources and are looking for ways to avoid the large capex involved in procuring infrastructure upfront. Even though there are several benefits like cost savings, faster time to market, etc., from this model, there are a few challenges listed below that are preventing wide scale adoption of public clouds.

Security - The biggest blockade is the possible security issues due to multi-tenant nature of public clouds. There is security and privacy concerns with sharing same physical hardware with unknown parties that need to addressed.

Control over IT – No direct control of IT infrastructure hence putting up mission critical, data sensitive applications is a potential risk.

Leveraging Existing Investment - Most large organizations that have already invested in their own data centers would see a need to leverage those investments as an important criterion in adopting cloud computing.

Corporate Governance and Auditing: Performing governance and auditing activities with the corporate data abstracted in the public cloud poses challenges, which are yet to be addressed. The limitation or law for storing data across different national boundaries etc.

Outage – There are several outages of public cloud vendor and that creates a chaos to really want to adapt to public cloud for their mission critical applications.

Incident history -

Microsoft Azure: malfunction - 22 h outage on March 13/14, 2008 in Windows Azure,

S3 outage: authentication service overload - 2 h outage on Feb 15, 2008 leading to unavailability,

S3 outage: Single bit error leading to 6–8 h outage on July 20, 2008 gossip protocol blowup,

FlexiScale: core network failure 18 h outage on Oct 31, 2008

The most up-to-date AWS outage around 48 h.

Well, the most important positive about private cloud offer a critical additional benefit: TRUST. This ability to offer elastic computing without sacrificing security or control that is driving many businesses to moving to this delivery model for IT services.

Initially, most private clouds will be made up almost entirely of internal resources. A private cloud can combine both external and internal cloud resources to meet the needs of an application system, and that combination, which is totally under enterprise control using unified management, can change moment by moment. This is also called as Hybrid cloud. With a private cloud, enterprises can run processes internally and externally, having established the private cloud as the control point for workloads. With control through a unified management tool and a user-centric view, the private cloud thus enables IT to make the best decisions about whether to use internal or external resources, or both, and allows that decision to be made on a real-time basis to meet user service needs.

The movement toward cloud computing began for the enterprises with data center virtualization and consolidation of server, storage, and network resources to reduce redundancy and wasted space and equipment with measured planning of both architecture (including facilities allocation and design) and process.

Below are major three stages to cloudify your organization. (There are several stages in between like requirement gathering, building service offerings & service catalogs upon gathered information, security, software compliance, information security compliance, defining billing model/charge back system etc)

Stage 1: Consolidation and Virtualization

Consolidation is a critical application of virtualization, enabling IT departments to regain control of distributed resources by creating shared pools of standardized resources that can be rationalized and centrally managed. Many IT departments already are consolidating under-utilized computing resources by running multiple applications on a single physical server with virtualization technology from IBM AIX, Sun-Oracle Zones/Container & LDOMs, and VMware, Linux KVM etc.

Stage 2: Automation and Optimized Virtualization

In this stage, virtualization optimizes IT resources and increases IT agility, thus speeding time-to market for services. Through automation, Data Centers systematically remove manual labor requirements for the run-time operation of the data center. To create a cloud service, self-service and metering (feedback about the cost of the resources allocated) are offered in addition to automation.

Stage 3: Federation

Linking disparate cloud computing infrastructures with one another by connecting their individual management infrastructures allows disparate cloud IT resources and capabilities-capacity, monitoring, and management-to be shared, much like power from a power grid. It also enables unified metering and billing, one-stop self-service provisioning, and the movement of application loads between clouds, since federation can occur across data center and organization boundaries, with cloud internetworking. Cloud internetworking is the network technology enabling the linkage of disparate cloud systems in a way that accommodates the unique nature of cloud computing and the running of IT workload.

Hence after stage 3 our private cloud should look like as follows.

There is much more to discuss, write on this topic however for this blog entry I’ve explained a very high level overview on implementing private cloud thoughts, I hope this helps.

Cloud Computing Fun with OpenQRM

noreply@blogger.com (Nilesh Joshi) — Sat, 18 Jun 2011 06:06:00 +0000

As promised, I'm sharing OpenQRM deployment & Configuration details.

OpenQRM is an open-source Data-center management platform. Its fully plug-able architecture focuses on automatic, rapid- and appliance-based deployment, monitoring, high-availability, cloud computing and especially on supporting and conforming multiple virtualization technologies. OpenQRM is a single-management console for the complete IT-infra structure and provides a well defined API which can be used to integrate third-party tools as additional plug-ins.

I've been working on OpenQRM since past couple of days and I found this tool quite useful & ultimately cool. According to me, this tool is best fit for any small size IT infrastructure setup however if OpenQRM deployed with a proper planing and with little more customization then it's a very good choice for any medium size IT infrastructure setup to manage data center automation with Cloud computing feature.

Few observations from my side -

Cloud Billing module is available despite of it's being Open Source product. I'm seeing this as the only tool which provides billing facility while being Open Source
Hypervisor supported as management & managed platform – KVM, Xen, Citrix, VMware (VMware with few restrictions like version, security compromises etc)
Hardware independent, no vendor locking
NOT just a Cloud product, Cloud is part of it – it also provides Single Console Hypervisor management (KVM, Xen, Citrix, VMware)
Plug-able Monitoring framework using Zabbix
High availability support
Physical system provision via cloud
Public cloud provisioning (Hybrid Cloud is also possible using Eucalyptus & Amazon)
Number of Storage types are possible like Netapps, iSCSI, LVM, File-based, LVM NFS server & many more good feature.

I think it's worth to see how OpenQRM can help you so go get OpenQRM and try it. It's worth to play around this product; it's fun to work on this product & possibly deploy it for better infrastructure management.

Well, here you go for an Embed version of document -

OpenQRM Implementation Document PSL

Currently I'm working on IBM Service Delivery Manager (ISDM) cloud platform. ISDM has been build upon integration of IBM products namely -

IBM Tivoli Provisioning Manager
IBM Tivoli Service Automation Manager
IBM Tivoli Usage & Accounting Manager
IBM Tivoli Monitoring

A wonderful product to work on, I'll soon post more details about ISDM work & my experiences. Currently, I've been successfully completed ISDM integration with VMware & KVM hypervisors and enabled VM provisioning,metering & overall VM life cycle management. Lot more things in progress like building up software catalog, Image Library with well designed images and much more.

Cloud Computing with Eucalyptus Installing & Configuring Eucalyptus 2.0.0 on CentOS 5.5

noreply@blogger.com (Nilesh Joshi) — Sun, 26 Dec 2010 18:27:00 +0000

After a long time... Yes, really after a long time but there is a reason to that. The reason is, I recently started working on Cloud Computing & relatively I'm pretty new to this area and hence was little busy understanding the concepts and preparing for some POCs & possible deployments.

In my opinion, overall getting started with Cloud Administration is easy if your UNIX, Virtualization & Networking concepts are clear & strong.

As my first cloud assignment I started working with an Open Source tool known as Eucalyptus (Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems)

I found Eucalyptus very useful and though to share "Eucalyptus Installation & Configuration Documentation" with community for future reference.

Note: This document is a private document hence printing/downloading/copying of this document is restricted.

Before jump into "How-To" lets get introduced to the tool -

Eucalyptus is an open-source software platform that implements IaaS-style cloud computing using the existing Linux-based infrastructure found in the modern data center. It is interface compatible with Amazon's AWS making it possible to move workloads between AWS and the data center without modifying the code that implements them. Eucalyptus also works with most of the currently available Linux distributions including Ubuntu, Red Hat Enterprise Linux (RHEL), CentOS, SUSE Linux Enterprise Server (SLES), openSUSE, Debian and Fedora. Similarly, Eucalyptus can use a variety of virtualization technologies including VMware, Xen, and KVM to implement the cloud abstractions it supports.

Eucalyptus Feature Highlights:

Support for Amazon AWS (EC2, S3, and EBS)
Includes Walrus: an Amazon S3 interface-compatible storage manager
Added support for elastic IP assignment
Web-based interface for cloud configuration
Image registration and image attribute manipulation
Configurable scheduling policies and SLAs
Support for multiple hypervisor technologies within the same cloud

Benefits of Eucalyptus:

Build a private cloud that enables you to “cloud-burst” into Amazon AWS
Allows a cloud to be easily deployed on all types of legacy hardware and software
Customers can leverage the development strength of our worldwide user community
Eucalyptus is compatible with multiple distributions of Linux
Eucalyptus also supports the commercial Linux distributions: Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES)

Benefits of Eucalyptus for IT administrators:

Delivers a self-service provisioning IT infrastructure to end users that require IT resources quickly
Maintains existing infrastructure with no additional capital expense and reduces operating expense
Keeps critical data behind the firewall
Technology is an overlay to the existing hardware and software infrastructure, not a replacement
Avoids lock-in to a 3rd party public cloud vendor
Enables easy transitions back and forth between private and public clouds

Well, I hope now everyone know what the tool is all about, why to make use of it and benefits of deploying this tool. Now let's jump into document & have a happy Eucalyptus configuration.

Direct URL to Documentation - http://www.scribd.com/full/45951224?access_key=key-10u1jf5m7ld06ys640uf

Here you go, An Embedded Version -

Installing Configuring Eucalyptus 2.0.0 on CentOS 5.5

Hope this helps!

Currently, I'm working on a POC for building private cloud using tool known as "OpenQRM" and I'm almost completed it... I'm hoping to share a nice document on this subject soon! Till then enjoy & have a Happy Christmas!!!