UNIX my first love...

Dynamically Adding a RAW device to a Solaris zone.

noreply@blogger.com (Nilesh Joshi) — Fri, 10 Feb 2012 20:06:00 +0000

Dynamically Adding a RAW device to a Solaris zone.

By default it is not possible to add raw device to zone without taking a reboot on zone however there is a famous saying => "There is always a way out for those clever enough to find it". So I'm, just kidding, don't take it seriously. :)

I found a little hack to accomplish the objective of adding raw device to zone without rebooting it. Here is a way out -

1) Add the device to the zonecfg

#zonecfg -z barvozone1
zonecfg:barvozone1> add device
zonecfg:barvozone1:device> set match=/dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0
zonecfg:barvozone1:device> end
zonecfg:barvozone1>exit

2) use the mknod command to create the device in the zones dev folder

#ls -l /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0

lrwxrwxrwx   1 root     root          67 Feb 18 15:34 /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0 -> ../../devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

#ls -l /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

crw-r-----   1 root     sys      118, 128 Mar  5 23:55 /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

# cd /barvozone1/zonepath/dev

# mknod c3t60050768018A8023B8000000000000F0d0s0 c 118 128

That's it. The raw device is now visible within zone and now you can start with your stuffs without any downtime. Isn't it cool?

Identify ASM Disks from OS interface

noreply@blogger.com (Nilesh Joshi) — Wed, 08 Feb 2012 07:07:00 +0000

There are several ways to identify ASM disks from the database interface however sometimes or in fact most of the times it might be the case that the System Engineer/Administrator doesn't have access to Oracle (ASM instance) or he may not aware of database operations so in that case how to recognize that a particular disk is being used and under control of ASM.

There are several occasions we need to order SAN disks/LUNs to either expand volume group, zpool or metadevice to increase a filesystem within it. Now consider, there are already few devices/disks available on system and looks free but if you are unsure whether a specific device/disk/LUN is being used by ASM, you can look at the header of the device. The od (octal dump) command can be used. E.g. od --read-bytes=128 --format=c displays the 1st 128 bytes of a device in character format.

For Solaris distributions –

root:aczfild3s:/var/tmp # od -c -N 128 /dev/rdsk/c5t6006016051402800F07C0E23B0CFDF11d0s0
0000000 \0 202 001 001 \0 \0 \0 \0 200 \0 \0 017 A 200 327 272
0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \n 020 \0 \0 \0 017 001 003 V 4 S _ P E R _
0000120 D A T A 0 1 _ 0 0 1 5 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 V 4 S _ P E R _
0000160 D A T A 0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

This is a ASM disk as the ORCLDISK string exists in the header. The ASM disk name is V4S_PER_DATA01_0015.

For Linux distributions –

# od --read-bytes=128 --format=c /dev/dm-39
0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 w 374 211 343
0000020 244 } 030 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \0 \0 020 \n \0 \0 001 003 N C A S _ D I S
0000120 K 1 _ 0 0 0 0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 N C A S _ D I S
0000160 K 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

The ASM disk name is NCAS_DISK1_0000.

For AIX distributions –

AIX is one of the “SENSIBLE” distributions I must say. Why? Check this out –

In Linux or Oracle-Sun Solaris the native volume manager or utilities around it does not identify if the disk is being used by ASM and it simply allow to format/erase the disk and allow to add the device to inline filesystem or volume group however AIX native volume manager is a step ahead & smart - disks being used by Oracle ASM cannot be also used with AIX LVM. The main reason for this is because the Oracle ASM software puts information on the raw disk to identify it as an Oracle disk, which wipes out any existing PVID or VGDA information.

If an attempt is made to use LVM on a disk already in an Oracle ASM disk group the following errors will be seen:

root> extendvg -f apps hdiskpower10
0516-1339 extendvg: Physical volume contains some 3rd party volume group.
0516-1397 extendvg: The physical volume hdiskpower10, will not be added to the volume group.
0516-792 extendvg: Unable to extend volume group.

root> mkvg -y lparvg hdiskpower128
0516-1339 mkvg: Physical volume contains some 3rd party volume group.
0516-1397 mkvg: The physical volume hdiskpower128, will not be added to the volume group.
0516-862 mkvg: Unable to create volume group.

LVM knows to search for the Oracle identifier on these disks, and will fail the operation. You can manually check for the identifier by running the lquerypv command.

# lquerypv -h /dev/hdiskpower7
00000000 00820101 00000000 80000000 F7BE76D7 ..............v.00000010 00000000 00000000 00000000 00000000
................00000020 4F52434C 4449534B 00000000 00000000 ORCLDISK........00000030 00000000 00000000 00000000 00000000 ................00000040 0A100000 00000103 44415441 5F303030 ........DATA_00000000050 30000000 00000000 00000000 00000000 0...............00000060 00000000 00000000 44415441 00000000 ........DATA....00000070 00000000 00000000 00000000 00000000 ................00000080 00000000 00000000 44415441 5F303030 ........DATA_00000000090 30000000 00000000 00000000 00000000 0...............000000A0 00000000 00000000 00000000 00000000 ................000000B0 00000000 00000000 00000000 00000000 ................000000C0 00000000 00000000 01F64610 3B306C00 ..........F.;0l. 000000D0 01F64610 3B9C7C00 02001000 00100000 ..F.;. .........000000E0 0001BC80 0000C800 00000002 00000001 ................000000F0 00000002 00000002 00000000 00000000 ................

Hope that helps.

Oracle-Solaris Patching using Live Upgrade

noreply@blogger.com (Nilesh Joshi) — Mon, 03 Oct 2011 09:34:00 +0000

In today's post I'll be talking about a procedure for patching Solaris server having zones installed on it using Live Upgrade functionality.

The Solaris OS Recommended Patch Cluster provides critical Solaris OS Security, Data Corruption, and System Availability fixes & hence it is advisable to patch your Solaris systems twice in a year (atleast), as per Oracle-Sun's Critical Patch Update release schedule, I prefer to execute patch cycle for my environment in end of April and sometime late October every year.

Oracle-Sun CPUs are released on the Tuesday closest to the 17th of January, April, July, and October –

See - http://www.oracle.com/technetwork/topics/security/alerts-086861.html

In my environment, I use Live Upgrade to patch our Solaris systems. Reason behind using Live Upgrade for patching purpose are -

1. Create a copy of the system environment; that is, a copy of the root (/) file system

2. Live Upgrade has build-in feature for splitting the mirrors of an SVM mirrored root (detach, attach, preserve options on lucreate) hence low overhead to deal with SVM mirror break stuffs separately etc.

3. Less downtime (not more than 15-20 mins) and minimal risk.

4. Better back out option. In case something breaks after patching revert to old BE and be at stage from where started, again that doesn’t take much downtime and safe option.

5. The most appropriate option for those Solaris servers who have zones/containers installed on it.

There might be many more benefits out there, however I find above benefits best fit for my purpose.

So to summarize, all tasks except the reboot can be accomplished on an operational production system; the impact on any running process is minimal. Live Upgrade is a combination of maximizing system availability when applying changes and minimizing risk by offering the ability to reboot to a known working state (your original environment).

Well. let's see how to do it in real life, in my current environment we have many servers which uses Solaris Volume Manager as their primary volume manager to manage the disk and data. So, let's take a look at patching procedure to patch servers who have SVM installed and configured on it along with zones installed on it sitting on ZFS filesystem.

Let us grab the output of metastat to understand metadevice placement -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23 d13
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d100             m   10GB d20 d10
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Alright, my / is on d100 and /var is on d103. Let us create an alternative boot environment out of it.

# lucreate -c Sol10 -n Sol10pu -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d20:detach,attach -m /var:/dev/md/dsk/d3:ufs,mirror –m /var:/dev/md/dsk/d23:detach,attach

Here I'm trying to create a metadevice d0 representing / UFS filesystem having a sub-mirror d20 (sub-mirror d20 first gets detach from d100 and then attach to d0). Same thing applicable for /var filesystem and it's meta device configuration.

In above command I'm creating a new boot environment called Sol10pu using option “-n”, option “-m” Specifies the vfstab information for a new UFS-based BE.

NOTE: The -m option is not supported for BEs based on ZFS file systems.

NOTE: In case you're performing upgrade and patching in one go then point to be ponder - Before upgrading, you must install the Oracle Solaris Live Upgrade packages from the release to which you are upgrading. New capabilities are added to the upgrade tools, so installing the new packages from the target release is important. Example, you need to upgrade from Oracle Solaris 10 update 4 to Oracle Solaris update 8, so you must get the Oracle Solaris Live Upgrade packages from the Oracle Solaris update 8 DVD.

Once above command finishes, you will see you meta device configuration changed as follows -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

d0 and d3 has one sub-mirror and d100 and d100 has one sub-mirror associated.

Also you will be able to see two boot environments on your Solaris system -

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      yes    yes       no     -
Sol10pu                    yes      no     no        yes    -

Fine, so now we have 2 boot environments with us and we are going to patch the alternate BE (sol10pu) using a patching tool called PCA, BTW - I use PCA (Patch Check Advance) tool to apply patches to our Solaris systems. PCA has been setup to download patches via local web proxy to access outside systems.

PCA setup tips can be found at - http://www.par.univie.ac.at/solaris/pca/usage.html

What PCA needs in case setting it up -

- PERL distribution
- At least one server which is internet facing (this server will then act as a proxy to rest of servers)
- Patch cross-reference file called patchdiag.xref (latest one always while patching)
- Valid Oracle support (MOS) user ID and password
- If at all required, some wrapper scripts to PCA

To do so, let us mount the alternate BE on mount point say /a

# lumount Sol10pu /a
/a

Now I'll create a temporary directory to download the missing & required patches,

# mkdir -p /patchman/patches
My next job is to generate patch_order file,

# /bin/pca missingrs -R /a -H --format "%p-%c" > /patchman/patches/patch_order

Where -R stands for Alternative root directory
Where -H stands for Don't display descriptive headers
Where --format stands for Set output format to FORMAT

And go get them, download -

# /bin/pca missingrs -R /a -d -P /patchman/patches/

Where -d stands for download patches
Where -P stands for Patch download directory

Unmount the ABE -

# luumount /a

Now if you populate the /patchman/patches directory then you will see the list of patches in there.

Unzip all those patches -

# for i in `ls *.zip`;do
unzip $i
rm $i
done

Okay, at this stage we are ready upgrade ABE with patches available -

# cd /patchman/patches; luupgrade -n Sol10pu -s /patchman/patches -t `cat patch_order`

NOTE: Reactive patching may occasionally be necessary to address break-and-fix issues so in this case you can use LU with something like -

Apply single patch to ABE -

# luupgrade -n Sol10pu -s /patchman/patches -t

This will update patches on global as well as non-global zones.

Once the patches are installed it will automatically un-mounts the ABE sol10pu mounted on mount point /a.

Now it's time to activate the ABE sol10pu which just been patched using Live Upgrade utility.

# luactivate Sol10pu
A Live Upgrade Sync operation will be performed on startup of boot environment .
**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Enter the PROM monitor (ok prompt).

2. Change the boot device back to the original boot environment by typing:

     setenv boot-device /pci@0/pci@0/pci@2/scsi@0/disk@0,0:a

3. Boot to the original boot environment by typing:

     boot

**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

# init 6
updating /platform/sun4v/boot_archive
SYSTEM GOING DOWN!!!!

NOTE: Live upgrade always uses init 6 or shutdown commands. Halt and reboot commands will create big time bang, be aware!!!

Once system is up it should show new kernel patch version.

Great, it's been a week after patching and application, DB owners are happy with patching stuffs and now we need to perform post patching stuffs upon certain confirmations.

POST PATCHING WORK TO DO - A Week LATER....
=========================================

Now a week later I need to delete the old boot environment and rebuild the metadevices to be in mirror layout.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      no     no        yes    -
Sol10pu                    yes      yes    yes       no     -

# ludelete Sol10
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

So, the meta devices are looking like as below -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Now clear the d100 & d103 mirrors.

# metaclear d100 d103
d100: Mirror is cleared
d103: Mirror is cleared

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1
d13              s   10GB c1t1d0s3
d10              s   10GB c1t1d0s0

Next attach the sub-mirrors d10 & d13 to metadevices d0 and d3 respectively.

# metattach d0 d10
d0: submirror d10 is attached

# metattach d3 d13
d3: submirror d13 is attached

Hence my final meta device placement looks like as follows -

# metastat -c
d32              p 1.0GB d4
d33              p 1.0GB d4
d36              p   40GB d4
d35              p 1.0GB d4
d34              p 4.0GB d4
d60              p   16GB d4
d30              p 1.0GB d4
d31              p 1.0GB d4
    d4           m 100GB d14 d24
        d14      s 100GB c1t0d0s4
        d24      s 100GB c1t1d0s4
d3               m   10GB d23 d13 (resync-25%)
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d0               m   10GB d20 d10 (resync-45%)
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

That's it. Now your done with patching your Solaris server and zones deployed on it.

Migrating VERITAS Volumes from one storage array to another storage array

noreply@blogger.com (Nilesh Joshi) — Tue, 20 Sep 2011 11:56:00 +0000

Background:

This week I got a task to Migrate Veritas disk groups & their volumes on a specific server from one storage array to new storage array due to old storage array is ruining out of capacity.

To execute this task I split this into 3 phases -

1. Responsibilities with Storage

     a. Target server has to be zoned to new Storage system
     b. Provide LUNs/capacity to server (total capacity required is 3.5 TB 58x60G & 2x15G LUNs)

2. Responsibilities with me

      a. Migrate Veritas disk groups to new storage array
            A) Label all newly added SAN disks (may be using some scripted method) & make sure SAN disks are visible under VxVM
            B) Initialize all those new SAN disks with VxVM
            C) Add the SAN disk to disk group
            D) Mirror the volumes
            E) Verify if sync is completed
            F) Verify from vxprint if you see a new plex added to designated volumes
            G) If all is well then go ahead and detach old plex
            H) Once plex is disassociated from designated volumes then delete the old plex
            I) Verify if data group(s) & their volumes are on new storage array.
     b. Remove disks associated with old storage from Veritas configuration

3. Responsibilities with Storage

     a. Remove disks associated with old storage from server
     b. Take care of redundant paths

Back out -

     a. Have server full backup handy.

Execution -

List the disk groups needs to be migrated to new storage array.

# vxdg list
NAME STATE ID
xxxx_dg enabled,cds 1279726733.18.xxxxx
localswdg enabled,cds 1279726567.16.xxxxx
nass3_dg enabled 1074844579.1535.nassau3

Get the true picture of your VxVM configuration. Save output of this command for future reference

# vxprint -hrt

Now that Storage has attached SAN disks to servers HBA we need to label them and get them to VERITAS control.

List the existing Disks & new disks detected.

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online <<<< All UNIX# disks are from old storage arrary
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto - - online invalid   <<<< All emc_* disks are from new storage arrary
emc_clariion0_1705 auto - - online invalid
emc_clariion0_1706 auto - - online invalid
emc_clariion0_1707 auto - - online invalid
emc_clariion0_1708 auto - - online invalid
emc_clariion0_1709 auto - - online invalid
emc_clariion0_1710 auto - - online invalid
emc_clariion0_1713 auto - - online invalid
emc_clariion0_1800 auto - - online invalid
emc_clariion0_1801 auto - - online invalid
emc_clariion0_1803 auto - - online invalid
emc_clariion0_1804 auto - - online invalid
emc_clariion0_1805 auto - - online invalid
emc_clariion0_1806 auto - - online invalid
emc_clariion0_1807 auto - - online invalid
emc_clariion0_1808 auto - - online invalid
emc_clariion0_1809 auto - - online invalid
emc_clariion0_1810 auto - - online invalid
emc_clariion0_3700 auto - - online invalid

[...]

emc_clariion0_5809 auto - - online invalid
emc_clariion0_5858 auto - - online invalid

Initialize the disks with VxVM and add them to appropriate disk group.

For a single disk -

# vxdisksetup -i emc_clariion0_5858 format=sliced

For multiple disks

# vxdisk list | awk '{print $1}' | grep -i emc > /tmp/EMC_disks
# for d in `cat /tmp/EMC_disks` ; do vxdisksetup -i $d format=sliced; done

Add disks to disk group

#!/bin/sh
#for e.g. vxdg -g nass3_dg adddisk nass3_dg01=emc_clariion0_1704
DG=nass3_dg
DISKS=`vxdisk list | awk '{print $3"="$1}' | grep -i emc`
#Output of command vxdisk list | awk '{print $3"="$1}' | grep -i emc is look like as = nass3_dg01=emc_clariion0_1704
for d in $DISKS;
do vxdg -g $DG adddisk $DISKS;
done

OR you can also use vxdiskadm menu based command to perfrom this activity.

Now you should see something like -

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto:sliced nass3_dg01 nass3_dg online
emc_clariion0_1705 auto:sliced nass3_dg02 nass3_dg online
emc_clariion0_1706 auto:sliced nass3_dg03 nass3_dg online
emc_clariion0_1707 auto:sliced nass3_dg04 nass3_dg online
emc_clariion0_1708 auto:sliced nass3_dg05 nass3_dg online
emc_clariion0_1709 auto:sliced nass3_dg06 nass3_dg online
emc_clariion0_1710 auto:sliced nass3_dg07 nass3_dg online
emc_clariion0_1713 auto:sliced nass3_dg08 nass3_dg online
emc_clariion0_1800 auto:sliced nass3_dg09 nass3_dg online
emc_clariion0_1801 auto:sliced nass3_dg10 nass3_dg online
emc_clariion0_1803 auto:sliced nass3_dg11 nass3_dg online
emc_clariion0_1804 auto:sliced nass3_dg12 nass3_dg online
emc_clariion0_1805 auto:sliced nass3_dg13 nass3_dg online
emc_clariion0_1806 auto:sliced nass3_dg14 nass3_dg online
emc_clariion0_1807 auto:sliced nass3_dg15 nass3_dg online
emc_clariion0_1808 auto:sliced nass3_dg16 nass3_dg online
emc_clariion0_1809 auto:sliced nass3_dg17 nass3_dg online
emc_clariion0_1810 auto:sliced nass3_dg18 nass3_dg online
emc_clariion0_3700 auto:sliced nass3_dg19 nass3_dg online

[...]

emc_clariion0_5809 auto:sliced nass3_dg59 nass3_dg online

Now that we have added new disks to the appropriate datagroups, next task to mirror the volumes.

NOTE: To decide how many disks are require to mirroe a perticular volume is very easy, put those many disks which will meet the current size of volume.

# vxassist -g nass3_dg mirror db_TESTDB_vol alloc=nass3_dg05,nass3_dg02,nass3_dg06,nass3_dg07,nass3_dg08,nass3_dg09,nass3_dg10,nass3_dg11,nass3_dg12,nass3_dg13,nass3_dg14,nass3_dg15,nass3_dg16,nass3_dg17,nass3_dg18,nass3_dg19,nass3_dg20,nass3_dg21,nass3_dg22,nass3_dg23,nass3_dg24,nass3_dg25,nass3_dg26,nass3_dg27,nass3_dg28,nass3_dg29,nass3_dg30,nass3_dg31,nass3_dg32,nass3_dg33,nass3_dg34,nass3_dg35,nass3_dg36,nass3_dg37,nass3_dg38,nass3_dg39,nass3_dg40,nass3_dg41,nass3_dg42,nass3_dg43,nass3_dg44,nass3_dg45,nass3_dg46,nass3_dg47,nass3_dg48,nass3_dg49,nass3_dg50,nass3_dg51,nass3_dg52,nass3_dg53,nass3_dg54,nass3_dg55

Repeat the previous step for rest of the data groups and volumes.

Likewise you can add disks to the remaining volumes and then we have to mirror them.

Check sync progress using -

# vxtask -l list
Task: 5912 RUNNING
Type: ATCOPY
Operation: PLXATT Vol db_TESTDB_vol Plex db_TESTDB_vol-02 Dg nass3_dg
Started: Fri Sep 16 16:05:24 2011
Throttle: 0
Progress: 4.27% 268713984 of 6291456000 Blocks
Work time: 29 minutes, 30 seconds (11:01:11 remaining)

Verify from vxprint , you should see a new plex added to db_TESTDB_vol volume

# vxprint -qthg nass3_dg db_TESTDB_vol

If everything looks good, then detach/disassociate & remove old plex, in short - break the mirror. (Before doing so get application owner consent)

# vxmend -g nass3_dg off db_TESTDB_vol-01
# vxplex -g nass3_dg -o rm dis db_TESTDB_vol-01

Repeat the previous two steps for rest of the plexes.

Well, by this time we can say we are done so verify if datagroup(s) & their volumes are on new storage arrary.

# vxprint -qthg nass3_dg db_TESTDB_vol
v db_TESTDB_vol - ENABLED ACTIVE 6291456000 SELECT db_TESTDB_vol-02 fsgen
pl db_TESTDB_vol-02 db_TESTDB_vol ENABLED ACTIVE 6291456000 STRIPE 3/128 RW
sd nass3_dg05-01 db_TESTDB_vol-02 nass3_dg05 0 125754880 0/0 emc_clariion0_1708 ENA
sd nass3_dg08-01 db_TESTDB_vol-02 nass3_dg08 0 125754880 0/125754880 emc_clariion0_1713 ENA
sd nass3_dg11-01 db_TESTDB_vol-02 nass3_dg11 0 125754880 0/251509760 emc_clariion0_1803 ENA
sd nass3_dg14-01 db_TESTDB_vol-02 nass3_dg14 0 125754880 0/377264640 emc_clariion0_1806 ENA
sd nass3_dg17-01 db_TESTDB_vol-02 nass3_dg17 0 125754880 0/503019520 emc_clariion0_1809 ENA
sd nass3_dg20-01 db_TESTDB_vol-02 nass3_dg20 0 125754880 0/628774400 emc_clariion0_3702 ENA
sd nass3_dg23-01 db_TESTDB_vol-02 nass3_dg23 0 125754880 0/754529280 emc_clariion0_3705 ENA
sd nass3_dg26-01 db_TESTDB_vol-02 nass3_dg26 0 125754880 0/880284160 emc_clariion0_3708 ENA
sd nass3_dg29-01 db_TESTDB_vol-02 nass3_dg29 0 125754880 0/1006039040 emc_clariion0_3800 ENA
sd nass3_dg32-01 db_TESTDB_vol-02 nass3_dg32 0 125754880 0/1131793920 emc_clariion0_3803 ENA

[...]

sd nass3_dg43-01 db_TESTDB_vol-02 nass3_dg43 0 125754880 2/1594132480 emc_clariion0_5704 ENA
sd nass3_dg46-01 db_TESTDB_vol-02 nass3_dg46 0 125754880 2/1719887360 emc_clariion0_5707 ENA
sd nass3_dg50-01 db_TESTDB_vol-02 nass3_dg50 0 125754880 2/1845642240 emc_clariion0_5800
ENA
sd nass3_dg53-01 db_TESTDB_vol-02 nass3_dg53 0 125754880 2/1971397120 emc_clariion0_5803 ENA

Yes, volume has been moved to new storage.

Now we are ready to remove disks associated with old storage from Veritas-configuration.

# vxdg -g nass3_dg rmdisk fls02c4_nass3_dg

Ask storage team to detach/remove the old disks permanently from the server.

This is overall procedure to migrate your data groups & their volumes from one storage to another storage.

Disable Large Segment Offload (LSO) in Solaris 10

noreply@blogger.com (Nilesh Joshi) — Tue, 30 Aug 2011 08:37:00 +0000

In this blog article, I will share my understanding on Large Segment Offload (LSO). I got a task to disable the LSO on few of the servers (include zones).

Let's first understand what is LSO stands for and what is the purpose of using LSO.

As you see above LSO stands for Large Segment Offload.

TCP Offload Engine is an embryonic technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards, hence the no CPU cycle and kernel time will get consumed.

LSO saves valuable CPU cycles by allowing the network protocol stack to handle large segments instead of the traditional model of MSS (TCP Maximum Segment Size) sized segments. In the traditional network stack, the TCP layer segments the outgoing data into the MSS sized segments and passes them down to the driver. This becomes computationally expensive with 10 GigE networking because of the large number of kernel functional calls required for every MSS segment. With LSO, a large segment is passed by TCP to the driver, and the driver or NIC hardware does the job of TCP segmentation (LSO offload the segmentation job on Layer 4 to the NIC driver). An LSO segment may be as large as 64 KByte. The larger the LSO segment, better the CPU efficiency since the network stack has to work with smaller number of segments for the same throughput.

So in simple words, use LSO for better network performance while reducing processor (CPU) utilization.

Segmentation is needed if a full TCP segment does not fit into the Ethernet Maximum Transmission Unit (MTU) size. With LSO, TCP segments do not need to get split in software implementation, this is done on the interface card hardware instead. Being much more effective, this improves the network performance while reducing the workload on the CPUs. LSO is most helpful for 10 Gigabit Ethernet network interfaces and on systems with slow CPU threads or lack of CPU resources.

Solaris LSO can be used if all of the three conditions are met :

1.The TCP/IP stack integrates LSO,

2.The Network Interface Card hardware supports it (for e.g. drivers like e1000g,ixgb,ixgbe etc),
3.The driver for this network card is capable of handling it.

Sadly, in most of the cases LSO seems to be not working that well hence it leads to disable the LSO support. Here is the ways to disable the LSO.

Ways to disable LSO -

Disable LSO by adding the following line in the /kernel/drv/e1000g.conf file (I’m using the e1000g interface/driver hence the file that I'm using is /kernel/drv/e1000g.conf) :

lso_enable=0,0,0,0,0,0,0,0;

After making the changes reboot is required or else if reboot is not possible then you can use ndd utility/command to disable it on a temporary basis and not persist across the reboot.

Using ndd you can disable it as shown below -

# ndd -set /dev/ip ip_lso_outbound 0

Also if you don't want to reboot the server after modifying the file /kernel/drv/e1000g.conf you can simply unplumb all of your e1000g interfaces with ifconfig, do "update_drv e1000g" to reload the .conf file, and then replumb and reconfigure the interfaces with ifconfig however still if I'm going to unplumb the network interfaces then eventually I'll be disturbing the services so reboot is the best option.

I had to disable the LSO as our application folks were experiencing slowness in their web application (response time etc.) It looks like LSO cause unstable connections & hence there are few observations like dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately application folks observed slowness in their web application, NFS stuffs etc.

Installing Perl modules on Solaris 10 - An experience

noreply@blogger.com (Nilesh Joshi) — Fri, 26 Aug 2011 14:35:00 +0000

In recent times I got a task to perform few Perl modules installation. To build the module in question (Compress::Raw::Zlib), I downloaded the module from CPAN (Comprehensive Perl Archive Network), verified that the MD5 checksum was correct, and used the following steps to compile the module:

# perl Makefile.PL
# make
# make install

The ‘make Makefile.PL’ completed successfully, but the make failed with the following errors:

# make
cp lib/Compress/Raw/Zlib.pm blib/lib/Compress/Raw/Zlib.pm
AutoSplitting blib/lib/Compress/Raw/Zlib.pm (blib/lib/auto/Compress/Raw/Zlib)
/usr/bin/perl /usr/perl5/5.8.4/lib/ExtUtils/xsubpp -typemap /usr/perl5/5.8.4/lib/ExtUtils/typemap -typemap typemap Zlib.xs > Zlib.xsc && mv Zlib.xsc Zlib.c
cc -c -I./zlib-src -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO -xO3 -xspace -xildoff -DVERSION=\"2.037\" -DXS_VERSION=\"2.037\" -KPIC "-I/usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE" -DNO_VIZ -DGZIP_OS_CODE=3 -DUSE_PPPORT_H Zlib.c
/usr/ucb/cc: language optional software package not installed
*** Error code 1
make: Fatal error: Command failed for target `Zlib.o'

Above output indicate an error related to compiler & it is clearly showing that it is looking for cc complier however I was trying to build the module using gcc and not the cc (Sun studio) compiler but it seems like by default modules Makefile is referring to Sun Studio compiler and not the gcc complier. I do not have Sun Studio compiler on our systems & as this is being license software I can not even simply install it on any server due to compliance restrictions.

After doing a little bit research on the issue, I found that there is a way to fix this problem. If you want to use gcc to build all Perl modules on a system, you can permanently remove the Sun Studio compiler references by adjusting the “cccdlflags” and “optimize” variables in /usr/perl5/5.8.4/lib/sun4-solaris-64int/Config.pm:

root@XXXX:/usr/perl5/5.8.4/lib/sun4-solaris-64int# egrep '(KPIC|O3)' Config.pm

cccdlflags='-KPIC'
optimize='-xO3 -xspace -xildoff'

After removing the above entries make was successful with gcc complier.

There is yet another method to do the Perl module installation even not performing above changes to Config.pm. You can simply build Perl module using –

- /usr/perl5/bin/perlgcc Makefile.PL
- /usr/sfw/bin/gmake
- /usr/sfw/bin/gmake test
- /usr/sfw/bin/gmake install

What we were discussing in above is how to install Perl modules manually and how to tackle with complier related issues. There is yet another automated method to install Perl module and which is most preferred and quick method to do it.

Simply export the http_proxy environment variable with your proxy server name or IP address and valid port number so that you can directly reach to the internet via proxy.

# http_proxy=myproxy:8080; export http_proxy
# perl -MCPAN -e shell

If this is the first time this script is run it will ask a lot of questions and store the results in Config.pm

Then simply run –

1. Start the CPAN shell script: /usr/perl5/bin/perl -MCPAN -e shell.
2. Install a module, e.g.: install CGI::Session
3. Exit the shell script: exit

To check the already installed modules execute –

# perl -e 'use HTML::Parser;'

If nothing is returned, Perl was able to locate the module. Otherwise, you will see “Can't locate HTML/Parser.pm in @INC”

One more experience to share here. I installed Threads Perl module however user was not able to execute his scripts due to current Perl distribution was not build or complied with “useithreads” configuration option hence he requested to recompile Perl distribution with threads option enable.

Here is a string that I use to compile Perl distribution with threads option.

# sh Configure -Dusethreads -Dprefix=/usr/perl5/5.14.1

With new Perl compilation user was able to execute his scripts with threads module.

Hope this document will help to those who are new to this work.

IBM’s Private Cloud Computing Platform (ISDM)

noreply@blogger.com (Nilesh Joshi) — Sat, 30 Jul 2011 17:41:00 +0000

In last post I've discussed about Cloud Computing at a broad level and as I said would be writing the brief article on IBM private cloud platform. Since past few months I've been involved in evaluating IBM private cloud technology known as ISDM (IBM Integrated Service Delivery Manager) and herein I'm writing about my own understandings and experiences about IBM private cloud PoC that recently deployed in our test environment.

The below document gives very well information about IBM private cloud solution. I hope you will find this document helpful.

A Review of IBM Private Cloud

Cloud computing – Revisited

noreply@blogger.com (Nilesh Joshi) — Sat, 02 Jul 2011 19:46:00 +0000

Well, before I write something about IBM Cloud Computing platform (IBM Integrated Service Delivery Manager) I would like to explain few of my understandings about Cloud Computing & its implementation factors.

I’m sure almost everyone knows about cloud models/types which are namely public cloud, private cloud, hybrid cloud & community cloud also these all clouds have specific service delivery models in particular, IT-as-a-Service, Platform-as-a-service & Software-as-a-Service.

Today, I would like to write an article on implementation factor(s) of cloud computing. Since past 8-9 months have been working on various cloud technology platforms/products and quite a few POC environments to test out & make out which product best suits to an organization specific requirements. There is no ready made cloud product out there which will able meet all of your organization requirements hence you need to take a call whether customizing the solution or altering the organization policies, structure i.e. whatever requirement. Trust me testing the technology is an easy job however to fix on what is the best fit product for an organization is a “not easy” job as this is something which is beyond just a technology & cloud implementers, decision makers and technical people should look at it from business viewpoint too anyways this is not a topic of discussion here.

Let’s start this discussion with traditional IT system management portfolio & service delivery model. In ideal scenario a particular software development project gets a requirement to develop a product and resources required to deliver that product, the project manager or project lead starts identifying the resource requirements and ask IT department to procure necessary systems, OS, software, databases, toolkits or provision necessary systems, OS, software, databases via available IT infrastructures which takes significant amount of time since person who driving the project has to get many things approved by higher management, software compliance team, information security team etc then procurement start which takes several weeks or he raise a request to IT department to provision systems with necessary OS, software, databases which also takes several days and so on. Then only actual application, product development starts, tested & deployed on that infrastructure. We can see several challenges in this model as described below –

Need large Capex - Large investments required to procure the infrastructure required for any product development.

Poor utilization of resources - Application usage is not going to be constant yet the infrastructure is provisioned for peak demand hence the infrastructure remains under-utilized for a major part of the time.

Slow Time-to-Market - This model of procuring and provisioning infrastructure usually requires significant time and reduces the agility of an organization in creating new business solutions.

Now we can clearly see how traditional IT system management portfolio & service delivery model prove expensive in current competitive, difficult economic market.

Also on the other hand, still nothing is accountable as such. For system administrators it’s becomes a challenge to showcase or prove that a particular system or systems are under utilized which is dedicated to a specific functional group and those can be shared among the other groups. In such cases cost grows, complexities increases as number of hardware resources and software resources increases, data center space, cooling cost increases, requires more man power to handle the infrastructure, lack of standardization due to silos & much more to talk about.

Another point of view, Over 70% of IT budgets in a typical Data Centre goes just to "keeping the lights on", especially keeping the “green” lights on! Conversely, only a small portion of each dollar spent on IT today creates a direct business benefit. Since Data Center IT assets become obsolete approximately every 5 years, the vast majority of IT investment is spent on upgrading various pieces of infrastructure and providing redundancy and recover-ability: activities that consume approximately 60 to 80% of IT expenditures without necessarily providing optimal business value or innovation.

To overcome on above challenges to a certain level virtualization & consolidation has played a major role however IT has started seen many challenges with virtualization environment like VM sprawl, costing etc consequently to overcome on those challenges & challenges of overall service delivery methodology Cloud Computing technology evolved.

Cloud Computing is a model of service delivery and access where dynamically scalable and virtualized resources are provided as a service over the Internet.

Cloud Computing offers an alternative approach that profoundly transforms the way in which information and services are consumed and provided and can enable businesses to:

Lower costs by using energy and resources more efficiently

Enhance agility, growth, and profitability

Simplify operations and management

Ensure elastic and trusted collaboration between various groups which results into visibility & standardized, smooth operations.

Faster time to market

On-demand elastic, dynamic infrastructure & lot more.

Cloud computing addresses many of the challenges of IT silos: inefficiencies, high costs, and ongoing support and maintenance concerns, as well as increasing user demand for services.

Evolution towards Cloud

Both "private" and "public" cloud computing is based on qualities such as self-service, pay-as-you-go charge-back, on-demand provisioning, and the appearance of unbounded scalability.

Public cloud has its own benefits and challenges; everybody knows the benefits so we will see what all challenges of public cloud –

Public clouds like Amazon AWS, Microsoft Azure, Google AppEngine offer infrastructure and platforms as services over the internet. In public clouds, resources and costs are shared by users who use them over the internet on pay per use model.

This model appeals especially to startups and small organizations that have not invested in hardware resources and are looking for ways to avoid the large capex involved in procuring infrastructure upfront. Even though there are several benefits like cost savings, faster time to market, etc., from this model, there are a few challenges listed below that are preventing wide scale adoption of public clouds.

Security - The biggest blockade is the possible security issues due to multi-tenant nature of public clouds. There is security and privacy concerns with sharing same physical hardware with unknown parties that need to addressed.

Control over IT – No direct control of IT infrastructure hence putting up mission critical, data sensitive applications is a potential risk.

Leveraging Existing Investment - Most large organizations that have already invested in their own data centers would see a need to leverage those investments as an important criterion in adopting cloud computing.

Corporate Governance and Auditing: Performing governance and auditing activities with the corporate data abstracted in the public cloud poses challenges, which are yet to be addressed. The limitation or law for storing data across different national boundaries etc.

Outage – There are several outages of public cloud vendor and that creates a chaos to really want to adapt to public cloud for their mission critical applications.

Incident history -

Microsoft Azure: malfunction - 22 h outage on March 13/14, 2008 in Windows Azure,

S3 outage: authentication service overload - 2 h outage on Feb 15, 2008 leading to unavailability,

S3 outage: Single bit error leading to 6–8 h outage on July 20, 2008 gossip protocol blowup,

FlexiScale: core network failure 18 h outage on Oct 31, 2008

The most up-to-date AWS outage around 48 h.

Well, the most important positive about private cloud offer a critical additional benefit: TRUST. This ability to offer elastic computing without sacrificing security or control that is driving many businesses to moving to this delivery model for IT services.

Initially, most private clouds will be made up almost entirely of internal resources. A private cloud can combine both external and internal cloud resources to meet the needs of an application system, and that combination, which is totally under enterprise control using unified management, can change moment by moment. This is also called as Hybrid cloud. With a private cloud, enterprises can run processes internally and externally, having established the private cloud as the control point for workloads. With control through a unified management tool and a user-centric view, the private cloud thus enables IT to make the best decisions about whether to use internal or external resources, or both, and allows that decision to be made on a real-time basis to meet user service needs.

The movement toward cloud computing began for the enterprises with data center virtualization and consolidation of server, storage, and network resources to reduce redundancy and wasted space and equipment with measured planning of both architecture (including facilities allocation and design) and process.

Below are major three stages to cloudify your organization. (There are several stages in between like requirement gathering, building service offerings & service catalogs upon gathered information, security, software compliance, information security compliance, defining billing model/charge back system etc)

Stage 1: Consolidation and Virtualization

Consolidation is a critical application of virtualization, enabling IT departments to regain control of distributed resources by creating shared pools of standardized resources that can be rationalized and centrally managed. Many IT departments already are consolidating under-utilized computing resources by running multiple applications on a single physical server with virtualization technology from IBM AIX, Sun-Oracle Zones/Container & LDOMs, and VMware, Linux KVM etc.

Stage 2: Automation and Optimized Virtualization

In this stage, virtualization optimizes IT resources and increases IT agility, thus speeding time-to market for services. Through automation, Data Centers systematically remove manual labor requirements for the run-time operation of the data center. To create a cloud service, self-service and metering (feedback about the cost of the resources allocated) are offered in addition to automation.

Stage 3: Federation

Linking disparate cloud computing infrastructures with one another by connecting their individual management infrastructures allows disparate cloud IT resources and capabilities-capacity, monitoring, and management-to be shared, much like power from a power grid. It also enables unified metering and billing, one-stop self-service provisioning, and the movement of application loads between clouds, since federation can occur across data center and organization boundaries, with cloud internetworking. Cloud internetworking is the network technology enabling the linkage of disparate cloud systems in a way that accommodates the unique nature of cloud computing and the running of IT workload.

Hence after stage 3 our private cloud should look like as follows.

There is much more to discuss, write on this topic however for this blog entry I’ve explained a very high level overview on implementing private cloud thoughts, I hope this helps.

Cloud Computing Fun with OpenQRM

noreply@blogger.com (Nilesh Joshi) — Sat, 18 Jun 2011 06:06:00 +0000

As promised, I'm sharing OpenQRM deployment & Configuration details.

OpenQRM is an open-source Data-center management platform. Its fully plug-able architecture focuses on automatic, rapid- and appliance-based deployment, monitoring, high-availability, cloud computing and especially on supporting and conforming multiple virtualization technologies. OpenQRM is a single-management console for the complete IT-infra structure and provides a well defined API which can be used to integrate third-party tools as additional plug-ins.

I've been working on OpenQRM since past couple of days and I found this tool quite useful & ultimately cool. According to me, this tool is best fit for any small size IT infrastructure setup however if OpenQRM deployed with a proper planing and with little more customization then it's a very good choice for any medium size IT infrastructure setup to manage data center automation with Cloud computing feature.

Few observations from my side -

Cloud Billing module is available despite of it's being Open Source product. I'm seeing this as the only tool which provides billing facility while being Open Source
Hypervisor supported as management & managed platform – KVM, Xen, Citrix, VMware (VMware with few restrictions like version, security compromises etc)
Hardware independent, no vendor locking
NOT just a Cloud product, Cloud is part of it – it also provides Single Console Hypervisor management (KVM, Xen, Citrix, VMware)
Plug-able Monitoring framework using Zabbix
High availability support
Physical system provision via cloud
Public cloud provisioning (Hybrid Cloud is also possible using Eucalyptus & Amazon)
Number of Storage types are possible like Netapps, iSCSI, LVM, File-based, LVM NFS server & many more good feature.

I think it's worth to see how OpenQRM can help you so go get OpenQRM and try it. It's worth to play around this product; it's fun to work on this product & possibly deploy it for better infrastructure management.

Well, here you go for an Embed version of document -

OpenQRM Implementation Document PSL

Currently I'm working on IBM Service Delivery Manager (ISDM) cloud platform. ISDM has been build upon integration of IBM products namely -

IBM Tivoli Provisioning Manager
IBM Tivoli Service Automation Manager
IBM Tivoli Usage & Accounting Manager
IBM Tivoli Monitoring

A wonderful product to work on, I'll soon post more details about ISDM work & my experiences. Currently, I've been successfully completed ISDM integration with VMware & KVM hypervisors and enabled VM provisioning,metering & overall VM life cycle management. Lot more things in progress like building up software catalog, Image Library with well designed images and much more.

Cloud Computing with Eucalyptus Installing & Configuring Eucalyptus 2.0.0 on CentOS 5.5

noreply@blogger.com (Nilesh Joshi) — Sun, 26 Dec 2010 18:27:00 +0000

After a long time... Yes, really after a long time but there is a reason to that. The reason is, I recently started working on Cloud Computing & relatively I'm pretty new to this area and hence was little busy understanding the concepts and preparing for some POCs & possible deployments.

In my opinion, overall getting started with Cloud Administration is easy if your UNIX, Virtualization & Networking concepts are clear & strong.

As my first cloud assignment I started working with an Open Source tool known as Eucalyptus (Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems)

I found Eucalyptus very useful and though to share "Eucalyptus Installation & Configuration Documentation" with community for future reference.

Note: This document is a private document hence printing/downloading/copying of this document is restricted.

Before jump into "How-To" lets get introduced to the tool -

Eucalyptus is an open-source software platform that implements IaaS-style cloud computing using the existing Linux-based infrastructure found in the modern data center. It is interface compatible with Amazon's AWS making it possible to move workloads between AWS and the data center without modifying the code that implements them. Eucalyptus also works with most of the currently available Linux distributions including Ubuntu, Red Hat Enterprise Linux (RHEL), CentOS, SUSE Linux Enterprise Server (SLES), openSUSE, Debian and Fedora. Similarly, Eucalyptus can use a variety of virtualization technologies including VMware, Xen, and KVM to implement the cloud abstractions it supports.

Eucalyptus Feature Highlights:

Support for Amazon AWS (EC2, S3, and EBS)
Includes Walrus: an Amazon S3 interface-compatible storage manager
Added support for elastic IP assignment
Web-based interface for cloud configuration
Image registration and image attribute manipulation
Configurable scheduling policies and SLAs
Support for multiple hypervisor technologies within the same cloud

Benefits of Eucalyptus:

Build a private cloud that enables you to “cloud-burst” into Amazon AWS
Allows a cloud to be easily deployed on all types of legacy hardware and software
Customers can leverage the development strength of our worldwide user community
Eucalyptus is compatible with multiple distributions of Linux
Eucalyptus also supports the commercial Linux distributions: Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES)

Benefits of Eucalyptus for IT administrators:

Delivers a self-service provisioning IT infrastructure to end users that require IT resources quickly
Maintains existing infrastructure with no additional capital expense and reduces operating expense
Keeps critical data behind the firewall
Technology is an overlay to the existing hardware and software infrastructure, not a replacement
Avoids lock-in to a 3rd party public cloud vendor
Enables easy transitions back and forth between private and public clouds

Well, I hope now everyone know what the tool is all about, why to make use of it and benefits of deploying this tool. Now let's jump into document & have a happy Eucalyptus configuration.

Direct URL to Documentation - http://www.scribd.com/full/45951224?access_key=key-10u1jf5m7ld06ys640uf

Here you go, An Embedded Version -

Installing Configuring Eucalyptus 2.0.0 on CentOS 5.5

Hope this helps!

Currently, I'm working on a POC for building private cloud using tool known as "OpenQRM" and I'm almost completed it... I'm hoping to share a nice document on this subject soon! Till then enjoy & have a Happy Christmas!!!

Moving to POWER7 from SPARC

noreply@blogger.com (Nilesh Joshi) — Tue, 19 Oct 2010 16:56:00 +0000

Last weekend I was reading a very interesting thread about moving to POWER7 from SPARC. I found it very interesting hence thought to share it with you.

There is an attention-grabbing discussion going on under LinkedIn. The subject is –

“Moving to POWER7 from SPARC”

I think this is quite exciting topic or rather very strategic decision to make so I decided to have a brief summary note on it.

The discussion starts with question –

“My employer is considering seriously moving to POWER7 from SPARC as we retire EOSL (End of Service Life) hardware. Has anyone considered such a move or made the move from POWER to SPARC? “

Very difficult to answer isn’t it? Yes, it is!

I’m just trying to summarize the comments came from singular experts.

- There is a very valid point I agree upon - I would suggest not changing everything unless there is a very compelling reason. Point to ponder that if there is a particular need to run AIX, you can just integrate an AIX servers or cluster into your data centre without changing everything else which will be less overhead and sensible decision. Though this is not a technical point however first thought in my mind if my management ask me to do so.

- Datacenter power usage, cooling and space point of view - If one particular is struggling with power, cooling and space issue @ datacenter then consider mixing T- and M-series SPARC systems. T-series are great as power savers, they work best with highly parallel applications. M-series work great on single-threaded apps. The T-series are very efficient on power. However they do not do well on single threaded applications. For those you need to stick with M series.

- Another point - cost factor point of view - Solaris has x86 (Intel/AMD architecture), which is an option we don't have with AIX. For raw processing power and large memory footprints, Solaris 10 on Nehalem Intel is very motivating. You don't get all the RAS features of SPARC hardware, but if you have a load balanced applications or edge layer you can move there it can be a great fit. Also Solaris Supports x64 CPUs for excellent price/performance.

- The very vital point - The recent SPARC T3 servers aka “Rainbow Falls” that were announced last week at Oracle OpenWorld, that POWER7 isn't as desirable of a platform. Considering that a SPARC T3-4 can perform as well and in many benchmarks better than a 4 socket POWER7 box, but at considerably lower TCO, I don't see the point in the IBM/AIX/POWER route.

The SPARC T3 processor has the following specifications:

- 16 Cores x 8 Threads = 128 Threads Running at 1.65Ghz
- 2 x Execution Units per Core with 4 x Threads Each
- 1 x Floating Unit and 1 x Crypto Unit per Core
- 6MB of Shared L2 Cache
- 2 x DDR3 Memory Controllers
- 6 x Coherency Links for Glue-Less SMP up to 4 Sockets
- 2 x 10GbE NIU Controllers
- 2 x PCI-E 2.0 Controllers -> 8GB/s Bi-Directional I/O Each

Not done yet! Here is more details of various flavors of T3.

- Licensing factor - IBM will charge you for licenses left and right for each feature, especially on the virtualization front (LPARs, MPARs, and WPARs,), not to mention all the external components (HMCs, etc.). As where you can use Oracle VM Server for SPARC (LDoms) for free with the server and only pay for a RTU and support for S10 or S11 once for the whole machine (you can have hundreds or thousands of guests for no additional charge)! And don't forget that Solaris Containers are free and available on x86 and SPARC. Plus they can be used in LDoms(T-Series) and Dynamic System Domains (M-Series) for free! Oracle core licensing factor on SPARC T3 is 0.25

- AIX has no equivalent technology to ZFS.

- Solaris can scale to >64 CPUs to solve extremely large problems.

- Point to be noted - AIX on POWER is a good platform, I don't want to badmouth AIX. But here are the two biggest issues with the platform:

1. Costs
2. Finding enough AIX folks to support you! [It doesn’t mean that out there lot of resources available for supporting Solaris, I mean "GOOD RESOURCES"…]

- Punch line - SPARC is not dead, actually is more alive than ever and Solaris is the most advanced OS in the market: why change ?

- Virtualization point of view – AIX got LiveMotion! But not Sun BUT---BUT there are ways. Lets discuss it in details.

You can migrate your LDoms in a cold or warm migration, look out for LDoms 1.3 Admin guide in the chapter "Migrating Logical Domains", in particular the section on "Migrating an Active Domain" on page 129. (Oracle VM Server for SPARC 2.0). With the warm migration, the LDom is suspended and moved within seconds->minutes depending on the size and network bandwidth. As for rebooting service domains, if you're on something like a T5240 or T5440, you can use external boot storage (SAN/JBOD/iSCSI) to keep the second service domain up and have enough PCI-E slots for redundancy. By splitting your redundancy (network and storage) between them, your LDom guests will continue running with IPMP and MPXIO. So no down-time. FYI, even if you reboot the Primary domain without a secondary service domain, the guests will continue to run and wait for the Primary to return, so all is not lost. The only way you'd lose all of your LDoms is if you lose power or reset the SC. You can also use Solaris Cluster to automate the migration or fail-over of LDom guests.

The bad thing about the IBM Power VM setup is the following:

1. High-Over Head! The VIO overhead can easily consume over 40%-50% of your resources! As where on a T-series with LDoms, it's one core for the Primary domain and less than 10% overhead for the networking and storage I/O virtualization. The CPU threads are partitioned at the hypervisor and CPU level. RAM is virtualized at the hypervisor and MMU level. And on the M-series with Dynamic Domains you have 0% overhead because the CPU, Memory, and I/O are electrically partitioned on the centerplane. Even Solaris Containers have extremely low overhead, usually less than 5%. So I can get more out of my Solaris servers than you can in the IBM PowerVM world.

2. If there is a fault on a CPU or memory module, you can take out multiple LPAR/WPARs. As where on M-series with Dynamic Domains, faults like that would only affect the Domain the CMU module was on. You can even do a mirrored Memory configuration on the M-Series to be fully fault tolerant. Not to mention that the Dynamic Domains are electrically isolated, something you don't get anywhere else.

3. Costs, you have to get licenses to enable PowerVM features and they add up quickly. As where LDoms and Dynamic Domains are free with the hardware.

NOTE: In AIX WPAR (equivalent to Sun Container) can be migrated on the fly BUT on Solaris Containers cannot be migrated on the fly BUT not to worry it is something Sun-Oracle is working on and we'll probably see down the road. As of now for live migrating something around, I would use an LDom for now!

Not biased about AIX POWER architecture as this is a very vital decision to make and at the same time very difficult situation to conclude on but taking help of above points what I can think of – I’ll stick to SPARC architecture!

Your comments on this will be very much appreciated.

Solaris Flash Archives

noreply@blogger.com (Nilesh Joshi) — Thu, 14 Oct 2010 10:18:00 +0000

Flash images or flar image is very useful in situations where you need cloning/imaging or crashed server recovery. The flarcreate command creates a flash archive. A flash archive can be created on a system that is running a UFS root file system or a ZFS root file system. A flash archive of a ZFS root pool contains the entire pool hierarchy except for the swap and dump volumes and any excluded datasets. The swap and dump volumes are created when the flash archive is installed.

NOTE: By default, the flarcreate command ignores items that are located in "swap" partitions.

Let's see how we can work with flar image creation.

Create the archive:

For UFS:

# flarcreate -n "Solaris 10 10/09 build" -S -c -x /var/tmp/ /var/tmp/S10-1009.ufs.archive.sun4u-`date +'%Y%m%d%H%M'`

For ZFS:

# flarcreate -n "Solaris 10 10/09 build" -S -c /var/tmp/S10-1009.zfs.archive.sun4u-`date +'%Y%m%d%H%M'`

Where -

The "-n Solaris 10 10/09 build" implants a name into the FLAR image. The name should be something unique and meaningful to better identify it as the FLAR image for the system.

The "-x /var/tmp/" option causes the /var/tmp/ directory and its contents to be excluded from the FLAR image since it will not be needed in the FLAR image.

-S option causes to skip the disk space check and do not write archive size data to the archive. Without -S, flarcreate builds a compressed archive in memory before writing the archive to disk, to determine the size of the archive. The result of the use of -S is a significant decrease in the time it takes to create an archive.

-c Tells flar to compress the archive as it's writing it.

E.g. -

# time flarcreate -n "Solaris 10 10/09 build" -S -c /var/tmp/S10-1009.zfs.archive.sun4u-`date '+%m-%d-%y'`

Full Flash
Checking integrity...
Integrity OK.
Running precreation scripts...
Precreation scripts done.
Creating the archive...
Archive creation complete.
Running postcreation scripts...
Postcreation scripts done.

Running pre-exit scripts...
Pre-exit scripts done.

real 19m58.57s
user 13m42.99s
sys 1m55.48s

# ls -l /var/tmp/S10-1009.zfs.archive.sun4u*
-rw-r--r-- 1 root root 5339709933 Oct 14 04:54 /var/tmp/S10-1009.zfs.archive.sun4u-10-14-10

# flar info /var/tmp/S10-1009.zfs.archive.sun4u-10-14-10
archive_id=2f27a01690ce4fcaf398e638fcdcb66e
files_archived_method=cpio
creation_date=20101014093417
creation_master=XXXXXX
content_name=Solaris 10 10/09 build
creation_node=XXXXXXXX
creation_hardware_class=sun4u
creation_platform=SUNW,Sun-Fire-V240
creation_processor=sparc
creation_release=5.10
creation_os_name=SunOS
creation_os_version=Generic_142900-09
rootpool=rpool
bootfs=rpool/ROOT/s10s_u8wos_08a_Pre-patch
snapname=zflash.101014.04.10
files_compressed_method=compress
content_architectures=sun4c,sun4d,sun4m,sun4u,sun4s,sun4us
type=FULL

Also we can have a small shell script to create flar image -

#!/bin/sh
echo
echo Enter image name, i.e. Solaris build e.g. S10-1009.ufs.archive.sun4v
read ANS
echo "Image Name: ${ANS}" > /etc/image_catalog
echo "Image Created on: `date`" >> /etc/image_catalog
echo "Image Created by: `/usr/ucb/whoami` on `hostname`" >> /etc/image_catalog

#
# Clean up wtmpx so that new machine won't have last logins
#
cat /dev/null > /var/adm/wtmpx
#
# Now create flar, excluding -x /var/tmp/
#
flarcreate -n ${ANS} -c -a `/usr/ucb/whoami` -x /var/tmp/ /var/tmp/${ANS}_`date +'%Y%m%d%H%M'`

Oracle Solaris 10 9/10 Release = Oracle Solaris 10 Update 9 released on Sept 8, 2010

noreply@blogger.com (Nilesh Joshi) — Thu, 09 Sep 2010 04:34:00 +0000

Yesterday, September 8, 2010 Oracle officially announced Oracle Solaris 10 9/10, Oracle Solaris Cluster 3.3 and Oracle Solaris Studio12.2. For now we will be concentrating on Oracle Solaris 10 9/10.

Here is a history of Solaris 10 Update releases –

Solaris 10 3/05 = Solaris 10 (FCS == First Customer Ship)
Solaris 10 01/06 = Solaris 10 Update 1
Solaris 10 06/06 = Solaris 10 Update 2
Solaris 10 11/06 = Solaris 10 Update 3
Solaris 10 8/07 Release = Solaris 10 Update 4
Solaris 10 5/08 Release = Solaris 10 Update 5
Solaris 10 10/08 Release = Solaris 10 Update 6
Solaris 10 5/09 Release = Solaris 10 Update 7
Solaris 10 10/09 Release = Solaris 10 Update 8 >>>> Currently we are here.
Oracle Solaris 10 9/10 Release = Oracle Solaris 10 Update 9 >>>> New Release

So what Solaris 10 U9 includes, let’s take quick tour –

There are some drastic changes under this update –

The most awaited, Oracle Solaris Containers now provide enhanced “P2V” (Physical to Virtual) capabilities to allow customers to seamlessly move from existing Oracle Solaris 10 physical systems to virtual containers quickly and easily. At our project we developed a custom method to perform p2v from Solaris8/9 to Solaris 10 Container and to be honest we was really looking forward for this feature.
Host ID Emulation - Migration of a physical Solaris 10 machine into a Zone with support for the HostID will allow more network management platforms to be virtualized while still retaining their licensing features.
Oracle 11g Release 2 Support
Networking and database optimizations for Oracle Real Application Clusters (Oracle RAC).
Increased reliability for virtualized Solaris instances when deployed using Oracle VM for SPARC, also known as Logical Domains.

ZFS Changes –

ZFS device replacement enhancements - namely autoexpand
some changes to the zpool list command
Holding ZFS snapshots
Triple parity RAID-Z (raidz3)
The logbias property
Log device removal - at last
ZFS storage pool recovery
New ZFS system process – In this release, each storage pool has an associated process, zpool-poolname
Splitting a mirrored ZFS storage pool (zpool split)

For more information - http://dlc.sun.com/pdf/821-1840/821-1840.pdf

So what you’re waiting for let grab one for testing - http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html

HTH,

Tx,

--Nilesh--

The most annoying error - cannot mount '/home': directory is not empty

noreply@blogger.com (Nilesh Joshi) — Tue, 31 Aug 2010 14:15:00 +0000

Was working on migrating few SVM metadevice based filesystems to ZFS filesystem on Solaris 10 U8 and for the reason I have created ZFS filesystems with temporary mountpoint under /mnt like:

# zfs create –o mountpoint=/mnt/home –o quota=2g rpool/home

After creating this filesystem I’ll sync the data from metadevice based file system to ZFS based filesystem residing under /mnt like:

# rsync -axP --delete /home/ /mnt/home/

Once the data migrated I’ll un-mount the metadevice for /home and rearrange the mount point for ZFS filesystem back to /home using:

# zfs set mountpoint=/home rpool/home

While performing above command execution I get the error –

# zfs mount -a
cannot mount '/home': directory is not empty

This error may occur due to various reasons like one but not the last is auto mount daemon and so on which I’m not aware of… :)

I got the rid of this pesky issue performing “Overlay mount”

# zfs mount -O rpool/home

BTW then what is overlay mount? – Overlay mount allow the file system to be mounted over an existing mount point, making the underlying file system inaccessible. If a mount is attempted on a pre-existing mount point without setting this flag, the mount will fail, producing the error "device busy" or in ZFS terms “directory is not empty”.

This solution seems to be working just fine for me but I cannot guaranty that same will work for you as I said before there are various factors that leads to such issue.

HTH

alloc: /: file system full

noreply@blogger.com (Nilesh Joshi) — Mon, 09 Aug 2010 11:41:00 +0000

Today I came across with a strange issue of file system full with error -

Aug 9 00:17:41 server1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full

When I looked at the top disk space consumers I found nothing useful.

# df -h | sort -rnk 5

/dev/md/dsk/d0 3.0G 2.9G 0K 100% /
/dev/md/dsk/d3 2.0G 1.5G 404M 80% /var
/dev/md/dsk/d30 469M 330M 93M 79% /opt
/dev/md/dsk/d6 992M 717M 215M 77% /home
/dev/md/dsk/d33 752M 494M 198M 72% /usr/local/install
[...]

After doing a du on whole filesystem I can see it is showing 2.5G only and df showing 2.9G consumed space.

# du -shd /
2.5G

I realized few days back I came across same issue on ZFS filesystem hosting oracle DB and below understanding helped me there.

Normally, If filesystem is full, then look around in the directories that will be hidden by mounted filesystems in higher init states or see if any files that are eating up the disk space, in case if you get nothing useful from this exercise then one of the things to check is the open files and consider what has been cleaned up. Sometimes, if an open file is emptied or unlinked from the directory tree the disk space is not de-allocated until the owning process has been terminated or restarted. The result is an unexplainable loss of disk space. If this is the cause a reboot would clear it up. If you can't reboot consider any process that would be logging to that partition as a suspect and check all of your logs for any entries that imply rapid errors in a process.

In my case, reboot was not possible as the server caused file system full

# lsof +aL1 /

lsof: WARNING: access /.lsof_server1: No such file or directory
lsof: WARNING: created device cache file: /.lsof_server1
lsof: WARNING: can't write to /.lsof_server1: No space left on device
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
scp 16472 root 4r VREG 85,0 238616064 0 119696 / (/dev/md/dsk/d0)
scp 22154 root 4r VREG 85,0 238213120 0 119677 / (/dev/md/dsk/d0)

Where:

``+L1'' will select open files that have been unlinked. A specification of the form ``+aL1 '' will select unlinked open files on the specified file system.

I got the processes ID's via lsof, after verifying the processes I killed them and suddenly it has released ~450MB space.

# df -kh | sort -rnk 5
/dev/md/dsk/d0 3.0G 2.5G 418M 86% /
/dev/md/dsk/d3 2.0G 1.5G 406M 80% /var
/dev/md/dsk/d30 469M 331M 91M 79% /opt
/dev/md/dsk/d6 992M 717M 215M 77% /home
/dev/md/dsk/d33 752M 494M 198M 72% /usr/local/install

Hope this helps.

UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath

noreply@blogger.com (Nilesh Joshi) — Wed, 07 Jul 2010 20:33:00 +0000

UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath.

Few weeks ago I made a mistake, it was height of stupidity as an experienced and sensible system administrator. Few weeks back while migrating my UFS based system to ZFS I landed up in a situation where it made me upset for a long time. Below is the brief history on problem.

I'm currently working on UFS to ZFS migration on systems having zones which are already migrated to ZFS zonepath/zone root.

In last upgrade/migration, the mistake I made is - I deleted the snapshots/clones made by LiveUpgrade for zone's zonepath/zoneroot.

The story behind is - When we create a ABE which will boot to ZFS LiveUpgrade program creates a snapshot and that snapshot creates a clone of /[zonename]/zonepath as shown below -

[...]
Creating snapshot for zone1/zonepath on mailto:zone1/zonepath@s10s_u8wos_08a].
Creating clone for [zone1/zonepath@s10s_u8wos_08a] on zone1/zonepath-s10s_u8wos_08a.
Creating snapshot for zone2/zonepath on mailto:zone2/zonepath@s10s_u8wos_08a].
Creating clone for [zone2/zonepath@s10s_u8wos_08a] on zone2/zonepath-s10s_u8wos_08a.
[...]
After creating ABE for ZFS we can luactivate it and boot into ZFS.

# luactivate s10s_u8wos_08a

Till this point everything Worked fine.

# init 6
# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u8 yes no no yes -
s10s_u8wos_08a yes yes yes no -

From above output you can see I'm in ZFS boot environment.

Now the real stupidity stunt starts onwards below portion. Unfortunately and lack of study before task execution, I deleted snapshots/clones using -

# zfs list -H -t snapshot | cut -f 1
zone1/zonepath-Sol10u8@s10s_u8wos_08a
zone2/zonepath-Sol10u8@s10s_u8wos_08a
zone3/zonepath-Sol10u8@s10s_u8wos_08a

# for snapshot in `zfs list -H -t snapshot
cut -f 1`; do zfs destroy -R -f $snapshot;done

From this point onward I lost my zonepaths and could not able to boot my zones. WOW... Horrible Sunday starts here!

After few weeks after I found the answer as described below -

Solution -

Thanks to Alex and Julien for their suggestions and help.

I removed/destroyed the snapshots/clones meant for new ZFS BE

After this point onward tried mounting the UFS BE but due to unclean zonepaths it failed and left nebulous mouintpoints. To clear those mount points you rebooted the server.
This point onward, after analyzing the ICF* files decided to re-created the zonepath using old UFs BE. Command zpool history helped me here.

# zfs clone zone1/zonepath-d100@s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:rbe=s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:zn=zone1 zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=off zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=on zone1/zonepath-s10s_u8wos_08a
# zfs rename zone1/zonepath-s10s_u8wos_08a zone1/zonepath
# zfs set mountpoint =/zone1/zoenpath zone1/zonepath

likewise one by one recreated/recovered zonepaths snapshots/clones for all the zones which were at fault.

This way, zonepath backed again and fall back to UFS BE went soft however the main thing here, was the patching needs to be re-done here as the zonepath were re-created using unpatched UFS BE.

After backout, deleted the ZFS BE & this point onwards I created new ZFS BE and patched it once again.

I hope no one is stupid like me to make such a mistakes but if made then here you go!

ZFS Revisited

noreply@blogger.com (Nilesh Joshi) — Tue, 06 Jul 2010 15:12:00 +0000

Understanding ZFS & ZFS ARC/L2ARC

It's being almost a year that I’m working on ZFS Filesystem Administration and just finished migrating all of our Solaris servers from UFS root to ZFS root, so now OS data and Application data resides on ZFS. The intention to write this document is to have some handy notes about ZFS & it’s cache mechanism which always plays a vital role in terms of system performance!

This is just a revisit the features/techniques of this great filesystem known as ZFS.

Let’s start from assumptions. In last one year I’ve experienced that there were some assumption that I and others were carrying but before we jump into anything assumption queue should get clear so let’s do that.

Some assumptions about ZFS –

• Tuning ZFS is Evil (TRUE)
• ZFS doesn’t require tuning (FALSE)
• ZFS is a memory hog (TRUE)
• ZFS is slow (FLASE)
• ZFS won’t allow corruption (FLASE)

Alright then, now we clear on few of assumptions and now known to the facts! Let’s take a look at few important features of ZFS –

• ZFS is the world’s first 128-bit file system and as such has a huge capacity.
• Capacity wise Single filesystems 512TB+ (theoretical 264 devices * 264 bytes)
• Trillions of files in a single file system (theoretical 248 files per fileset)
• ZFS is Transaction based, copy-on-write filesystem, so no "fsck" is required
• High Internal data redundancy (New RAID level called RAIDz)
• End-to-end checksum of all data/metadata with Strong algorithm (SHA-256) but CPU consuming sometimes! So can be turn off on data and ON on metadata. Checksums are used to validate blocks.
• Online integrity verification and reconstruction
• Snapshots, filesets, compression, encryption facilities and much more!

What is the ZFS filesystem capacity?

264 — Number of snapshots of any file system
248 — Number of entries in any individual directory
16 EB (264 bytes) — Maximum size of a file system
16 EB — Maximum size of a single file
16 EB — Maximum size of any attribute
256 ZB (278 bytes) — Maximum size of any zpool
256 — Number of attributes of a file (actually constrained to 248 for the number of files in a ZFS file system)
264 — Number of devices in any zpool
264 — Number of zpools in a system
264 — Number of file systems in a zpool

How about data redundancy in ZFS?

ZFS supports following RAID configurations

> Stripes (RAID-0)
> Mirroring (RAID-1)
> RAID-Z (Similar to RAID-5)
> RAID-Z2 (Double parity, similar to RAID-6)

Hey, what is it…WOW no FSCK needed? HOW?

Yes, it’s true. ZFS does not need fsck to correct filesystem errors/corruptions as due to COW behavior. ZFS maintains its records as a tree of blocks. Every block is accessible via a single block called the “uber-block”. When you change an existing block, instead of getting overwritten a copy of the data is made and then modified before being written to disk this is Copy on Write (COW). This ensures ZFS never overwrites live data. This guarantees the integrity of the file system as a system crash still leaves the on disk data in a completely consistent state. There is no need for fsck. Ever.

ZFS is self healing data feature?

Yes, provided that –

• If a Bad Block is found ZFS can repair it so long as it has another copy
• RAID-1 - ZFS can “heal” bad data blocks using the mirrored copy
• RAID-Z/Z2 - ZFS can “heal” bad data blocks using parity

Also note that self healing is avail for ZFS metadata but not to actual application data.

ZFS Best Practices –

• Tune recordsize only on fixed records DB files
• Mirror for performance
• 64-bit kernel (allows greater ZFS caches)
• configure swap (don't be scared by low memory)
• Don't slice up devices (confuses I/O scheduler)
• Isolate DB log writer if that is critical (use few devices)
• Separate Root pool (system's identify) and data pools (system's function)
• Keep pool below 80% full (helps COW)
• Usage of snapshots/clones for backup/DR purpose

Okay then, let’s talk about ARC first.

ARC stands for “Adaptive/Adjustable Replacement Cache” - ARC is a very fast cache located in the server’s memory (RAM). The amount of ARC available in a server is usually all of the memory except for 1GB or 3/4th of main memory whichever is greater so simple calculation shows 3GB on your 4GB machine.

Originally, ARC concepts has been first described and invented by two IBM researchers Megiddo and Modha in November 2003 but ZFS ARC is significantly modified version of original ARC design.

The major differences of ZFS ARC is –

• The ZFS ARC is variable in size and can react to the available memory & maybe that’s the reason it is called as “Adjustable Replacement Memory”. It can grow in size when memory is available or it can shrink in size when memory is needed for other processes/jobs.
• The designed proposed by Megiddo and Modha assumes the block size should be same but ZFS ARC works with multiple block sizes.
• Under ZFS ARC you can lock pages in the cache to excuse them from the removal. This prevents the cache to remove pages, that are currently in use. This feature is not in original ARC design.

ZFS ARC stores ZFS data and metadata information from all active storage pools in physical memory (RAM) by default as much as possible, except 1 GB of RAM or 3/4th of main memory BUT I would say this is just a thumb rule or theoretical rule and depending on the environment tuning needs to be done for better system performance. Consider limiting the maximum ARC memory footprint in the following situations:

• When a known amount of memory is always required by an application. Databases often fall into this category.
• On platforms that support dynamic reconfiguration of memory boards, to prevent ZFS from growing the kernel fence in onto all boards.
• A system that requires large memory pages might also benefit from limiting the ZFS cache, which tends to breakdown large pages into base pages.
• Finally, if the system is running another non-ZFS file system, in addition to ZFS, it is advisable to leave some free memory to host that other file system's caches.

The ARC grows and consumes memory on the theory that no need exists to return data to the system while there is still plenty of free memory. When the ARC has grown and near to go exceed memory pressure exists, for example, when a new application starts up, then the ARC releases its hold on memory. Lastly and very important to note that ZFS is not designed to steal memory from applications, ZFS is very innocent!

By default, UFS uses page caching managed virtual memory system however ZFS does not use the page caching except few type of files! ZFS use ARC. There can be only one ARC per system however caching policy can be change per dataset basis.

As I said before, to make sure application/databases has enough dedicated memory available you need to perform tuning or capping on ARC.

# prtconf | grep Mem
Memory size: 98304 Megabytes

# grep zfs /etc/system
set zfs:zfs_arc_min = 1073741824
set zfs:zfs_arc_max = 17179869184

So here, I’ve 96GB total physical memory and ARC has capped at 16G. So around ~17% has been capped for arc_max and 1G for arc_min

The following command gives the current memory size in bytes that is used by ZFS cache:

# kstat zfs::arcstats:size
module: zfs instance: 0
name: arcstats class: misc
size 15325351536

The thumb rules for tuning ARC will be –

• know your future application memory requirements say if it required 20% memory in overall then it makes sense to cap the ARC such that it does not consume more than the remaining 80% of memory.
• Understand and feel your existing applications well. If application known or indeed uses large memory pages then putting cap on ARC will be a bottleneck to that application as limiting the ARC prevents ZFS from breaking up the pages and fragmenting the memory. Limiting the ARC preserves the availability of large pages.
• It’s certainly not easy to tune ARC and need quite a bit deep understanding of applications and their needs though it always helps to have mentioned scripts handy and added to your tools/script depot - arc_summary.pl (By Ben Rockwood) & arcstat.pl (By Neelakanth Nadgir)

Now let’s have a small discussion about L2ARC

L2ARC is a new layer between Disk and the cache (ARC) in main memory for ZFS. It uses dedicated storage devices to hold cached data. The main role of this cache is to boost the performance of random read workloads. The intended L2ARC devices include 10K/15K RPM disks like short-stroked disks, solid state disks (SSD), and other media with substantially faster read latency than disk.

L2ARC, or Level 2 caching in ARC makes it possible to use a disk in addition to RAM for caching reads. This improves read performance, which can otherwise be slow because of the fragmentation caused by the copy-on-write model used by ZFS.

In actual L2ARC cache is supported in Solaris 10 update 6 however as far as my knowledge L2ARC is well supported and functional in Solaris 10 Update 8 and onwards.

The L2ARC attempts to cache data from the ARC before it is evicted so the L2ARC populates its cache by periodically reading data from the tail of the ARC. The data in the tail of the ARC is the data that hasn’t been used for while. I’m still reading through and understanding on how exactly L2ARC works and how it relates to ARC and all that stuffs.

Am I really need L2ARC?

It all depends on the situation, if you have a system where a lot of data is being read frequently and you want better performance, then yes. But if not it still won’t hurt performance, only improve it.

There is lot more things to talk about & to share however you can also find this information from where I found like go through Sun officials books, Solarisinternal.com and some great blogs!

It was a great ZFS year and I’m glad that Sun (now Oracle) folks worked and working hard to make it as a Legend in filesystem arena!

Swap is in use by Live Upgrade!!!

noreply@blogger.com (Nilesh Joshi) — Fri, 02 Jul 2010 20:12:00 +0000

Just now working on few ZFS post migration cleanup task like creating new ZFS filesystems which are currently on UFS SVM devices, rsync them, ludelete the UFS BE and finally add empty disk to the rpool. While working on this I encountered an error while deleting the swap space holding one of the soft partition.

I was in a process of clearing the metadevices and one of the soft partition holding 32G of swap space and system was not allowing me to delete it. The error I was getting was –

# swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d34 85,34 16 67108848 67108848

# swap -d /dev/md/dsk/d34
/dev/md/dsk/d34: Not enough space

This was obvious because system was using this swap as a active swap.

# top -c
last pid: 5639; load avg: 3.78, 3.40, 3.41; up 25+15:39:04
1327 processes: 1258 sleeping, 64 zombie, 5 on cpu
CPU states: 91.0% idle, 2.5% user, 6.5% kernel, 0.0% iowait, 0.0% swap
Memory: 96G phys mem, 14G free mem, 32G swap, 32G free swap

Then I was wondering, where is my ZFS swap volume gone? Why system is not using this volume? So I tried making it active using swap -a but I failed to do it & system gave me below message.

# swap -a /dev/zvol/dsk/rpool/swap
/dev/zvol/dsk/rpool/swap is in use for live upgrade -. Please see ludelete(1M).

Okay, so this was first time ever that happened to me. Well after scratching my head on wall for a while I got the answer.

The swap -a attempt might fail if the swap area is already listed in /etc/vfstab or is in use by Live Upgrade. In this case, use the swapadd feature instead.

# /sbin/swapadd

# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,1 16 2097136 2097136

# top -clast pid: 13969; load avg: 2.34, 2.66, 2.84; up 25+16:07:28
1321 processes: 1255 sleeping, 64 zombie, 2 on cpu
CPU states: % idle, % user, % kernel, % iowait, % swap
Memory: 96G phys mem, 12G free mem, 64G swap, 64G free swap

All right then, sometimes it’s good to scratch your head on wall for a while… :) isn’t it?

HTH

Oracle 11.1.0.7.0 bug

noreply@blogger.com (Nilesh Joshi) — Fri, 25 Jun 2010 17:47:00 +0000

This week, two days back I hit a bug in Oracle 11.1.0.7.0

If you get following error while database start up or if your database crashed due to SAN unavailability, unexpected server crash, power outage etc and if you're starting database and get following error.

ORA-27167: Attempt to determine if Oracle binary image is stored on remote server failed
ORA-27300: OS system dependent operation:parse_df failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: parse failed

$ sqlplus '/as sysdba'
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Jun 24 07:27:29 2010
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-00445: background process "MMNL" did not start after 120 seconds
This error indicate that you have hit a bug 6813883 on oracle 11.1.0.7 version [Metalink article id: 784754.1]

After installing patch 6813883 using Opatch utility database startup went very well and now databases are running like a Piece of cake!

Hope this helps someone...

Modify number of CPUs from a pool/pset while it is running

noreply@blogger.com (Nilesh Joshi) — Thu, 03 Jun 2010 14:12:00 +0000

Sometimes you need to modify number of CPU's for a perticular pool. In this case you only need to transfer #CPUs from the pset pset_default to the pset of your pool.

In my case I've two psets available with me -

# poolcfg -dc info

[... Long Lines of Output ...]

pset oracle_pset
                       int pset.sys_id 1
                       boolean pset.default false
                      uint pset.min 4
                       uint pset.max 4
                       string pset.units population
                       uint pset.load 5563
                       uint pset.size 4
                       string pset.comment

pset pset_default
                        int pset.sys_id -1
                        boolean pset.default true
                        uint pset.min 1
                        uint pset.max 65536
                        string pset.units population
                        uint pset.load 574
                        uint pset.size 12
                        string pset.comment

[... Long Lines of Output ...]

So here you can see that I've 2 processor sets named oracle_pset having 4 CPUs & pset_default having 12 CPUs. Now the situation is, application/DB demands more CPU capacity than currently I have. So in this case you can modify number of CPUs from running pool/pset.

Here is a method to do so -

Save you current configuration

# pooladm -s

Modify the CPU's using "-d" - -d operates directly on the kernel state.

# poolcfg -dc 'modify pset oracle_pset ( uint pset.min = 6 ; uint pset.max = 6)'

Transfer 2 CPUs from pset_default to oracle_pset

# poolcfg -dc 'transfer 2 from pset pset_default to oracle_pset' OR if you want a specific processor then - # poolcfg -dc 'transfer to pset oracpu_pset ( cpu 5)'

Update the configuration in /etc/pooladm.conf file.

# pooladm -c

# poolcfg -dc info

[... Long Lines of Output ...]

pset oracpu_pset

                          int pset.sys_id 1
                          boolean pset.default false
                          uint pset.min 6
                          uint pset.max 6
                          string pset.units population
                          uint pset.load 2009
                          uint pset.size 6
                         string pset.comment

pset pset_default

                          int pset.sys_id -1
                          boolean pset.default true
                          uint pset.min 1
                          uint pset.max 65536
                          string pset.units population
                          uint pset.load 498
                          uint pset.size 10
                          string pset.comment

[... Long Lines of Output ...]

Runtime example –

$vmstat 5
kthr memory page disk faults cpu

r b w swap free re mf pi po fr de sr sd sd sd sd in sy cs us sy id0 0 0 36536976 6040672 244 1343 0 0 0 0 0 0 0 0 0 1105 17598 1840 16 14 70
0 0 0 36529296 6040216 96 341 0 0 0 0 0 2 2 0 0 1034 12900 1268 13 8 79
0 0 0 36531896 6039864 367 1412 0 0 0 0 0 0 0 0 0 1145 33910 1625 16 15 69

processors removed: 4, 5
3 0 0 39565360 8833624 597 2343 0 0 0 0 0 7 7 0 0 1379 33522 5416 24 14 62
0 0 0 36533816 6040080 516 1999 0 0 0 0 0 0 1 0 0 823 15979 1893 21 17 62
7 0 0 36500632 6013496 788 4785 0 0 0 0 0 3 3 0 0 1029 69589 1848 37 24 39
16 0 0 36370632 5920912 2258 14106 0 0 0 0 0 3 6 0 0 1819 165593 3616 62 38 0
15 0 0 36466944 5978160 911 4854 0 0 0 0 0 9 9 0 0 2095 298114 3968 67 32 0
8 0 0 36579424 6058944 511 3664 0 0 0 0 0 0 0 0 0 1690 238234 4102 57 30 13
processors added: 4, 5
[see the perfromance boost up, look at processor idle]
3 0 0 39565240 8833520 626 2467 0 0 0 0 0 7 7 0 0 9437 34315 7514 16 11 73
1 0 0 36645344 6103968 163 630 0 0 0 0 0 0 0 0 0 1243 14485 1989 15 9 76
1 0 0 36612504 6079352 205 1985 0 0 0 0 0 0 0 0 0 1393 86756 1705 28 15 57
0 0 0 36648152 6102368 199 1015 0 0 0 0 0 0 0 0 0 1247 17551 1909 18 10 72
0 0 0 36659264 6114368 46 672 0 0 0 0 0 6 6 0 0 1154 11101 1670 13 8 80
1 0 0 36668248 6118136 297 1124 0 0 0 0 0 2 1 0 0 1493 37868 3793 18 16 66
1 0 0 36674216 6121112 48 430 0 0 0 0 0 0 0 0 0 1268 12972 2075 14 9 77
2 0 0 36677736 6124456 317 1402 0 0 0 0 0 0 0 0 0 1414 18159 2336 17 10 73
1 0 0 36673120 6120576 365 1361 0 0 0 0 0 0 0 0 0 1360 17413 2494 14 10 75
1 0 0 36678104 6124032 217 713 0 0 0 0 0 1 1 0 0 1107 12223 1781 14 12 74

Hope this helps!

Migrating zones between sun4u and sun4v systems

noreply@blogger.com (Nilesh Joshi) — Fri, 14 May 2010 10:07:00 +0000

Migrating zones between sun4u and sun4v systems
============================================

I've recently started off with my new project which is a mixture of UFS --> ZFS migration, Zones/Containers migration from one host to another & patching. The real challenge is I've to do it with minimum downtime & have to be "real fast & accurate" at execution.

As I've already started with this project so before jump into project I done some detail study on few subjects related to this project so thought of publishing my findings on my blog.

The first question came in my mind is - If the zone is residing on V890 i.e sun4u arch & I've to move it to SPARC-Enterprise-T5120 i.e. sun4v arch then is it supported & if yes then how it can be done? Below paragraph talks about it.

A recent (not that recent) RFE to make attach work across sun4u and sun4v - 6576592 RFE: zoneadm detach/attach should work between sun4u and sun4v architecture.
Starting with the Solaris 10 10/08 release, zoneadm attach with the -u option also enables migration between machine classes, such as from sun4u to sun4v.

Note for Solaris 10 10/08: If the new host has later versions of the zone-dependent packages and their associated patches, using zoneadm attach with the -u option updates those packages within the zone to match the new host. The update on attach software looks at the zone that is being migrated and determines which packages must be updated to match the new host. Only those packages are updated. The rest of the packages, and their associated patches, can vary from zone to zone.

This option also enables automatic migration between machine classes,such as from sun4u to sun4v.

Okay now when I'm all clear with this doubt so let's move ahead with looking at how to do the migration & what all steps are involved to do so.

Overview -

Migrating a zone from one system to another involves the following steps:

1. Detaching the Zone. This leaves the zone on the originating system in the "configured" state. Behind the scenes, the system will generate a "manifest" of the information needed to validate that the zone can be successfully attached to a new host machine.

2. Data Migration or if your zones are on SAN then re-zone those LUNs. At this stage we may choose to move the data or rezone the storage LUNs which represents the zone to a new host system.

3. Zone Configuration. at this stage we have to create the zone configuration on the new host using zonecfg command.

4. Attaching & if required update (-u) the zone. This will validate that the host is capable of supporting the zone before the attach can succeed. The zone is left in the "installed" state.

5. Boot the zone & have a fun as here you completes the zone migration.

Let's talk more about point #2.

How to Move the zonepath to a new Host?

There are several ways to create an archive of the zonepath. You can use the cpio or pax commands/utilities to archive your zonepath.

There are also several ways to transfer the archive to the new host. The mechanism used to transfer the zonepath from the source host to the destination depends on the local configuration. One can go for SCP, FTP or if it's on ZFS then zfs send/receive etc.

In some cases, such as a SAN, the zonepath data might not actually move. The SAN might simply be reconfigured so the zonepath is visible on the new host. This is what we do in our environment & that's the reason I prefer to have zoneroot on SAN.

Try before you do

Starting from Solaris 10 5/08, You can perform a trial run before the zone is moved to the new machine by using the “no execute” option,-n.

Here is the details how it actually works -

The zoneadm detach subcommand is used with the -n option to generate a manifest on a running zone without actually detaching the zone. The state of the zone on the originating system is not changed. The zone manifest is sent to stdout.

Then we can direct this output to a file or pipe it to a remote command to be immediately validated on the target host. The zoneadm attach subcommand is used with the -n option to read this manifest and verify that the target machine has the correct configuration to host the zone without actually doing an attach.

The zone on the target system does not have to be configured on the new host before doing a trial-run attach.

E.g.
gz1_source:/
# uname -m
sun4u

gz1_dest:/
# uname -m
sun4v

gz1_source:/
# zoneadm list -icv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
7 zone1 running /zone1/zonepath native shared

gz1_source:/
# zoneadm -z zone1 detach -n | ssh gz1_dest zoneadm attach -n -

The validation is output to the source host screen, which is stdout.

I hope this information will help me to get started with project work.

Amazon EC2 pricing models

noreply@blogger.com (Nilesh Joshi) — Wed, 12 May 2010 16:29:00 +0000

Cloud computing is changing the way IT resources are utilized. Now a days cloud computing is one of the Innovated technology platform & emerging trend for IT industry & certainly not limited to it.

Cloud computing is a simple idea, but it can have a huge impact on business.

There are many vendors for enabling/providing cloud computing solutions like Amazon, VMWare, Rackspace and many more.

Today we will focus on Amazon EC2 (Amazon Elastic Compute Cloud).

Amazon is a leading public cloud computing provider, AWS (Amazon Web Services) falls in the infrastructure as a service (IaaS) space, providing on demand service using virtual server instances with unique IP addresses and blocks of storage.

It's a news for me and may be for many of us that AWS opening a data center in Singapore to make its entry in the Asia Pacific region has created a lot of interest about Amazon Elastic Compute Cloud (Amazon EC2) in India.

So just to take a quick tour of AWS offerings - AWS provides various components/services like Amazon EC2, Amazon Simple Storage Service (Amazon S3), Amazon SimpleDB, Amazon Relational Database Service (Amazon RDS).

As AWS is stepping in to Asia Pacific region I'm very excited to know what will be the pricing model for their service offerings. So let's take a look at their pricing model.

Amazon EC2 pricing models follows a "pay-as-you-go" model (as in any other cloud computing model) however the flexibility is at it's par in case of Amazon EC2 pricing model.

When it comes to Amazon EC2 pricing models, the instances are grouped into three families:
1. Standard,
2. High-Memory
3. High-CPU.

Amazon EC2 pricing for each of these instances is as follows:

1. Standard Instances -- This model have memory to CPU ratios suitable for most general purpose applications. This Amazon EC2 pricing model ranges from $0.12 per hour to $ 0.96 per hour, for services running on Windows infrastructure. For infrastructure running on Linux and UNIX, Amazon EC2 pricing starts with $0.095 per hour to $0.76 per hour.

2. High-Memory instances -- This Amazon EC2 pricing model offers larger memory sizes for high throughput applications, including database and memory caching applications. This is priced at $ 0.62 per hour to $2.88 per hour for Windows based infrastructure. For Linux/UNIX based instances in this pricing model of Amazon EC2, prices range from $0.57 per hour to $2.68 per hour.

3. High-CPU instances: In this Amazon EC2 pricing model, proportionally more CPU resources are consumed than memory (RAM), and as a a result it targets compute-intensive applications. This Amazon EC2 suite has been priced at $0.29 per hour to $1.16 per hour for Windows based infrastructure. For Linux or UNIX based infrastructure, this Amazon EC2 pricing model charges from $0.19 per hour to $0.76 per hour.

When choosing Amazon EC2 pricing types, organizations should consider characteristics of their application with regards to resource utilization. Accordingly, they should select the optimal instance family and size.

Now we have understood the pricing model & now next question comes in heart is how about support model so here we go -

here are two AWS Premium Support offerings—Gold and Silver. The following is the breakup of the services.

Gold Support includes:

-Business day support (6 a.m. to 6 p.m.)
-24x7x365 coverage.
-One-on-one support via web-based ticketing system.
-One-on-one support via telephone.
-1 hour maximum response time for "Urgent issues".
-Guaranteed response time for non-urgent issues.
-Client-side diagnostic tools.
-Named support contacts.

Silver Support includes:

-Business day support (6 a.m. to 6 p.m.).
-One-on-one support via web-based ticketing system.
-Guaranteed response time for non-urgent issues.
-Client-side diagnostic tools.
-Named support contacts.

I hope it's worth knowing all this information as a IT Infrastructure professional.

Also as we all know pictures talks louder & who agree to it then here is a very good video for understanding cloud computing in simple language.

http://www.youtube.com/watch?v=XdBd14rjcs0

Hope this article helps...

Creating CPU resource pool & Processor set in Solaris 10

noreply@blogger.com (Nilesh Joshi) — Mon, 10 May 2010 08:52:00 +0000

Resource pools are used for partitioning server resources. It's a workload management framework.

I mostly work on Solaris servers with Oracle database hosted on it. Limited to subject, for my requirement I create CPU Pools to Support Oracle Licensing. All Global Zone servers running oracle containers should have at least one oracle cpu pool. The containers should be "bound" to this pool.

Oracle Licensing offered in two forms as far as know - CPU based & User based licenses.

How to create pools?

# pooladm -e       <<< The pools facility is not active by default when Solaris starts. pooladm -e explicitly activates the pools facility.

OPTION: If you wish to enbale resource pool fuction via SMF then just execute -

# /usr/sbin/svcadm enable svc:/system/pools:default

# pooladm -s      <<< Save the current configuration to /etc/pooladm.conf

# pooladm             <<< Shows current running pools configuration

Create Processor Set -

# poolcfg -c 'create pset oracle_pset (uint pset.min=4; uint pset.max=4)'     <<< pset.min & pset.max is consider the hardware thread.

Create resource pool -

# poolcfg -c 'create pool oracle_pool'     <<< Create pool

Associate resource pool & Processor Set -

# poolcfg -c 'associate pool oracle_pool (pset oracle_pset)'       <<< Associate pool with pset

NOTE: The global zones scheduler should be set to use FSS.
Set the default scheduling class to FSS:

# dispadmin -d FSS

# poolcfg -c 'modify pool oracle_pool (string pool.scheduler="FSS")'     <<< Enable FSS on pool

# pooladm -c      <<< Activate the configuration (After executing this command you can see /etc/pooladm.conf has been modified with current configuration)

Now once you're done creating & associating the pset & resource pool next thing to do is configure the Non-Global Zone configuration.

# zonecfg -z zone1
zonecfg:zone1> set pool=oracle_pool
zonecfg:zone1> exit

Once the zone configuration has been altered then you can bind the resource pool to zone using -

# poolbind -p oracle_pool -i zoneid

This procedure demonstrate on how to create CPU/Processor sets, resource pools, binding them to containers like activities.

Hope this will help.

FATAL: system is not bootable, boot command is disabled

noreply@blogger.com (Nilesh Joshi) — Fri, 07 May 2010 16:35:00 +0000

In system administration job normally no news is a good news... Today I was working with Solaris MPxIO on V890 server model & due to some unknown MPxIO misconfiguration under /kernel/drv/fp.conf - the whole system got messed up & I left with option of rebuilding the whole system.

Here is the crash pattern after MPxIO misconfiguration -

Rebooting with command: boot

Boot device: /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w21000014c3dbf465,0:a File and args:
SunOS Release 5.10 Version Generic_142900-01 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
NOTICE: error reading device label

NOTICE:

***************************************************
* This device is not bootable!                    *
* It is either offlined or detached or faulted.   *
* Please try to boot from a different device.     *
***************************************************

NOTICE: spa_import_rootpool: error 19
Cannot mount root on /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w21000014c3dbf465,0:a
fstype zfs
panic[cpu7]/thread=180e000: vfs_mountroot: cannot mount root
000000000180b950 genunix:vfs_mountroot+358 (800, 200, 0, 1872800, 189b400, 18cbc00)

Thank god that the system was not rolled out to production yet.

When I realized the system is crashed & I may need to boot the system into single user mode for maintenance I logged onto the SC and got to my console, and type boot as one does.

{1} ok boot
FATAL: system is not bootable, boot command is disabled
Ohh.. no.. What the mess...

There are many errors which you never imagine or unseen throughout your tiny professional life & I hit this one today, for the first time in my 6 years of professional life.

Just in case you happen to hit this horrible error then here is the fix -

set auto-boot? to false, reset the box, and then set it to true and finally boot as shown below -

{1} ok setenv auto-boot? false
auto-boot? = false
{1} ok reset-all
SC Alert: Host System has Reset
Sun Fire V890, No Keyboard
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.22.34, 65536 MB memory installed, Serial #XXXXX.
Ethernet address X:XX:XX:XX:XX:XX, Host ID: XXXXX.
{1} ok setenv auto-boot? true

auto-boot? = true
{1} ok boot net - install nowin

.......... lots of output ........

This will rebuild your system now.

One advice - BE ALWAYS CAREFUL WHILE WORKING WITH MPxIO. YOUR BEST FRIEND MAY TURN INTO WORST ENEMY IF YOU HURT HIM...

Mounting an ISO image on Solaris Systems

noreply@blogger.com (Nilesh Joshi) — Thu, 22 Apr 2010 17:03:00 +0000

Now a days it's not necessary to create CD's for the downloaded ISO images & for cost cutting reasons IT staff does not allow it without manager approval. Instead of burning the image to a CD-ROM to access its contents, it is easy to mount the image directly into the file system with the help of lofiadm and mount commands -

lofiadm administers lofi, the loopback file driver. lofi allows a file to be associated with a block device. That file can then be accessed through the block device. This is useful when the file contains an image of some file system (such as a floppy or CD-ROM image), because the block device can then be used with the normal system utilities for mounting, checking or repairing filesystems.

Given an ISO image in /home/nilesh_rjoshi/sol-10-ccd-GA-SPARC-iso, a loopback file device (/dev/lofi/1) is created with the following command -

# lofiadm -a /home/nilesh_rjoshi/sol-10-ccd-GA-SPARC-iso /dev/lofi/1

The lofi device creates a block device version of the file:

# lofiadm
Block Device File
/dev/lofi/1 /home/nilesh_rjoshi/sol-10-ccd-GA-SPARC-iso

And then mount the block device on /mnt with the following command:

# mount -F hsfs -o ro /dev/lofi/1 /mnt

All the above commands can be combined into a single command as follows:

# mount -F hsfs -o ro `lofiadm -a /export/home/techno/sol-10-ccd-GA-x86-iso.iso` /mnt

# ls /mnt

components installer README volstart

Once done using the files from CD image, unmount the lofi block device

# umount /mnt

Finally remove the association of loopback file device as follows:

# lofiadm -d /dev/lofi/1

# lofiadm
Block Device File

HTS