Stuff...

Solaris 11, Oracle 11g & EM12C

2012-01-30T09:24:00.000-05:00

I've just started using Solaris 11, this is my first ever install of it.

Solaris 11

Downloaded it, mounted it as lofi, and then created an LDOM by booting off the ISO. I've not tried the Automated Installer (AI) yet.

I realize that there isn't much information here, but really, there wasn't much to do.

Add ISO to LDOM:

ldm add-vdsdev /sol11/sol-11-1111-text-sparc.iso sol11iso@primary-vds0
ldm add-vdisk id=1 sol11iso sol11iso@primary-vds0 orcem12c

Show Disks on LDOM Ok prompt:

{0} ok show-disks
a) /virtual-devices@100/channel-devices@200/disk@1
b) /virtual-devices@100/channel-devices@200/disk@0
c) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/disk@1 has been selected.

Boot from Disk 1 (as thats what the ID is for the ISO). Also choose slice "f"

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1:f

Follow the installer along and give appropriate credentials. A user will need to be created, as root login is disabled by default.

To allow root login:

1. Change "PermitRootLogin" to "yes" in the /etc/ssh/sshd_config file.
2. Comment out "#CONSOLE=/dev/console" in the /etc/default/login file to allow non-console root login.
3. Switch the role of the root user "rolemod -K type=normal root"

Oracle 11g (11.2.0.3 Install)

First things first, rebuild the Package index and then get the X packages installed:

pkg rebuild-index

Test to make sure you have all the required packages

pkginfo -i SUNWarc SUNWbtool SUNWhea SUNWlibm SUNWlibms SUNWpool SUNWpoolr SUNWsprot SUNWtoo SUNWuiu8 SUNWfont-xorg-core SUNWfont-xorg-iso8859-1 SUNWmfrun SUNWxorg-client-programs SUNWxorg-clientlibs SUNWxwfsw SUNWxwplt

Install packages that are needed.

pkg install compatibility/packages/SUNWxwplt SUNWmfrun SUNWarc SUNWhea SUNWlibm

pkg install x11/xclock

Add Swap if you dont have enough (You'll need about 16Gb depending upon the RAM you've allocated to the LDOM).

zfs create -V 14G swpool/swap14G
swap -a /dev/zvol/dsk/swpool/swap14G

Edit the /etc/vfstab to add the new swapfile.

/dev/zvol/dsk/swpool/swap14G - - swap - no -

I found that Timezone wasn't set. In Solaris 11, all the properties have moved to SMF.

root@orcem12c:~# svcs -a | grep environ
online 17:45:30 svc:/system/environment:init
root@orcem12c:~# svccfg -s system/environment:init
svc:/system/environment:init> listprop environment/TZ astring
svc:/system/environment:init> setprop environment/TZ=US/Eastern
svc:/system/environment:init> validate
svc:/system/environment:init> end

Install Oracle 11g. I choose to use the file storage instead of ASM for the Db disks.

Create new Pool for /opt/app/Oracle & /opt/app/oradata (to hold all the Software & Db files respectively)

zpool create -f -m /opt/app/Oracle swpool c2d1s0
zpool create -f -m /opt/app/oradata dbpool c2d2s0

Requirements for the Db:

1. 2G SGA.
2. Redo log files must atleast be 300Mb with 3 Redo logs.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo01.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo02.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo03.log') Size 300M;

SQL> alter system switch logfile;

If adding logfiles, you can drop the old ones. If the Db complains that it cant drop a logfile (if you need to), run "alter system checkpoint". You cannot drop a logfile in "Current" status.

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC
---------- ---------- ---------- ---------- ---------- ---------- ---
STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------------- ------------- --------- ------------ ---------
4 1 13 314572800 512 1 NO
INACTIVE 1089635 29-JAN-12 1089642 29-JAN-12

5 1 14 314572800 512 1 NO
CURRENT 1089642 29-JAN-12 2.8147E+14

6 1 0 314572800 512 1 YES
UNUSED 0 0

I had to make the following changes:

SQL> alter system set session_cached_cursors=300 scope=spfile;

System altered.

SQL> alter system set job_queue_processes=20;

System altered.

SQL> alter system set shared_pool_size=600M scope=spfile;

System altered.

EM12C (Cloud control) Install

Once I made all the above Db changes, I was able to get through the installer using the "Simple" option.

The only error message after all this, was that there wasn't an EM tablespace with an autoextend datafile. Since I didnt create a tablespace, so I proceeded.

Once the install finishes, I logged in with https://:port/em. Login with the user sysman.

Ops Center DHCP Hell....

2011-10-18T17:57:00.000-04:00

If you've set up Ops Center and are wondering why the machines wont jump with DHCP, make sure you choose "ISC" instead of Solaris as your DHCP server.

Also a good link to follow is this note OpsCenter and ISC DHCP server blog!

At Oracle World...Past 2 days...

2011-10-05T10:28:00.002-04:00

There were quite a few announcements at Oracle world! None of them interesting!

Enterprise Manager 12c has been released.

Ops Center is free

Solaris 11 is almost there, we can download an early release and install it. The big shocker - there is no way to go from 10 to 11 without a backup, reinstall and restore!\

More to come as I have time to write up.

Release of Solaris (Sparc or x86?)

2011-09-30T11:12:00.000-04:00

I got a handy tip from an Oracle person the other day....to make sure what solaris release and version the solaris cd is or jumpstart location is, check the SUNWsolnm/reloc/etc/release file.

[root@host1]# cat SUNWsolnm/reloc/etc/release
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
[root@host1]#

TFTP Boot error Access Violation

2011-09-30T10:10:00.001-04:00

When accessing a tftp server clients receive "Error code 2: Access violation"
due to ownership & permission of the /tftpboot directory.

Steps to Follow
in.tftpd runs as user nobody. nobody needs rwx permissions to the /tftpboot
directory
chmod 766 /tftpboot
chown nobody:other /tftpboot

Product
Network Management/Conn

Oracle 11g RAC in LDOMs (Oracle VM for Sparc)

2011-09-16T17:21:00.000-04:00

Setup
1. 2 T3-1B blades on different chassis. Patch the firmware. Install LDOM 2.1 software.
2. SAN Connected storage via MPXIO (EMC SAN)
3. Solaris 10/9 installed with End User Option.

Create install User

useradd -u 0 -o -g 1 -c "Install user" -d / -s /bin/true install

Patch LDOM.

Download the following Patches from oracle
• 117837-05: C++ compiler optimizer patch
• 117846-08: C++ compiler Optimization patch
• 118682-01
• 127111-02 SunOS 5.10: libc patch (this should already exist if LDOM is patched)
• 137111-04 SunOS 5.10: kernel patch (this should already exist if LDOM is patched)

Profile Settings

• On a new system, set the following in /etc/profile
a. MANPATH=/usr/man:/usr/local/man:/usr/sfw/man:$MANPATH

• Create the oracle required users & groups

groupadd -g 102 dba
groupadd -g 103 oinstall
projadd -p 102 -c "Oracle Project" group.dba
useradd -u 101 -g dba -G oinstall -s /usr/bin/bash -c "Oracle User" -m –p 102 oracle

• Set the Oracle password

passwd -r files oracle

• Add the following entries to /etc/system for a 4G SGA or less. Shmmax should always be greater than SGA size.

set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967296
set shmsys:shminfo_shmmni=100
set rlim_fd_max=65636
set rlim_fd_cur=4096

Reboot the system after the above /etc/system have been made.

Alternate way

Run prctl to make changes to the existing install so that a reboot is not required. These should be run as root. These changes don’t persist after a reboot.
prctl -n project.max-shm-memory -v 4gb -r -i project user.root
prctl -n project.max-sem-ids -v 256 -r -i project user.root

Project Settings

Make changes to the project so that changes persist for shm memory & sem ids (4G or less SGA). These should be run as root.

projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,256,deny)" group.dba

/etc/project looks like this after changes:

[root@orc11gtest]# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:102:Oracle Project:oracle::project.max-sem-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Verify the changes applied to the oracle user as below.

[oracle@orc11gtest]$ prctl -n project.max-shm-memory -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 4.00GB - deny -
[oracle@orc11gtest]$ prctl -n project.max-sem-ids -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-sem-ids
privileged 256 - deny

Patch Check:

patchadd -p | grep 125555-03
patchadd -p | grep 140796-01
patchadd -p | grep 140899-01
patchadd -p | grep 141016-01
patchadd -p | grep 139555-08
patchadd -p | grep 141414-02 (Replaced by 141444-09)
patchadd -p | grep 141736-05
patchadd -p | grep 120753-06
patchadd -p | grep 139574-03
patchadd -p | grep 141414-10
patchadd -p | grep 141736-05

I'm using Ops center to create LDOMs to make my life a little easier. If you're doing this, the Vnets sometimes will be different between the 2 blades. Go into the "/guests//servercfg.xml" and fix the

< idx >3< /idx >

number to match on both nodes.

NTP Configuration
Make sure to set the /etc/inet/ntp.conf with slewalways and disable p11.

server < serverip >
driftfile /etc/ntp.drift
slewalways yes
disable pll

Grid Software Install

ON each Control Domain -- add the disks for OVD, OCR & Data to the LDOMs. Make sure to add them in the same order, and set the same ID. A good way to make sure everything is the same on both nodes, is to also volume label the disks on the control / IO domain before assigning it to the LDOM.

Have xwin running and kick off the installer. Before you kick off the installer, its a good idea to run cluster verify tool.

./runcluvfy.sh stage -pre crsinst -n orczone1,orczone2 -verbose

Install any packages that are missing, fix the swap or any such thing, the MTUs etc, and you're ready to roll.

To add Swap Space when using ZFS

To create swap space on a ZFS volume (created 2 2g volumes):

zfs create -V 2gb rpool/swap2g
zfs create -V 2gb orcinstall/swap2g

Add it to swap

swap -a /dev/zvol/dsk/rpool/swap2g
swap -a /dev/zvol/dsk/orcinstall/swap2g

Add it to /etc/vfstab

/dev/zvol/dsk/rpool/swap2g - - swap - no -
/dev/zvol/dsk/orcinstall/swap2g - - swap - no -

Install / Troubleshooting Notes

1. A group called "DATA" was created by a previous install, and even though I zapped the disks, the group didnt go away. To drop the diskgroup (which wouldn't mount or come online) I had to do this:
drop diskgroup data force including contents;
2. If you're having trouble with ulimits and such, this link: Max user processes is pretty helpful.

Oracle 11g RAC in a zone - Partial instructions.

2011-09-14T09:10:00.001-04:00

This is a post thats a WIP. I will be adding to this as I keep finding new things, and hopefully a full step by step doc by the end of this.

I had to stop this because of the requirement of EXCLUSIVE IPs for the non-global zone. You can only have one Exclusive interface per vlan tag per server.

That is, if you have VLAN tag 3 for your public interface, and you want to create another RAC cluster on the same blade, you'll have to get another vlan from your network team. This seems unworkable, as all our Dbs are in the same VLAN.

I've just begun setting up 2 T3-Blades to use zones for Oracle RAC cluster.

Setup:
2 T3-1B Sparc blades with 64Gb of RAM.
Oracle 11gR2 downloaded
Oracle 11gR2 Grid Download
OS installed with "Full Install with OEM"
OS patched with the latest patch cluster.
Use EMC SAN for disk

MPXIO
Enable MPXIO if you're not using any vendor provided software to connect to the SAN
Using MPXIO to connect to the EMC CX-500
change "mpxio-disable" to "no" in ./kernel/drv/fp.conf

Disks
Using ZFS for the global zone with internal disks.

All Oracle zones will be created using mirrored disks from the SAN.

I intend on creating a zone, configuring the basic setup (DNS, oracle user, oracle download copied over) and take

[root@ch0bl7qadb1w]# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
orcpool 248G 11.0G 237G 4% ONLINE -
rpool 278G 7.99G 270G 2% ONLINE -
[root@ch0bl7qadb1w]# zpool status -v orcpool
pool: orcpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
orcpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t60060160733014002C5FAEED7BD9E011d0 ONLINE 0 0 0
c0t60060160733014002D5FAEED7BD9E011d0 ONLINE 0 0 0

errors: No known data errors

Creating Zones
Create the zone (orczone1), boot it and configure the DNS and such.

Zones have an annoying habit of setting auto-home. I normally run a script to disable auto-home. To allow the creation of the /home/oracle directory below, it's required to disable autohome.

Create the oracle project with group membership of dba.

projadd -p 102 -c "Oracle Project" group.dba

Create the dba (id: 102) oinstall (id: 103) groups

Create the Oracle, Oragrid users, the assign the oracle user belong to the oracle project.

useradd -u 101 -G dba -g oinstall -s /usr/bin/bash -m -p 102 oracle
useradd -u 103 -G dba -g oinstall -s /usr/bin/bash -m oragrid

Verify that the users are created appropriately

[root@orczone1]# id -a oracle
uid=101(oracle) gid=103(oinstall) groups=102(dba)
[root@orczone1]# id -a oragrid
uid=103(oragrid) gid=103(oinstall) groups=102(dba)

According to OFA guidelines, Oracle going to be installed in
/opt/app/oracle

Oracle install also recommends that only ORACLE_BASE is set. I'm also going to be installing Grid software and Db software as the same user.

Bash Profile for the oracle User

PS1="[\u@\h]$ "
ORACLE_BASE=/opt/app/oracle
LD_LIBRARY_PATH=/usr/lib:/lib:$LD_LIBRARY_PATH
export SQLPATH=/home/oracle

export ORACLE_BASE ORACLE_SID LD_LIBRARY_PATH

export PATH=$CRS_HOME/bin:$ORACLE_HOME/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/local/bin:/bin/sbin:$PATH

umask 022
ulimit -n 65536

export EDITOR=vi
export editor=vi
export PS1

NTP
My /etc/inet/ntp.conf file is this (in the Global Zone):

server 10.27.1.254
driftfile /etc/ntp.drift
slewalways yes
disable p11

Grid Software install

1. Create /opt/app/11.2.0/grid on both nodes (orczone1 & orczone2).
2. Create /opt/app/oracle on both nodes
3. I created the "app" directory and let oracle have rwx permissions on it.
4. Add the SCAN Ips as well as the virtual IPs to DNS

Orczone1.cctdev.com
Orczone1-vip.cctdev.com

Orczone2.cctdev.com
Orczone2-vip.cctdev.com

Orczone-cluster.cctdev.com -- this was the SCAN IP.

5. Set the following in /etc/system for the global zone, to avoid annoyances in the non-global zone. I use these blades only for Oracle, so it works for me!

set rlim_fd_max=65636
set rlim_fd_cur=4096

5. During install of GRID software, oracle will try to plumb IPs and if you're not using Exclusive IPs in the non-global zone, it will fail. This is where I stopped and switched to LDOMs.

Issue Log:

One of the things that are needed are a Public & Private Interface.

When plumbing them, make sure that the private interface stays private and does not get added to your "default gateway". If that happens, traffic has a tendency to get "lost" and when you try to reboot zones or even the server, you may lose connectivity with other nodes. The way I did this, was to not add a defrouter in the zonecfg for the private interface.

When creating a Zone for the Private interface, do not set the default router for that NIC unless you're using VLAN tagging and such.

The netstat -rn should look something similar to this

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.25.20.5 UG 1 27
default 10.25.20.5 UG 1 4 e1000g2
10.0.0.0 10.26.6.202 U 1 16 igb0
10.25.20.0 10.25.23.137 U 1 3 e1000g2
192.168.204.0 192.168.204.7 U 1 0 e1000g0
224.0.0.0 10.25.23.137 U 1 0 e1000g2
127.0.0.1 127.0.0.1 UH 5 92 lo0

cacaoadm not working

2011-08-24T16:01:00.002-04:00

I was trying to install Oracle Enterprise Ops Center and I kept getting a weird error which I couldn't find any information online about.

[root@host1:/]# /usr/sbin/cacaoadm
Cannot perform firstime initialisation and configuration.

Even reinstalling cacao didnt help (the packages were SUNWcacaodtrace & SUNWcacaort ).

Turns out the cacao.properties file was completely hosed. Here's where the file is located /etc/cacao/instances/default/private/cacao.properties

I ended up copying it from a different install, and it's working!!

OEM 10g not starting ...Fix!

2011-07-02T13:07:00.003-04:00

If you have Oracle Enterprise Manager that very annoyingly wont restart, though it's been perfectly working before you shut it down, and opmnctl complains that it cant start the HTTP server, you're probably running into the file size limit of the Apache logs.

[oragrid@admin30w]$ tail -20 /opt/oragrid/oms10g/Apache/Apache/logs/error_log
[Sat Jul 2 01:43:20 2011] [error] (79)Value too large for defined data type: could not open transfer log file /opt/oragrid/oms10g/Apache/Apache/logs/access_log.

If you see that message, it implies that the access log file is too large, typically over 2Gb. OEM by default doesn't rotate logs, so you need to either set it up, or rotate the logs manually.

oragrid@admin30w]$ ls -l /opt/oragrid/oms10g/Apache/Apache/logs/access_log
-rw-r---- 1 oragrid oinstall 2147483881 Apr 30 21:58 /opt/oragrid/oms10g/Apache/Apache/logs/access_log

To fix it, I just did a

cat /dev/null > /opt/oragrid/oms10g/Apache/Apache/logs/access_log

You can also set up log rotation this way.
Note 339819.1 from Metalink will give you detailed steps on the multiple ways to setup log rotation. However, this is supposed to be fixed in 11g.

1. Stop the OMS:

cd /opmn/bin
opmnctl stopall
2. Take a backup of the files :
/sysman/config/httpd_em.conf
/sysman/config/httpd_em.conf.template

NOTE: You must make the changes to both httpd_em.conf AND httpd_em.conf.template because any 'emctl secure' operation results in the http_em.conf file being regenerated using http_em.conf.template and any changes that had been made to httpd_em.conf earlier will be lost.
3. Edit the /sysman/config/httpd_em.conf and modify:

ErrorLog /Apache/Apache/logs/error_log
TransferLog /Apache/Apache/logs/access_log

TO

ErrorLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/error_log 43200"
TransferLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/access_log 43200"

where is the full path to your OMS install directory.
4. Edit the /sysman/config/httpd_em.conf.template and modify:

ErrorLog &ORACLE_HOME&/Apache/Apache/logs/error_log
TransferLog "&ORACLE_HOME&/Apache/Apache/logs/access_log"

To

ErrorLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/error_log 43200"
TransferLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/access_log 43200"
5. Delete the existing access_log and error_log files in the /Apache/Apache/log directory.

6. Re-start the OMS
cd /opmn/bin
opmnctl startall

Registering a Sunone 7 node to the admin server

2011-06-02T15:55:00.002-04:00

We have a number of SunONE 7 (now renamed to iPlanet) servers and the documentation online is fast fading, thanks to Oracle.

To register a node after install:

webserver7/bin

/wadm --host= --port= --user=admin

It will prompt for a password.
wadm> register-node

That should register the host successfully.

If you have renamed your server, navigate to the admin-server/config directory and fix the server.xml with the new hostname.

Tablespace Utilization in Oracle

2011-02-24T17:14:00.003-05:00

Taken from: http://vsbabu.org/oracle/sect03.html
Usage

TABLESPACE USAGE NOTES:
Tablespace Name - Name of the tablespace
Bytes Used - Size of the file in bytes
Bytes Free - Size of free space in bytes
Largest - Largest free space in bytes

select a.TABLESPACE_NAME,
a.BYTES bytes_used,
b.BYTES bytes_free,
b.largest,
round(((a.BYTES-b.BYTES)/a.BYTES)*100,2) percent_used
from
(
select TABLESPACE_NAME,
sum(BYTES) BYTES
from dba_data_files
group by TABLESPACE_NAME
)
a,
(
select TABLESPACE_NAME,
sum(BYTES) BYTES ,
max(BYTES) largest
from dba_free_space
group by TABLESPACE_NAME
)
b
where a.TABLESPACE_NAME=b.TABLESPACE_NAME
order by ((a.BYTES-b.BYTES)/a.BYTES) desc

Files modified between times

2011-02-24T16:53:00.002-05:00

Sometimes, I get this task to find out files that changed between 2 dates. I never seem to remember it, so I figured it out (googling) and here it is. Hopefully someone else also can use it!

olddate="201001010001"
newdate="201012312359"
touch -t $olddate ./tmp/oldfile
touch -t $newdate ./tmp/newfile
find /path/to/directory -type f -newer ./tmp/oldfile ! -newer ./tmp/newfile

Good Bye 2010, Hello 2011

2011-01-17T00:06:00.007-05:00

Where did 2010 go? Everything almost seems to be a blur...I spent almost 350 days working last year I think (yes, that includes, nights, weekends etc).

I'm going to try to recount some happy times ...

January as always is a happy time, as I celebrate my anniversary with my lovely wife. This time, we decided to go to Las Vegas as our friends were staying there and celebrate. A couple of nights at their house, and one night at MGM Grand. It was a good trip! Here is a picture I could find, of giving my wife a surprise gift. I had my Brother In Law bring a pair of ear rings from India, and then send it to my friends place in Las Vegas without telling Lavanya :) (Yes, thats her acting surprised :D)

The rest of the year moved quickly, as we announced that we were going to have our second child!

Ofcourse, who can forget the monstrous winter of 2009 / 2010!

Time quickly flew by after that. We had my uncle, aunt & mom come from India in May / June (I think). They went on an US Tour and Alaska cruise and returned.
We went to a great airshow in May. This was our first time ever, and unfortunately we were a little late, but got to see some awesome acrobatics. Next time, I think we're going to go there for the whole day. Pranav had a blast!

In June, Lavanya had her surprise "Seemantham" or Baby shower at our neighbours house.

Then my In-laws arrived in August to help with our upcoming kid.

September rolled around, and my second son -- Vaibhav Chandra decided he wanted to be born before his due date, and he was born on Sept 3rd.

My parents flew down on the 10th, and so our house was full with family and joy!

On a side note, I flew so often to Toronto this year, it was crazy. I think I made 6 trips in about a span of 3 months, often staying there a week at a time.

After all the excitement was over, we all went to Minneapolis for a week to visit my brother in law (I flew in from Toronto and returned to Toronto !)

November came around, and my birthday passed silently this year, without a party or much fan fare! I think its owed to my little one :)

December came around, and time just flew. I was super busy with work, going to Toronto once that month as well I think.

2010 was definitely a year to remember because of the birth of Vaibhav, but not many things else, as it was a year of constant work and no play! I look forward to a great 2011!

Solaris Jumpstart - setting up the ISO

2010-12-27T16:03:00.005-05:00

Step 1: Download the Solaris OS from Oracle
wget --http-user= --http-password= --no-check-certificate --output-document=sol-10-u9-ga-sparc-dvd-iso.zip http://download.oracle.com/otn/solaris/10/sol-10-u9-ga-sparc-dvd-iso.zip &

a. wget may be installed in /usr/sfw/bin, so check that location as well, before you go hunting for wget.
b. This command will seem like it's hanging, but tail the output file, and you'll see the download status.

Step2:
Unzip the file.

Step 3:
Mount the ISO
a. Create a block device for the ISO:

[root@gpadev-sol-x86:/export/depot]# lofiadm -a /export/depot/sol-10-u9-ga-sparc-dvd.iso
/dev/lofi/1

b. Mount it.
[root@gpadev-sol-x86:/export/depot]# mount -F hsfs /dev/lofi/1 /mnt

Step 3

a. Create a directory where you want the media to reside at.
mkdir -p /export/depot/s10upd9

b. Use the Solaris Tools to copy over the Install media to the directory you want. Dont just to a cp!

[root@gpadev-sol-x86:/export/depot/s10upd9]# cd /mnt/Solaris_10/Tools/
[root@gpadev-sol-x86:/mnt/Solaris_10/Tools]# ls
add_install_client Boot dial Installers rm_install_client setup_install_server

c. Use the "setup_install_server" command
[root@gpadev-sol-x86:/mnt/Solaris_10/Tools]# ./setup_install_server -w /export/depot/s10upd9 /export/depot/s10upd9
Verifying target directory...
Calculating the required disk space for the Solaris_10 product
Calculating space required for the installation boot image
Copying the CD image to disk...
Copying Install Boot Image hierarchy...
Copying /boot netboot hierarchy...
Install Server setup complete

Step 4:
Add to the Jumpstart server the location.

Step 5:
Remove the lofi device.

lofiadm -d /dev/lofi/1

Verify:
lofiadm

Sun Hardware handy tools

2010-12-27T11:59:00.004-05:00

I came across ipmitool recently, and it is really handy when you need information about your machine (Serial Number, amount of memory etc)

Eg:
Whats the Serial Number of the Sun Blade?

ipmitool -H blade ILO IP-U root fru | grep "Product Serial"

Whats the Serial Number of the Sun Chassis
ipmitool -H chassis ilo ip -U root fru | grep "Product Serial"

How much memory do I have? (I had just installed some 4Gb dimms)
ipmitool -H blade ilo ip -U root fru | grep 4096MB | wc -l
Password:
24

i.e 24 dimms installed.

Oracle Grid control -- Removing EM Targets cleanly

2010-12-10T19:46:00.003-05:00

I was installing Grid Control agents on some of my middleware boxes, and ofcourse, the hostname got messed up when I did the silent install. My number of attempts to remove the entry from Grid control had no use. A quick google search revealed, I had to connect to the EMREP database on the Grid Control Server and remove it.

On the Grid Control Server

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
emrep

Get a list of all the targets...

SQL> select target_name from mgmt_targets where target_type='oracle_emd';

Remove the offending target:
SQL> exec mgmt_admin.cleanup_agent('offendingTarget:3872');

After a minute or more, voila, "PL/SQL procedure successfully completed."

Refresh my Grid Control screen, and it's all gone.

Got to give credit where due: I found it here

Solaris things...

2010-09-22T11:23:00.012-04:00

I've been working with Solaris a lot, and we recently started with ZFS as well. Perhaps I'm missing something, but "top" & prstat dont report the correct amount of Memory free (RAM) on the machine when using zfs and neither does vmstat.

After some google searching, I found this helpful blog and thought I'd report. Thanks to "milek.blogspot.com" for this:

#!/bin/bash
#########
# Copied from http://milek.blogspot.com/2006/09/how-much-memory-does-zfs-consume.html
# to get correct reading of Free Memory
#

FREE=$(($(pagesize)*$(kstat -p unix:0:system_pages:freemem|awk '{print $NF}')))
ZIO_BUF=$(kstat -p zfs:0:arcstats:size|awk '{print $NF}')
FREE_MEM=$(((FREE+ZIO_BUF)/1024))
echo "Free Memory: $FREE_MEM MB"
FREE_MEM=$(((FREE+ZIO_BUF)/1024/1024))
echo "Free Memory: $FREE_MEM GB"

Next one was backups. We use Solaris 8 Branded Zones heavily to migrate off from our old Solaris environment. I started using ufsdump for backups, since this was not ZFS.

Well, trying to restore using ufsrestore, caused a ton of problems.

The simple

zoneadm -z s8zone install -a dumpfile

would give me weird errors such as " Sanity Check: Missing etc at /mnt"

Decided to use a ufsrestore, and that didnt help since zoneadm didn't know how to get it up.

Long story short, I did the following

1. Do a ufsrestore on to the disk
2. Used Zonecfg to create the configuration
3. Manually edited the /etc/zones/index file to modify the entry to match the backup version (with new name).

The index file would look like this:

global:installed:/
hostname:installed:/export/zones/app30qa:some id
hostname:installed:/export/zones/s8qa2flar:some id

This made zoneadm think that it was "installed" and I was finally able to boot up.

11 years....

2010-09-09T18:54:00.002-04:00

Whoa! I just realized, that this past August, it's been a full 11 years since I landed in the good ol' US of A in 1999. The twists and turns life has taken is interesting to say nonetheless. However, there is a small part of me that has regrets....

I didn't go back to India as I once thought I would by this time, but more than that, the fact that I'vent been able to be an entrepreneur in this country is hitting me the hardest. I keep thinking of going back, but life keeps taking turns. The worst part however, is the American immigration system that allows illegals to do what they please, but the ones like me who are following the law, be at the end of the line to get a Green Card. Bah!

The positives are definitely there too....better quality of life. A nice townhome, a lovely wife, two wonderful sons now (the second one just born) are all positives....

Now, if I can just get through the sleepless nights and refocus on being back on the entrepreneur path, it would just be so awesome.

Sucky Airlines

2010-05-13T15:48:00.003-04:00

A friend of mine recently commented on how awesome Southwest is, and how sucky the rest of the airlines are on his blog post here

Today, the WSJ had an article regarding Frequent Flyer miles and how easy it is to redeem them. Results are not surprising. Here's the link: "The Road to Redemption"

US Air & Delta were the suckiest of the lot. Southwest came up on top.

I hate almost all US based carriers, but United, Delta & US Air take top spots. For international travel, travelling by a US based carrier is like throwing away money to get tortured (atleast the few trips I've been on)

What have I been doing?

2010-03-11T21:26:00.002-05:00

Well...the past few months have been grueling at work (and still continues to be). My friends keep telling me to take a break, and its high time I do, but unfortunately...I'm addicted to it (for now)...

So...what have I done / learnt this past few months (starting November) with work?

Well...designed, architected (with a colleague) and currently leading the build of a Highly available Geo Clustered Oracle RAC cluster with Weblogic running on Sun Blades.

Whats cool about it....well ..everything is :-)

1. Implementing virtualization with Sun LDOMs & Zones.

2. Implemented a Sun Cluster to provide highly available Network File Storage (I keep thinking this would have been incredibly useful at my previous job, since we used to only NFS mount our shared storage area) instead of spending tons of money on redundant NAS devices (filers as some people like to call them in the Sun world)

3. Created an infrastructure so that any new Solaris or Linux machines can be quickly imaged with Jumpstart.

4. Implemented Sun directory server for LDAP with replication.

5. Java memory troubleshooting....

and many more...but those are some of the highlights.

On the personal front though....

We celebrated our 6 year anniversary...

and ofcourse we announced that we're expecting another child due in September!

Sun Cluster 3.2 with LDOM 1.3

2010-02-24T10:03:00.006-05:00

I had to get a Sun Cluster 3.2 to create a HA-NFS on LDOM 1.3. LDOM 1.3 and Sun Cluster 3.2 now fully support Sun Clusters on Guest Domains not just the I/O Domain. This makes it incredibly useful.

I faced plenty of issues, and hoping that this post will serve as a guide to those who need to do it.

Setup:

1. 2 Physical T-6340 blades.
2. Control Domain that also serves as our I/O Domain.
3. 1 Guest Domain on each Server for Sun Cluster.
3. EMC Powerpath installed on the Control Domain

Storage:

Managed by EMC Powerpath

The devices will be seen as : emcpowerXa

1 LUN for each Sun Cluster domain

1 Shared LUN between the 2 physical blades

I want to note STRONGLY that you should first try using the powerdevices for the LDOM & Sun Cluster, and only if that doesnt work use the cXtXdX devices. Each configuration may be different and you may have better luck or a patch that I did not have at the time of my trial.

powermt display dev= will give you the underlying cXtXdX number (in the I/O Domain).

Present all devices in MPGroups to the LDOM (even for boot disk since we'll be using SVM with Sun Cluster) as cXtXdX devices.

3 disks to be presented in this example

1. Boot Disk (mpgroup: bdskgroup)

2. Quorum Disk (mpgroup: quroumgroup)

3. Shared Disk for storage (mpgroup: cludskgroup)

(Make sure to give an id to each of the vdisks that are presented to the LDOM so that they always are presented in the same order even on a reboot)

For each Sun Cluster LDOM present the VNETs from the VSWs in the order they're cabled for the private interconnect. This is important.

I used cross over cables.

Jumpstart and boot up the LDOM.

If you've done everything correctly you should see 3 devices when you do "format".

Install the Sun Cluster software with the NFS packages. Patch it appropriately.

I used the below instructions to create the cluster, however, its far easier in 3.2. Just create the metadb by configuring the SVM (instructions in link below) and then run scinstall. Once thats done run clsetup and follow the instructions for the NFS Service.

http://napobo3.lk.net/sc/dual-node_HA_NFS_cluster_setup.html

Copied from the above link:
2 nodes (hostname: node1, node2) with installed Solaris 10 01/06, patches, Sun Cluster, NFS agent for Sun Cluster, VxFS
Both nodes are connected to the FC SAN storage, 8 storage LUNs are mapped to each node.

Configure SVM
On both nodes
Create the 25MB partition on the boot disk (s7)
Create the SVM database replica
metadb -afc 3 c0d0s7 (c0t0d0s7 on sparc)

On one node (node1)
- Create disk sets :
metaset -s nfs1 -a -h node1 node2
metaset -s nfs1 -t -f
metaset -s nfs1 -a /dev/did/rdsk/d2 /dev/did/rdsk/d3 /dev/did/rdsk/d4 /dev/did/rdsk/d5
metainit -s nfs1 d1 4 1 /dev/did/rdsk/d2s0 1 /dev/did/rdsk/d3s0 1 /dev/did/rdsk/d4s0 1 /dev/did/rdsk/d5s0
metastat -s nfs1 -p >> /etc/lvm/md.tab

metaset -s nfs2 -a -h node2 node1
metaset -s nfs2 -t -f
metaset -s nfs2 -a /dev/did/rdsk/d6 /dev/did/rdsk/d7 /dev/did/rdsk/d8 /dev/did/rdsk/d9
metainit -s nfs2 d1 4 1 /dev/did/rdsk/d6s0 1 /dev/did/rdsk/d7s0 1 /dev/did/rdsk/d8s0 1 /dev/did/rdsk/d9s0
metastat -s nfs2 -p >> /etc/lvm/md.tab
scp /etc/lvm/md.tab node2:/tmp/md.tab
ssh node2 'cat /tmp/md.tab >> /etc/lvm/md.tab'

- Create VxFS on shared devices -- I didnt do this.. I just did a newfs.
mkfs -F vxfs /dev/md/nfs1/rdsk/d1
mkfs -F vxfs /dev/md/nfs2/rdsk/d1

On both nodes
- Create the directories
mkdir -p /global/nfs1
mkdir -p /global/nfs2

http://docs.sun.com/app/docs/doc/819-0912/chapter1-1?l=en&a=view is the quick start guide, is some help as well (especially with the metadb setup), but they talk about LOCAL devices as HA-NFS, and that isn't want we're doing. :-)

iPad

2010-01-28T17:03:00.002-05:00

Yaay...the new Tablet. The name sucks, seriously iPAD? Sounds like a feminine hygiene product (thats what everyone else is saying too :P )

Anyways...I read through the specs and I really wanted to think it was cool, but...
1. Wheres the USB port?

Why the hell would I want to connect to my computer for everything? If you're making a revolutionary and magical device, wouldn't you want people to use it even without a bloddy computer to get my files? So, if I am in class and the other guy has his documents on a USB drive, how is he supposed to magically send it to my iPad?

2. No Flash Support? If I want to have a full media web browser, then I certainly want flash! Heck, you can't even log an Oracle support ticket without flash now! LAME.

3. Lame resolution. 1024 x 768 makes it sound like an age old monitor. Bah.

Whats cool is the price point. If it had a USB port, I'd probably pre order it and find a hack to get flash working...but now I'm really thinking if I need the device, even though it would have been an awesome replacement for my dieing laptop.

Travel in the US

2010-01-28T16:58:00.003-05:00

I recently had to travel to Toronto (after the Christmas day attempted attack) and boy do I hate the airlines even more. I also travelled to Las Vegas which was mildly better but not by much... Where do I start?

1. Charging for checking in bags? Totally blows. Why would people travel without their f'ing clothes?
2. Airport security -- WTF. Why is America so scared? I think it's more political than actual fact. Seriously....on the way back from Toronto I wasn't allowed to carry my laptop bag, had to check it in (it was a bloddy Swiss Backpack).
2.1 The security check at the gate (no not the first x-ray check, one more before all the gates that lead to ANY US bound flights). They were Canadians, so they atleast smiled. Took out every single thing from my pocket and double signed my boarding pass and then someone else checked the double signature....seriously???
3. I hate hate hate all the bloddy airlines for their lack of professional service. Just hate them. We really need a new airline.

I think the Republicans should shut the f**k up and not create a hysteria. Sure some idiot tried to set off a firecracker, but wtf, seriously. Just really dumb. Makes us appear more scared than brave.

Wow..2009 is over

2009-12-28T15:16:00.013-05:00

With 2009 coming to an end, I thought i'd write a little on what went on this year. Perhaps for me to reflect in the coming years....

January marked the 5th Year of me being married. Time does fly. The days munged by to April where my Son turned 4 years old. We got him a bike to start off...

and he got plenty more gifts at his birthday party...His nanamma (grandmother) was here as well which made it all the more fun.

I changed jobs in October and then left for my holiday trip to India.
We had an exciting trip to India this time. We stopped over in Dubai on the way there. We spent 3 days, seeing the sights and sounds of Dubai. The iconic Burj Al Dubai:

The gold souk, the indoor ski place, all exciting stuff. Pranav had a blast as well. The one thing that sucked in Dubai was it's beach near the Hotel Atlantis. The water was all murky and pretty disgusting. It was surprising that a 7 star hotel had such a crappy beach, but I guess thats what you get when you over build. However, the city itself was excellent.

One of the displays at Dubai Gold Souk:

One thing that did catch me offguard was, you had to pay for water at the hotel. Most hotels, when you sit down, bring you a bottle of water, and at the end they put it on your bill as well. There is no concept of "free" water.

I also tasted an unbelievably tasty Chocolate Praline Cake. A small piece was about 25 Dhirams (~$6.5), but was amazing!

Dubai is known for it's malls. But, they're really trying to outdo one another there. The Dubai Mall has one of the best musical fountains in the world:

Not to forget, we also went for Dune bashing, followed by a Dune buggy ride & a belly dance with arabic food (was mostly Indian. The only Arabic dish there was Khoobs (bread) and Hummus)

Then our India trip was fun as well.
My Dad turned 60, and as tradition would have it, we celebrated by calling our near and dear friends & family along with the traditional rituals.

Following that we went to Tirpuati, Vijayanagram (the small town my dad is from, and met many of our relatives and some of his childhood friends) and my dad's aunt who pretty much brought him up. Then we went to Vizag, a beautiful coastal city, with a beautiful sea shore.

Then I spent Diwali with my in-laws, back to Chennai with my grandparents and then it was time for me to get back to work. Ofcourse, I probably gained about 5 lbs with all the eating (more like stuffing :) ) the delicious food back home.

Once I came back from India though work got really busy. We're bringing about 60 servers online for a very critical application and managing it all is as awesome as it's tiring. Yes I am a geek and I enjoy working on Sun / Oracle Hardware / Software :).

Somewhere along the way I did turn 31. My wife threw an awesome surprise Birthday bash for me.

Sixth Sense technology....AMAZING

2009-12-26T02:19:00.002-05:00

Ok...I feel really bad that I haven't seen this video from TED for so long.

Pranav Mistry talk at TED

Please check it out. This is amazing.

"Why use a tablet computer, why not just a piece of paper"?
"Why get stuck to only paper, how about moving that to a computer for more work"?

All those and more answered....