Stuff...

Oracle EBS 12.2 Installer

2014-08-17T20:41:00.001-04:00

I have just been trying to install EBS Suite 12.2 with 11.2.0.3/11.2.0.4 Grid. It is the worst ever, most painful install ever.

The number of files to be downloaded, the config changes for the servers and just the complexity makes me wonder how many people who've posted that they've done this install to be true!

Here is what I'm trying to do.

2 Node RAC
1 Node Application Tier (I want 2, but starting with 1).

This post will be updated to something sane, once I get through it...though - I'm posting just incase, someone can be saved some of the pain.

System Config
Oracle Linux x86_64
Yum Repo:
Oracle Public and Oracle Addons

[public_ol6_latest]

name=Oracle Linux $releasever Latest ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1

[public_ol6_addons]

name=Oracle Linux $releasever Addons ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1

RPMs Used:
yum install oracle-rdbms-server-11gR2-preinstall
yum install oracle-ebs-server-R12-preinstall
.>
Other config:
Fix the hosts file to have only the following
127.0.0.1 localhost
FQDN DN

For the oracle hosts, when trying to install the Database with the EBS installer, cluvfy will most likely fail because of the network check, dns check or something else (11.2.0.4 cluvfy will successfully complete, the issue is only with 11.2.0.3)

NSLookup Fix

To temporarily fix the nslookup error:
# mv /usr/bin/nslookup /usr/bin/nslookup.orig
# echo '#!/bin/bash
/usr/bin/nslookup.orig $*
exit 0' > /usr/bin/nslookup
# chmod a+x nslookup

Network Check
Once I confirmed that everything was fine with the 11.2.0.4 cluvfy, modify the runcluvfy in the /TechInstallMedia/database/grid directory
The last line, change it as below

# Return the status of executed cluster verification command

exit 0 # new add

#exit $EXEC_STATUS

Oracle Database Appliance - My case for Solaris

2014-07-10T14:46:00.001-04:00

I just got to try out the Oracle database appliance (ODA). It's supposed to be quick and easy to setup and get it running. We've run into some intial hiccups, but this is a POC box and I'm not going to give Oracle grief here.

Here are my quick thoughts

Pros:
1. Extremely easy to setup (hardware perspective)
2. Very cost effective.
3. Comes with lots of Storage and now allows expansion into another Storage Shelf.

Cons:

1. Only Oracle Linux. No Solaris.
2. Only one ASM across all the databases.
3. Patching has to be done in a particular order. You can't simply patch Db, you need to patch hardware, OS and such, before you get to the Db patching (There is a hierarchy)
4. When you start patching, depending upon what you're patching, ALL Databases need to be shutdown. This is very hard to do, especially when you have multiple applications using Dbs hosted on the ODA.

So - where is Solaris and why you ask.

1. Solaris 10 / 11 allows you to run Oracle RAC clusters in Zones.
2. With Solaris, you can have Zones running your Oracle RAC, and multiple Oracle RAC instances can be spun up, especially with MULTIPLE ASMs running. This will immediately allow you to patch your infrastructure one node at a time.
3. The bonded connections with Solaris 11 makes it incredibly simple to work with. This link specifically: Network Virtualization High Availability made me think of ODA on Solaris a lot more.
4. Solaris 11 allows you to add disks to zones without a reboot (a big Solaris 10 limitation).
5. Zones allow you to CPU cage.
6. The Hardware is all based off the Sun systems.
7. You can use ZFS for all OS disks.
8. You can get away from the clunky OVM DOM0 world that the ODA appliance uses now.
9. You can effortlessly do virtualization and use the VM Templates.
10. CloudFS partition would be effortless too - just make a new ZPOOL and you're done!
11. All this can be tied into Oracle Enterprise Manager & Oracle Ops Center for the hardware.
12. All the patching can be made so much more effortless.

Chef - my journey with the Infrastructure tool - Part 1 of hopefully many more...

2014-04-08T15:03:00.000-04:00

Chef is a Infrastructure in Code service, which allows you to define your infrastructure in different files (recipes, Cookbooks, resources etc) and by using Chef - you can build all your servers the exact same way. The advantage I see with Chef is - it has inbuilt knowledge on how to work with a number of different OS's - so Solaris could have pkgadd, RHEL yum or CentOS apt-get and these would automatically be determined by Chef.

Chef is written in Ruby, and you need to know just some basics of Ruby to get started with Chef. However, even if you dont know it - you can keep following along - or you can learn what you need for Chef here.

I highly recommend going through the screencasts available at OpsCode - here.

So - here's where I am - I got my workstation (a Mac) setup for Chef. I decided if I was going to learn it, the best spot to try things out is the Amazon cloud - it'll cost me some money, but considering a lot of organizations are moving there, might as well get experience with that as well :)

Amazon does offer a Micro instance for free for 750 hrs - and yes thats what I'm trying to use.

So to get Started, I picked an UBUNTU image which is available on the Amazon Market place.

I played around with Chef - and got Chef talking between Amazon and my desktop + Enterprise Hosted Chef (free for less than 5 nodes).

Yes - I am getting to the meat now :).

My next target is to be able to Deploy an OS on the Amazon cloud using just Chef.

There's probably a ton of blogs out there with this - so I am adding to it. I'm hoping to detail any issues that I've run into as well as use this blog as my own documentation :)

One of the things to note is that : chef always installs in "/opt/chef" on *nix flavors. There are too many dependencies, and its not supported if its not there.

First : Install knife-ec2 plugin for Chef. This is assuming that has already been installed.




/opt/chef/embedded/bin/gem install knife-ec2

Next, get the Amazon EC2 key and access code - its available under your Amazon Account, "Security Credentials" & Access Keys.

You'll also need your Amazon key pair - you can create a new one, or use the existing one you have already. Its under "Network Security"& Key Pairs

this is what you'll need to put in the knife.rb file.




### The Key pair with .pem file name. (without the .pem)

knife[:aws_ssh_key_id] = 'KEY_ID' 



#### The Access key id and access key from Amazon account.
knife[:aws_access_key_id] = 'ACCESS_KEY_ID' 

knife[:aws_secret_access_key] = 'SECRET_ACCESS_KEY'

Make sure the .pem file has permissions of 400. Otherwise, the ssh-user wont allow the use of the .pem file.

Solaris 10 x86 - Grub not loading

2014-04-07T17:55:00.002-04:00

I have a Solaris 10x86 install which failed all attempts after a reboot with ZFS

After a lot of trial and error, I finally did the below from the Grub menu and it worked.

grub> find /sbin/init
grub> root (hd1,0,a) <<---this a="" command.="" find="" hd1="" is="" of="" output="" p="" the="">grub> kernel$ /platform/i86pc/multiboot -B $ZFS-BOOTFS
grub>module /platform/i86pc/boot_archive
grub> boot

An example of the menu.lst at /rpool/boot/grub/menu.lst

#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris 10 10/09 s10x_u8wos_08a X86
findroot (pool_rpool,0,a)
kernel$ /platform/i86pc/multiboot -B $ZFS-BOOTFS
module /platform/i86pc/boot_archive
#---------------------END BOOTADM--------------------
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris failsafe
findroot (pool_rpool,0,a)
kernel /boot/multiboot -s
module /boot/amd64/x86.miniroot-safe
#---------------------END BOOTADM--------------------

Chromecast

2013-07-31T23:02:00.000-04:00

This small device is just awesome! I got it, connected it to my AV Receiver, and setup took about 5 mins after that.

For $35 -
I got it before the Neflix subscription offer expired - so 3 months of Netflix
the Chromecast device
A HDMI extender
A charger for the Chrome cast with the USB cable.

This is a awesome device and I highly recommend it!

Downloading a sites certificate with openssl

2013-07-28T17:27:00.002-04:00

I keep running into an issue where I need to get a server certificate sometimes, and its not always accessible via the browser.

When I need it from the server - this handy comment on this site worked like a charm http://serverfault.com/questions/139728/how-to-download-ssl-certificate-from-a-website

echo -n | openssl s_client -connect HOST:PORTNUMBER | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$SERVERNAME.cert
That will save the certificate to /tmp/$SERVERNAME.cert.
You can use -showcerts if you want to download all the certificates in the chain. But if you just want to download the server certificate, there is no need to specify -showcertsecho -n gives a response to the server, so that the connection is released
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' removes information about the certificate chain. This is the preferred format to import the certificate into other keystores.

Thanks to the person who gave that comment, it has helped me immensley.

Weblogic - App having javax.net.ssl.SSLHandshakeException

2013-07-28T16:40:00.003-04:00

If you use Weblogic and have an application that connects to the outside world via HTTPS, you could be in for a treat.

If you have the DemoIdentityTrustStore configured or any keystore configured and the JDK keystore configured, it looks like Weblogic randomly decides which keystore for Trust to use.

We ran into an issue, where the application would randomly complain of invalid cert chain similar to this:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

After many many days of troubleshooting, wondering why even thoguh the java CACERTS had the correct certs, I decided to try accessing the site with the DemoTrust keystore instead. Whola...the error manifested itself again.

What is insanely annoying is that this is not repeatable in WLS. Every restart seems to have a different behavior.

To import a certificate you already have into the keystore - use the "keytool" utility.

cd $JAVA_HOME/jre/lib/security
keytool -import -alias VerisignC3PPCA -keystore cacerts -trustcacerts -file /root/getrootcert.cer

I found this great piece of code online (InstallCert.java) (below) which helped me troubleshoot.

If you want to use the Weblogic Identity store - below is some useful data. You will need to modify the below code to point to this JKS if you want to (change the "File file = new ... " block) . To add the certificate, you can do it using "keytool" as well if you have the certificates you care about.

This table was very neatly done at : http://itguykelly.wordpress.com/2010/05/20/default-weblogic-keystore-passwordpassphrase/ which I've borrowed.

Property	Value
Trust store location	%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password	DemoTrustKeyStorePassPhrase
Key store location	%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password	DemoIdentityKeyStorePassPhrase
Private key password	DemoIdentityPassPhrase

Property	Value
Trust store location	%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password	DemoTrustKeyStorePassPhrase
Key store location	%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password	DemoIdentityKeyStorePassPhrase
Private key password	DemoIdentityPassPhrase

About these ads

Copy and paste the below code into a file called "InstallCert.java"
javac InstallCert.java
java InstallCert.java

/*
* Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* - Neither the name of Sun Microsystems nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
* IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
* THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* Originally from:
* http://blogs.sun.com/andreas/resource/InstallCert.java
* Use:
* java InstallCert hostname
* Example:
*% java InstallCert ecc.fedora.redhat.com
*/

import javax.net.ssl.*;
import java.io.*;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

/**
* Class used to add the server's certificate to the KeyStore
* with your trusted certificates.
*/
public class InstallCert {

public static void main(String[] args) throws Exception {
String host;
int port;
char[] passphrase;
if ((args.length == 1) || (args.length == 2)) {
String[] c = args[0].split(":");
host = c[0];
port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
String p = (args.length == 1) ? "changeit" : args[1];
passphrase = p.toCharArray();
} else {
System.out.println("Usage: java InstallCert [:port] [passphrase]");
return;
}

File file = new File("jssecacerts");
if (file.isFile() == false) {
char SEP = File.separatorChar;
File dir = new File(System.getProperty("java.home") + SEP
+ "lib" + SEP + "security");
file = new File(dir, "jssecacerts");
if (file.isFile() == false) {
file = new File(dir, "cacerts");
}
}
System.out.println("Loading KeyStore " + file + "...");
InputStream in = new FileInputStream(file);
KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(in, passphrase);
in.close();

SSLContext context = SSLContext.getInstance("TLS");
TrustManagerFactory tmf =
TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
X509TrustManager defaultTrustManager = (X509TrustManager) tmf.getTrustManagers()[0];
SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
context.init(null, new TrustManager[]{tm}, null);
SSLSocketFactory factory = context.getSocketFactory();

System.out.println("Opening connection to " + host + ":" + port + "...");
SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
socket.setSoTimeout(10000);
try {
System.out.println("Starting SSL handshake...");
socket.startHandshake();
socket.close();
System.out.println();
System.out.println("No errors, certificate is already trusted");
} catch (SSLException e) {
System.out.println();
e.printStackTrace(System.out);
}

X509Certificate[] chain = tm.chain;
if (chain == null) {
System.out.println("Could not obtain server certificate chain");
return;
}

BufferedReader reader =
new BufferedReader(new InputStreamReader(System.in));

System.out.println();
System.out.println("Server sent " + chain.length + " certificate(s):");
System.out.println();
MessageDigest sha1 = MessageDigest.getInstance("SHA1");
MessageDigest md5 = MessageDigest.getInstance("MD5");
for (int i = 0; i < chain.length; i++) {
X509Certificate cert = chain[i];
System.out.println
(" " + (i + 1) + " Subject " + cert.getSubjectDN());
System.out.println(" Issuer " + cert.getIssuerDN());
sha1.update(cert.getEncoded());
System.out.println(" sha1 " + toHexString(sha1.digest()));
md5.update(cert.getEncoded());
System.out.println(" md5 " + toHexString(md5.digest()));
System.out.println();
}

System.out.println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");
String line = reader.readLine().trim();
int k;
try {
k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
} catch (NumberFormatException e) {
System.out.println("KeyStore not changed");
return;
}

X509Certificate cert = chain[k];
String alias = host + "-" + (k + 1);
ks.setCertificateEntry(alias, cert);

OutputStream out = new FileOutputStream("jssecacerts");
ks.store(out, passphrase);
out.close();

System.out.println();
System.out.println(cert);
System.out.println();
System.out.println
("Added certificate to keystore 'jssecacerts' using alias '"
+ alias + "'");
}

private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();

private static String toHexString(byte[] bytes) {
StringBuilder sb = new StringBuilder(bytes.length * 3);
for (int b : bytes) {
b &= 0xff;
sb.append(HEXDIGITS[b >> 4]);
sb.append(HEXDIGITS[b & 15]);
sb.append(' ');
}
return sb.toString();
}

private static class SavingTrustManager implements X509TrustManager {

private final X509TrustManager tm;
private X509Certificate[] chain;

SavingTrustManager(X509TrustManager tm) {
this.tm = tm;
}

public X509Certificate[] getAcceptedIssuers() {
throw new UnsupportedOperationException();
}

public void checkClientTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
throw new UnsupportedOperationException();
}

public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
this.chain = chain;
tm.checkServerTrusted(chain, authType);
}
}
}

Apache 64-bit on Solaris 10 & Weblogic

2013-07-08T12:03:00.001-04:00

Solaris 10 comes with Apache 32- bit by default. This was a surprise.

My task here is to get Apache 64-bit and then hook it up to Weblogic on the SSL channel. I will detail the Apache + Weblogic SSL communication. Setting up Weblogic to listen on SSL is a different post, that I may or may not write up :)

I will be installing apache at /usr/local/apache (Many folks like to install it at /usr/local/apache2 etc, but I'm keeping it simple here)

I have a staging directory to compile / install the required packages. This is a Whole Root zone, with Solaris 10 fully installed with all the packages on the Global Zone.

I need
0. Apache 2.2.4 -- I tried Apache 2.4 and as of this writing is not supported with the mod_wl.so library.
1. PCRE (Perl-Compatible Regular Expressions Library)
2. Apr
3. Apr-utils

A gist of what is required is available on Apache's documentation site ( Apache Documentation for Installation )

For PCRE, I downloaded it from: PCRE
Download APR & APR-UTIL from here.

Initial settings on the terminal

export CFLAGS="-m64"
export LDFLAGS="-L/usr/sfw/lib/sparcv9"

export LD_LIBRARY_PATH=/usr/sfw/lib/sparcv9:/usr/local/lib/sparcv9

the CFLAGS setting is really important, otherwise, you wont get it compiled in 64 bit. Make sure your LD_LIBRARY_PATH and LDFLAGS point to the accurate directories.

Compiling PCRE

This installs PCRE in /usr/local/pcre

/root/apache-compilation/pcre-8.33
./configure --disable-cpp CFLAGS="-m64 -g"
make
make install

Compiling APR

/root/apache-compilation/apr-1.4.8
./configure
make
make install

APR gets installed in /usr/local/apr. The CFLAGS setting set in the environment should carry on.

Compiling APR-UTIL

/root/apache-compilation/apr-util-1.5.2
./configure --with-apr=/usr/local/apr
make
make install

Compiling Apache
I use the prefix /usr/local/apache - but some people prefer to use "apache2". Make it your own. The prefix designates where apache gets installed.

/root/apache-compilation/httpd-2.2.24
./configure --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr --prefix=/usr/local/apache --enable-mods-shared=most --enable-ssl --with-ssl=/usr/sfw
make
make install

After the installation, I like to test to make sure its 64 bit (as I've gotten bitten a number of times).

# cd /usr/local/apache/modules
# file mod_vhost_alias.so
mod_vhost_alias.so: ELF 64-bit MSB dynamic lib SPARCV9 Version 1, dynamically linked, not stripped, no debugging information available

That tells you its a 64 Bit file.

To get Weblogic working, I copied the wls 1.1 plugins *.so files (which are only 64 bit) to the /usr/local/apache/modules directory. Then followed the instructions from Oracle here

Good luck. Comments are welcome.

Solaris 11....network interfaces..ugh!

2013-02-03T13:49:00.001-05:00

Ok - I'm not sure who was responsible for the Solaris 11 redesign - but someone should take him out and beat him with a stick.

Why on earth would you actually change how to configure the network interfaces and such and make it more complicated? Unix is a text based system for administrative activities. One would expect you make it EASIER not more complex.

For Eg:
To create a new tagged VLAN interface in Solaris 10

ifconfig *1000+networkid plumb - gives you the network interface

Solaris 11

root@solaris:~# dladm create-vlan -l net2 -v 11
root@solaris: ifconfig net11002 plumb netmask 255.255.255.0 broadcast + up

So basically you've doubled the commands to achieve the same damn thing.

A lot of things have gone this way. Apparently now, you cant have an already plumbed interface and can't add it to an ipmp group either (IP Multipathing). So if I had 2 interfaces and needed to add it to an ipmp group, with one of them having the IP which I'm working from - its a lot more convoluted.

I'm thinking may be I should just stick with Solaris 10 for eternity. Who the hell does such stuff (Linux is no better - but come on!)

2011 -- a (delayed) recap...

2012-12-31T19:08:00.000-05:00

I just realized, I didnt write up my recap for 2011. Tells you how busy my year has been and how fast time has flown that it's April 30th, 2012.

The year was so busy, I dont remember much.

January 2011 rolled around -- we had our little one in Sept 2010, so time was really flying. I barely noticed it as I was working constantly, and even from the delivery room I should mention. Our project at work, finally went live in January, delayed, but went live.

Roll forward to March where we did a stability release for the product, which made my life so much saner. I wasn't being called in the middle of the night to troubleshoot the product anymore!

We went to July / Aug India in 2011, so it was really great to meet all my family and celebrate. I worked out of the Mumbai office for a week which was interesting to the least. Definitely a change in culture, and tone compared to working in the US. Especially true, since I was working UK hours, so that I could talk to my colleagues back home, and still go back to the hotel at a decent hour! Mumbai has changed significantly, yet landing in Mumbai, the view from the air gave me a shock of my life. Not to mention just walking down the road from my hotel, I found a bicycle repair guy living in literally a treehouse, and he had a ladder going up to his "house". What was unbelievable was, he had 2 kids who were running around!!! Really -- gives you a pause to think - when we the educated and probably middle class think many times about the future of our children - he doesnt - he just has his kids as a matter of fact!

August was great, as we celebrated Aarathi's birthday (my niece). I haven't been around for one her birthdays since 1999 - so this was excellent. Lots of fun! She'll probably get married soon, so this was one of the last birthdays we were all able to play pranks on and have a "kiddish" time :)

We also had a couple of ceremonies for Vaibhav ( the new one :) ). Those occasions brought together all our friends and relatives at one place, and was great to meet all of them.

We came back to the US, and then it's back to the grind. Pranav started First Grade, Vaibhav turned one in September, and we did his birthday party at our friend Suresh's community hall. That was fun too. My Brother-in-law, sister-in-law and my in-laws were all here. So it was enjoyable and fun. One of the most memorable moments was when Vaibhav barely a year old was being carried by our friends and he pointed to a photograph of his mom on the camera and pointed his finger out back to the hall -- his intention " take my to my mom!". That was unbelievable. The power of kids without even saying a word :)

Ofcourse, we also celebrated my wife's birthday in September.

My birthday was low key again in November. December rolled around and we're off into 2012.

2012 is almost over...and boy am I glad....

2012-12-31T19:07:00.002-05:00

I try to do this every year...where I think about the year...so here goes this one!

Started out with my wife having surgery for a relatively minor nail issue, turned out into a 3 week saga. The year then dragged on with work - where Travelex Global Business Payments separated from Travelex, and we were busy integrating with Western Union.

Ofcourse, I can't forget our 8 years anniversary we hit on Jan 24th. Thats always the pleasant beginning for me, where I think about the wonderful time of the last 8 years with my wife.

This March - unbelievably - we got our green card!

The next memorable event was in April when Pranav turned 7. A small party at home with his friends - which we all enjoyed.

My parents came to the US in June - since my dad retired in 2011 December, he's been getting used to not having to get up and get to work everyday! He moved back to India, to set up our house in Chennai. Then in the US on June 8th. Mom & Dad stayed till 15th October - so thats always fun! Vaibhav got used to his grandparents, and Pranav had a blast. Finally Lavanya & I were able to take a weekend and go off to by ourselves. That was a short and fun weekend we had the weekend of the 15th Sept, 2012! This was also our way of specially celebrating Lavanya's birthday :). We had a family Birthday for her after!

We celebrated Vaibhav turning 2 in a small fashion as well - we expect we'll be celebrating his 3rd birthday in a much bigger fashion.

We went to Diya on Oct 12th to celebrate Dad's birthday and then they started getting ready to head back to Chennai!

Ofcourse, right after they came, I had to have surgery for a pinched nerve on the right hand - bummer - and then right after they left Nov 7th - the same thing on the left hand - big bummer.

Then came by Birthday - Lavanya gave me a Mac Book Pro for my birthday - that I enjoy very much!

There is also something that happened in December, that I can write about, but it definitely changed my perspective about how much my work is appreciated! It was a good feeling!

This year has been one of the most roller coaster years I've ever had!!! Ever! Most of it has to do with family health issues....started with my wife, then me twice!!! Three surgeries this year is enough! I'm hoping to have a much more pleasant 2013.

If you're reading this - Happy New Year to you too!

PS: Photos hopefully to follow soon....

Network Interfaces with Vlan Tags in Solaris 11

2012-11-06T16:22:00.002-05:00

If you're trying to plumb a NIC with a VLAN tag in Solaris 11, its not straight forward anymore!

First, Look for the interfaces that you have.

root@solaris:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown igb1
net3 Ethernet unknown 10000 half qlge3
net5 Ethernet unknown 10000 half qlge2
net0 Ethernet unknown 0 unknown igb0
net2 Ethernet unknown 10000 full qlge0
net9 Ethernet unknown 10000 half qlge1

This is a brand new install and T4 blade with the QLogic 10Gb CNA cards. I had to figure out manually which one of these were connected to what.

However, first things first. To use a Tagged VLAN for the network interface, first use dladm create-vlan to create a new network interface.

root@solaris:~# dladm create-vlan -l net2 -v 11
root@solaris:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 unknown --
net3 phys 1500 unknown --
net5 phys 1500 unknown --
net0 phys 1500 up --
net2 phys 1500 up --
net9 phys 1500 up --
net11002 vlan 1500 up net2

Next plumb the interface

root@solaris:~# ifconfig net11002 plumb netmask 255.255.255.0 broadcast + up

You should now be able to SSH to the IP (if the default gateway is set right). Run netstat -rn to see the default gateway. If it's not set - "route add default " will do the trick.

If it is the wrong port, then you can just as easily run

root@solaris:~# dladm delete-vlan net11002

This will remove the new interface.

Solaris 11, Oracle 11g & EM12C

2012-01-30T09:24:00.000-05:00

I've just started using Solaris 11, this is my first ever install of it.

Solaris 11

Downloaded the text installer, mounted it as lofi, and then created an LDOM by booting off the ISO. I've not tried the Automated Installer (AI) yet.

I realize that there isn't much information here, but really, there wasn't much to do.

Add ISO to LDOM:

ldm add-vdsdev /sol11/sol-11-1111-text-sparc.iso sol11iso@primary-vds0
ldm add-vdisk id=1 sol11iso sol11iso@primary-vds0 orcem12c

Show Disks on LDOM Ok prompt:

{0} ok show-disks
a) /virtual-devices@100/channel-devices@200/disk@1
b) /virtual-devices@100/channel-devices@200/disk@0
c) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/disk@1 has been selected.

Boot from Disk 1 (as thats what the ID is for the ISO). Also choose slice "f"

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1:f

Follow the installer along and give appropriate credentials. A user will need to be created, as root login is disabled by default.

To allow root login:

1. Change "PermitRootLogin" to "yes" in the /etc/ssh/sshd_config file.
2. Comment out "#CONSOLE=/dev/console" in the /etc/default/login file to allow non-console root login.
3. Switch the role of the root user "rolemod -K type=normal root"

Oracle 11g (11.2.0.3 Install)

Prerequisite

Before you start the install of EM 12c, update the Solaris 11 packages, or the OEM 12c installer will probably throw an error.

If this is a fresh install of Solaris 11, make sure to update the Publisher. The directions are at the Oracle Solaris 11 Certificate Site.

Once you download the certificates then do the following (from Oracle's site)

Download the provided key and certificate files, called Oracle_Solaris_11_Support.key.pem andOracle_Solaris_11_Support.certificate.pem using the buttons above. Don't worry if you get logged out, or lose the files. You can come back to this site later and re-download them. We'll assume that you downloaded these files into your Desktop folder,~/Desktop/.
Use the following comands to make a directory inside of /var/pkg to store the key and certificate, and copy the key and certificate into this directory. The key files are kept by reference, so if the files become inaccessible to the packaging system, you will encounter errors. Here is how to do it:
```
 $ sudo mkdir -m 0755 -p /var/pkg/ssl
 $ sudo cp -i ~/Desktop/Oracle_Solaris_11_Support.key.pem /var/pkg/ssl

 $ sudo cp -i ~/Desktop/Oracle_Solaris_11_Support.certificate.pem /var/pkg/ssl
 
```

Add the publisher:

 $ sudo pkg set-publisher \
            -k /var/pkg/ssl/Oracle_Solaris_11_Support.key.pem \
            -c /var/pkg/ssl/Oracle_Solaris_11_Support.certificate.pem \
            -G '*' -g https://pkg.oracle.com/solaris/support/ solaris

Check your publisher settings, there should be no unrelated mirrors set up. To check for any set up mirrors invoke the following command:
```
 $ pkg publisher solaris | grep Mirror
 
 
```
If the output is empty you are all set. If not remove unrelated mirrors by running:
```
 $ sudo pkg set-publisher -M http://mirror1.x.com -M http://mirror2.y.com ... solaris
 
 
```
To see the packages supplied by this publisher, try:
```
 $ pkg list -a 'pkg://solaris/*'
```

Next update packages. Oracle site lists the directions : here.

Steps are as follows:

First, we use pkg update with the --accept option to update a small set of system packages and then we reboot:
```
# pkg update --accept
# reboot
```
(SPARC only) If you are running on a SPARC system and have any Oracle Solaris Zones installed, you will need to perform an additional step at this stage. For each zone installed on the system, you will need to remove thepkg:/system/ldoms/ldomsmanager package as follows:
```
# for z in `zoneadm list`; do zlogin $z pkg uninstall ldomsmanager; done
```
Next, update the IPS package itself:
```
# pkg update pkg:/package/pkg
```
Last, update the rest of the system packages and reboot the system.

To update, we use the --accept flag to pkg update to agree to and accept the terms of the licenses that are being updated. We also use the --be-name flag to provide a new name to the boot environment that will be created.
```
# pkg update --be-name s11.1ga --accept
# reboot
```

First things first, rebuild the Package index and then get the X packages installed:

pkg rebuild-index

Test to make sure you have all the required packages

pkginfo -i SUNWarc SUNWbtool SUNWhea SUNWlibm SUNWlibms SUNWpool SUNWpoolr SUNWsprot SUNWtoo SUNWuiu8 SUNWfont-xorg-core SUNWfont-xorg-iso8859-1 SUNWmfrun SUNWxorg-client-programs SUNWxorg-clientlibs SUNWxwfsw SUNWxwplt

Install packages that are needed.

pkg install compatibility/packages/SUNWxwplt SUNWmfrun SUNWarc SUNWhea SUNWlibm

pkg install x11/xclock

Add Swap if you dont have enough (You'll need about 16Gb depending upon the RAM you've allocated to the LDOM).

zfs create -V 14G swpool/swap14G
swap -a /dev/zvol/dsk/swpool/swap14G

Edit the /etc/vfstab to add the new swapfile.

/dev/zvol/dsk/swpool/swap14G - - swap - no -

I found that Timezone wasn't set. In Solaris 11, all the properties have moved to SMF.

root@orcem12c:~# svcs -a | grep environ
online 17:45:30 svc:/system/environment:init
root@orcem12c:~# svccfg -s system/environment:init
svc:/system/environment:init> listprop environment/TZ astring
svc:/system/environment:init> setprop environment/TZ=US/Eastern
svc:/system/environment:init> validate
svc:/system/environment:init> end

Install Oracle 11g. I choose to use the file storage instead of ASM for the Db disks.

Create new Pool for /opt/app/Oracle & /opt/app/oradata (to hold all the Software & Db files respectively)

zpool create -f -m /opt/app/Oracle swpool c2d1s0
zpool create -f -m /opt/app/oradata dbpool c2d2s0

Requirements for the Db:

1. 2G SGA.
2. Redo log files must atleast be 300Mb with 3 Redo logs.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo01.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo02.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo03.log') Size 300M;

SQL> alter system switch logfile;

If adding logfiles, you can drop the old ones. If the Db complains that it cant drop a logfile (if you need to), run "alter system checkpoint". You cannot drop a logfile in "Current" status.

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC
---------- ---------- ---------- ---------- ---------- ---------- ---
STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------------- ------------- --------- ------------ ---------
4 1 13 314572800 512 1 NO
INACTIVE 1089635 29-JAN-12 1089642 29-JAN-12

5 1 14 314572800 512 1 NO
CURRENT 1089642 29-JAN-12 2.8147E+14

6 1 0 314572800 512 1 YES
UNUSED 0 0

I had to make the following changes:

SQL> alter system set session_cached_cursors=300 scope=spfile;

System altered.

SQL> alter system set job_queue_processes=20;

System altered.

SQL> alter system set shared_pool_size=600M scope=spfile;

System altered.

EM12C (Cloud control) Install

Once I made all the above Db changes, I was able to get through the installer using the "Simple" option.

The only error message after all this, was that there wasn't an EM tablespace with an autoextend datafile. Since I didnt create a tablespace, so I proceeded.

Once the install finishes, I logged in with https://:port/em. Login with the user sysman.

Ops Center DHCP Hell....

2011-10-18T17:57:00.000-04:00

If you've set up Ops Center and are wondering why the machines wont jump with DHCP, make sure you choose "ISC" instead of Solaris as your DHCP server.

Also a good link to follow is this note OpsCenter and ISC DHCP server blog!

At Oracle World...Past 2 days...

2011-10-05T10:28:00.002-04:00

There were quite a few announcements at Oracle world! None of them interesting!

Enterprise Manager 12c has been released.

Ops Center is free

Solaris 11 is almost there, we can download an early release and install it. The big shocker - there is no way to go from 10 to 11 without a backup, reinstall and restore!\

More to come as I have time to write up.

Release of Solaris (Sparc or x86?)

2011-09-30T11:12:00.000-04:00

I got a handy tip from an Oracle person the other day....to make sure what solaris release and version the solaris cd is or jumpstart location is, check the SUNWsolnm/reloc/etc/release file.

[root@host1]# cat SUNWsolnm/reloc/etc/release
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
[root@host1]#

TFTP Boot error Access Violation

2011-09-30T10:10:00.001-04:00

When accessing a tftp server clients receive "Error code 2: Access violation"
due to ownership & permission of the /tftpboot directory.

Steps to Follow
in.tftpd runs as user nobody. nobody needs rwx permissions to the /tftpboot
directory
chmod 766 /tftpboot
chown nobody:other /tftpboot

Product
Network Management/Conn

Oracle 11g RAC in LDOMs (Oracle VM for Sparc)

2011-09-16T17:21:00.000-04:00

Setup
1. 2 T3-1B blades on different chassis. Patch the firmware. Install LDOM 2.1 software.
2. SAN Connected storage via MPXIO (EMC SAN)
3. Solaris 10/9 installed with End User Option.

Create install User

useradd -u 0 -o -g 1 -c "Install user" -d / -s /bin/true install

Patch LDOM.

Download the following Patches from oracle
• 117837-05: C++ compiler optimizer patch
• 117846-08: C++ compiler Optimization patch
• 118682-01
• 127111-02 SunOS 5.10: libc patch (this should already exist if LDOM is patched)
• 137111-04 SunOS 5.10: kernel patch (this should already exist if LDOM is patched)

Profile Settings

• On a new system, set the following in /etc/profile
a. MANPATH=/usr/man:/usr/local/man:/usr/sfw/man:$MANPATH

• Create the oracle required users & groups

groupadd -g 102 dba
groupadd -g 103 oinstall
projadd -p 102 -c "Oracle Project" group.dba
useradd -u 101 -g dba -G oinstall -s /usr/bin/bash -c "Oracle User" -m –p 102 oracle

• Set the Oracle password

passwd -r files oracle

• Add the following entries to /etc/system for a 4G SGA or less. Shmmax should always be greater than SGA size.

set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967296
set shmsys:shminfo_shmmni=100
set rlim_fd_max=65636
set rlim_fd_cur=4096

Reboot the system after the above /etc/system have been made.

Alternate way

Run prctl to make changes to the existing install so that a reboot is not required. These should be run as root. These changes don’t persist after a reboot.
prctl -n project.max-shm-memory -v 4gb -r -i project user.root
prctl -n project.max-sem-ids -v 256 -r -i project user.root

Project Settings

Make changes to the project so that changes persist for shm memory & sem ids (4G or less SGA). These should be run as root.

projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,256,deny)" group.dba

/etc/project looks like this after changes:

[root@orc11gtest]# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:102:Oracle Project:oracle::project.max-sem-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Verify the changes applied to the oracle user as below.

[oracle@orc11gtest]$ prctl -n project.max-shm-memory -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 4.00GB - deny -
[oracle@orc11gtest]$ prctl -n project.max-sem-ids -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-sem-ids
privileged 256 - deny

Patch Check:

patchadd -p | grep 125555-03
patchadd -p | grep 140796-01
patchadd -p | grep 140899-01
patchadd -p | grep 141016-01
patchadd -p | grep 139555-08
patchadd -p | grep 141414-02 (Replaced by 141444-09)
patchadd -p | grep 141736-05
patchadd -p | grep 120753-06
patchadd -p | grep 139574-03
patchadd -p | grep 141414-10
patchadd -p | grep 141736-05

I'm using Ops center to create LDOMs to make my life a little easier. If you're doing this, the Vnets sometimes will be different between the 2 blades. Go into the "/guests//servercfg.xml" and fix the

< idx >3< /idx >

number to match on both nodes.

NTP Configuration
Make sure to set the /etc/inet/ntp.conf with slewalways and disable p11.

server < serverip >
driftfile /etc/ntp.drift
slewalways yes
disable pll

Grid Software Install

ON each Control Domain -- add the disks for OVD, OCR & Data to the LDOMs. Make sure to add them in the same order, and set the same ID. A good way to make sure everything is the same on both nodes, is to also volume label the disks on the control / IO domain before assigning it to the LDOM.

Have xwin running and kick off the installer. Before you kick off the installer, its a good idea to run cluster verify tool.

./runcluvfy.sh stage -pre crsinst -n orczone1,orczone2 -verbose

Install any packages that are missing, fix the swap or any such thing, the MTUs etc, and you're ready to roll.

To add Swap Space when using ZFS

To create swap space on a ZFS volume (created 2 2g volumes):

zfs create -V 2gb rpool/swap2g
zfs create -V 2gb orcinstall/swap2g

Add it to swap

swap -a /dev/zvol/dsk/rpool/swap2g
swap -a /dev/zvol/dsk/orcinstall/swap2g

Add it to /etc/vfstab

/dev/zvol/dsk/rpool/swap2g - - swap - no -
/dev/zvol/dsk/orcinstall/swap2g - - swap - no -

Install / Troubleshooting Notes

1. A group called "DATA" was created by a previous install, and even though I zapped the disks, the group didnt go away. To drop the diskgroup (which wouldn't mount or come online) I had to do this:
drop diskgroup data force including contents;
2. If you're having trouble with ulimits and such, this link: Max user processes is pretty helpful.

Oracle 11g RAC in a zone - Partial instructions.

2011-09-14T09:10:00.001-04:00

This is a post thats a WIP. I will be adding to this as I keep finding new things, and hopefully a full step by step doc by the end of this.

I had to stop this because of the requirement of EXCLUSIVE IPs for the non-global zone. You can only have one Exclusive interface per vlan tag per server.

That is, if you have VLAN tag 3 for your public interface, and you want to create another RAC cluster on the same blade, you'll have to get another vlan from your network team. This seems unworkable, as all our Dbs are in the same VLAN.

I've just begun setting up 2 T3-Blades to use zones for Oracle RAC cluster.

Setup:
2 T3-1B Sparc blades with 64Gb of RAM.
Oracle 11gR2 downloaded
Oracle 11gR2 Grid Download
OS installed with "Full Install with OEM"
OS patched with the latest patch cluster.
Use EMC SAN for disk

MPXIO
Enable MPXIO if you're not using any vendor provided software to connect to the SAN
Using MPXIO to connect to the EMC CX-500
change "mpxio-disable" to "no" in ./kernel/drv/fp.conf

Disks
Using ZFS for the global zone with internal disks.

All Oracle zones will be created using mirrored disks from the SAN.

I intend on creating a zone, configuring the basic setup (DNS, oracle user, oracle download copied over) and take

[root@ch0bl7qadb1w]# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
orcpool 248G 11.0G 237G 4% ONLINE -
rpool 278G 7.99G 270G 2% ONLINE -
[root@ch0bl7qadb1w]# zpool status -v orcpool
pool: orcpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
orcpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t60060160733014002C5FAEED7BD9E011d0 ONLINE 0 0 0
c0t60060160733014002D5FAEED7BD9E011d0 ONLINE 0 0 0

errors: No known data errors

Creating Zones
Create the zone (orczone1), boot it and configure the DNS and such.

Zones have an annoying habit of setting auto-home. I normally run a script to disable auto-home. To allow the creation of the /home/oracle directory below, it's required to disable autohome.

Create the oracle project with group membership of dba.

projadd -p 102 -c "Oracle Project" group.dba

Create the dba (id: 102) oinstall (id: 103) groups

Create the Oracle, Oragrid users, the assign the oracle user belong to the oracle project.

useradd -u 101 -G dba -g oinstall -s /usr/bin/bash -m -p 102 oracle
useradd -u 103 -G dba -g oinstall -s /usr/bin/bash -m oragrid

Verify that the users are created appropriately

[root@orczone1]# id -a oracle
uid=101(oracle) gid=103(oinstall) groups=102(dba)
[root@orczone1]# id -a oragrid
uid=103(oragrid) gid=103(oinstall) groups=102(dba)

According to OFA guidelines, Oracle going to be installed in
/opt/app/oracle

Oracle install also recommends that only ORACLE_BASE is set. I'm also going to be installing Grid software and Db software as the same user.

Bash Profile for the oracle User

PS1="[\u@\h]$ "
ORACLE_BASE=/opt/app/oracle
LD_LIBRARY_PATH=/usr/lib:/lib:$LD_LIBRARY_PATH
export SQLPATH=/home/oracle

export ORACLE_BASE ORACLE_SID LD_LIBRARY_PATH

export PATH=$CRS_HOME/bin:$ORACLE_HOME/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/local/bin:/bin/sbin:$PATH

umask 022
ulimit -n 65536

export EDITOR=vi
export editor=vi
export PS1

NTP
My /etc/inet/ntp.conf file is this (in the Global Zone):

server 10.27.1.254
driftfile /etc/ntp.drift
slewalways yes
disable p11

Grid Software install

1. Create /opt/app/11.2.0/grid on both nodes (orczone1 & orczone2).
2. Create /opt/app/oracle on both nodes
3. I created the "app" directory and let oracle have rwx permissions on it.
4. Add the SCAN Ips as well as the virtual IPs to DNS

Orczone1.cctdev.com
Orczone1-vip.cctdev.com

Orczone2.cctdev.com
Orczone2-vip.cctdev.com

Orczone-cluster.cctdev.com -- this was the SCAN IP.

5. Set the following in /etc/system for the global zone, to avoid annoyances in the non-global zone. I use these blades only for Oracle, so it works for me!

set rlim_fd_max=65636
set rlim_fd_cur=4096

5. During install of GRID software, oracle will try to plumb IPs and if you're not using Exclusive IPs in the non-global zone, it will fail. This is where I stopped and switched to LDOMs.

Issue Log:

One of the things that are needed are a Public & Private Interface.

When plumbing them, make sure that the private interface stays private and does not get added to your "default gateway". If that happens, traffic has a tendency to get "lost" and when you try to reboot zones or even the server, you may lose connectivity with other nodes. The way I did this, was to not add a defrouter in the zonecfg for the private interface.

When creating a Zone for the Private interface, do not set the default router for that NIC unless you're using VLAN tagging and such.

The netstat -rn should look something similar to this

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.25.20.5 UG 1 27
default 10.25.20.5 UG 1 4 e1000g2
10.0.0.0 10.26.6.202 U 1 16 igb0
10.25.20.0 10.25.23.137 U 1 3 e1000g2
192.168.204.0 192.168.204.7 U 1 0 e1000g0
224.0.0.0 10.25.23.137 U 1 0 e1000g2
127.0.0.1 127.0.0.1 UH 5 92 lo0

cacaoadm not working

2011-08-24T16:01:00.002-04:00

I was trying to install Oracle Enterprise Ops Center and I kept getting a weird error which I couldn't find any information online about.

[root@host1:/]# /usr/sbin/cacaoadm
Cannot perform firstime initialisation and configuration.

Even reinstalling cacao didnt help (the packages were SUNWcacaodtrace & SUNWcacaort ).

Turns out the cacao.properties file was completely hosed. Here's where the file is located /etc/cacao/instances/default/private/cacao.properties

I ended up copying it from a different install, and it's working!!

OEM 10g not starting ...Fix!

2011-07-02T13:07:00.003-04:00

If you have Oracle Enterprise Manager that very annoyingly wont restart, though it's been perfectly working before you shut it down, and opmnctl complains that it cant start the HTTP server, you're probably running into the file size limit of the Apache logs.

[oragrid@admin30w]$ tail -20 /opt/oragrid/oms10g/Apache/Apache/logs/error_log
[Sat Jul 2 01:43:20 2011] [error] (79)Value too large for defined data type: could not open transfer log file /opt/oragrid/oms10g/Apache/Apache/logs/access_log.

If you see that message, it implies that the access log file is too large, typically over 2Gb. OEM by default doesn't rotate logs, so you need to either set it up, or rotate the logs manually.

oragrid@admin30w]$ ls -l /opt/oragrid/oms10g/Apache/Apache/logs/access_log
-rw-r---- 1 oragrid oinstall 2147483881 Apr 30 21:58 /opt/oragrid/oms10g/Apache/Apache/logs/access_log

To fix it, I just did a

cat /dev/null > /opt/oragrid/oms10g/Apache/Apache/logs/access_log

You can also set up log rotation this way.
Note 339819.1 from Metalink will give you detailed steps on the multiple ways to setup log rotation. However, this is supposed to be fixed in 11g.

1. Stop the OMS:

cd /opmn/bin
opmnctl stopall
2. Take a backup of the files :
/sysman/config/httpd_em.conf
/sysman/config/httpd_em.conf.template

NOTE: You must make the changes to both httpd_em.conf AND httpd_em.conf.template because any 'emctl secure' operation results in the http_em.conf file being regenerated using http_em.conf.template and any changes that had been made to httpd_em.conf earlier will be lost.
3. Edit the /sysman/config/httpd_em.conf and modify:

ErrorLog /Apache/Apache/logs/error_log
TransferLog /Apache/Apache/logs/access_log

TO

ErrorLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/error_log 43200"
TransferLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/access_log 43200"

where is the full path to your OMS install directory.
4. Edit the /sysman/config/httpd_em.conf.template and modify:

ErrorLog &ORACLE_HOME&/Apache/Apache/logs/error_log
TransferLog "&ORACLE_HOME&/Apache/Apache/logs/access_log"

To

ErrorLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/error_log 43200"
TransferLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/access_log 43200"
5. Delete the existing access_log and error_log files in the /Apache/Apache/log directory.

6. Re-start the OMS
cd /opmn/bin
opmnctl startall

Registering a Sunone 7 node to the admin server

2011-06-02T15:55:00.002-04:00

We have a number of SunONE 7 (now renamed to iPlanet) servers and the documentation online is fast fading, thanks to Oracle.

To register a node after install:

webserver7/bin

/wadm --host= --port= --user=admin

It will prompt for a password.
wadm> register-node

That should register the host successfully.

If you have renamed your server, navigate to the admin-server/config directory and fix the server.xml with the new hostname.

Tablespace Utilization in Oracle

2011-02-24T17:14:00.003-05:00

Taken from: http://vsbabu.org/oracle/sect03.html
Usage

TABLESPACE USAGE NOTES:
Tablespace Name - Name of the tablespace
Bytes Used - Size of the file in bytes
Bytes Free - Size of free space in bytes
Largest - Largest free space in bytes

select a.TABLESPACE_NAME,
a.BYTES bytes_used,
b.BYTES bytes_free,
b.largest,
round(((a.BYTES-b.BYTES)/a.BYTES)*100,2) percent_used
from
(
select TABLESPACE_NAME,
sum(BYTES) BYTES
from dba_data_files
group by TABLESPACE_NAME
)
a,
(
select TABLESPACE_NAME,
sum(BYTES) BYTES ,
max(BYTES) largest
from dba_free_space
group by TABLESPACE_NAME
)
b
where a.TABLESPACE_NAME=b.TABLESPACE_NAME
order by ((a.BYTES-b.BYTES)/a.BYTES) desc

Files modified between times

2011-02-24T16:53:00.002-05:00

Sometimes, I get this task to find out files that changed between 2 dates. I never seem to remember it, so I figured it out (googling) and here it is. Hopefully someone else also can use it!

olddate="201001010001"
newdate="201012312359"
touch -t $olddate ./tmp/oldfile
touch -t $newdate ./tmp/newfile
find /path/to/directory -type f -newer ./tmp/oldfile ! -newer ./tmp/newfile

Good Bye 2010, Hello 2011

2011-01-17T00:06:00.007-05:00

Where did 2010 go? Everything almost seems to be a blur...I spent almost 350 days working last year I think (yes, that includes, nights, weekends etc).

I'm going to try to recount some happy times ...

January as always is a happy time, as I celebrate my anniversary with my lovely wife. This time, we decided to go to Las Vegas as our friends were staying there and celebrate. A couple of nights at their house, and one night at MGM Grand. It was a good trip! Here is a picture I could find, of giving my wife a surprise gift. I had my Brother In Law bring a pair of ear rings from India, and then send it to my friends place in Las Vegas without telling Lavanya :) (Yes, thats her acting surprised :D)

The rest of the year moved quickly, as we announced that we were going to have our second child!

Ofcourse, who can forget the monstrous winter of 2009 / 2010!

Time quickly flew by after that. We had my uncle, aunt & mom come from India in May / June (I think). They went on an US Tour and Alaska cruise and returned.
We went to a great airshow in May. This was our first time ever, and unfortunately we were a little late, but got to see some awesome acrobatics. Next time, I think we're going to go there for the whole day. Pranav had a blast!

In June, Lavanya had her surprise "Seemantham" or Baby shower at our neighbours house.

Then my In-laws arrived in August to help with our upcoming kid.

September rolled around, and my second son -- Vaibhav Chandra decided he wanted to be born before his due date, and he was born on Sept 3rd.

My parents flew down on the 10th, and so our house was full with family and joy!

On a side note, I flew so often to Toronto this year, it was crazy. I think I made 6 trips in about a span of 3 months, often staying there a week at a time.

After all the excitement was over, we all went to Minneapolis for a week to visit my brother in law (I flew in from Toronto and returned to Toronto !)

November came around, and my birthday passed silently this year, without a party or much fan fare! I think its owed to my little one :)

December came around, and time just flew. I was super busy with work, going to Toronto once that month as well I think.

2010 was definitely a year to remember because of the birth of Vaibhav, but not many things else, as it was a year of constant work and no play! I look forward to a great 2011!