SysAdmin - AiX, Linux, Windows & VMware

Setting up NIM Server

2012-06-15T07:06:00.001-07:00

Setting up NIM Master server: used AIX 7 but pretty much works the same for AIX 5 & 6
Install NIM Filesets(Client, Master and SPOT)
# smitty install_latest

Create Volume Group, loglv, and filesystem for NIM resources
# mkvg -f -S -s 128 -y nim_vg hdisk3
# mklv -t jfs2log -y nim_jfs2log nim_vg 1
# logform /dev/nim_jfs2log
# mklv -t jfs2 -y nim_base nim_vg 2
# crfs -v jfs2 -d /dev/nim_base -m /nim -p rw -A yes -u NIM
# mount /nim

Create Filesystem for Installation Images
# mklv -t jfs2 -y nim_images nim_vg 100
nim_images
# crfs -v jfs2 -d /dev/nim_images -m /nim/images -p rw -A yes -u NIM
File system created successfully.
13106596 kilobytes total disk space.
New File System size is 26214400
# mount /nim/images

Create Filesystem for LPP Sources
# mklv -t jfs2 -y nim_lppsrc nim_vg 100
nim_lppsrc
# crfs -v jfs2 -d /dev/nim_lppsrc -m /nim/lppsource -p rw -A yes -u NIM
File system created successfully.
13106596 kilobytes total disk space.
New File System size is 26214400
# mount /nim/lppsource

Creat Filesystem for SPOTS
# mklv -t jfs2 -y nim_spot nim_vg 30
nim_spot
# crfs -v jfs2 -d /dev/nim_spot -m /nim/spot -p rw -A yes -u NIM
File system created successfully.
3931836 kilobytes total disk space.
New File System size is 7864320
# mount /nim/spot

Verify the Filesystems Created
# df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          917504    341240   63%    11455    21% /
/dev/hd2        16646144    329032   99%    49881    54% /usr
/dev/hd9var      1179648    356512   70%     8470    17% /var
/dev/hd3         1966080   1949800    1%       99     1% /tmp
/dev/fwdump       262144    261448    1%        4     1% /var/adm/ras/platform
/dev/hd1          131072    103728   21%       25     1% /home
/dev/hd11admin     262144    261384    1%        7     1% /admin
/proc                  -         -    -         -     - /proc
/dev/hd10opt      917504    540312   42%     7173    11% /opt
/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump
/dev/nim_base     524288    523488    1%        7     1% /nim
/dev/nim_images   26214400 26209744    1%        4     1% /nim/images
/dev/nim_lppsrc   26214400 26209744    1%        4     1% /nim/lppsource
/dev/nim_spot    7864320   7862464    1%        4     1% /nim/spot

Initialize NIM
To initiatize nim on the server with IP: 10.62.137.109 & NetMask: 255.255.255.0, run the following
# nimconfig -a netname=net_10_62_137 -a pif_name=en0 -a netboot_kernel=64 -a c>
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 6946994.

Upon Initializing NIM, some base resources are created as shown below
# lsnim
master           machines        master
boot             resources       boot
nim_script       resources       nim_script
net_10_62_137     networks        ent

Create Basic NIM Resources
Following basic resources must be created.

an LPP source (NIM object of type lppsource)
a SPOT (NIM object of type spot)
NIM scripts (NIM objects of type script and fb_script)

Creating LPP Source from AIX 7 DVD
For creating an LPP source from an AIX 7.1 installation DVD.We copy the files to our target filesystem (/nim/lppsource) using bffcreate. Since we want to be able to serve more than one LPP source we decide to store the files in a subfolder LPP71_TL00:
# mkdir /nim/lppsource/LPP71_TL00
# smitty bffcreate

I have downloaded the TL filesets under /datafs/ for AIX 7100-01-00-1140.. We now copy the filesets into the LPP source. Please note the target directory should be /nim/lppsource/LPP71_TL00/installp/ppc

# smitty bffcreate

I have downloaded Service pack: 7100-01-01-1141 and ran smitty bffcreate again for updating service pack
# smitty bffcreate

Define LPP Source
# cd /nim/lppsource/LPP71_TL00/installp/ppc
# inutoc .
# nim -o define -t lpp_source \
> -a server=master \
> -a location=/nim/lppsource/LPP71_TL00 \
> -a comments='Full AIX 7100-01-03 Source' \
> lpp_aix710

Preparing to copy install images (this will take several minutes)...

Now checking for missing install images...
All required install images have been found. This lpp_source is now ready.

Save some space by removing superseeded filesets : You can remove Language software by selecting yes if you don't want to keep language software.

Create SPOT from LPP source:

Create Installation Image
This is an option step needed if you want to install servers from mksysb image instead of using lpp source. If you want to define mksysb resource, create mksysb from existing AIX7 installation, copy it over to NIM Master. If you do not have any AIX 7 in your environment, then create an installation from lppsource on a spare lpar, customize it and create mksysb or create mksysb from the nim master before configuring the nim master..
After you have a mksysb image of AIX 7, you can define the mksysb nim resource as below
# nim -o define -t mksysb \
-a server=master \
-a comments='MKSYSB AIX 7 Installation - Created <DATE> \
-a location=/nim/install_mksysbimages/mksysb_aix7 \
mksysb_aix7

Create Scripts
If you have customization scripts like first boot script called fboot_aix7.sh and a postinstall script called pboot_aix7.sh in place, then we have to define two more NIM resources:

Firstboot Script# vi fboot_aix7.sh
# chmod ug+x fboot_aix7.sh
# nim -o define -t fb_script \
            -a server=master \
            -a comments='Firstboot script for AIX 7' \
            -a location=/nim/scripts/fboot_aix7.sh \
          fb_aix7

Postinstall Script
# vi pboot_aix7.sh
# chmod ug+x pboot_aix7.sh
# nim -o define -t script \
            -a server=master \
            -a comments='Postinstall script for AIX 7 installations' \
            -a location=/nim/scripts/pboot_aix7.sh \
          script_aix7_post

This completes the NIM Master setup, then do the followingon Nim Master to
Define a client: use smitty nim->Perform Nim Admin Tasks->Manage Machines->Define a Machine or command line as below.
#nim -o define -t standalone -a platform=chrp -a netboot_kernel=64 -a if1="net_10_62_137 -a cable_type1=tp <client>" <client>
Allocate resources:
# nim -o allocate -a spot=spot_aix710 -a lpp_source=LPP71_TL00 <ClientName>
Initialize lpp installation:
nim -o bos_inst -a spot=spot_aix710 -a lpp_source=LPP71_TL00 -a no_client_boot=yes -a accept_licenses=yes <client>
If the installation Fails and you have to restart installation, deallocate all resources before reallocating
# nim -Fo deallocate -a subclass=all <ClientName>

Other useful commands and resources

remove resource:
nim -o remove spot_aix710
add filesets to a spot
smitty nim_inst_latest
Configuring the NIM master and creating basic installationresources from the command line
IBM Redbook: NIM from A to Z

Update DNS Server Entries on Multiple Servers

2012-05-01T06:10:00.001-07:00

Recently we had upgraded our Windows 2000 Domain Controllers with Windows 2008 Domain Controllers and had to replace the DNS Servers listed in Network Properties across all the member servers running windows 2000, Windows 2003 & Windows 2008 Servers. I created the following script to replace DNS Entries.. Replace "a.b.c.d" with the actual dns server IP's in the script, create a file "d:\DNS_Order\servers.txt" with the server names listed in it. I ran it from command line as below: test.txt will record the output from the script which lists the DNS Entries of each and every server before and after completing the replacement..

C:\ cscript dns_replace_Serversfile.vbs >test.txt

dns_replace_Serversfile.vbs
Const ForReading = 1
Set objNetwork = CreateObject("Wscript.Network")
'Open the File System Object to Feed the Script
Set objFSO = CreateObject("Scripting.FileSystemObject")

'This line needs to be customized for the servers.txt localtion text file
Set objTextFile = objFSO.OpenTextFile ("d:\DNS_Order\servers.txt", ForReading)

'Starts the process reading thru the input file line by line until EOF
Do Until objTextFile.AtEndOfStream
strComputer = objTextFile.ReadLine
Wscript.Echo ""
Wscript.Echo "********************** Gathering Data On **********************"
Wscript.Echo " Computer Name From TXT File : "& strComputer
Wscript.Echo "********************** Gathering Data On **********************"

On Error Resume Next

arrNewDNSServerSearchOrder = Array("a.b.c.d", "a.b.c.d1", "a.b.c.d2", "a.b.c.d3")

Set objWMIService = GetObject("winmgmts:" _
& "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colNicConfigs = objWMIService.ExecQuery _
("SELECT * FROM Win32_NetworkAdapterConfiguration WHERE IPEnabled = True")

WScript.Echo VbCrLf & "Computer: " & strComputer

For Each objNicConfig In colNicConfigs
WScript.Echo VbCrLf & " Network Adapter " & objNicConfig.Index
WScript.Echo "    DNS Server Search Order - Before:"
If Not IsNull(objNicConfig.DNSServerSearchOrder) Then
    For Each strDNSServer In objNicConfig.DNSServerSearchOrder
      WScript.Echo "        " & strDNSServer
    Next
End If
intSetDNSServers = _
   objNicConfig.SetDNSServerSearchOrder(arrNewDNSServerSearchOrder)
If intSetDNSServers = 0 Then
    WScript.Echo "    Replaced DNS server search order list."
Else
    WScript.Echo "    Unable to replace DNS server search order list."
End If
Next

WScript.Echo VbCrLf & String(80, "-")

Set colNicConfigs = objWMIService.ExecQuery _
("SELECT * FROM Win32_NetworkAdapterConfiguration WHERE IPEnabled = True")

For Each objNicConfig In colNicConfigs
WScript.Echo VbCrLf & " Network Adapter " & objNicConfig.Index
WScript.Echo "    DNS Server Search Order - After:"
If Not IsNull(objNicConfig.DNSServerSearchOrder) Then
    For Each strDNSServer In objNicConfig.DNSServerSearchOrder
      WScript.Echo "        " & strDNSServer
    Next
End If
Next
Loop

Update AIX 7.1 Service Packs

2012-04-26T09:41:00.000-07:00

Make sure that all current filesets are commited by running installp -s or lslpp -l
# installp -s
installp: No filesets were found in the Software
Vital Product Database in the APPLIED state.
To cleanup the repository images of your installation directory. If you have committed all the fixes and don't need these files anymore..
#/usr/lib/instl/lppmgr -d /usr/sys/inst.images -l -b -x -r -e
lppmgr: Source table of contents location is /usr/sys/inst.images/.toc
lppmgr: Building table of contents in /usr/sys/inst.images ..
dsm.core.7.1.1.1.U
mcr.rte.7.1.1.1.U
security.pkcs11.7.1.1.1.U
sysmgt.cfgassist.7.1.0.15.U
======================== end list ==============================

Run the inutoc command, this creates the .toc file in Directory. If a .toc file already exists, it is recreated with new information. The default installation image Directory is/usr/sys/inst.images. The inutoc command adds table of contents entries in the .toc file for every installation image in Directory. To create .toc file for /usr/sys/inst.images
# inutoc /usr/sys/inst.images
Optional--Not needed to run --To rename the downloaded files to their fileset name, run the bffcreate command. After renaming the files, run the inutoc command again.
# bffcreate -c -d /usr/sys/inst.images
# inutoc /usr/sys/inst.images
Run smit update_all to install all updates from this package that apply to the installed filesets on your system, select the Installation Directory, make sure expand filesystem as required..
# smit update_all

Reboot the system if required

Update Firmware on 9111-285 (power5 System)

2012-03-26T08:43:00.000-07:00

Download Required Firmware - In this case Recommended Firmware is SF240-417
FTP the rpm file (01SF240_417_382.rpm) to the Server that Needs to be Upgraded

Run the command below to extract the flash image file in the rpm file:

rpm -Uvh --ignoreos /tmp/01SF240_202_201.rpm

Before installing check the existing firmware level. From AIX, use the command lsmcode. This command resides in the diagnostic directory. An example of the output of the lsmcode command is as follows:
----------------------------------------------------------------------------------------------------------
DISPLAY MICROCODE LEVEL 802811
IBM,9111-285

The current permanent system firmware image is SF240_202
The current temporary system firmware image is SF240_202
The system is currently booted from the temporary firmware image.

Use Enter to continue.
--------------------------------------------------------------------------------------------------------------
Next, run the update_flash command to upgrade firmware:
# ls /tmp/fwupdate
01SF240_417_382
# /usr/lpp/diagnostics/bin/update_flash -f /tmp/fwupdate/01SF240_417_382
The image is valid and would update the temporary image to SF240_417.
The new firmware level for the permanent image would be SF240_202.

The current permanent system firmware image is SF240_202.
The current temporary system firmware image is SF240_202.

***** WARNING: Continuing will reboot the system! *****

Do you wish to continue?
Enter 1=Yes or 2=No
SHUTDOWN PROGRAM
Mon Mar 26 11:01:35 EDT 2012
Stopping The LWI Nonstop Profile...
Waiting for The LWI Nonstop Profile to exit...
Waiting for The LWI Nonstop Profile to exit...
Stopped The LWI Nonstop Profile.
0513-044 The sshd Subsystem was requested to stop.
Wait for 'Rebooting...' before stopping.
Error logging stopped...
Advanced Accounting has stopped...
Process accounting stopped...
Stopping NFS/NIS Daemons
0513-044 The nfsd Subsystem was requested to stop.
0513-044 The biod Subsystem was requested to stop.
0513-044 The rpc.lockd Subsystem was requested to stop.
0513-044 The rpc.statd Subsystem was requested to stop.
0513-004 The Subsystem or Group, gssd, is currently inoperative.
0513-004 The Subsystem or Group, nfsrgyd, is currently inoperative.
0513-044 The rpc.mountd Subsystem was requested to stop.
0513-004 The Subsystem or Group, ypbind, is currently inoperative.

Connection to host lost.
After the System is back up, ran lsmcode and found that the system booted from the new Temporary Firmware as shown below
DISPLAY MICROCODE LEVEL 802811
IBM,9111-285

The current permanent system firmware image is SF240_202
The current temporary system firmware image is SF240_417
The system is currently booted from the temporary firmware image.

Use Enter to continue.
To move the Temporary Image

# /usr/lpp/diagnostics/bin/update_flash -c

Rename hdisk in AIX

2012-03-23T06:35:00.000-07:00

Recently we had an issue with 2 hdisks going bad in AIX LPAR, both in different volume groups (hdisk15 & hdisk78)., The Volume groups had disks mirrored (hdisk0 to hdisk39 mirrored to hdisk40 to hdisk79). After replacing the drives, found that the hdisks names got swapped as shown below before and after replacement. Since I do not want to create any confusion in future, I had to rename them back to original state before adding them to voulme groups as below..

Before Replacing

#lscfg -vl hdisk15
hdisk15 U5791.001.992055M-P2-T5-L11-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

#lscfg -vl hdisk78
hdisk78 U5791.001.9920546-P2-T6-L10-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

After Replacing

#lscfg -vl hdisk78
hdisk78 U5791.001.992055M-P2-T5-L11-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

#lscfg -vl hdisk15
hdisk15 U5791.001.9920546-P2-T6-L10-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

Swapping Back to Original State

Removed both the drives

#rmdev -dl hdisk78

#rmdev -dl hdisk15

Ran cfgmgr in the order I wanted

#cfgmgr -l scsi3

#cfgmgr -l scsi23

Verified that they are swapped back

#lsdev -Cc disk |grep hdisk15

hdisk15 Available 0P-08-00-11,0 16 Bit LVD SCSI Disk Drive

$ lsdev -Cc adapter |grep 0P-08

scsi3 Available 0P-08 Wide/Ultra-3 SCSI I/O Controller

#lscfg -vl hdisk15
hdisk15 U5791.001.992055M-P2-T5-L11-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

#lsdev -Cc disk |grep hdisk78

hdisk78 Available 0t-08-00-10,0 16 Bit LVD SCSI Disk Drive

# lsdev -Cc adapter |grep 0t-08

scsi23 Available 0t-08 Wide/Ultra-3 SCSI I/O Controller

#lscfg -vl hdisk78
hdisk78 U5791.001.9920546-P2-T6-L10-L0 16 Bit LVD SCSI Disk Drive (7
3400 MB)

Replace Faulty Disk in a SCSI Mirrored VG in AIX

2012-03-23T05:59:00.000-07:00

Found a Faulty SCSI Disk in a volume group on one of the LPAR in p595, Volume Group has 40 SCSI Disks mirrored and one Logical Volume and it is marked stale.,
Following are the steps taken to replace the Faulty Disk

Error Report Details

# errpt |more

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
EAA3D429 0321085312 U S LVDD PHYSICAL PARTITION MARKED STALE
EAA3D429 0321084712 U S LVDD PHYSICAL PARTITION MARKED STALE

16F35C72 0321044812 P H hdisk78 DISK OPERATION ERROR
16F35C72 0321023812 P H hdisk78 DISK OPERATION ERROR

$ errpt -a -j F7DDA124 |more
---------------------------------------------------------------------------
LABEL: LVM_SA_PVMISS
IDENTIFIER: F7DDA124

Date/Time: Wed Mar 21 04:48:29 EDT 2012
Sequence Number: 99327
Machine Id: 00CFEFAF4C00
Node Id: test01t
Class: H
Type: UNKN
WPAR: Global
Resource Name: LVDD
Resource Class: NONE
Resource Type: NONE
Location:

Description
PHYSICAL VOLUME DECLARED MISSING

Probable Causes
POWER, DRIVE, ADAPTER, OR CABLE FAILURE

Detail Data
MAJOR/MINOR DEVICE NUMBER
8000 0013 0000 0051
SENSE DATA
00CF EFAF 0000 4C00 0000 0113 D553 FECC 00CF EFAF 9018 3EC4 0000 0000 0000 0000

$ errpt -a -j 16F35C72 |more
---------------------------------------------------------------------------
LABEL: DISK_ERR2
IDENTIFIER: 16F35C72

Date/Time: Wed Mar 21 04:48:29 EDT 2012
Sequence Number: 99325
Machine Id: 00CFEFAF4C00
Node Id: test01t
Class: H
Type: PERM
WPAR: Global
Resource Name: hdisk78
Resource Class:
Resource Type:
Location:
VPD:
Manufacturer................IBM
Machine Type and Model......ST373454LC
FRU Number..................00P2685
ROS Level and ID............43373137
Serial Number...............0005D90D
EC Level....................H13092
Part Number.................26K5280
Device Specific.(Z0)........000004129F00013E
Device Specific.(Z1)........0721C717
Device Specific.(Z2)........0002
Device Specific.(Z3)........05179
Device Specific.(Z4)........0001
Device Specific.(Z5)........22
Device Specific.(Z6)........H13092

Description
DISK OPERATION ERROR

Probable Causes
DASD DEVICE

Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS

$ lscfg -vl hdisk78

hdisk78 U5791.001.9920546-P2-T6-L10-L0 16 Bit LVD SCSI Disk Drive (7

3400 MB)

Manufacturer................IBM

Machine Type and Model......ST373454LC

FRU Number..................00P2685

ROS Level and ID............43373137

Serial Number...............0005D90D

EC Level....................H13092

Part Number.................26K5280

Device Specific.(Z0)........000004129F00013E

Device Specific.(Z1)........0721C717

Device Specific.(Z2)........0002

Device Specific.(Z3)........05179

Device Specific.(Z4)........0001

Device Specific.(Z5)........22

Device Specific.(Z6)........H13092

LSPV LVM Details

$ lspv -l hdisk78

hdisk78:

LV NAME LPs PPs DISTRIBUTION MOUNT POINT

testlv 545 545 110..109..109..109..108 /testview

16F35C72 0321044812 P H hdisk78 DISK OPERATION ERROR

#lsvg -l testview2vg

testview2vg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

testlv jfs2 10890 21780 40 open/stale /testview

loglv13 jfs2log 1 2 2 open/syncd N/A

#lslv -m testlv ----- This command gives the paired disks

test:/testview
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0111 hdisk1 0111 hdisk75
0002 0111 hdisk2 0111 hdisk77
0003 0111 hdisk5 0111 hdisk78
0004 0111 hdisk6 0111 hdisk79
0005 0111 hdisk7 0111 hdisk76
0006 0111 hdisk8 0110 hdisk52
0007 0111 hdisk9 0110 hdisk53
0008 0111 hdisk11 0110 hdisk68
0009 0111 hdisk12 0110 hdisk69
0010 0111 hdisk13 0110 hdisk70
0011 0111 hdisk16 0110 hdisk71
0012 0111 hdisk18 0110 hdisk72

#/usr/sbin/unmirrorvg testview2vg hdisk78 ------This unmirrors all the mirrored drives
#/usr/sbin/reducevg testview2vg hdisk78 ---Remove the failed disk from Volume Group

#lspv -l hdisk78 ----Verify that the disk is not used
0516-320 : Physical volume 00cfefaf90183ec40000000000000000 is not assigned to
a volume group.

Identify & Replace Procedure

#diag -->Task Selection->Hot Plug Task->SCSI and SCSI RAID Hot Plug Manager->Identify a Device Attached to a SCSI Hot Swap Enclosure Device-> select the device...once identified Select
Replace/Remove a Device Attached to an SCSI Hot Swap Enclosure Devic-->Select the Device--> it will mark the device in replace mode-> Have CE replace the device-> hit enter in the screen--> it will mark the device populated -----> then select Configure Added/Replaced Devices--> this will run cfgmgr

Assign the Drive to Volume Group and Remirro...

#/usr/sbin/extendvg 'testview2vg' 'hdisk78'
#/usr/sbin/unmirrorvg testview2vg hdisk46 hdisk47 hdisk48 hdisk49 hdisk50 hdisk51 hdisk52 hdisk53 hdisk68 hdisk69 hdisk70 hdisk71 hdisk72 hdisk73 hdisk74 hdisk75 hdisk76 hdisk77 hdisk78 hdisk79
#nohup /usr/sbin/syncvg -testlv00 & ---This will sync the Volumes, will take few hours depending on the size

Replacing Fiber Adapter (HBA) in IBM AIX 5.2.0 on p595

2011-11-07T21:17:00.000-08:00

One of the lpar running on p595 servers had Started reporting Errors on one of the two Fiber Adapters (fscsi1) in Error report, the errors appeared as Type:TEMP as shown below.
825849BF   1106105211 T H fcs1           ADAPTER ERROR
B8FBD189   1106105211 T S fscsi1         SOFTWARE PROGRAM ERROR
LABEL:          FSCSI_ERR6
IDENTIFIER:     B8FBD189

Date/Time:       Sun Nov 6 10:52:31 EST
Sequence Number: 872005
Machine Id:      XXXXXX
Node Id:        XXXXXXX
Class:           S
Type:            TEMP
Resource Name:   fscsi1
LABEL:          FCS_ERR2
IDENTIFIER:     825849BF

Date/Time:       Sun Nov 6 10:52:31 EST
Sequence Number: 873473
Machine Id:      XXXXXXXX
Node Id:         XXXXXXX
Class:           H
Type:            TEMP
Resource Name:   fcs1
Resource Class: adapter
Resource Type:   df1000fa
Location:        U5791.001.992083W-P1-C05-T1
VPD:

Description
SOFTWARE PROGRAM ERROR

Probable Causes
ADAPTER MICROCODE
SOFTWARE PROGRAM
SOFTWARE DEVICE DRIVER

Failure Causes
ADAPTER MICROCODE
SOFTWARE PROGRAM
SOFTWARE DEVICE DRIVER

        Recommended Actions
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE
This server was running DB2 and DB2 crashed around the same time when we started getting these alerts. Found that DB2 crashed due to super block corruption in the filesystem and DB2 was unable to create files..,Unmounted the Filesystem, corrected superblock corruption by copying secondary superblock.

dd count=1 bs=4k skip=31 seek=1 if=/dev/LVXX of=/dev/LVXX
dd: 1+0 records in.
dd: 1+0 records out.

Had contacted SAN Team to look into any SAN issues and they did not find any issues on SAN.
Found that it works for a while and goes into degraded state and eventually into Failed state.
Placed a service call with IBM to replace the Adapter and replacing the adapter fixed the issue.. Steps followed to replace the Fiber Adapter..

datapath query adapter --- To identify the faulty or degraded adapter

Adpt# Name State Mode Select Errors Paths Active
0 fscsi1 FAILED ACTIVE 2059041211 711 20 0
1 fscsi3 NORMAL ACTIVE 231765446 37 20 18

datapath remove adapter 0 --- To remove the adapter from SDD
rmdev -Rdl fcs1 ---To remove adapter from ODM. This will also remove all child devices ( hdiskXX). This step is not needed as the following step will do this for you, hotplug mgr will do that for you, having the device still there allows you to identify it easier
diag, choose Task Selection, Hot Plug Task, PCI Hot Plug Manager.... tasks to replace the adapter
lscfg -vl fcs1 ---To identify new wwn. Have SAN Team assign LUN's to this adapter
cfgmgr -vl fscsi1 --- Run cfgmgr to rediscover the deleted paths to LUNs
cfgmgr -v
addpaths ---add paths to SDD
datapath query device ----verify all LUNS have same number of active paths
datapath query adapter

AiX Commads

2011-08-06T00:20:00.000-07:00

ssa_ela -h 7 ; To get SRN Status for ssa (output
ssa0 SRN 45002
enclosure0 SRN 80221)
lscfg -vlpdisk3 -- To get details of pdisk3
lsdev –C –c pdisk –s ssar –H –To list all pdisks or use smit Devices->SSA Disks—>SSA Physical Disks->List All Defined SSA Physical Disks
lsdev –C –t hdisk –c disk –s ssar –H – To list SSA Logical Disks or use smit Devices->SSA Disks—> SSA Logical Disks->List All Defined SSA Logical Disks
smit -> SSA Dsks—>SSA Physical Disks ->Show Physical to Logical SSA Disk Relationship or--for I in $(lsdev –CS1 –c pdisk –ssar –F name); do; echo “$i: “$ssaxlate –l $i);done
lsdev -P -H --To list all devices in the Predefined Devices object class with column headers, type--The system displays a message similar to the following:(class type subclass description--logical_volume vgtype vgsubclass Volume group )
To Remove pdisk3 follow the steps below

pdisk3, which is hdisk5
go into diags
Task Selection (Diagnostics, Advanced Diagnostics, Service Aids, etc.
SSA Service Aids
Set Service Mode
<ServerName>:pdisk3
Do Identify on pdisk3 to find out where it's located

Once you find it backout to the service mode menu again and do
Set or Reset Service Mode on pdisk3

This is when the drive light should come on solid and can be replaced
Pull old drive out and put new drive in

Then
rmdev -l pdisk3 -d
rmdev -l hdisk5 -d

cfgmgr

lscfg -vl pdisk3

You should see the new pdisk3

RHEL 4-- NFS Setup

2011-07-25T12:22:00.000-07:00

I was trying to setup NFS share to Share folder (/NFS_PROJECT/SPECS) on one of our server(hostname: nfsserver) and have it Automounted on the client both running RHEL4ES
Checks and Setups On the NFS Server:

#rpm -qa nfs-utils ---To confirm if NFS Service is installed on the server, in the response shown below, I found it installed, if it was not installed, install NFS: yum install nfs-utils
nfs-utils-1.0.6-93.EL4
#service nfs status ---To confirm that NFS Services are running, they are running as shown below. If the services are not running, start NFS by using the command: /sbin/service nfs start
rpc.mountd (pid 26598) is running...
nfsd (pid 26594 26593 26592 26591 26588 26587 26586 26585) is running...
rpc.rquotad (pid 26581) is running...

Edit the /etc/exports file and include the Folder to be shared, following are the details of /etc/exports after the change:

/NFS_PROJECT/SPECS nfsclient1(rw,root_squash) nfsclient2(rw,root_squash)

/usr/sbin/exportfs -ra ---To export the added contents of the /etc/exports file. Reexport all directories. It synchronizes /var/lib/nfs/xtab with /etc/exports. It removes entries in /var/lib/nfs/xtab which are deleted from /etc/exports, and remove any entries from the kernel export table which are no longer valid.

Following Actions taken on NFS Client: Hostname: nfsclient2

Check the automount.master file for the key files, on client2 found /misc is labelled and found key file /etc/auto.misc. Edited the automount.misc and following is the entry after the edit

nfstester -fstype=nfs,rsize=8192,wsize=8192,ro,soft nfssrever:/NFS_PROJECT/SPECS

That's it, configuration is done..check it out by

cd misc

cd tester --- The NFS Share got mounted as /misc/tester..

Useful Links

2011-06-22T12:10:00.000-07:00

IBM xSeries Manuals
AIX Virtual User Group-Wiki
AIX NIM Setup Guide
AIX Health Check Blog
pSeries Info Center
Linux_Slackware
Electronics
Demartek_iSCSI_Deployment_Guide.html
Windows Scripting - Users..
IBM Education Assistant

Windows Active Directory Migration from Windows 2000 to Windows 2008

2011-06-13T10:32:00.000-07:00

Reference Link#1 for Domain Controllers Migration from Windows 2000 to Windows 2008 R2

Reference Link#2 for Domain Controllers Migration from Windows 2000 to Windows 2008 R2

Reference Link#3 for Domain Controllers Migration from Windows 2000 to Windows 2008 R2

Current Hardware: Hardware: Intel Pentium III Processor, 767.52 MB RAM (.7 GB), 2x16 GB HDD

Current Roles on these Servers:

· Active Directory

· DNS for the Domain environment

Tasks:

· Extend the Schema master (Active Directory) so that Windows 2008 can be a member server

Copy adprep32.exe from Windows 2008 DVD to d:\adprep directory on the Windows 2000 DC's
Login to All Domain Controllers one at a time and run the following steps. This will prepare Active Directory Schema for Windows 2008 R2
adprep32.exe /forestprep
adprep32.exe /domainprep

Screen Output for adprep

ADPREP WARNING:

Before running adprep, all Windows 2000 Active Directory Domain Controllers in the forest should be upgraded to Windows 2000 Service Pack 4 (SP4) or later.

[User Action]

If ALL your existing Windows 2000 Active Directory Domain Controllers meet this requirement, type C and then press ENTER to continue. Otherwise, type any other key and press ENTER to quit.

Opened Connection to DCTEST

Note: Schema update from sch17(windows 2000) to sch 47 for Windows 2008 R2 & 44 for Windows 2008

For windows 2000 we might get error msgs

To resolve the issue - http://support.microsoft.com/kb/329194

· Create the Schema Update Allowed DWORD value in the following registry key, and then set the value to 1:

HKEY LOCAL MACHINE\System\CurrentControlSet\Services\NTDS\Parameters

For additional information about how to do this, click the following article number to view the article in the Microsoft Knowledge Base:

216060 Registry Modification Required to Allow Write Operations to Schema

· Create and run the Ipsec_acl_fix.vbs script. To do so:

a. Option Explicit

c. '**************************************************

d. '* ADS_RIGHTS_ENUM

e. '**************************************************

f. Const ADS_RIGHT_DS_READ_PROP = &H10&

g. const ADS_RIGHT_DS_LIST = &H4&

h. Const ADS_RIGHT_DS_LIST_OBJECT = &H80&

i. Const READ_CONTROL = &H20000&

j. Const ADS_RIGHT_DS_WRITE_PROP = &H20&

k. Const ADS_RIGHT_DS_CONTROL_ACCESS = &H100&

l. Const ADS_RIGHT_DS_CREATE_CHILD =&H1&

m. Const ADS_RIGHT_DS_DELETE_CHILD = &H2&

n. Const WRITE_OWNER = &H80000&

o. Const WRITE_DAC = &H40000&

p. Const DELETE = &H10000&

q. Const ADS_RIGHT_DS_DELETE_TREE = &H40&

r. Const ADS_RIGHT_DS_SELF = &H8&

t. '**************************************************

u. '* ADS_ACETYPE_ENUM

v. '**************************************************

x. Const ADS_ACETYPE_ACCESS_ALLOWED = &H0&

z. '**************************************************

aa. '* ADS_ACEFLAGS_ENUM

bb. '**************************************************

cc.

dd. Const CONTAINER_INHERIT_ACE = &H2&

ee. Const OBJECT_INHERIT_ACE = &H1&

ff.

gg. Const ADS_ACEFLAG_INHERIT_ACE = &H2&

hh.

ii. '**************************************************

jj. '* Check if a "IP Security" container exists and correct the

kk. '* Acls for the container

ll. '* 0 -> container exists, Success

mm. '* 1 -> container doesn't Exist

nn. '* 2 -> other failures

oo. '*

pp. '**************************************************

qq. Function FixupIpsecContainerAcls(InputDomainDN, DomainShortName)

rr.

ss. Dim x, deletedAce, Dacl, hResult, ace1, RemoveFlags, sd, ace

tt.

uu. hResult = 0

vv.

ww. Set x = GetObject( "LDAP://cn=IP Security,cn=system," & InputDomainDN)

xx. if Err.Number <> 0 Then

yy. MsgBox "Container cn=IP Security,cn=system," & InputDomainDN & "does not exist"& " Error :" & Err.Number, vbCritical

zz. hResult = 1

aaa. End If

bbb.

ccc. Set sd = x.Get("nTSecurityDescriptor")

ddd. if Err.Number <> 0 Then

eee. MsgBox "Could not get nTSecurityDescriptor for cn=IP Security,cn=system," & InputDomainDN & " Error :" & Err.Number, vbCritical

fff. hResult = 2

ggg. End If

hhh.

iii. deletedAce = FALSE

jjj.

kkk. Set Dacl = sd.DiscretionaryAcl

lll.

mmm. For Each ace In Dacl

nnn. If(ace.AceType = ADS_ACETYPE_ACCESS_ALLOWED) Then

ooo. if (ace.AceFlags = 0) Then

ppp. if(ace.Trustee = "NT AUTHORITY\Authenticated Users") Then

qqq. RemoveFlags = 0

rrr. RemoveFlags = ADS_RIGHT_DS_READ_PROP Or ADS_RIGHT_DS_LIST Or ADS_RIGHT_DS_LIST_OBJECT Or READ_CONTROL

sss. If (ace.AccessMask AND RemoveFlags) <> 0 Then

ttt. Dacl.RemoveAce ace

uuu. deletedAce = TRUE

vvv. End If

www. End If

xxx. End If

yyy. End If

zzz. Next

aaaa.

bbbb. if deletedAce = FALSE Then

cccc. MsgBox "ACE to remove rights for Authenticated users was not found or was changed from the default installation", vbExclamation

dddd. End If

eeee.

ffff. set ace1 = CreateObject("AccessControlEntry")

gggg.

hhhh. ace1.AceType = ADS_ACETYPE_ACCESS_ALLOWED

iiii. ace1.AceFlags = CONTAINER_INHERIT_ACE OR OBJECT_INHERIT_ACE

jjjj. ace1.AccessMask = ADS_RIGHT_DS_READ_PROP OR ADS_RIGHT_DS_LIST OR ADS_RIGHT_DS_LIST_OBJECT OR READ_CONTROL

kkkk. ace1.Trustee = DomainShortName &"\Domain Computers"

llll.

mmmm. Dacl.AddAce ace1

nnnn. sd.DiscretionaryAcl = Dacl

oooo.

pppp. x.Put "nTSecurityDescriptor", Array(sd)

qqqq. x.SetInfo

rrrr. if Err.Number <> 0 Then

ssss. MsgBox "There was an Error Adding ACls for cn=IP Security"& " Error :" & Err.Number, vbCritical

tttt. hResult = 2

uuuu. End If

vvvv.

wwww. FixupIpsecContainerAcls = hResult

xxxx.

yyyy. End Function

zzzz.

aaaaa.

bbbbb. '**************************************************

ccccc. '* Clear the ACLs for all the IPSec objects so that

ddddd. '* they inherit Acls from their container.

eeeee. '*

fffff. '* 0 -> container exists, Success

ggggg. '* 1 -> other failures

hhhhh. '*

iiiii. '**************************************************

jjjjj. Function ClearIpsecObjectAcls(InputDomainDN)

kkkkk.

lllll. Dim ZeroAcl, obj(22), i

mmmmm.

nnnnn. set ZeroAcl = CreateObject("AccessControlList")

ooooo.

ppppp. obj(0) ="LDAP://cn=ipsecPolicy{72385230-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

qqqqq. obj(1) ="LDAP://cn=ipsecISAKMPPolicy{72385231-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

rrrrr. obj(2) ="LDAP://cn=ipsecNFA{72385232-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

sssss. obj(3) ="LDAP://cn=ipsecNFA{59319BE2-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

ttttt. obj(4) ="LDAP://cn=ipsecNFA{594272E2-071D-11D3-AD22-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

uuuuu. obj(5) ="LDAP://cn=ipsecNFA{6A1F5C6F-72B7-11D2-ACF0-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

vvvvv. obj(6) ="LDAP://cn=ipsecPolicy{72385236-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

wwwww. obj(7) ="LDAP://cn=ipsecISAKMPPolicy{72385237-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

xxxxx. obj(8) ="LDAP://cn=ipsecNFA{59319C04-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

yyyyy. obj(9) ="LDAP://cn=ipsecPolicy{7238523C-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

zzzzz. obj(10) ="LDAP://cn=ipsecISAKMPPolicy{7238523D-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

aaaaaa. obj(11) ="LDAP://cn=ipsecNFA{7238523E-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

bbbbbb. obj(12) ="LDAP://cn=ipsecNFA{59319BF3-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

cccccc. obj(13) ="LDAP://cn=ipsecNFA{594272FD-071D-11D3-AD22-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

dddddd. obj(14) ="LDAP://cn=ipsecNegotiationPolicy{59319BDF-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

eeeeee. obj(15) ="LDAP://cn=ipsecNegotiationPolicy{59319BF0-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

ffffff. obj(16) ="LDAP://cn=ipsecNegotiationPolicy{59319C01-5EE3-11D2-ACE8-0060B0ECCA17},cn=IP Security,cn=system," & InputDomainDN

gggggg. obj(17) ="LDAP://cn=ipsecNegotiationPolicy{72385233-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

hhhhhh. obj(18) ="LDAP://cn=ipsecNegotiationPolicy{7238523F-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

iiiiii. obj(19) ="LDAP://cn=ipsecNegotiationPolicy{7238523B-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

jjjjjj. obj(20) ="LDAP://cn=ipsecFilter{7238523A-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

kkkkkk. obj(21) ="LDAP://cn=ipsecFilter{72385235-70FA-11D1-864C-14A300000000},cn=IP Security,cn=system," & InputDomainDN

llllll.

mmmmmm. for i=0 To 21

nnnnnn. Dim x, hResult, sd, ace

oooooo.

pppppp. hResult = 0

qqqqqq.

rrrrrr. set x = GetObject(obj(i))

ssssss. if Err.Number = 0 Then

tttttt. Set sd = x.Get("nTSecurityDescriptor")

uuuuuu. if Err.Number <> 0 Then

vvvvvv. MsgBox "nTSecurityDescriptor could not be retrieved from " & obj(i) & " Error :" & Err.Number, vbCritical

wwwwww. hResult = 1

xxxxxx. End If

yyyyyy.

zzzzzz. set ZeroAcl = sd.DiscretionaryAcl

aaaaaaa. For Each ace In ZeroAcl

bbbbbbb. ZeroAcl.RemoveAce ace

ccccccc. Next

ddddddd. sd.DiscretionaryAcl = ZeroAcl

eeeeeee. x.Put "nTSecurityDescriptor", Array(sd)

fffffff. x.SetInfo

ggggggg. if Err.Number <> 0 Then

hhhhhhh. MsgBox "Error setting Acls for " & obj(i) & " Error :" & Err.Number, vbCritical

iiiiiii. hResult = 1

jjjjjjj. End If

kkkkkkk. End If

lllllll. Next

mmmmmmm.

nnnnnnn. ClearIpsecObjectAcls = hResult

ooooooo.

ppppppp. End Function

qqqqqqq.

rrrrrrr. Function FixupSchemaObjectAcls(InputDomainDN)

sssssss.

ttttttt. Dim ZeroAcl, obj(6), i, container, deletedDAAce, deletedSYAce, RemoveFlags, deletedAUAce, hResult

uuuuuuu.

vvvvvvv. hResult = 0

wwwwwww.

xxxxxxx. set ZeroAcl = CreateObject("AccessControlList")

yyyyyyy.

zzzzzzz. container = "CN=Schema, CN=Configuration," & InputDomainDN

aaaaaaaa. obj(0) ="LDAP://cn=Ipsec-Base," & container

bbbbbbbb. obj(1) ="LDAP://cn=Ipsec-Filter," & container

cccccccc. obj(2) ="LDAP://cn=Ipsec-ISAKMP-Policy," & container

dddddddd. obj(3) ="LDAP://cn=Ipsec-Negotiation-Policy," & container

eeeeeeee. obj(4) ="LDAP://cn=Ipsec-NFA," & container

ffffffff. obj(5) ="LDAP://cn=Ipsec-Policy," & container

gggggggg.

hhhhhhhh. for i = 0 To 5

iiiiiiii.

jjjjjjjj. Dim x, sd

kkkkkkkk.

llllllll. Set x = GetObject( obj(i) )

mmmmmmmm. if Err.Number <> 0 Then

nnnnnnnn. MsgBox obj(i) & "does not exist"& " Error :" & Err.Number, vbCritical

oooooooo. hResult = 1

pppppppp. Else

qqqqqqqq. sd = x.Get("defaultSecurityDescriptor")

rrrrrrrr. x.Put "defaultSecurityDescriptor", "D:"

ssssssss. x.SetInfo

tttttttt. if Err.Number <> 0 Then

uuuuuuuu. MsgBox "Error setting defaultsecurityDescriptor for " & obj(i) & " Error :" & Err.Number, vbCritical

vvvvvvvv. hResult = 1

wwwwwwww. End If

xxxxxxxx. End If

yyyyyyyy. Next

zzzzzzzz.

aaaaaaaaa. FixupSchemaObjectAcls = hResult

bbbbbbbbb.

ccccccccc. End Function

ddddddddd.

eeeeeeeee. Function IsSchemaUpdateAllowed()

fffffffff.

ggggggggg. Dim WshShell, bKey

hhhhhhhhh.

iiiiiiiii. Set WshShell = WScript.CreateObject("WScript.Shell")

jjjjjjjjj. bKey = WshShell.RegRead("HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Schema Update Allowed")

kkkkkkkkk.

lllllllll. IsSchemaUpdateAllowed = bKey

mmmmmmmmm.

nnnnnnnnn. End Function

ooooooooo.

ppppppppp. '**************************************************

qqqqqqqqq. '* MAIN

rrrrrrrrr. '**************************************************

sssssssss.

ttttttttt. Dim Info

uuuuuuuuu. Dim dnsName, domainDN

vvvvvvvvv. Dim regOk, retVal1, retVal2, retVal3

wwwwwwwww.

xxxxxxxxx. MsgBox "This script attempts to correct the ACLs of IP Security related objects and schema in Active Directory. For this, it needs Schema changes be allowed through a registry Key."& vbcrlf & "Please read Q<KB #> and Q216060 for more details."

yyyyyyyyy.

zzzzzzzzz. regOk = IsSchemaUpdateAllowed()

aaaaaaaaaa. if regOk <> 1 Then

bbbbbbbbbb. MsgBox "The registry key to allow schema updates is not set. Please read Q216060 for more information." & vbcrlf & "The Script will stop processing. The ACL corrections have not been made.", vbExclamation

cccccccccc. Else

dddddddddd. Set Info = CreateObject("AdSystemInfo")

eeeeeeeeee. dnsName = Info.DomainDNSName

ffffffffff. domainDN = Replace(dnsName, ".", ",dc=")

gggggggggg. domainDN = "dc=" & domainDN

hhhhhhhhhh.

iiiiiiiiii. retVal1 = FixupIpsecContainerAcls(domainDN, Info.DomainShortName)

jjjjjjjjjj.

kkkkkkkkkk. retVal2 = ClearIpsecObjectAcls(domainDN)

llllllllll.

mmmmmmmmmm. retVal3 = FixupSchemaObjectAcls(domainDN)

nnnnnnnnnn.

oooooooooo. If retVal1 = 0 And retVal2 = 0 And retVal3 = 0 Then

pppppppppp. MsgBox "The ACL corrections for IPSec Objects on domain " & Info.DomainShortName & " Completed successfully." & vbcrlf & "You may now reset/delete the registry key that allows schema updates per Q216060"

qqqqqqqqqq. Else

rrrrrrrrrr. MsgBox "The ACL corrections for IPSec Objects on domain " & Info.DomainShortName & " Completed with some errors" & vbcrlf & "If you are not going to retry the operation, you may reset/delete the registry key that allows schema updates per Q216060"

ssssssssss. End If

tttttttttt. End If

uuuuuuuuuu. On the File menu, click Save, click All Files in the Save as type box, type ipsec_acl_fix.vbs in the File name box, specify a location where you want to save the file, and then click Save.

vvvvvvvvvv. Quite Notepad.

wwwwwwwwww. Run the Ipsec_acl_fix.vbs script from a command prompt.

Delete the Schema Update Allowed registry value that you created in step 1.

Note You can also use this procedure to modify the permissions of the IPSec policy of a Windows Server 2003 domain that you upgraded from Windows 2000, in the situation where the permissions of policy objects were not modified during the upgrade operation.

After the above script run the adprep /forestprep it should come up without any errors.

· Install new Windows 2008 Server and join to our Production Domain Controller as Member Server

· Transfer the Active Directory & DNS role to the new Windows 2008 Server and bring the new Windows 2008 server as one of the new Directory server.

· Once we checked & confirmed that the new Domain controller server is good we can start the other server.

· Back out plan – if the server didn’t work as we expected, bring down the server. There is no impact/outage to any of the services as we have two domain controllers and we are migrating only one Domain controller at a time.

· Once the new domain controller is migrated all the users will be authenticated via Windows 2008

VMware Setup

2011-03-02T11:29:00.001-08:00

Planning to setup VMware vSphere 4.1 environment.

Too many open files, redhat 7.2 Server: cannot open shared libraries

2010-11-03T15:17:00.000-07:00

Had an issue with Redhat 7.2 Linux Server, when I tried to login to the server this morning,I was getting an error " Unable to open shared libraries", turned out that the system had too many open files. I was able to login to the server as root but couldn't increase number of open files through ulimit command. This is a production server and was not in a position to reboot it. Increased the limit using : echo "10000" > /proc/sys/fs/max-file and everything started working fine..