THE UNIX PROMPT # blog

EDAC: Which DIMM?

2011-03-01T13:19:00.000+11:00

EDAC (Error Detection and Correction) messages are designed to provide information about hardware problems with the system memory. Some of these messages are correctable, and some are uncorrectable.
EDAC is documented at http://www.kernel.org/doc/Documentation/edac.txt

Our HP hardware running RHEL5 , We often get DIMMs in our servers going bad with the following errors in syslog:

 EDAC k8 MC1: general bus error: participating processor(local node origin), time-out(no timeout)

 memory transaction type(generic read), mem or i/o(mem access), cache level(generic)

 EDAC MC1: CE page 0x103ca78, offset 0xf88, grain 8, syndrome 0x9f65, row 1, channel 0, label "":

 k8_edac

 EDAC MC1: CE - no information available: k8_edac Error Overflow set

 EDAC k8 MC1: extended error code: ECC chipkill x4 error

EDAC is Error Detection and Correction, it will try to detect and correct hardware problems. In this case it appears that chipkill is detecting the problem and correcting it. Any significant hardware problems may not be experienced in the short term; however, it is recommend to have the DIMMs checked and replace the faulty one.

To locate which DIMM is having the issue, going back to the EDAC errors above I saw on my server's console, MC1 (Memory Controller 1) means CPU1, row 1 is referred to as csrow1 (Chip-Select Row 1) in the Linux EDAC documentation, and channel 0 means memory channel 0.

EDAC MC1: CE page 0x103ca78, offset 0xf88, grain 8, syndrome 0x9f65, row 1, channel 0, label "": k8_edac

            Channel 0       Channel 1

    ===================================

    csrow0  | DIMM_A0       | DIMM_B0 |

    csrow1  | DIMM_A0       | DIMM_B0 |

    ===================================

    ===================================

    csrow2  | DIMM_A1       | DIMM_B1 |

    csrow3  | DIMM_A1       | DIMM_B1 |

    ===================================

Of course, there are actually two DIMM slots called DIMMA0 on my server (one for each CPU), but again the MC1 error corresponds to CPU1, which is listed under "Bank Locator" in the output of dmidecode:

 # dmidecode -t memory | grep DIMMA0 -B9 -A8

 Handle 0x002E, DMI type 17, 27 bytes.

  Memory Device

        Array Handle: 0x002B

        Error Information Handle: Not Provided

        Total Width: 72 bits

        Data Width: 64 bits

        Size: 4096 MB

        Form Factor: DIMM

        Set: None

        Locator: DIMMA0

        Bank Locator: CPU0

        Type: DDR2

        Type Detail: Synchronous

        Speed: 533 MHz (1.9 ns)

        Manufacturer:  

        Serial Number:  

        Asset Tag:  

        Part Number:  

 --

 Handle 0x003E, DMI type 17, 27 bytes.

 Memory Device

        Array Handle: 0x002B

        Error Information Handle: Not Provided

        Total Width: 72 bits

        Data Width: 64 bits

        Size: 4096 MB

        Form Factor: DIMM

        Set: None

        Locator: DIMMA0

        Bank Locator: CPU1

        Type: DDR2

        Type Detail: Synchronous

        Speed: 533 MHz (1.9 ns)

        Manufacturer:  

        Serial Number:  

        Asset Tag:  

        Part Number:

(On my workstation, dmidecode actually shows the Part Number and Serial Number for my DIMMs, which is very useful.)
In addition to looking at errors on the console and in logs, you can also see errors per MC/CPU, row/csrow, and channel by examining /sys/devices/system/edac. In my case the errors were only on MC1, csrow1, channel 0:

# grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count

 /sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow0/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow1/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow2/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow2/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow3/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow3/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow4/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow4/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow5/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow5/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow6/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow6/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow7/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc0/csrow7/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0 

 /sys/devices/system/edac/mc/mc1/csrow0/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:6941652

 /sys/devices/system/edac/mc/mc1/csrow1/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow2/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow2/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow3/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow3/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow4/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow4/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow5/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow5/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow6/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow6/ch1_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow7/ch0_ce_count:0

 /sys/devices/system/edac/mc/mc1/csrow7/ch1_ce_count:0

java: Alert on java crashes

2010-11-29T09:00:00.002+11:00

The last couple of days I have noticed a number of irregular core dumps created in my system core file dump location /var/core

 -rw-------   1 root    root  2529790203 Nov 10 11:55 core_host1_java_1094_300_1289350401_28578

 -rw-------   1 root    root  2564932547 Nov 15 13:06 core_host1_java_1094_300_1289786684_1664

 -rw-------   1 root    root  2498732827 Nov 17 17:29 core_host1_java_9092_300_1289975232_5664

 -rw-------   1 root    root  2525420387 Nov 19 12:08 core_host3_java_1094_300_1290128885_16234

Depending on how you've setup your core file dump pattern, you can determine which process/application user account its comming from by just reading the file core name. eg

 # coreadm|grep pattern

    global core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p

    init core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p

 %n ; system node name uname -n

 %f ; executable filename

 %u ; uid

 %g ; gid

 %t ; time in seconds since 1970,1,1.

 %p ; PID

My core dump process is coming from a java process. Bugs can occur in a Java runtime environment and most administrators would want to get notified.
If you need to take a corrective action and diagnose further, you will need to be alerted at the time of incident.
The Java runtime has a number of useful options that can be used for this purpose. The first option is “-XX:OnOutOfMemoryError”, which allows a command to be run when the runtime environment incurs an out of memory condition. When this option is combined with the logger command line utility:

 java -XX:OnOutOfMemoryError=”logger Java process %p encountered an OOM condition” …

Syslog entries will be generated each time an Out Of memory (OOM) event occurs.

Another useful option is “-XX:OnError”, which allows a command to be run when the runtime environment incurs a fatal error (i.e., a hard crash). When this option is combined with the logger utility:

 java -XX:OnError=”logger -p Java process %p encountered a fatal condition” …

Syslog entries will be generated when a fatal event occur.

The options above allow you to run one or more commands when these errors are encountered, so you could chain together a utility (logger or mail) to generate alerts, and maybe a restarter script to start a new Java process.

zones: Physical to Virtual (P2V) Migration

2010-11-22T09:00:00.000+11:00

Since the release of Solaris 10 Update 9 I was interested in trying out its new capabilities. One of which was to move from an existing Oracle Solaris 10 Physical system to a virtual container quickly and easily on a separate system. And also the Host ID migration feature which it also allows.

In this post I will provide a demonstration of P2V , migrating a physical Solaris system into a zone, using Host ID migration, keeping in mind in this example both systems are built from a Solaris 10 Update 9 release and is a ZFS on root build.

1. Collect the system information you wish to be migrated into a zone.
Obtain the hostname:

 # hostname

Obtain the hostid:

 # hostid

Obtain the root passwd
View what software being run on the system if necessary
Check the network configuration on the system:

 # ifconfig -a    

View the storage utilized for example the contents of /etc/vfstab
View the amount of local disk storage in use, and determine the target host has enough space for the install
Examine the /etc/system of any specific or unique changes you may wish to keep or change once the environment has been virtualized.

2. Use the flarcreate command to create a flash archive image of the system you wish to migrate.
Login as root to the source system and change to the root directory.

 # cd /

Run the flarcreate using the following options:
Note: Here I have used the option to compress using the -c , used the archived method with cpio, you can use pax if you wish, then supplied the content name identifier of the archive using the -n option (best practice s10u9-system-name), also I have used the -y option because this system has a separate /var dataset and is ZFS on root based, then supplied the path destination of the flar archive creation.
When the flar creation is running be sure to monitor and review any errors thoroughly.

 # flarcreate -c -L cpio -n name -y /var /path/to/flar

3. Transfer the flar archive created to the destination target host.

4. Create a new zone configuration on the target host:
Note: I don't want to inherit any packages from the global or any loop-back file systems, so I have created a whole root zone configuration and provided a new ip-address as I don't want it to conflict with the existing migrated host as its still alive.

Also you will see I have added a hostid entry in the zone configuration, when applications are migrated from a physical Solaris system into a zone on a new system, the hostid changes to be the hostid of the new machine. In some cases, applications depend on the original hostid, and it is not possible to update the application configuration. In these cases, the zone can be configured to use the hostid of the original system. This is done by setting a zonecfg property to specify the hostid as shown below. The value used should be the output of the hostid command as run on the original system previously.

 # zonecfg -z hostname

 hostname: No such zone configured

 Use 'create' to begin configuring a new zone.

 zonecfg:hostname> create -b

 zonecfg:hostname> set autoboot=true

 zonecfg:hostname> set zonepath=/zones/hostname

 zonecfg:hostname> set bootargs="-m verbose"

 zonecfg:hostname> set hostid=84###375

 zonecfg:hostname> add net

 zonecfg:hostname:net> set physical=bge0

 zonecfg:hostname:net> set address=ip-address

 zonecfg:hostname:net> end

 zonecfg:hostname> verify

 zonecfg:hostname> commit

 zonecfg:hostname> exit

5. Installing the zone on the target system using the flar archive created. Become root user and install the configured zone using the install -a option and the path to the flar archive created. Noticed I have used the -p option becuase I want to preserve the system identity, the zone will have the same identity as the system used to create the image. You can use the -u to sys-unconfig the zone.
Best practice make sure you tail the zone installation log file and ensure no errors are found.

 # zoneadm -z hostname install -p -a /path/to/flar

 A ZFS file system has been created for this zone.

       Log File: /var/tmp/hostname.install_log.IQaGnI

     Installing: This may take several minutes...

 Postprocessing: This may take a while...

    Postprocess: Updating the zone software to match the global zone...

    Postprocess: Zone software update complete

    Postprocess: Updating the image to run within a zone

         Result: Installation completed successfully.

       Log File: /zones/hostname/root/var/log/hostname.install17462.log

6. Boot the zone into single user mode and login via the console and of course use the root password from your migrated host. Make any necessary checks.

 # zoneadm -z hostname boot -s

 # zlogin -C hostname 

From here you can see the P2V is complete, hostname and data has been kept intact including the host-id.
Now you can decide whether you need to make any further necessary changes such as hostname, network configuration etc etc. And of course be prepared to boot the migrated zone for a live production environment.

news: Oracle Solaris 11 Express Download Available

2010-11-17T13:49:00.001+11:00

Solaris 11 Express 2010.11 is now available for download .
You will also find an overview and documentation available at the following link:
Click Here:

Solaris 11 Express now allows administrators to test and deploy within their enterprise environments and greatly simplify their day to day operations. It contains many different technology innovations that are not available in Oracle Solaris 10 such as new package management tools and utilities, built-in network virtualization, and support for the latest hardware platforms. To see the list of features that
are new to Oracle Solaris 11 Express please Clicke Here:

Oracle Solaris 11 Express is the latest release of the OracleSolaris operating system. This release is the path forward for developers, end-users and partners using previous generations of OpenSolaris releases.
This release will provide administrators to access the latest technology and innovation that will form
a future Oracle Solaris 11 which will be released sometime in 2011.

Here is a youtube video tutorial if you wish to test it out in your virtualbox setup. Enjoy
http://www.youtube.com/watch?v=r5hlrqlQAIc

inetd: Disable inetd Connection Logging for individual Services

2010-11-15T09:00:00.006+11:00

Noticed a large number of unwanted constant connection messages in my system messages file.

 Aug 31 18:36:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19080] from ip-address 45632

 Aug 31 18:36:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19081] from ip-address 45633

 Aug 31 18:40:35 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19288] from ip-address  48640

 Aug 31 18:40:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19290] from ip-address 48641

 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19333] from ip-address  48653

 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19333] from ip-address 48653

 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19334] from ip-address 48654

 Aug 31 18:45:51 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19543] from ip-address 48714

 Aug 31 18:45:52 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19544] from ip-address 48715

 Aug 31 18:50:09 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19781] from ip-address 48786

 Aug 31 18:50:09 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19782] from ip-address 48787

 Aug 31 18:57:59 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[24199] from ip-address 48871

The above is coming from my Veritas Netbackup network connection daemon which is constantly filling up my messages file during its nightly backup procedure.

If inetd is running, the "tracing" feature can be used to log information about the source of any network connections seen by the daemon. Rather than disabling inetd tracing for all services, the administrator has the option of disabling tracing for individual services with inetadm -m svcname tcp_trace=FALSE , where is the name of the specific service that should use tracing.

1. The following command will display the properties for the vnetd service.

 # inetadm -l svc:/network/vnetd/tcp:default

 SCOPE    NAME=VALUE

          name="vnetd"

          endpoint_type="stream"

          proto="tcp"

          isrpc=FALSE

          wait=FALSE

          exec="/usr/openv/bin/vnetd"

          user="root"

 default  bind_addr=""

 default  bind_fail_max=-1

 default  bind_fail_interval=-1

 default  max_con_rate=-1

 default  max_copies=-1

 default  con_rate_offline=-1

 default  failrate_cnt=40

 default  failrate_interval=60

 default  inherit_env=TRUE

 default  tcp_trace=TRUE

 default  tcp_wrappers=TRUE

 default  connection_backlog=10

2. The following command will disable tracing for the vnetd service

 # inetadm -m svc:/network/vnetd/tcp:default tcp_trace=FALSE

3. Confirm the changes using the display option again.

  
 # inetadm -l svc:/network/vnetd/tcp:default

 SCOPE    NAME=VALUE

          name="vnetd"

          endpoint_type="stream"

          proto="tcp"

          isrpc=FALSE

          wait=FALSE

          exec="/usr/openv/bin/vnetd"

          user="root"

 default  bind_addr=""

 default  bind_fail_max=-1

 default  bind_fail_interval=-1

 default  max_con_rate=-1

 default  max_copies=-1

 default  con_rate_offline=-1

 default  failrate_cnt=40

 default  failrate_interval=60

 default  inherit_env=TRUE

          tcp_trace=FALSE

 default  tcp_wrappers=TRUE

 default  connection_backlog=10

news: Solaris 11 Express Summit

2010-11-10T13:14:00.000+11:00

The slides are now available for the presentations at the Oracle Solaris 11 Express Summit at the LISA Conference which was hosted on Tuesday Nov 9th.
The event showcased Oracle Solaris 11 Express, targeting System Administrators and Architects.

Here are the slides available for the following presentations:

Slide 1 Introduction to Oracle Solaris 11 Express, Markus Flierl
Slide 2 Image Packaging System, Bart Smaalders
Slide 3 Deploying Oracle Solaris 11 in the Enterprise, Dave Miner
Slide 4 Advances in Solaris Networking with Crossbow and Beyond, Nicolas Droux
Slide 5 Oracle Solaris Containers in Oracle Solaris 11 Express, Dan Price
Slide 6 ZFS Features in Oracle Solaris Express, Cindy Swearingen
Slide 7 New Security Features in Oracle Solaris 11 Express, Glenn Faden
Slide 8 Deploying Applications Using SMF and Other Solaris 11 Features, Liane Praza

The stream of videos have been recorded and can be viewed at the following link: Click here

JASS: Auditing & Controlling Output Logs

2010-11-08T09:00:00.019+11:00

You can configure the Solaris Security Toolkit audit option to report or omit banners and messages.

You might want to eliminate pass messages (JASS_LOG_SUCCESS variable) from the output so you can report and focus only on fail messages (JASS_LOG_FAILURE variable).

If the logging variable is set to 0, then no output is generated for messages of that type. Conversely, if the logging variable is set to 1, then messages are displayed. The default action for each of these variables is to display the output.

All Banner Output, This parameter controls the display of banner messages. These messages are
usually surrounded by separators comprised of either equal sign (“=”) or dash (“-”)
characters.

 JASS_LOG_BANNER

[ERR], This parameter controls the display of error messages. If set to 0, no error messages will be generated.

 JASS_LOG_ERROR

[FAIL] This parameter controls the display of failure messages. If set to 0, no failure messages will be generated.

 JASS_LOG_FAILURE

[NOTE] This parameter controls the display of notice messages. If set to 0, no notice messages will be generated.

 JASS_LOG_NOTICE

[PASS] This parameter controls the display of success or passing status messages. If set to 0, no success messages will be generated.

 JASS_LOG_SUCCESS

[WARN] This parameter controls the display of warning messages. If set to 0, no warning messages will be generated.

 JASS_LOG_WARNING

Using these options is very useful when you only need to view specific messages. By setting these options, you can minimize output, yet still focus on areas you deem critical. For example, by setting all logging variables to 0 except for JASS_LOG_FAILURE (leave it at the default of 1), the audit reports only on failures
generated by the logFailure function.

 # JASS_LOG_FAILURE=1

 # JASS_LOG_FAILURE=1

 # JASS_LOG_NOTICE=0

 # JASS_LOG_SUCCESS=0

 # JASS_LOG_WARNING=0

 # export JASS_LOG_WARNING JASS_LOG_SUCCESS JASS_LOG_NOTICE JASS_LOG_FAILURE

 # ./jass-execute -a secure.driver -V 2

 update-at-deny [FAIL] User test is not listed in

 /etc/cron.d/at.deny.

 update-at-deny [FAIL] Audit Check Total : 1 Error(s)

 update-inetd-conf [FAIL] Service ftp is enabled in

 /etc/inet/inetd.conf.

 update-inetd-conf [FAIL] Service telnet is enabled in

 /etc/inet/inetd.conf.

 update-inetd-conf [FAIL] Service rstatd is enabled in

 /etc/inet/inetd.conf.

 update-inetd-conf [FAIL] Audit Check Total : 3 Error(s)

Here I have provided a Jass Auditing script that can be run on a weekly, monthly , yearly however you may choose. The audit will alert on any system changes via Email set by the MAIL_LIST variable. The script requires Repository directory under /opt/SUNWjass.

jass-audit.sh

 #!/usr/bin/ksh

 HOST=`hostname`

 TIMESTAMP=`date +%H%M.%d%m`

 SPOOL="/opt/SUNWjass"

 L_LOG="$SPOOL/Repository/Jass_Audit.$TIMESTAMP"

 L_OUT="$SPOOL/Repository/Jass_Audit.$TIMESTAMP.OUT"

 MAIL_LIST=""

 JASS_LOG_FAILURE=1

 JASS_LOG_NOTICE=0

 JASS_LOG_SUCCESS=0

 JASS_LOG_WARNING=0

 export JASS_LOG_WARNING JASS_LOG_SUCCESS JASS_LOG_NOTICE JASS_LOG_FAILURE

 $SPOOL/bin/jass-execute -a server-secure.driver -V 2 -o $L_LOG

 if [ -f $L_LOG ]; then

    ERR=`grep FAIL $L_LOG|wc -l`

     if [ $ERR -ne 0 ]; then

      echo "Solaris Security Log: AUDIT (${HOST}) $TIMESTAMP" > $L_OUT

      echo "" >> $L_OUT

      echo "File : "$L_LOG" " >> $L_OUT

      echo "========================================================"  >> $L_OUT

      echo "Failures : " >> $L_OUT

      grep FAIL $L_LOG | egrep -v Error >> $L_OUT

      echo "========================================================"  >> $L_OUT

      tail -12 $L_LOG >> $L_OUT

      mailx -s "Solaris Security Toolkit Log: AUDIT (${HOST})" $MAIL_LIST < $L_OUT

     else

      exit 0

     fi

 fi

Output Example:

 Solaris Security Log: AUDIT (host-name) 1301.0211

 File : /opt/SUNWjass/Repository/Jass_Audit.1301.0211 

 ========================================================

 Failures : 

 update-at-deny                 [FAIL] User test is not listed
in /etc/cron.d/at.deny.

 ========================================================

 server-secure.driver           [SUMMARY] Results Summary for AUDIT
run of server-secure.driver

 server-secure.driver           [SUMMARY] The run completed with a
total of 84 scripts run.

 server-secure.driver           [SUMMARY] There was a Failure  in   1
Script

 server-secure.driver           [SUMMARY] There were  Errors  
in   0 Scripts

 server-secure.driver           [SUMMARY] There was a Warning  in   1
Script

 server-secure.driver           [SUMMARY] There were  Notes   
in  19 Scripts

 server-secure.driver           [SUMMARY] Failure Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-failures.txt

 server-secure.driver           [SUMMARY] Warning Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-warnings.txt

 server-secure.driver           [SUMMARY] Notes Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-notes.txt

Veritas: Displaying the Status of VVR Links

2010-11-01T09:00:00.013+11:00

The status of VVR links can be displayed by running the folloing commands, as root, on the appropriate host.

Log in to the primary host of the relevant VVR link and su to root
To view the status of the link, run the command

 # /usr/sbin/vxrlink -g disk-group-name status rlink-name

Or the following command to update every 5 seconds

 # /usr/sbin/vrlink -g disk-group-name -i 5 status rlink-name

Another option to show link status is to use the following vradmin command. This command displays more information than the previous one, including SRL size, current mode, and the amount of time (if any) that replication is behind.

 # /usr/sbin/vradmin -g disk-group-name repstatus rvg-name

To view the amount of traffic being sent over the network for a VVR Link, run the following command, notice how I use the option "stats" instead of "status" here.

 # /usr/sbin/vrlink -g disk-group-name -i 5 stats rlink-name

JASS: Enable Veritas Netbackup Services in JASS

2010-10-27T09:00:00.031+11:00

Once you have installed JASS you will need to edit specific files so to customize based on individual site needs.

In this case if you have installed Veritas Netbackup and your host is hardened with JASS you will experience issues with Netbackup services and its TCP wrappers.

You will need to tell JASS to enable and open all Netbackup services related deamons and ports.

From the SUNWjass folder you will need to edit the following files:

 ./Files/etc/hosts.allow-server:

 ./Drivers/server-secure.driver:

 ./Drivers/finish.init:

Grant access to the following Netbackup services, add the following lines to JASS ./Files/etc/hosts.allow-server file:

 bpcd:   ALL

 vnetd:  ALL

 vopied: ALL

 bpjava-msvc:    ALL

Update the Finish script and enable the following Netbackup Services. In the ./Drivers/finish.init file locate the JASS_SVCS_ENABLE section and add the following services string in the if statement.

Before:

 if [ -z "${JASS_SVCS_ENABLE}" ]; then

    JASS_SVCS_ENABLE=""

 fi

 export JASS_SVCS_ENABLE/logs/error_log

After:

 if [ -z "${JASS_SVCS_ENABLE}" ]; then

    JASS_SVCS_ENABLE="

       svc:/network/vnetd/tcp:default

       svc:/network/bpjava-msvc/tcp:default

       svc:/network/bpcd/tcp:default

       svc:/network/vopied/tcp:default"

 fi

 export JASS_SVCS_ENABLE

Update the secure-driver script and also enable the following Netbackup services. In the ./Drivers/server-secure.driver locate the JASS_SVCS_ENABLE variable and add the Netbackup deamons and services

Before:

 JASS_SVCS_ENABLE="${JASS_SVCS_ENABLE} dtspc rstatd 100155 

 svc:/network/rpc/rstat:default

 svc:/network/rpc/smserver:default "

After:

 JASS_SVCS_ENABLE="${JASS_SVCS_ENABLE} dtspc rstatd 100155 bpcd bpjava-msvc vnetd vopied

 svc:/network/rpc/rstat:default

 svc:/network/rpc/smserver:default

 svc:/network/vnetd/tcp:default

 svc:/network/bpjava-msvc/tcp:default

 svc:/network/bpcd/tcp:default

 svc:/network/vopied/tcp:default "

Run JASS hardening to complete the configuration changes.

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

veritas: Disk mirroring using Veritas Volume Manager

2010-10-25T09:00:00.014+11:00

This paper will present a short introduction to mirroring two disks using Veritas Volume Manager (VxVM)

In this case you will see that two separate set of Storage Array disks can be mirrored across a single disk group without interruptions.

The following vxprint shows the contents of disk group dgprdnfs. It has a number of small sets of disks and from a single controller/storage c3's . The disks a set across three sets of concatenated volumes volprdnfs1 volprdnfs2 & volprdnfs3.

 # vxprint -qhtg dgprdnfs

 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c3t0d52      c3t0d52s2    auto     2048     53028096 -

 dm c3t0d53      c3t0d53s2    auto     2048     53028096 -

 dm c3t0d54      c3t0d54s2    auto     2048     53028096 -

 dm c3t0d55      c3t0d55s2    auto     2048     53028096 -

 dm c3t0d122     c3t0d122s2   auto     2048     53026176 -

 dm c3t0d123     c3t0d123s2   auto     2048     53026176 -

 dm c3t0d124     c3t0d124s2   auto     2048     53026176 -

 dm c3t0d125     c3t0d125s2   auto     2048     53026176 -

 dm c3t0d126     c3t0d126s2   auto     2048     53026176 -

 dm c3t0d127     c3t0d127s2   auto     2048     53026176 -

 dm c3t0d128     c3t0d128s2   auto     2048     53026176 -

 dm c3t0d129     c3t0d129s2   auto     2048     53026176 -

 dm c3t0d130     c3t0d130s2   auto     2048     53026176 -

 dm c3t0d131     c3t0d131s2   auto     2048     53026176 -

 dm c3t0d132     c3t0d132s2   auto     2048     53026176 -

 dm c3t0d133     c3t0d133s2   auto     2048     53026176 -

 dm c3t0d203     c3t0d203s2   auto     2048     53026176 -

 dm dgprdnfs01   c3t0d12s2    auto     2048     53028096 -

 dm dgprdnfs02   c3t0d13s2    auto     2048     53028096 -

 dm dgprdnfs03   c3t0d14s2    auto     2048     53028096 -

 dm dgprdnfs04   c3t0d18s2    auto     2048     53028096 NOHOTUSE

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen

 pl volprdnfs1-01 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW

 sd dgprdnfs01-01 volprdnfs1-01 dgprdnfs01 0    53028096 0         c3t0d12  ENA

 sd dgprdnfs02-01 volprdnfs1-01 dgprdnfs02 0    9886464  53028096  c3t0d13  ENA

 sd dgprdnfs02-03 volprdnfs1-01 dgprdnfs02 19772928 33255168 62914560 c3t0d13 ENA

 sd dgprdnfs04-01 volprdnfs1-01 dgprdnfs04 0    39544576 96169728  c3t0d18  ENA

 sd c3t0d52-01   volprdnfs1-01 c3t0d52 0        53028096 135714304 c3t0d52  ENA

 sd c3t0d53-01   volprdnfs1-01 c3t0d53 0        53028096 188742400 c3t0d53  ENA

 sd c3t0d54-01   volprdnfs1-01 c3t0d54 0        53028096 241770496 c3t0d54  ENA

 sd c3t0d55-01   volprdnfs1-01 c3t0d55 0        50630912 294798592 c3t0d55  ENA

 sd c3t0d126-02  volprdnfs1-01 c3t0d126 53024   48247168 345429504 c3t0d126 ENA

 sd c3t0d127-01  volprdnfs1-01 c3t0d127 0       53026176 393676672 c3t0d127 ENA

 sd c3t0d128-01  volprdnfs1-01 c3t0d128 0       53026176 446702848 c3t0d128 ENA

 sd c3t0d129-01  volprdnfs1-01 c3t0d129 0       53026176 499729024 c3t0d129 ENA

 sd c3t0d130-01  volprdnfs1-01 c3t0d130 0       53026176 552755200 c3t0d130 ENA

 sd c3t0d131-01  volprdnfs1-01 c3t0d131 0       53026176 605781376 c3t0d131 ENA

 sd c3t0d132-01  volprdnfs1-01 c3t0d132 0       53026176 658807552 c3t0d132 ENA

 sd c3t0d133-01  volprdnfs1-01 c3t0d133 0       53026176 711833728 c3t0d133 ENA

 sd dgprdnfs04-02 volprdnfs1-01 dgprdnfs04 39544576 13483520 764859904 c3t0d18 ENA

 sd c3t0d55-02   volprdnfs1-01 c3t0d55 50630912 1898016  778343424 c3t0d55  ENA

 sd c3t0d123-03  volprdnfs1-01 c3t0d123 132064  52894112 780241440 c3t0d123 ENA

 sd c3t0d124-03  volprdnfs1-01 c3t0d124 130320  52895856 833135552 c3t0d124 ENA

 sd c3t0d125-02  volprdnfs1-01 c3t0d125 65168   52961008 886031408 c3t0d125 ENA

 sd c3t0d126-03  volprdnfs1-01 c3t0d126 48300192 4725984 938992416 c3t0d126 ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen

 pl volprdnfs2-01 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW

 sd dgprdnfs02-02 volprdnfs2-01 dgprdnfs02 9886464 9886464 0       c3t0d13  ENA

 sd dgprdnfs03-01 volprdnfs2-01 dgprdnfs03 0    53028096 9886464   c3t0d14  ENA

 sd c3t0d203-01  volprdnfs2-01 c3t0d203 0       52428800 62914560  c3t0d203 ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen

 pl volprdnfs3-01 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW

 sd c3t0d122-01  volprdnfs3-01 c3t0d122 0       53026176 0         c3t0d122 ENA

To start the mirroring process I have been given a set of of new disks. These sets of disks need to be equal or large enough to complete the mirroring process. Notice how these disks are on a separate controller/storage c4's.

 # vxdisk -o alldgs list|grep c4

 c4t2d1s2     auto:none       -            -            online invalid

 c4t2d2s2     auto:none       -            -            online invalid

 c4t2d3s2     auto:none       -            -            online invalid

Add new disks for mirroring to the same disk group as source.

 # vxdg -g dgprdnfs adddisk c4t2d1=c4t2d1s2

 # vxdg -g dgprdnfs adddisk c4t2d2=c4t2d2s2

 # vxdg -g dgprdnfs adddisk c4t2d3=c4t2d3s2

 # vxdisk -o alldgs list|grep c4

 c4t2d1s2     auto:cdsdisk    c4t2d1       dgprdnfs     online

 c4t2d2s2     auto:cdsdisk    c4t2d2       dgprdnfs     online

 c4t2d3s2     auto:cdsdisk    c4t2d3       dgprdnfs     online

Execute and Create mirror specifying the new disks to mirror each volume.

 # vxassist -g dgprdnfs -b mirror volprdnfs1 alloc=c4t2d1

 # vxassist -g dgprdnfs -b mirror volprdnfs2 alloc=c4t2d2

 # vxassist -g dgprdnfs -b mirror volprdnfs3 alloc=c4t2d3

After the above steps the following vxprint shows the new set of c4 disks attached to dgprdnfs disk group. Notice the sectored size of each disk, which must equal or be larger than the total volume size will are mirroring.

With the above vxassist command along with the mirror option we have executed the mirroring process, notice it has created a new plex for each volume volprdnfs1-02 volprdnfs2-02 volprdnfs3-02, each new plex is in TEMPRMSD state , which means an attached new temporary plex state, its copying the volume contents to the existing new plex, it wont be considered enabled once the process has completed.

 # vxprint -qhtg dgprdnfs

 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c3t0d52      c3t0d52s2    auto     2048     53028096 -

 dm c3t0d53      c3t0d53s2    auto     2048     53028096 -

 dm c3t0d54      c3t0d54s2    auto     2048     53028096 -

 dm c3t0d55      c3t0d55s2    auto     2048     53028096 -

 dm c3t0d122     c3t0d122s2   auto     2048     53026176 -

 dm c3t0d123     c3t0d123s2   auto     2048     53026176 -

 dm c3t0d124     c3t0d124s2   auto     2048     53026176 -

 dm c3t0d125     c3t0d125s2   auto     2048     53026176 -

 dm c3t0d126     c3t0d126s2   auto     2048     53026176 -

 dm c3t0d127     c3t0d127s2   auto     2048     53026176 -

 dm c3t0d128     c3t0d128s2   auto     2048     53026176 -

 dm c3t0d129     c3t0d129s2   auto     2048     53026176 -

 dm c3t0d130     c3t0d130s2   auto     2048     53026176 -

 dm c3t0d131     c3t0d131s2   auto     2048     53026176 -

 dm c3t0d132     c3t0d132s2   auto     2048     53026176 -

 dm c3t0d133     c3t0d133s2   auto     2048     53026176 -

 dm c3t0d203     c3t0d203s2   auto     2048     53026176 -

 dm c4t2d1       c4t2d1s2     auto     2048     964654848 -

 dm c4t2d2       c4t2d2s2     auto     2048     125821696 -

 dm c4t2d3       c4t2d3s2     auto     2048     62909696 -

 dm dgprdnfs01   c3t0d12s2    auto     2048     53028096 -

 dm dgprdnfs02   c3t0d13s2    auto     2048     53028096 -

 dm dgprdnfs03   c3t0d14s2    auto     2048     53028096 -

 dm dgprdnfs04   c3t0d18s2    auto     2048     53028096 NOHOTUSE

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen

 pl volprdnfs1-01 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW

 sd dgprdnfs01-01 volprdnfs1-01 dgprdnfs01 0    53028096 0         c3t0d12  ENA

 sd dgprdnfs02-01 volprdnfs1-01 dgprdnfs02 0    9886464  53028096  c3t0d13  ENA

 sd dgprdnfs02-03 volprdnfs1-01 dgprdnfs02 19772928 33255168 62914560 c3t0d13 ENA

 sd dgprdnfs04-01 volprdnfs1-01 dgprdnfs04 0    39544576 96169728  c3t0d18  ENA

 sd c3t0d52-01   volprdnfs1-01 c3t0d52 0        53028096 135714304 c3t0d52  ENA

 sd c3t0d53-01   volprdnfs1-01 c3t0d53 0        53028096 188742400 c3t0d53  ENA

 sd c3t0d54-01   volprdnfs1-01 c3t0d54 0        53028096 241770496 c3t0d54  ENA

 sd c3t0d55-01   volprdnfs1-01 c3t0d55 0        50630912 294798592 c3t0d55  ENA

 sd c3t0d126-02  volprdnfs1-01 c3t0d126 53024   48247168 345429504 c3t0d126 ENA

 sd c3t0d127-01  volprdnfs1-01 c3t0d127 0       53026176 393676672 c3t0d127 ENA

 sd c3t0d128-01  volprdnfs1-01 c3t0d128 0       53026176 446702848 c3t0d128 ENA

 sd c3t0d129-01  volprdnfs1-01 c3t0d129 0       53026176 499729024 c3t0d129 ENA

 sd c3t0d130-01  volprdnfs1-01 c3t0d130 0       53026176 552755200 c3t0d130 ENA

 sd c3t0d131-01  volprdnfs1-01 c3t0d131 0       53026176 605781376 c3t0d131 ENA

 sd c3t0d132-01  volprdnfs1-01 c3t0d132 0       53026176 658807552 c3t0d132 ENA

 sd c3t0d133-01  volprdnfs1-01 c3t0d133 0       53026176 711833728 c3t0d133 ENA

 sd dgprdnfs04-02 volprdnfs1-01 dgprdnfs04 39544576 13483520 764859904 c3t0d18 ENA

 sd c3t0d55-02   volprdnfs1-01 c3t0d55 50630912 1898016  778343424 c3t0d55  ENA

 sd c3t0d123-03  volprdnfs1-01 c3t0d123 132064  52894112 780241440 c3t0d123 ENA

 sd c3t0d124-03  volprdnfs1-01 c3t0d124 130320  52895856 833135552 c3t0d124 ENA

 sd c3t0d125-02  volprdnfs1-01 c3t0d125 65168   52961008 886031408 c3t0d125 ENA

 sd c3t0d126-03  volprdnfs1-01 c3t0d126 48300192 4725984 938992416 c3t0d126 ENA

 pl volprdnfs1-02 volprdnfs1  ENABLED  TEMPRMSD 943718400 CONCAT   -        WO

 sd c4t2d1-01    volprdnfs1-02 c4t2d1  0        943718400 0        c4t2d1   ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen

 pl volprdnfs2-01 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW

 sd dgprdnfs02-02 volprdnfs2-01 dgprdnfs02 9886464 9886464 0       c3t0d13  ENA

 sd dgprdnfs03-01 volprdnfs2-01 dgprdnfs03 0    53028096 9886464   c3t0d14  ENA

 sd c3t0d203-01  volprdnfs2-01 c3t0d203 0       52428800 62914560  c3t0d203 ENA

 pl volprdnfs2-02 volprdnfs2  ENABLED  TEMPRMSD 115343360 CONCAT   -        WO

 sd c4t2d2-01    volprdnfs2-02 c4t2d2  0        115343360 0        c4t2d2   ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen

 pl volprdnfs3-01 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW

 sd c3t0d122-01  volprdnfs3-01 c3t0d122 0       53026176 0         c3t0d122 ENA

 pl volprdnfs3-02 volprdnfs3  ENABLED  TEMPRMSD 53026176 CONCAT    -        WO

 sd c4t2d3-01    volprdnfs3-02 c4t2d3  0        53026176 0         c4t2d3   ENA

Monitor the synchronization process using the vxtask command.

 # vxtask list

 TASKID  PTID TYPE/STATE    PCT   PROGRESS

    161           ATCOPY/R 21.47% 0/53026176/11386880 PLXATT volprdnfs3 volprdnfs3-02 dgprdnfs

    163           ATCOPY/R 05.95% 0/115343360/6862848 PLXATT volprdnfs2 volprdnfs2-02 dgprdnfs

    164           ATCOPY/R 00.52% 0/943718400/4921344 PLXATT volprdnfs1 volprdnfs1-02 dgprdnfs

Once the synchronization process has completed, you are safe to remove the old plexes from the original source copy, or in this case the plexes which contain the c3's subdisks.

 # vxplex -g dgprdnfs -o rm dis volprdnfs1-01

 # vxplex -g dgprdnfs -o rm dis volprdnfs2-01

 # vxplex -g dgprdnfs -o rm dis volprdnfs3-01

Now the mirroring is complete, the following vxprint shows the end result. Disk group dgprdnfs is no longer dependent on controller c3 subdisks and now runs on a new set of storage subdisks c4's .

 # vxprint -qhtg dgprdnfs

 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c4t2d1       c4t2d1s2     auto     2048     964654848 -

 dm c4t2d2       c4t2d2s2     auto     2048     125821696 -

 dm c4t2d3       c4t2d3s2     auto     2048     62909696 -

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen

 pl volprdnfs1-02 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW

 sd c4t2d1-01    volprdnfs1-02 c4t2d1  0        943718400 0        c4t2d1   ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen

 pl volprdnfs2-02 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW

 sd c4t2d2-01    volprdnfs2-02 c4t2d2  0        115343360 0        c4t2d2   ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen

 pl volprdnfs3-02 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW

 sd c4t2d3-01    volprdnfs3-02 c4t2d3  0        53026176 0         c4t2d3   ENA

lsof: Use lsof to Find PID on Port

2010-10-20T09:00:00.008+11:00

No need for large scripts to find process ID with Port. If you have the lsof binary available in your system including tr, you can easily find the process ID using the UDP or TCP socket.

In this example will will locate the process ID using the TCP 1080 port.

 # lsof -i tcp:1080 -Fp | tr -d p

veritas: VCS Java Console Logs showing wrong date & timezone

2010-10-18T09:00:00.032+11:00

I have a number of Veritas clustered environments that I maintain. I monitor these clusters through the Veritas Cluster Manager JAVA console running on my Windows 7 workstation.

For the last couple of weeks, I have noticed the VCS logs where showing the wrong date and time. Here is a screen shot showing the time I logged in. At the time it was 1 day and almost 12 hours behind.

Beginning in 2007 the U.S.A. and other countries changed the way Daylight Savings Time is scheduled. The Symantec provided Java Runtime Environments (JREs), VRTSjre, and VRTSjre15, are affected by the changes in DST. This will cause the time reported internally by the JRE to be off by 1 hour for 4 weeks each year, causing incorrect date and time processing for Symantec applications relying on these JREs. The new DST rules will went into effect in March 2007. To comply with the DST changes, updates to the JREs must be provided. This is done by either by applying an update to your Symantec product or with a JRE update tool provided by the vendor as described below.
This affects VRTSjre (Java 1.4), VRTSjre15 (Java 1.5), and 3.2 / 3.3 versions of Veritas Enterprise Administrator (VEA) on all platforms currently supported by these components.

1. To investigate this issue you will need to ensure first if the primary VRTSvcs log from the host is also not showing the wrong date and time:

 Log location: /var/VRTSvcs/log/engine.log_A

2. Also you can double check by running an X Windows session , exporting your DISPLAY variables and run the hagui locally from the primary host. Then ensure the VCS logs dont also show the wrong date and time.

 # DISPLAY=ip-address:0;export DISPLAY

 # hagui &

By running through the above two basic steps you can confirm which JRE session is having the issue.

Either from your Workstation or the Primary Host.

In my Case the issue was coming from my Windows 7 Workstation.

The following steps will show you how to apply the Java Runtime Environment Timezone Database Update tool to the VRTSjre and Veritas Enterprise Administrator for Daylight Saving Time (DST) changes.

1. You will need to download the latest tzupdater tool to update my JRE timezone running on your workstation.

You should be able to download it from the following link: http://java.sun.com/javase/tzupdater_README.html

2. You must log in using a valid Sun Online account to download. There is an option to register and create a new account if needed. Log in using your Sun Online account

3. Once you are logged in, agree to the license. This is required to download the tool

4. Click on the Java Standard Edition (SE) download section, and click on the timezone tool download.

5. Once downloaded extract the files to a temporary directory on your workstation,

6. Update the implementation of the JRE as follows:
For Windows systems, bring up a command prompt session and run the following:

 # cd C:\Program Files\Common Files\VERITAS Shared\VRTSjre\jre1.5\bin\

 java -jar C:\javatz\tzupdater.jar -f -bc -v

To verify:

 # java -jar C:\javatz\tzupdater.jar -t -v

 # cd c:\Program Files\VERITAS\VERITAS Object Bus\jre\bin

 java -jar C:\javatz\tzupdater.jar -f -bc -v 

To verify:

 # java -jar C:\javatz\tzupdater.jar -f -bc -v 

7. Once you have applied the update to the Java Runtime Environment , you can start a new Veritas Cluster Management JAVA console and ensure the VCS logs have the correct date and time.

8. If you continue to experience issues I suggest to uninstall the Veritas Cluster Management JAVA console, upgrade to the latest version which at time of this post is 5.1.00.2.

5.1 Veritas Cluster Management Console version is backwards compatible. And can be downloaded from the following location:
http://www.symantec.com/business/support/index?page=content&id=TECH78415

JASS: Permanently Disabling Services using JASS

2010-10-14T09:00:00.021+11:00

The following will provide some basic instructions on how to disable services on systems using the Solaris Security Toolkit (JASS)

1. Determine what services you wish to disable that you may no longer need. Especially on a WEB farm architecture you would wish to harden/disable uneeded OS services that could potentially be a security threat.

Here is an example of what you can consider.

all NFS services (client AND server)
autofs/automounter
cde-login and all cde/graphical login components
drd (only needed for logical domains)
Sun ServiceTags (They are usually used to provide discovery of systems and software on the LAN, but I think this is not appropriate for a web host)
stosreg and sthwreg services.

For example I have disabled: (on top of the usual set of JASS FTP TELNET etc,etc) the following.

CDE and font server

svc:/application/graphical-login/cde-login

svc:/network/rpc/cde-calendar-manager

svc:/network/rpc/cde-ttdbserver

svc:/application/cde-printinfo

Logical Domains (T series only)

 svc:/platform/sun4v/drd

NFS and automounter

svc:/network/nfs/cbd

svc:/network/nfs/client

svc:/network/nfs/server

svc:/network/nfs/mapid

svc:/network/nfs/status

svc:/network/nfs/nlockmgr

svc:/network/nfs/rquota

svc:/system/filesystem/autofs

Service Tags

svc:/network/stdiscover:default

svc:/network/stlisten:default

Removal of the above should leave a nice clean OS with minimal external ports open.

2. To disable such services you will need to use the JASS_SVCS_DISABLE JASS variable in /opt/SUNWjass/Drivers/user.init Use the package-provided file user.init.SAMPLE as a template

 # cd /opt/SUNWjass/Drivers

 # cp user.init.SAMPLE user.init

 # vi user.init 

Add the following lines of services to disable.

 JASS_SVCS_DISABLE="${JASS_SVCS_DISABLE}

 svc:/network/telnet:default

 svc:/network/ftp:default

 svc:/application/management/webmin:default

 svc:/system/webconsole:console

 svc:/application/management/snmpdx:default

 svc:/application/management/dmi:default

 svc:/application/management/wbem:default

 svc:/application/graphical-login/cde-login

 svc:/network/rpc/cde-calendar-manager

 svc:/network/rpc/cde-ttdbserver

 svc:/application/cde-printinfo

 svc:/application/font/fc-cache:default

 svc:/platform/sun4v/drd:default

 svc:/network/nfs/cbd

 svc:/network/nfs/client

 svc:/network/nfs/server

 svc:/network/nfs/mapid

 svc:/network/nfs/status

 svc:/network/nfs/nlockmgr

 svc:/network/nfs/rquota

 svc:/system/filesystem/autofs

 svc:/network/stdiscover:default

 svc:/network/stlisten:default "

 export JASS_SVCS_DISABLE

3. To apply the changes run jass-execute in apply mode again and reboot:

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver 

JASS: Installing the Solaris Security Toolkit JASS

2010-10-11T09:00:00.028+11:00

The Solaris Security Toolkit provides a flexible way to harden a Solaris system, making it more secure from malicious attack. This software may be installed during a unattended Solaris JumpStart install, or installed after Solaris is installed and booted.

The degree of hardening depends on your architectural requirements. I've been happily using JASS for the last year now, I have my own specific JASS build packages for various environments, I also run audits on a weekly basis to ensure there have been no malicious system or software changes.

Pre-canned scripts provide various levels of hardening & auditing.
The server-secure.driver leaves frequently-used server services open. The following discusses customizing the server-secure.driver to your site-specific needs. Once customized, your systems can be hardened in an automated way using one or more configurations you established.

Here I'll give enough details to get you started. I'm only covering interactive use in this example.

1. First, download the SUNWjass 4.2.0 package, available at http://www.sun.com/software/security/jass/. You don't need the MD5 (SUNBEmd5) or fixmodes (SUNBEfixm) packages, as the functionality is incorporated in Solaris 10.

2. If SUNWjass is already installed, remove it with pkgrm (back up any modified files first)

3. Uncompress and install the package to /opt/SUNWjass

 # uncompress SUNWjass.pkg.Z

 # pkgadd -d SUNWjass.pkg

4. Run a driver in "apply" mode. In this example, we use server-secure.driver This takes a few minutes.

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

5. Check the summary output for failures and errors:

 [SUMMARY] Results Summary for APPLY run of server-secure.driver

 [SUMMARY] The run completed with a total of 85 scripts run.

 [SUMMARY] There were  Failures in   0 Scripts

 [SUMMARY] There were  Errors   in   0 Scripts

 [SUMMARY] There were  Warnings in   2 Scripts

 [SUMMARY] There were  Notes    in  68 Scripts

6. Reboot and login again:

7. You can verify the previous run of jass-execute and manually run audit of the system

 # /opt/SUNWjass/bin/jass-execute -a server-secure.driver

This takes a few minutes and produces a summary at the end:

 [SUMMARY] Results Summary for AUDIT run of server-secure.driver

 [SUMMARY] The run completed with a total of 85 scripts run.

 [SUMMARY] There was a Failure  in   1 Script

 [SUMMARY] There were  Errors   in   0 Scripts

 [SUMMARY] There was a Warning  in   1 Script

 [SUMMARY] There were  Notes    in  20 Scripts

 [SUMMARY] Failure Scripts listed in:

         /var/opt/SUNWjass/run/20050721092746/jass-script-failures.txt

Verify there are no failures. If any failures are found as in this case, look at the script output to see if there are any unexpected problems. In the example above, I see the failure is from set-root-home-dir.aud because I provided and created a custom .profile script:

 [FAIL] Template /root/.profile does not match target on system.

You can fix it by removing my custom .profile file, or modifying the .profile in Solaris Security Toolkit (under /opt/SUNWjass/Files).I prefer the later.
Change the root .profile to your desire from the following location /opt/SUNWjass/Files/root/.profile
Then run another hardening:

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

Then run another audit to confirm there are no errors this time.

 # /opt/SUNWjass/bin/jass-execute -a server-secure.driver

 [SUMMARY] Results Summary for AUDIT run of server-secure.driver

 [SUMMARY] The run completed with a total of 85 scripts run.

 [SUMMARY] There was a Failure  in   0 Script

 [SUMMARY] There were  Errors   in   0 Scripts

 [SUMMARY] There was a Warning  in   1 Script

 [SUMMARY] There were  Notes    in  20 Scripts

coreadm: dumpadm: Crash Dump Defined

2010-10-08T09:00:00.032+11:00

A core file is a point-in-time copy (snapshot) of the RAM allocated to a process.

Two types of core files:

1)Per-process core file - owned by user mode 600

2)Global core file - not created by default ; owned by root mode 600

To change the Core File Configuration use the coreadm command to modify /etc/coreadm.conf file.

 # coreadm -p pattern pid

 # coreadm -i pattern ;survives reboot

 # coreadm -e global/process/global-setid/proc-setid/log ;enable options

 # coreadm -d ;disables corefile option

 # coreadm -u ;updates

 # coreadm -g ;sets the global core file name pattern

 %p ; PID

 %u ; uid

 %g ; gid

 %f ; executable filename

 %n ; system node name uname -n

 %m ; machine hardware name = uname -m

 %t ; time in seconds since 1970,1,1.

 %d ; executable file directory/name

 %z ; zonename

 %% ; % itself

Examples:

 # coreadm -p core.%f.%p $$ ; $$ pid of the current shell

 # coreadm -p $HOME/corefiles/%n.%f.%p $$

 # coreadm -g /var/core/core.%f.%p -e global ;

 # coreadm 228 507

 # coreadm -p /var/core/usr/bin ; listing corefiles for pid

 # coreadm -G all -g /var/core/%d/%f %p %n

The dumpadm, stores a dump of memory, the /etc/dumpadm.conf stores coredump configurations. Don't edit this file. use dumpadm command instead.

The default dump area is the swap space

/var/crash/host-name/vmcore.0 - memory content

/var/crash/host-name/unix.0 - symbol table

savecore command is a utility that saves crushdump into a file on reboot.

 # dumpadm

 # dumpadm -c curproc -d swap ; Modify the dump configuration so it dumps Kernel memory pages and its

 processes to swap memory

 # dumpadm -n ; turn off savecore

 # dumpadm -u ; update kernel from /etc/dumpadm.conf

 # dumpadm -y ; turn on savecore, default

 # dumpadm -c ; specify the dump content - kernel,all or curproc

 # dumpadm -d ; specify the dump device

 # dumpadm -m minK ; set a mininum space savecore should reserve,normally,in /var/crash/host1/

 filesystem.

 # dumpadm -s savecore_dir ; specify the directory where savecore saves 

 # dumpadm -r root_dir ; specify the relative root dir, default /

dumpadm Examples Tests

 # dumpadm

 Dump content: kernel pages

 Dump device: /dev/dsk/c0t0d0s1 (swap)

 Savecore directory: /var/crash/host-name

 Savecore enabled: yes

 # dumpadm -d /dev/dsk/c1t0d0s5

 Dump content: kernel pages

 Dump device: /dev/dsk/c1t0d0s5 (dedicated)

 Savecore directory: /var/crash/host-name

 Savecore enabled: yes

 # sync

 # savecore -L

 dumping to /dev/dsk/c1t0d0s5, offset 65536, content: kernel

 100% done: 11679 pages dumped, compression ratio 3.14, dump succeeded  

 System dump time: Tue Dec 5 13:21:05 2006

 Constructing namelist /var/crash/host-name/unix.0

 Constructing corefile /var/crash/host-name/vmcore.0

 100% done: 11679 of 11679 pages saved

 # cd /var/crash/host-name

 -rw-r--r-- 1 root root 1201176 Dec 5 13:21 unix.0

 -rw-r--r-- 1 root root 97640448 Dec 5 13:21 vmcore.0

 # file vmcore.0

 vmcore.0: SunOS 5.10 Generic_118822-25 64-bit SPARC crash dump from host-name

coreadm Examples Tests

 # coreadm

 global core file pattern:

 global core file content: default

 init core file pattern: core

 init core file content: default

 global core dumps: disabled

 per-process core dumps: enabled

 global setid core dumps: disabled

 per-process setid core dumps: disabled

 global core dump logging: disabled

 # mkdir /var/core

 # coreadm -e global -g /var/core/core.%f.%p

 # coreadm -e log

 # coreadm

 global core file pattern: /var/core/core.%f.%p

 global core file content: default

 init core file pattern: core

 init core file content: default

 global core dumps: enabled

 per-process core dumps: enabled

 global setid core dumps: disabled

 per-process setid core dumps: disabled

 global core dump logging: enabled

 # mkdir /var/tmp/dir

 # cd /var/tmp/dir

 # pwd

 /var/tmp/dir

  # ps

 PID TTY TIME CMD

 1094 pts/6 0:00 ps

 1056 pts/6 0:00 ksh

 # kill 8 1056

 # ls /var/core

 core.ksh.729

 core.ksh.893

 /var/core/core.ksh.729: ELF 32-bit MSB core file SPARC Version 1, from 'ksh'

 /var/core/core.ksh.893: ELF 32-bit MSB core file SPARC Version 1, from 'ksh'

 # tail /var/adm/messages

 Dec 5 13:21:08 host-name genunix: [ID 851671 kern.notice] dump succeeded

 Dec 5 13:21:29 host-name savecore: [ID 748169 auth.error] saving system crash dumpin /var/crash

 /host-name/*.0

 Dec 5 13:26:29 host-name genunix: [ID 603404 kern.notice] NOTICE: core_log: ksh[893] core dumped:

 /var/core/core.ksh.893

 Dec 5 13:27:23 host-name genunix: [ID 603404 kern.notice] NOTICE: core_log: ksh[729] core dumped:

 /var/core/core.ksh.729

news: Oracle Solaris 10 Update 9 released

2010-10-04T09:00:00.024+11:00

September 8, 2010 Oracle officially announced Oracle Solaris 10 9/10, Oracle Solaris Cluster 3.3 and Oracle Solaris Studio12.2. For now we will be concentrating on Oracle Solaris 10 9/10.

So what Solaris 10 U9 includes, let’s take quick tour -

There are some drastic changes under this update -

The most awaited, Oracle Solaris Containers now provide enhanced “P2V” (Physical to Virtual) capabilities to allow customers to seamlessly move from existing Oracle Solaris 10 physical systems to virtual containers quickly and easily. At our project we developed a custom method to perform p2v from Solaris8/9 to Solaris 10 Container and to be honest we was really looking forward for this feature.
Host ID Emulation - Migration of a physical Solaris 10 machine into a Zone with support for the HostID will allow more network management platforms to be virtualized while still retaining their licensing features.
Oracle 11g Release 2 Support
Networking and database optimizations for Oracle Real Application Clusters (Oracle RAC).
Increased reliability for virtualized Solaris instances when deployed using Oracle VM for SPARC, also known as Logical Domains.

ZFS Changes -

ZFS device replacement enhancements - namely autoexpand
some changes to the zpool list command
Holding ZFS snapshots
Triple parity RAID-Z (raidz3)
The logbias property
Log device removal - at last
ZFS storage pool recovery
New ZFS system process – In this release, each storage pool has an associated process, zpool-poolname
Splitting a mirrored ZFS storage pool (zpool split)

For more information - http://dlc.sun.com/pdf/821-1840/821-1840.pdf

Download - http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html

XSCF: Upgrade of XCP firmware

2010-09-30T09:00:00.029+10:00

This section explains how to update the firmware on Sun Sparc Enterpise M Series.

The steps will follow the preferred method from upgrading from 1060 firmware to 1093 firmware.The three major steps are :

- shut down to the ok prompt (init 0)

- XCP import in the system

- upgrade the XCP firmware - This will include an XSCF reset

- boot the system

Note – XCP: Abbreviation for XSCF Control Package. XCP is a package that has the control programs of hardware that configures a computing system. The XSCF firmware and the OpenBoot PROM firmware are included in the XCP file. The firmware update functions provided by XSCF are used to manage XCP.

Firmware update using the XSCF Shell, Use the following commands to update the firmware:
■ getflashimage command: Imports firmware to this system.
■ flashupdate command: Downloads the firmware to flash memory and applies the XSCF firmware.
■ poweron command or reset command: Applies the OpenBoot PROM firmware.
■ version command: Displays the firmware version.

1. Once you have the system shutdown to the ok prompt , enter into the XSCF prompt.

2. Before updating the firmware, be sure to check the XCP version in the current system. Be aware of which version your upgrading from as steps will differ if there is a large version gap.

 XSCF> version -c xcp -v

 XSCF#0 (Active )

 XCP0 (Current): 1060

 OpenBoot PROM : 01.30.0000

 XSCF          : 01.06.0001

 XCP1 (Reserve): 1060

 OpenBoot PROM : 01.30.0000

 XSCF          : 01.06.0001

3. Confirm the list of the firmware program files that are still on the system using the getflashimage command.

 XSCF> getflashimage -l

 Existing versions:

 Version                Size  Date

 FFXCP1060.tar.gz   49053148  Tue Feb 26 19:29:49 EST 2008

4. Use the following getflashimage command to specify the firmware program file and import XCP to the system.
Login a remote ftp server specifying the user name and host name that requires authentication password, then, import the new version firmware program (tar.gz). Ensure that your firmware program file is located in the home directory of the user your going to connect as.

 XSCF> getflashimage -u user-name ftp://ip-address/FFXCP1093.tar.gz

 Existing versions:

 Version                Size  Date

 FFXCP1060.tar.gz   49053148  Tue Feb 26 19:29:49 EST 2008

 Warning: About to delete existing versions.

 Continue? [y|n]: y

 Removing FFXCP1060.tar.gz.

 Password:

   0MB received

   1MB received

   2MB received

 .......

  39MB received

  40MB received

 Download successful: 41859 Kbytes in 56 secs (784.888 Kbytes/sec)

 Checking file...

 MD5: f2dc08a4bd43061ea84c0172a6380c94

5. Confirm the list of the firmware program file you downloaded is now on the system using the getflashimage command.

 XSCF> getflashimage -l

 Existing versions:

 Version                Size  Date

 FFXCP1093.tar.gz   42863796  Thu Sep 23 14:09:40 EST 2010

6. Use the flashupdate command to confirm whether your able to update the new firmware version.

 XSCF> flashupdate -c check -m xcp -s 1093

 XCP update is possible with domains up 

7. Use the flashupdate command to update the firmware. Once complete the the XSCF will reset and the current session will disconnect, connect again once the XSCF has been restored.

 XSCF> flashupdate -c update -m xcp -s 1093

 The XSCF will be reset. Continue? [y|n] :y

 Checking the XCP image file, please wait a minute

 XCP update is started (XCP version=1093:last version=1060)

 OpenBoot PROM update is started (OpenBoot PROM version=02160000)

 OpenBoot PROM update has been completed (OpenBoot PROM version=02160000)

 XSCF update is started (XSCFU=0,bank=1,XCP version=1093:last version=1060)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=00:version=01090003:last version=01060000)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=00:version=01090003:last version=01060000)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=01:version=01090003:last version=01060001)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=01:version=01090003:last version=01060001)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=02:version=01080001:last version=01060000)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=02:version=01080001:last version=01060000)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=03:version=01090002:last version=01060000)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=03:version=01090002:last version=01060000)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=04:version=01090003:last version=01060001)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=04:version=01090003:last version=01060001)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=05:version=01090002:last version=01050000)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=05:version=01090002:last version=01050000)

 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element

 ID=07:version=01090001:last version=01060000)

 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware

 Element ID=07:version=01090001:last version=01060000)

 XSCF update has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060)

 XSCF is rebooting to update the reserve bank

8. Re-connect to the XSCF and log in again. To confirm that the XSCF firware update has finished, use the showlogs command with the monitor option. Ensure you see the "SCF:XCP update has been completed version=xxxx" message

 XSCF> showlogs monitor

 Sep 23 14:15:10 xscf1 monitor_msg: SCF:XCP update is started (XCP version=1093:last version=1060)

 Sep 23 14:15:49 xscf1 monitor_msg: SCF:XSCF download is started (XSCFU=0, bank=1, XCP

 version=1093:last version=1060, Firmware Element ID=00, version=01090003:last version=01060000)

 Sep 23 14:16:28 xscf1 monitor_msg: SCF:XSCF download has been completed (XSCFU=0, bank=1, XCP

 version=1093:last version=1060, Firmware Element ID=00, version=01090003:last version=01060000)

 Sep 23 14:16:41 xscf1 monitor_msg: SCF:XSCF download is started (XSCFU=0, bank=1, XCP

 version=1093:last version=1060, Firmware Element ID=01, version=01090003:last version=01060001)

 .......

 Sep 23 14:32:55 xscf1 monitor_msg: SCF:XCP update has been completed (XCP version=1093)

9. Confirm the version of the system firmware that is running is that of the firmware applied.

 XSCF> version -c xcp -v

 XSCF#0 (Active )

 XCP0 (Reserve): 1093

 OpenBoot PROM : 02.16.0000

 XSCF          : 01.09.0003

 XCP1 (Current): 1093

 OpenBoot PROM : 02.16.0000

 XSCF          : 01.09.0003

 OpenBoot PROM BACKUP

 #0: 01.30.0000

 #1: 02.16.0000

10. To complete the update restart the domain. Once the domain is running it will commence its boot sequence.

 XSCF> reset -d 0 por

 DomainID to reset:00

 Continue? [y|n] :y

 00 :Reset

 XSCF> showdomainstatus -a

 DID         Domain Status

 00          Initialization Phase

 01          -

 02          -

 03          -

 XSCF> showdomainstatus -a

 DID         Domain Status

 00          Running

 01          -

 02          -

 03          -

timezones: Daylight Savings

2010-09-28T09:00:00.017+10:00

Daylight Savings for this year starts on Sunday October 3. Are your systems ready to handle the time change automatically?

To check any systems you may be interested in, the easiest way is using the zdump command

 # zdump -v $TZ | grep 2010

 Australia/NSW  Thu Sep 23 05:08:57 2010 UTC = Thu Sep 23 15:08:57 2010 EST isdst=0

 Australia/NSW  Sat Apr  3 15:59:59 2010 UTC = Sun Apr  4 02:59:59 2010 EST isdst=1

 Australia/NSW  Sat Apr  3 16:00:00 2010 UTC = Sun Apr  4 02:00:00 2010 EST isdst=0

 Australia/NSW  Sat Oct  2 15:59:59 2010 UTC = Sun Oct  3 01:59:59 2010 EST isdst=0

 Australia/NSW  Sat Oct  2 16:00:00 2010 UTC = Sun Oct  3 03:00:00 2010 EST isdst=1

The first line is just the current date/time and can be ignored. the remaining entries show the entries from the time zone database, which determine when any time changes for this year will occur. The above shows that the system clock will go forward by 1 hour at 2:00 on Sunday Oct 3, which is correct.

veritas: Backing up the Veritas Cluster Server configuration

2010-09-27T09:00:00.018+10:00

Veritas cluster server stores custom agents and it’s configuration data as a series of files in /etc, /etc/VRTSvcs/conf/config and /opt/VRTSvcs/bin/ directories. Since these files are the life blood of the cluster engine, it is important to backup these files to ensure cluster recovery should disaster hit. VCS comes with the hasnap utility to simplify cluster configuration backups, and when run with the “-backup,” “-n,” “-f ,” and “-m ” options, a point in time snapshot of the cluster configuration will be written to the file passed to the “-f” option:

 # hasnap -backup -f clusterbackup.zip -n -m “Backup from March 25th 2007″  

 Starting Configuration Backup for Cluster foo 

 Dumping the configuration... 

 Registering snapshot "foo-2006.08.25-1156511358610" 

 Contacting host lnode1... 

 Error connecting to the remote host "lnode1" 

 Starting backup of files on host lnode2

 "/etc/VRTSvcs/conf/config/types.cf" ----> 1.0

 "/etc/VRTSvcs/conf/config/main.cf" ----> 1.0

 "/etc/VRTSvcs/conf/config/vcsApacheTypes.cf" ----> 1.0

 "/etc/llthosts" ----> 1.0

 "/etc/gabtab" ----> 1.0

 "/etc/llttab" ----> 1.0

 "/opt/VRTSvcs/bin/vcsenv" ----> 1.0

 "/opt/VRTSvcs/bin/LVMVolumeGroup/monitor" ----> 1.0

 "/opt/VRTSvcs/bin/LVMVolumeGroup/offline" ----> 1.0

 "/opt/VRTSvcs/bin/LVMVolumeGroup/online" ----> 1.0

 "/opt/VRTSvcs/bin/LVMVolumeGroup/clean" ----> 1.0

 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0

 "/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGroup.xml" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/fdsched" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/monitor" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/fdsetup.vxg" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/open" ----> 1.0

 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/RVGSnapshotAgent.pm" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/RVGSnapshot.xml" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/offline" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/online" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/attr_changed" ----> 1.0

 "/opt/VRTSvcs/bin/RVGSnapshot/clean" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/monitor" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/open" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/RVGPrimary.xml" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/offline" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/online" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/clean" ----> 1.0

 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0

 "/opt/VRTSvcs/bin/RVGPrimary/actions/fbsync" ----> 1.0

 "/opt/VRTSvcs/bin/triggers/violation" ----> 1.0

 "/opt/VRTSvcs/bin/CampusCluster/monitor" ----> 1.0

 "/opt/VRTSvcs/bin/CampusCluster/close" ----> 1.0

 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0

 "/opt/VRTSvcs/bin/CampusCluster/open" ----> 1.0

 "/opt/VRTSvcs/bin/CampusCluster/CampusCluster.xml" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/monitor" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/info" ----> 1.0

 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/RVG.xml" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/offline" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/online" ----> 1.0

 "/opt/VRTSvcs/bin/RVG/clean" ----> 1.0

 "/opt/VRTSvcs/bin/internal_triggers/cpuusage" ----> 1.0

 Backup of files on host lnode2 complete

 Backup succeeded partially

To check the contents of the snapshot, the unzip utility can be run with the “-t” option:

 # unzip -t clusterbackup.zip |more

 Archive: clusterbackup.zip 

 testing: /cat_vcs.zip OK 

 testing: /categorylist.xml.zip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/types.cf.zip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/main.cf.zip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/vcsApacheTypes.cf.z ip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/llthosts.zip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/gabtab.zip OK 

 testing: _repository__data/vcs/foo/lnode2/etc/llttab.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/vcsenv.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/monitor.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/offline.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/online.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/clean.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGro upAgent.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGro up.xml.zip OK 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/RVGSnapshot/fdsched.zip O K 

 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/RVGSnapshot/monitor.zip O K 

 ......

Since parts of the cluster configuration ran reside in memory and not on disk, it is a good idea to run “haconf -dump -makero” prior to running hasnap. This will ensure that the current configuration is being backed up, and will allow hasnap “-restore” to restore the correct configuration if disaster hits.

veritas: Veritas Volume Replicator (VVR) Commands

2010-09-24T09:00:00.005+10:00

Here are some links to Basic Veritas Volume Replicator (VVR) commands for your data replication management environments.

Basic VVR Commands:

veritas: rlinks in 'paused due to network disconnection' state

2010-09-20T09:00:00.020+10:00

Had a case open with Symantec with the following issue.

 # vradmin -g dgprdap2 repstatus rvg_prdap2

 Replicated Data Set: rvg_prdap2

 Primary:

   Host name:                  10.150.150.13

   RVG name:                   rvg_prdap2

   DG name:                    dgprdap2

   RVG state:                  enabled for I/O

   Data volumes:               1

   VSets:                      0

   SRL name:                   srl_prdap2

   SRL size:                   25.00 G

   Total secondaries:          1

 Secondary:

   Host name:                  10.150.150.16

   RVG name:                   rvg_prdap2

   DG name:                    dgprdap2

   Data status:                consistent, behind

   Replication status:         paused due to network disconnection

   Current mode:               asynchronous

   Logging to:                 SRL (213278 Kbytes behind, 0% full)

   Timestamp Information:      behind by 2668h 22m 7s

'vradmin repstatus' shows rlinks in 'paused due to network disconnection' state

Various attempts where made to try continue/attach or resume the replication but failed.

The following checks must be made to diagnose the problem.

Run the vrport command on both the primary and secondary nodes.

# /usr/sbin/vrport

Check communication on these ports, from primary to secondary and vice versa on each node:

# ping -p port-number host-name

If the ping command succeeds, try restarting the VVR daemons (both on primary and secondary) in the correct sequence:

Stop vradmin on secondary then on primary

# /usr/sbin/vxstart_vvr stop

Start vradmin on secondary then on primary

# /usr/sbin/vxstart_vvr start 

If for any unknown reason the above does not fix the issue you might need to try to restart the primary and secondary nodes. But if you encounter that a bounce does not fix issue it's a rare possibility that you might need to re-create the rvg's as it was in my case. Another unsolved VVR mystery.

veritas: Displaying and changing the ports used by VVR

2010-09-17T09:00:00.012+10:00

Use the vrport command to display, change or set the port numbers used by VVR. You may have to change the port numbers in the following cases:

To resolve a port number conflict with other applications.
To configure VVR to work in your firewall environment.
To configure VVR to work in your firewall environment when using UDP; to specify a restricted number of ports to replicate data between the Primary and the Secondary.

Port Used For Heartbeats

Use the vrport heartbeat command to display the port number used by VVR, for heartbeats. To change the heartbeat port number on a host, specify the port number with the vrport heartbeat command.
Use the vradmin changeip command to update the RLINKs with the new port information, and then restart the vxnetd daemon on the required system for the changes to take effect.
To display the port number used for heartbeats

 # vrport heartbeat

To change the port number used for heartbeats

 # vrport heartbeat port

This example shows how to change the replication heartbeat port on the host1. Follow the same steps to change the heartbeat port on secondary (host2).

Note:

VVR supports a configuration with different heartbeat port numbers on the primary and secondary.

To change the replication heartbeat port on host1 from 4145 to 5000

1. Use the vrport command to change the heartbeat port to 5000 on the required host.

 # vrport heartbeat 5000

2. Issue the vradmin changeip command without the newpri and newsec attributes.

 # vradmin -g hrdg changeip hr_rvg host2

3. Verify the changes to the local RLINK by issuing the following command on the required host:

 # vxprint -g hrdg -l rlk_host2_hr_rvg 

4. Stop the vxnetd daemon.

 # /usr/sbin/vxnetd stop

5. Restart the vxnetd daemon.

 # /usr/sbin/vxnetd

Port Used By vramind

To display the port number used by vradmind, use the vrport vradmind command. To change the vradmind port, specify the port number with the vrport vradmind command.
To display the port number used by vradmind

 # vrport vradmind

To change the port number used by vradmind

 # vrport vradmind port 

Note:

You must restart the server vradmind for this change to take effect. Make sure you change the port number on all the hosts in the RDS.

Port Used by in.vxrysyncd

To display the port numbers used by in.vxrsyncd, use the vrport vxrsyncd command. To change the port numbers used by in.vxrsyncd, specify the port number with the vrport vxrsyncd command.
To display the port number used by in.vxrsyncd

 # vrport vxrsyncd

To change the port number used by in.vxrsyncd

 # vrport vxrsyncd port 

Note:

You must restart the server in.vxrsyncd for this change to take effect. Make sure you change the port number on all the hosts in the RDS.

Ports Used To Replicate Date Using UDP

To display the ports used to replicate data when using UDP, use the vrport data command. To change the ports used to replicate data when using UDP, specify the list of port numbers to use with the vrport data command.

Each RLINK requires one UDP port for replication. Specify an unused, reserved port number that is less than 32768 so that there is no port conflict with other applications. The number of ports specified must be equal to or greater than the number of RLINKs on the system.

Note:

For systems using the TCP protocol for replication, you are not required to select any data port as the connection is established with the listener port on the remote host. The listener uses this port number which is numerically same as the UDP port used for heartbeat messages.

To display ports used to replicate data when using UDP

 # vrport data

To change ports used to replicate data when using UDP

For a system configured with one RLINK, use the following command:

 # vrport data port 

For a system configured with multiple RLINKs, you can specify either a range of port numbers or a list of port numbers or both.

To specify a range of port numbers, use the following command:

 # vrport data port1, port2, portlow-porthigh, ....  

For example:

 # vrport data 3400, 3405, 3500-3503, 5756-5760

Note:

To use the new port information, execute /usr/sbin/vxnetd, and then pause and resume all RLINKs.

veritas: Panic System On Disk Group Loss

2010-09-15T09:00:00.017+10:00

Be wary of the following DiskGroup resource attribute "PanicSystemOnDGLoss"

If you enable I/O fencing and set the DiskGroup attribute PanicSystemOnDGLoss to true, you'll get the desired failover behavior. The behavior you're seeing is by design and is intended to favor data integrity over availability.

The reason for halting the system is to ensure a failover takes place and there is no data corruption due to 2 hosts wanting to write to the shared storage.

I had experienced panic on one of my production servers and found the following example in the messages file:

 2009/07/23 09:37:57 VCS CRITICAL V-16-10001-1073 (cluster2) DiskGroup:mydg:monitorisk Group: mydg is disabled on system: cluster2. System will panic to migrate all service groups to another VCS node in system list

By default the PanicSystemOnDGLoss attribute is set to 1 (true).

The attribute will cause VCS to panic the system on sudden loss of the diskgroup, when imported by VCS. The resource will also need to be marked as "Critical" for the panic to occur. VCS will not panic the system if the resource is not marked critical.

VCS will then perform an evacuation of the resource and related service group to the next surviving node. If the surviving node is unable to online the resource, no further panics are induced. The clean procedure is called and VCS stops trying to online the resource until the fault is cleared.

lsof: alloc: /: file system full

2010-09-13T09:00:00.003+10:00

Have you seen this error?

 Aug 9 00:17:41 server1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full

When I looked at the top disk space consumers I found nothing useful.

 # df -h | sort -rnk 5

 /dev/md/dsk/d0 3.0G 2.9G 0K 100% /

 /dev/md/dsk/d3 2.0G 1.5G 404M 80% /var

 /dev/md/dsk/d30 469M 330M 93M 79% /opt

 /dev/md/dsk/d6 992M 717M 215M 77% /home

 /dev/md/dsk/d33 752M 494M 198M 72% /usr/local/install

 [...]

After doing a du on whole filesystem I can see it is showing 2.5G only and df showing 2.9G consumed space.

 # du -shd /

 2.5G

I realized few days back I came across same issue on ZFS filesystem hosting oracle DB and below understanding helped me there.

Normally, If filesystem is full, then look around in the directories that will be hidden by mounted filesystems in higher init states or see if any files that are eating up the disk space, in case if you get nothing useful from this exercise then one of the things to check is the open files and consider what has been cleaned up. Sometimes, if an open file is emptied or unlinked from the directory tree the disk space is not de-allocated until the owning process has been terminated or restarted. The result is an unexplainable loss of disk space. If this is the cause a reboot would clear it up. If you can't reboot consider any process that would be logging to that partition as a suspect and check all of your logs for any entries that imply rapid errors in a process.

In my case, reboot was not possible as the server caused file system full

 # lsof +aL1 /

 lsof  WARNING: access /.lsof_server1: No such file or directory

 lsof: WARNING: created device cache file: /.lsof_server1

 lsof: WARNING: can't write to /.lsof_server1: No space left on device

 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME

 scp 16472 root 4r VREG 85,0 238616064 0 119696 / (/dev/md/dsk/d0)

 scp 22154 root 4r VREG 85,0 238213120 0 119677 / (/dev/md/dsk/d0)

``+L1'' will select open files that have been unlinked. A specification of the form ``+aL1 '' will select unlinked open files on the specified file system.

I got the processes ID's via lsof, after verifying the processes I killed them and suddenly it has released ~450MB space.

 # df -kh | sort -rnk 5

 /dev/md/dsk/d0 3.0G 2.5G 418M 86% /

 /dev/md/dsk/d3 2.0G 1.5G 406M 80% /var

 /dev/md/dsk/d30 469M 331M 91M 79% /opt

 /dev/md/dsk/d6 992M 717M 215M 77% /home

 /dev/md/dsk/d33 752M 494M 198M 72% /usr/local/install

veritas: Remove a node from a cluster without interruptions

2010-09-09T09:00:00.017+10:00

Before making changes to the VERITAS Cluster Server (VCS) configuration, the main.cf file, make a good copy of the current main.cf. In this example, csvcs6 is removed from a two node cluster. Execute these commands on csvcs5, the system not to be removed.

1. Backup the configuration.

 # cp -p /etc/VRTSvcs/conf/config/main.cf    /etc/VRTSvcs/conf/config/main.cf.last_known.good

2. Check the current systems, group(s), and resource(s) status

 # hastatus  -sum  

 -- SYSTEM STATE 

 -- System               State                Frozen             

 A  csvcs5               RUNNING              0                     

 A  csvcs6               RUNNING              0                     

 -- GROUP STATE 

 -- Group           System               Probed     AutoDisabled    State 

 B  test_A          csvcs5               Y          N               ONLINE 

 B  test_A          csvcs6               Y          N               OFFLINE 

 B  test_B          csvcs6               Y          N               ONLINE 

 B  wvcs            csvcs5               Y          N               OFFLINE 

 B  wvcs            csvcs6               Y          N               ONLINE

Based on the outputs, csvcs5 and csvcs6 are the two nodes cluster. Service group test_A and service group wvcs are configured to run on both nodes. Service group test_B is configured to run on csvcs6 only.

Both service groups test_B and wvcs are online on csvcs6. Now it is possible to failover service group wvcs to csvcs5 if it is to be online.

hagrp -switch -to

 # hagrp  -switch  wvcs  -to  csvcs5

3. Check for service group dependency

 # hagrp -dep 

 Parent   Child     Relationship

 test_B    test_A    online global

4. Make VCS configuration writable

 # haconf -makerw

5. Unlink the group dependency if there is any. In this case, the service group test_B requires test_A.

hagrp -unlink

 # hagrp  -unlink  test_B  test_A

6. Stop VCS on csvcs6, the node to be removed.

hastop -sys

 # hastop  -sys  csvcs6

7. Check the status again, making sure csvcs6 is EXITED and the failover service group is online on running node.

 # hastatus -sum

 -- SYSTEM STATE

 -- System               State                Frozen            

 A  csvcs5               RUNNING              0                  

 A  csvcs6               EXITED               0                  

 -- GROUP STATE

 -- Group           System               Probed     AutoDisabled    State

 B  test_A          csvcs5               Y          N               ONLINE

 B  test_A          csvcs6               Y          N               OFFLINE

 B  test_B          csvcs6               Y          N               OFFLINE

 B  wvcs            csvcs5               Y          N               ONLINE

 B  wvcs            csvcs6               Y          N               OFFLINE

8. Delete csvcs6 from wvcs and test_A SystemList.

hagrp -modify SystemList -delete

 # hagrp  -modify  wvcs  SystemList  -delete  csvcs6 

 # hagrp  -modify  test_A  SystemList  -delete  csvcs6

9. Check all the resources belonging to the service group and delete all the resources from group test_B before removing the group.

hagrp -resources

 # hagrp  -resources  test_B

 jprocess

 kprocess

hares -delete

 # hares  -delete  jprocess

 # hares  -delete  kprocess

hagrp -delete

 # hagrp  -delete  test_B

10. Check the status again, making sure all the service groups are online on the other node. In this case csvcs5.

 # hastatus -sum

 -- SYSTEM STATE 

 -- System               State                Frozen             

 A  csvcs5               RUNNING              0                  

 A  csvcs6               EXITED               0                   

 -- GROUP STATE

 -- Group           System               Probed     AutoDisabled    State

 B  test_A          csvcs5               Y          N               ONLINE

 B  wvcs            csvcs5               Y          N               ONLINE

11. Delete system (node) from cluster, save the configuration, and make it read only.

 # hasys  -delete  csvcs6

 # haconf -dump -makero 

12. Depending on how the cluster is defined or the number of nodes in the cluster, it might be necessary to reduce the number for " /sbin/gabconfig -c -n # " in the /etc/gabtab file on all the running nodes within the cluster. If the # is larger than the number of nodes in the cluster, the GAB will not be auto seed.

To prevent VCS from starting after rebooting, do the following on the removed node (csvcs6):

1. Unconfigure and unload GAB

 # /sbin/gabconfig  -u

 # modunload -i `modinfo | grep gab | awk '{print $1}`

2. Unconfigure and unload LLT

 # /sbin/lltconfig  -U     

 # modunload -i `modinfo | grep llt | awk '{print $1}`

3. Prevent LLT, GAB and VCS from starting up in the future

 # mv  /etc/rc2.d/S70llt   /etc/rc2.d/s70llt

 # mv  /etc/rc2.d/S92gab   /etc/rc2.d/s92gab

 # mv  /etc/rc3.d/S99vcs   /etc/rc3.d/s99vcs

4. If it ** is not ** desired to be running VCS on this particular node again, all the VCS related packages and files can now be removed.

 # pkgrm VRTSperl

 # pkgrm VRTSvcs

 # pkgrm VRTSgab

 # pkgrm VRTSllt

 # rm /etc/llttab

 # rm /etc/gabtab

NOTE: Due to the complexity and variation of VCS configuration, it is not possible to cover all the possible situations and conditions of a cluster configuration in one technote. The above steps are essential for common configuration in most VCS setups and provide some idea how to deal with complex setups.