ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.

This can significantly affect performance on mission critical servers running Databases etc.

To identify how much memory uses:

# kstat -m zfs | grep size
 
        data_size                       18935877120
        hdr_size                        66041496
        l2_hdr_size                     0
        l2_size                         0
        other_size                      11310112
        size                            19013228728

Here “19013228728” (approx 18G) indicates the total memory used by ZFS.

Alternatively, the following mdb command show ZFS ARC usage:

# echo "::arc" | mdb -k|grep size
size                      =      2048 MB
hdr_size                  =  12493584
data_size                 = 2048608256
other_size                =  86475456
l2_size                   =         0
l2_hdr_size               =         0

It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more.

To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line

set zfs:zfs_arc_max=2147483648

where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.

1. Using fcinfo

In Solaris 10, you can use the “fcinfo” utility which is a 2 step process to identify the Luns presented to the server.

1. Find the HBA port WWN number on the server:

# fcinfo hba-port

This command will show the HBA Port WWN among other information like the Node WWN, Device name, manufacturer, firmware, driver details etc.

root@myserver:/root # fcinfo hba-port
HBA Port WWN: 21000022ff0553e9
OS Device Name: /dev/cfg/c5
Manufacturer: QLogic Corp.
Model: 371-4324-02
Firmware Version: 05.03.02
FCode/BIOS Version: BIOS: 2.02; fcode: 2.03; EFI: 2.01;
Serial Number: 0402R00-1102885753
Driver Name: qlc
Driver Version: 3.00p
Type: L-port
State: online
Supported Speeds: 2Gb 4Gb 8Gb
Current Speed: 4Gb
Node WWN: 20000022ff0553e9

Here 21000022ff0553e9 is the WWN number for the HBA port.

2. List SCSI targets for the remote port attached to the HBA port attached to the above.

# fcinfo -sl -p HBA-Port-WWN

root@myserver:/root # fcinfo remote-port -sl -p 21000022ff0553e9
Remote Port WWN: 202400a0b8744072
Active FC4 Types:
SCSI Target: yes
Node WWN: 200400a0b8744072
Link Error Statistics:
Link Failure Count: 1
Loss of Sync Count: 2
Loss of Signal Count: 72
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: SUN
Product: LCSM100_F
OS Device Name: /dev/rdsk/c5t202400A0B8744072d0s2
LUN: 2 Vendor: SUN
Product: LCSM100_F
OS Device Name: /dev/rdsk/c5t202400A0B8744072d2s2
LUN: 31 Vendor: SUN
Product: Universal Xport
OS Device Name: /dev/rdsk/c5t202400A0B8744072d31s2

Here, LUN0 and LUN2 are the actual disks and LUN31 is the Command device for the Storage controller

2. Using cfgadm

You can also find the above information using the “cfgadm” utility. One of the following commands should reveal the LUNS attached to the controller. Although the later works only on Solaris 10.

# cfgadm -al -o show_FCP_dev c5
 
or
 
# cfgadm -al -o show_SCSI_LUNS c5

where c5 is the controller. If you do not mentioned the controller, it then displays for every single disk controller on the server.

root@myserver:/root # cfgadm -al -o show_FCP_dev c5
Ap_Id Type Receptacle Occupant Condition
c5 fc-private connected configured unknown
c5::202400a0b8744072,0 disk connected configured unknown
c5::202400a0b8744072,2 disk connected configured unknown
c5::202400a0b8744072,31 disk connected configured unknown

root@myserver:/root # cfgadm -al -o show_SCSI_LUN c5
Ap_Id Type Receptacle Occupant Condition
c5 fc-private connected configured unknown
c5::202400a0b8744072,0 disk connected configured unknown
c5::202400a0b8744072,2 disk connected configured unknown
c5::202400a0b8744072,31 disk connected configured unknown

Share your thoughts, if you know more ways of doing this.

Attach a Block Device

sunsolaris# lofiadm -a /export/software/iso_image.iso

/dev/lofi/1

Mount the ISO Image block device

sunsolaris# mount -F hsfs -o ro /dev/lofi/1 /mnt

Mount the mounted ISO image to the Guest zone

sunsolaris# mount -F lofs /mnt /zones/myzone/root/mnt

Where /zones/myzone/root is the root of the Zone. The files should now be visible under /mnt on the guest zone.

/usr/gnu/bin

awk

basename

captoinfo

cat

chgrp

chmod

chown

chroot

cksum

clear

cmp

comm

cp

csplit

cut

date

dd

df

diff

diff3

diffmk

dirname

du

echo

egrep

env

eqn

expand

expr

factor

false

fgrep

find

fmt

fold

grep

grn

groups

head

hostid

id

indxbib

infocmp

infotocap

join

kill

link

ln

logname

ls

make

mkdir

mkfifo

mknod

mktemp

mv

ncurses5-config

neqn

nice

nl

nohup

nroff

od

paste

patch

pathchk

pic

pr

printf

pwd

refer

reset

rm

rmdir

sdiff

sed

sh

sleep

soelim

sort

split

stty

sum

sync

tail

tar

tee

test

tic

toe

touch

tput

tr

true

tset

tsort

tty

uname

unexpand

uniq

unlink

uptime

wc

which

who

whoami

xargs

yes

WARNING: BASE_CONFIG: JS_install_patchclust: Unable to locate patch 119545-03

patches_subdir=”patches”

# cd 10_x86_Recommended

# for patches in `cat patch_order`

# do

# ln -s patches/$patches .

# done

BASE_CONFIG: <<< 119545-03 (405 of 433) >>>

Let’s put it to work. tcpdrop solaris package is available for download from www.sunfreeware.com. Click one of the following links for Solaris 10 for x86 or SPARC.

tcpdrop (Solaris 10 – SPARC)

tcpdrop (Solaris 10 – x86)

Once downloaded unzip and install tcpdrop as follows:

# gunzip tcpdrop_solaris-20080516-sol10-x86-local.gz

# pkgadd -d tcpdrop_solaris-20080516-sol10-x86-local

The following packages are available:

1  SMCtcpdr     tcpdrop_solaris

(x86) 20080516

Select package(s) you wish to process (or ‘all’ to process

all packages). (default: all) [?,??,q]:

Processing package instance <SMCtcpdr> from </root/tcpdrop_solaris-20080516-sol10-x86-local>

tcpdrop_solaris(x86) 20080516

Ceri Davies

Using </usr/local> as the package base directory.

## Processing package information.

## Processing system information.

2 package pathnames are already properly installed.

## Verifying disk space requirements.

## Checking for conflicts with packages already installed.

## Checking for setuid/setgid programs.

Installing tcpdrop_solaris as <SMCtcpdr>

## Installing part 1 of 1.

/usr/local/bin/tcpdrop

/usr/local/doc/tcpdrop/OPENSOLARIS.LICENSE

/usr/local/doc/tcpdrop/README

/usr/local/man/man8/tcpdrop.8

[ verifying class <none> ]

Installation of <SMCtcpdr> was successful.

I’m testing on Solaris 10 9/10 (x86).

# cat /etc/release
Oracle Solaris 10 9/10 s10x_u9wos_14a X86
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010

I’ve established two ssh connections to the server.

# netstat -an|grep EST
192.168.1.66.22 192.168.1.65.49980 15872 51 49640 0 ESTABLISHED
192.168.1.66.22 192.168.1.65.50050 16640 0 49640 0 ESTABLISHED

Add /usr/local/bin to the PATH environment variable if it isn’t already there.

To drop a connection, the syntax for tcpdrop is

# tcpdrop <server address> <server listening port> <client address> <client source port>

In our case, to disconnect the 2nd connection,

# tcpdrop 192.168.1.66 22 192.168.1.65 50050
192.168.1.66:22 <-> 192.168.1.65:50050: dropped

where

192.168.1.66 is the local server IP

22 is the TCP port SSH server that we are listening for a connection

192.168.1.65 is the client IP

50050 is the source port of the clients ssh connection.

Let’s check netstat now:

# netstat -an|grep EST
192.168.1.66.22 192.168.1.65.49980 16896 67 49640 0 ESTABLISHED

There it is the connection is dropped!

Backup Server: 4.142.2.58: [0] The ‘shmget’ call failed with error number 22 (Invalid argument). Refer to your operating system documentation for further details.

To confirm, if maximum shared memory has been manually set,

# projects -l

user.asetest
projid : 101
comment: “Sybase ASE”
users  : (none)
groups : (none)
attribs: project.max-shm-memory=(privileged,4294967296,deny)

For instance, here we have set the Maximum Shared memory to 4GB to the user account which owns and runs the Sybase ASE. And, the last line “attribs: project.max-shm-memory=(privileged,4294967296,deny)” basically, sets the maximum shared memory for the user processes to 4GB (4294967296).

This is equivalent to doing

# cat /etc/projects

If something similar doesn’t exist then the ASE is using the system defaults.

In any case, we need to increase the Maximum Shared memory. In our case increasing it to 7GB fixed the problem.

To add a new Project entry,

# projadd -p 101 -U asetest -c “Sybase ASE” -K “max-shm-memory=(privileged,7GB,deny)” user.asetest

This would create a new project with ID 101, name user.asetest for user asetest and set the Maximum Shared Memory to 7GB.

To modify an existing Project entry,

# projmod -sK “project.max-shm-memory=(privileged,7GB,deny)” user.asetest

This should help!

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing hard reset on /SP failed
reset: No such inventory

or when you try to reset the ILOM password, you may get the error “Set: No such object value”
To resolve the problem, you can actually logon to the ILOM as the user “sunservice” with the password of the ILOM root user and then reboot the ILOM (equivalne to “reset /SP”) with the command “reboot” as follows. Clearly as you can see from this warning SUN doesn’t recommend it so use it at your own risk:

login as: sunservice
Using keyboard-interactive authentication.
Password:

Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

WARNING: The “sunservice” account is provided solely to allow
Sun Services to perform diagnosis and recovery tasks. Customer use of
the “sunservice” account may interfere with the correct operation of
ILOM and is not supported other than to perform recovery procedures as
documented by Sun Microsystems. Normal ILOM operations should always be
performed using the root account. Further usage of the “sunservice”
account implies your agreement with these terms.
[(flash)root@SERV-ILOM:~]# reboot
/sbin/reboot

Run the command

# ipmitool fru | grep “Product Serial”

where “FRU” is a Field Replaceable Unit. The problem here is it’s going to output too many serial numbers to confuse. The one we are interested is a 10 digit alpha-numeric number.

However, extending the command as

# ipmitool fru print |egrep “Mainboard|/SYS” | awk ‘{print $7}’
91)

Would give you the ID number of the “/SYS” or “Mainboard” (91 here) which mostly has the serial number of the system. We then run the “ipmitool” as follows with that ID number to find the system serial number.

# ipmitool fru print 91 |grep “Product Serial” | awk ‘{print $4}’

where “91″ was the ID number.