unix - linux - storage

Vmware - Netapp cloning : Rapid VM provisioning

2009-07-10T05:59:00.000-07:00

Checkout this demo of provisioning 5000 Vmware VMs using Netapp File cloning and volume flexclone technologies. Good one ..

Pocket guide for netapp commands

2009-07-05T00:09:00.001-07:00

This post contains the list of commands that will be most used and will come handy when managing or monitoring or troubleshooting a Netapp filer.

sysconfig -a : shows hardware configuration with more verbose information
sysconfig -d : shows information of the disk attached to the filer
version : shows the netapp Ontap OS version.
uptime : shows the filer uptime
dns info : this shows the dns resolvers, the no of hits and misses and other info
nis info : this shows the nis domain name, yp servers etc.
rdfile : Like "cat" in Linux, used to read contents of text files/
wrfile : Creates/Overwrites a file. Similar to "cat > filename" in Linux
aggr status : Shows the aggregate status
aggr status -r : Shows the raid configuration, reconstruction information of the disks in filer
aggr show_space : Shows the disk usage of the aggreate, WAFL reserve, overheads etc.
vol status : Shows the volume information
vol status -s : Displays the spare disks on the filer
vol status -f : Displays the failed disks on the filer
vol status -r : Shows the raid configuration, reconstruction information of the disks
df -h : Displays volume disk usage
df -i : Shows the inode counts of all the volumes
df -Ah : Shows "df" information of the aggregate
license : Displays/add/removes license on a netapp filer
maxfiles : Displays and adds more inodes to a volume
aggr create : Creates aggregate
vol create <volname> <aggrname> <size> : Creates volume in an aggregate
vol offline <volname> : Offlines a volume
vol online <volname> : Onlines a volume
vol destroy <volname> : Destroys and removes an volume
vol size <volname> [+|-]<size> : Resize a volume in netapp filer
vol options : Displays/Changes volume options in a netapp filer
qtree create <qtree-path> : Creates qtree
qtree status : Displays the status of qtrees
quota on : Enables quota on a netapp filer
quota off : Disables quota
quota resize : Resizes quota
quota report : Reports the quota and usage
snap list : Displays all snapshots on a volume
snap create <volname> <snapname> : Create snapshot
snap sched <volname> <schedule> : Schedule snapshot creation
snap reserve <volname> <percentage> : Display/set snapshot reserve space in volume
/etc/exports : File that manages the NFS exports
rdfile /etc/exports : Read the NFS exports file
wrfile /etc/exports : Write to NFS exports file
exportfs -a : Exports all the filesystems listed in /etc/exports
cifs setup : Setup cifs
cifs shares : Create/displays cifs shares
cifs access : Changes access of cifs shares
lun create : Creates iscsi or fcp luns on a netapp filer
lun map : Maps lun to an igroup
lun show : Show all the luns on a filer
igroup create : Creates netapp igroup
lun stats : Show lun I/O statistics
disk show : Shows all the disk on the filer
disk zero spares : Zeros the spare disks
disk_fw_update : Upgrades the disk firmware on all disks
options : Display/Set options on netapp filer
options nfs : Display/Set NFS options
options timed : Display/Set NTP options on netapp.
options autosupport : Display/Set autosupport options
options cifs : Display/Set cifs options
options tcp : Display/Set TCP options
options net : Display/Set network options
ndmpcopy <src-path> <dst-path> : Initiates ndmpcopy
ndmpd status : Displays status of ndmpd
ndmpd killall : Terminates all the ndmpd processes.
ifconfig : Displays/Sets IP address on a network/vif interface
vif create : Creates a VIF (bonding/trunking/teaming)
vif status : Displays status of a vif
netstat : Displays network statistics
sysstat -us 1 : begins a 1 second sample of the filer's current utilization (crtl - c to end)
nfsstat : Shows nfs statistics
nfsstat -l : Displays nfs stats per client
nfs_hist : Displays nfs historgram
statit : beings/ends a performance workload sampling [-b starts / -e ends]
stats : Displays stats for every counter on netapp. Read stats man page for more info
ifstat : Displays Network interface stats
qtree stats : displays I/O stats of qtree
environment : display environment status on shelves and chassis of the filer
storage show <disk|shelf|adapter> : Shows storage component details
snapmirror intialize : Initialize a snapmirror relation
snapmirror update : Manually Update snapmirror relation
snapmirror resync : Resyns a broken snapmirror
snapmirror quiesce : Quiesces a snapmirror bond
snapmirror break : Breakes a snapmirror relation
snapmirror abort : Abort a running snapmirror
snapmirror status : Shows snapmirror status
lock status -h : Displays locks held by filer
sm_mon : Manage the locks
storage download shelf : Installs the shelf firmware
software get : Download the Netapp OS software
software install : Installs OS
download : Updates the installed OS
cf status : Displays cluster status
cf takeover : Takes over the cluster partner
cf giveback : Gives back control to the cluster partner
reboot : Reboots a filer

If you are not aware of the complete details of these commands and need more information on these commands, refer the Netapp Data Ontap administration manual from now site.

More netapp blog posts at : http://unixfoo.blogspot.com/search/label/netapp

Xen : Bridging on bonded & trunked interfaces

2009-06-29T03:29:00.000-07:00

When you deploy some critical stuffs like Oracle DB, Oracle RAC DB etc on the Xen or Oracle virtual machine, you'll have to use bonded network interfaces and the vlan trunks. By default, Xen and Oracle VM doesn't support the xen bridges created on bonded-vlaned-interfaces. The below document gives you a clear idea on how to create and configure xen bridges on bonded and vlan trunked interfaces.

Xen wiki document on bonded-vlaned xen network configuration
Oracle VM server configuration for bonded & trunked interfaces

Dstat : Linux Performance analysis tool

2009-05-24T01:13:00.000-07:00

Dstat is a handy utility for monitoring systems during performance tuning tests, benchmarks or troubleshooting. It combines vmstat, iostat, ifstat, netstat information and more. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. One more great feature of Dstat is that it is written in python, modular & easy to extend, add your own counters as plugins. It also allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs You can download Dstat at : http://dag.wieers.com/home-made/dstat

Netapp Snapvault guide

2009-04-28T07:52:00.001-07:00

Netapp SnapVault is a heterogeneous disk-to-disk backup solution for Netapp filers and heterogeneous OS systems (Windows, Linux , Solaris, HPUX and AIX). Basically, Snapvault uses Snapshot technology to store online backups. In event of data loss or corruption on a filer, the backup data can be restored from the SnapVault filer with less downtime. It has significant advantages over traditional tape backups, like

Reduce backup windows versus traditional tape-based backup
Media cost savings
No backup/recovery failures due to media errors
Simple and Fast recovery of corrupted or destroyed data

Snapvault consists of major two entities – snapvault clients and a snapvault storage server. A snapvault client (Netapp filers and unix/windows servers) is the system whose data should be backed-up. The SnapVault server is a Netapp filer – which gets the data from clients and backs up data. For Server to Netapp Snapvault, we need to install Open System Snapvault client software provided by Netapp, on the servers. Using the snapvault agent software, the Snapvault server can pull and backup data on to the backup qtrees. SnapVault protects data on a client system by maintaining a number of read-only versions (snapshots) of that data on a SnapVault filer. The replicated data on the snapvault server system can be accessed via NFS or CIFS. The client systems can restore entire directories or single files directly from the snapvault filer. Snapvault requires primary and secondary license.

How snapvault works?

When snapvault is setup, initially a complete copy of the data set is pulled across the network to the SnapVault filer. This initial or baseline, transfer may take some time to complete, because it is duplicating the entire source data set on the server – much like a level-zero backup to tape. Each subsequent backup transfers only the data blocks that has changed since the previous backup. When the initial full backup is performed, the SnapVault filer stores the data on a qtree and creates a snapshot image of the volume for the data that is to be backed up. SnapVault creates a new Snapshot copy with every transfer, and allows retention of a large number of copies according to a schedule configured by the backup administrator. Each copy consumes an amount of disk space proportional to the differences between it and the previous copy.

Snapvault commands :

Initial step to setup Snapvault backup between filers is to install snapvault license and enable snapvault on all the source and destination filers.

Source filer – filer1

    filer1> license add XXXXX
    filer1> options snapvault.enable on
    filer1> options snapvault.access host=filer2

Destination filer – filer2

    filer2> license add XXXXX
    filer2> options snapvault.enable on
    filer2> options snapvault.access host=filer1

Consider filer2:/vol/snapvault_volume as the snapvault destination volume, where all backups are done. The source data is filer1:/vol/datasource/qtree1. As we have to manage all the backups on the destination filer (filer2) using snapvault – manually disable scheduled snapshots on the destination volumes. The snapshots will be managed by Snapvault. Disabling Netapp scheduled snapshots, with below command.

    filer2> snap sched snapvault_volume 0 0 0

Creating Initial backup: Initiate the initial baseline data transfer (the first full backup) of the data from source to destination before scheduling snapvault backups. On the destination filer execute the below commands to initiate the base-line transfer. The time taken to complete depends upon the size of data on the source qtree and the network bandwidth. Check “snapvault status” on source/destination filers for monitoring the base-line transfer progress.

    filer2> snapvault start -S filer1:/vol/datasource/qtree1 filer2:/vol/snapvault_volume/qtree1

Creating backup schedules: Once the initial base-line transfer is completed, snapvault schedules have to be created for incremental backups. The retention period of the backup depends on the schedule created. The snapshot name should be prefixed with “sv_”. The schedule is in the form of “[@][@]”.

On source filer:

For example, let us create the schedules on source as below - 2 hourly, 2 daily and 2 weekly snapvault . These snapshot copies on the source enables administrators to recover directly from source filer without accessing any copies on the destination. This enables more rapid restores. However, it is not necessary to retain a large number of copies on the primary; higher retention levels are configured on the secondary. The commands below shows how to create hourly, daily & weekly snapvault snapshots.

    filer1> snapvault snap sched datasource sv_hourly 2@0-22
    filer1> snapvault snap sched datasource sv_daily 2@23
    filer1> snapvault snap sched datasource sv_weekly 2@21@sun

On snapvault filer:

Based on the retention period of the backups you need, the snapvault schedules on the destination should be done. Here, the sv_hourly schedule checks all source qtrees once per hour for a new snapshot copy called sv_hourly.0. If it finds such a copy, it updates the SnapVault qtrees with new data from the primary and then takes a Snapshot copy on the destination volume, called sv_hourly.0. If you don’t use the -x option, the secondary does not contact the primary and transfer the Snapshot copy. It just creates a snapshot copy of the destination volume.

    filer2> snapvault snap sched -x snapvault_volume sv_hourly 6@0-22
    filer2> snapvault snap sched -x snapvault_volume sv_daily 14@23@sun-fri
    filer2> snapvault snap sched -x snapvault_volume sv_weekly 6@23@sun

To check the snapvault status, use the command "snapvault status" either on source or destination filer. And to see the backups, do a "snap list" on the destination volume - that will give you all the backup copies, time of creation etc.

Restoring data : Restoring data is as simple as that, you have to mount the snapvault destination volume through NFS or CIFS and copy the required data from the backup snapshot.

You can also try Netapp Protection manager to manage the snapvault backups either from OSSV or from Netapp primary storage. Protection manager is based on Netapp Operations manager (aka Netapp DFM). It is a client based UI, with which you connect to the Ops Manager and protect your storages.

Xen : Adding and removing NICs on virtual machines

2009-04-15T07:33:00.000-07:00

Virtual Network interfaces can be attached or detached from a guest virtual machine (domU) while the virtual machine is up and running. The below Xen networking steps gives details on those.

“xm network-list” is the command to display the virtual network interfaces attached to a the guest Xen virtual machine. In this example, the command lists 3 NICs for the virtual machine “vm455”. We’ll try to remove and add back the third vif.

[root@host203 ~]# xm network-list vm455
Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3E:47:AA:FD 0 4 12 2304 /2305 /local/domain/0/backend/vif/1/0
1 0 00:16:3E:22:33:44 1 4 13 2306 /2307 /local/domain/0/backend/vif/1/1
2 0 00:16:3E:40:22:12 2 4 14 2308 /2309 /local/domain/0/backend/vif/1/2
[root@host203 ~]#

To remove a NIC card from the virtual machine, use “xm network-detach”, with the corresponding options. In this example, the interface 3 is removed from the virtual machine. This command has to be executed on the Xen server.

[root@host203 ~]# xm network-detach vm455 2
[root@host203 ~]# xm network-list vm455
Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3E:47:AA:FD 0 4 12 2304 /2305 /local/domain/0/backend/vif/1/0
1 0 00:16:3E:22:33:44 1 4 13 2306 /2307 /local/domain/0/backend/vif/1/1
[root@host203 ~]#

Once the network is detached, you can only 2 virtual NICs using “xm network-list” on the Xen server. If you check the virtual machine, it will have only 2 NICS. Use ifconfig to check on the guest virtual machine.

To add a NIC to the virtual machine, use the command “xm network-attach”. Pass the MAC-id , bridge name and the script to create the VIF as arguments. The default vif creation script is “vif-bridge”. This is defined in /etc/xen/xend-config.sxp .

[root@host203 ~]# xm network-attach vm455 bridge=xenbr1 script=vif-bridge mac=00:16:3E:40:22:12
[root@host203 ~]# xm network-list vm455
Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3E:47:AA:FD 0 4 12 2304 /2305 /local/domain/0/backend/vif/1/0
1 0 00:16:3E:22:33:44 1 4 13 2306 /2307 /local/domain/0/backend/vif/1/1
2 0 00:16:3E:40:22:12 2 4 14 2308 /2309 /local/domain/0/backend/vif/1/2
[root@host203 ~]#

The xm network-attach/detach command adds and removes NIC on a live machine. To make the changes permanent, edit the virtual machine configuration file in /etc/xen and edit the vifs. The same steps apply to Oracle Virtual machine also.

GSmartControl : GUI for smartctl

2009-03-08T08:14:00.001-07:00

GSmartControl is a graphical user interface for Linux smartctl, which is a tool for querying and controlling S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) data on modern hard disk drives. You can read about smartctl at : http://unixfoo.blogspot.com/2007/11/disk-information-using-smartctl.html . It allows you to inspect the drive's S.M.A.R.T. data to determine its health, as well as run various tests on it.

GSmartControl features include

Automatically report and highlight any abnormal SMART information.
Ability to enable / disable SMART.
Ability to enable / disable Automatic Offline Data Collection - A short self-check that the drive will perform automatically every four hours with no impact on performance.
Ability to set global and per-drive options for smartctl.
Display drive identity, capabilities, attributes, error and self-test logs.
Perform SMART self-tests.
Ability to load smartctl output as a "virtual" device, which acts just like a real (read-only) device.
Works on most smartctl-supported operating systems.

The UI is very good and gives lot of information. Checkout the screenshots.

iperf : Network throughput measurement tool

2009-02-23T03:58:00.001-08:00

Iperf is a commonly used network testing tool that can create TCP and UDP data streams and measure the throughput of a network. Iperf allows the user to set various parameters that can be used for testing a network, or alternately for optimizing or tuning a network. Iperf has a client and server functionality, and can measure the throughput between the two ends, either unidirectonally or bi-directionally. You can download it in Sourceforce http://sourceforge.net/projects/iperf . Iperf is a client server program. In our example, let us start "server" iperf at location1 and let the "location2" be the client.

Server side : To start iperf as "server" service, use the "-s" flag.

[root@location1 ~]# iperf -s
---------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
---------------------------------------

To stop the server, send Ctrl+C.

Client side: For client side, iperf has to be initiated with "-c" option. There are lot of other options, which performs different kind of testing. Few examples are explained below.

1) Measuring throughput using default iperf option. This connects to the iperf server mentioned and finds out the network throughput.

[root@location2 ~]# iperf -c location1
---------------------------------------
Client connecting to location1, TCP port 5001
TCP window size: 16.0 KByte (default)
---------------------------------------
[ 3] local 201.87.4.3 port 41505 connected with 131.15.17.19 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 3] 0.0-10.1 sec 27.4 MBytes 22.8 Mbits/sec
[root@location2 ~]#

2) Measuring bidirectional throughput usind -d option.

[root@location2 ~]# iperf -c location1 -d
---------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
---------------------------------------
Client connecting to location1, TCP port 5001
TCP window size: 16.0 KByte (default)
---------------------------------------
[ 5] local 201.87.4.3 port 41518 connected with 131.15.17.19 port 5001
[ 4] local 201.87.4.3 port 5001 connected with 131.15.17.19 port 59284
[ ID] Interval       Transfer     Bandwidth
[ 5] 0.0-10.1 sec 27.2 MBytes 22.7 Mbits/sec
[ 4] 0.0-10.0 sec 25.2 MBytes 21.1 Mbits/sec
[root@location2 ~]#

3) Measuring multi-thread throughput. Use -P option to specify the number of parallel threads to be started.

[root@location2 ~]# iperf -c location1 -P 3
---------------------------------------
Client connecting to location1, TCP port 5001
TCP window size: 16.0 KByte (default)
---------------------------------------
[ 5] local 201.87.4.3 port 41523 connected with 131.15.17.19 port 5001
[ 4] local 201.87.4.3 port 41522 connected with 131.15.17.19 port 5001
[ 3] local 201.87.4.3 port 41521 connected with 131.15.17.19 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 5] 0.0-10.1 sec 27.9 MBytes 23.3 Mbits/sec
[ 4] 0.0-10.1 sec 27.9 MBytes 23.2 Mbits/sec
[ 3] 0.0-10.1 sec 27.8 MBytes 23.1 Mbits/sec
[SUM] 0.0-10.1 sec 83.6 MBytes 69.5 Mbits/sec
[root@location2 ~]#

3) Measuring throughput by disabling Nagle's Algorithm.

Nagle's algorithm, named after John Nagle, is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network. [ http://en.wikipedia.org/wiki/Nagle%27s_algorithm ]

[root@location2 ~]# iperf -c location1 -P 3 -N
---------------------------------------
Client connecting to location1, TCP port 5001
TCP window size: 16.0 KByte (default)
---------------------------------------
[ 3] local 201.87.4.3 port 41527 connected with 131.15.17.19 port 5001
[ 4] local 201.87.4.3 port 41528 connected with 131.15.17.19 port 5001
[ 5] local 201.87.4.3 port 41529 connected with 131.15.17.19 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 3] 0.0-10.1 sec 27.8 MBytes 23.1 Mbits/sec
[ 4] 0.0-10.1 sec 27.9 MBytes 23.2 Mbits/sec
[ 5] 0.0-10.1 sec 28.0 MBytes 23.2 Mbits/sec
[SUM] 0.0-10.1 sec 83.7 MBytes 69.6 Mbits/sec
[root@location2 ~]#

You can even run the client side on regular intervals using cron daemon and plot graphs (using mrtg or rrd) for the throughput. Hope this is helpful.

Vmware Performance tuning Tips

2009-02-12T01:14:00.001-08:00

These posts explain about Vmware performance tuning on Linux servers.

These links explains the tips to tune the disk io by setting read-aheads and memory based improvements by making changes to kernel parameters like vm.swapiness, vm.overcommit_memory, vm.dirty_background_ratio, vm.dirty_ratio. They also suggests some vmware config tweaks like mainMem.useNamedFile. Worth trying if you have Vmware performance issues. But dont try in production unless you are sure of what you are doing.

ZFS : Basic administration guide

2009-02-09T20:02:00.001-08:00

This ZFS guide provides an overview of ZFS and its administration commands that will be helpful for beginners.

ZFS Pool:

ZFS organizes physical devices into logical pools called storage pools. Both individual disks and array logical unit numbers (LUNs) that are visible to the operating system can be included in a ZFS pools. These pools can be created as disks striped together with no redundancy (RAID 0), mirrored disks (RAID 1), striped mirror sets (RAID 1 + 0), or striped with parity (RAID Z). Additional disks can be added to pools at any time but they must be added with the same RAID level.

ZFS Filesystem :

ZFS offers a POSIX-compliant file system interface to the Solaris/OpenSolaris operating system. ZFS file systems must be built in one and only one storage pool, but a storage pool may have more than one defined file system. ZFS file systems are managed & mounted through /etc/vfstab file. The common way to mount a ZFS file system is to simply define it against a pool. All defined ZFS file systems automatically mount at boot time unless otherwise configured.

Here are the basic commands for getting started with ZFS.

Creating Storage pool using "zpool create" :

bash-3.00# zpool create demovol raidz c2t1d0 c2t2d0
bash-3.00# zpool status
pool: demovol
state: ONLINE
scrub: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        demovol      ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c2t1d0   ONLINE       0     0     0
            c2t2d0   ONLINE       0     0     0

errors: No known data errors
bash-3.00#

"zfs list" will give the details of the pool and other zfs filesytems.

bash-3.00# zfs list
NAME                   USED AVAIL REFER MOUNTPOINT
demovol               1.00G 900G 38.1K /demovol
bash-3.00#

Creating File Systems : "zfs create" is used to create zfs filesytem.

bash-3.00# zfs create demovol/testing
bash-3.00# zfs list
NAME                   USED AVAIL REFER MOUNTPOINT
demovol               1.00G 900G 38.1K /demovol
demovol/testing       32.6K 900G 32.6K /demovol/testing
bash-3.00#

bash-3.00# ls /dev/zvol/dsk/demovol -- This should show you the disk file.

Setting Quota for the filesytem : Until Quota is set, the filesytem shows the total available space of the containter zfs pool.

bash-3.00# zfs set quota=10G emspool3/testing
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/testing         32.6K 10.0G 32.6K /demovol/testing

Creating a snapshot :

bash-3.00# zfs snapshot demovol/testing@snap21
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/testing         32.6K 10.0G 32.6K /demovol/testing
demovol/testing@snap21      0      - 32.6K -
bash-3.00#

Get all properties of a ZFS filesytem :

bash-3.00# zfs get all demovol/testing
NAME             PROPERTY         VALUE                  SOURCE
demovol/testing type             filesystem             -
demovol/testing creation         Mon Feb 9 9:05 2009 -
demovol/testing used             32.6K                  -
demovol/testing available        10.0G                  -
demovol/testing referenced       32.6K                  -
demovol/testing compressratio    1.00x                  -
demovol/testing mounted          yes                    -
demovol/testing quota            10G                    local
demovol/testing reservation      none                   default
demovol/testing recordsize       128K                   default
demovol/testing mountpoint       /demovol/testing       default
..

Cloning a ZFS filesystem from a snapshot :

bash-3.00# zfs clone demovol/testing@snap21 demovol/clone22
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/clone22             0 900G   32.6K /demovol/clone22
demovol/testing         32.6K 10.0G 32.6K /demovol/testing
demovol/testing@snap21      0      - 32.6K -
bash-3.00#

Performance IO Monitoring the ZFS storage pool:

bash-3.00# zpool iostat 1
               capacity     operations    bandwidth
pool         used avail   read write   read write
---------- ----- ----- ----- ----- ----- -----
demovol     4.95M 900G      0      0      0     35
demovol     4.95M 900G      0      0      0      0
demovol     4.95M 900G      0      0      0      0
demovol     4.95M 900G      0      0      0      0

Please refer to the man pages, zfs and zpool, for more detailed information. Additional documentation may be found at docs.sun.com and OpenSolaris ZFS Community.

EMC CLARiiON AX Demo

2009-02-02T20:00:00.001-08:00

Gotitsolutions.org has a great demo or EMC CLARiiON AX. You can see the demo here.

More details of EMC CLARiiON AX can be found here.

Xen performance monitoring

2009-01-29T06:58:00.001-08:00

Xen’s Hypervisor does not have an easy collection of performance counters. The management machine - “Domain-0″ is actually a privileged virtual machine, and thus - get its own small share of CPUs and RAM. Collecting performance information on it will lead to collecting performance information for a single VM, and not the whole bunch. “xentop” allows collection of information, however, combining this with Cacti , or any other SNMP-based collection tool is a bit tricky. A great solution is provided by Ian P. Christian in his blog post about Xen montoring and there is a script that collects the performance details.

Reference

OpenSource Tools for AIX

2009-01-22T03:37:00.001-08:00

Here are several websites that has OpenSource Tools for AIX. The "IBM AIX Toolbox" has the most tools, but not necessarily the most current versions. The other links are also good to note.

IBM AIX Toolbox for Linux Applications – http://www.ibm.com/servers/aix/products/aixos/linux/download.html
AIX Open Source – http://www.perzl.org/aix/
Hudson Valley CC - http://pware.hvcc.edu
Bull Open Software archive for AIX - http://www.bullfreeware.com
IBM Developerworks in general - http://www.ibm.com/developerworks/downloads/

Reference : AIXpert Blog

Linux Kernel panic reboot

2009-01-22T02:14:00.001-08:00

By default after a kernel panic Linux just waits there for a sysadmin to hit the restart or powercycle button. This is because of the value set on "kernel.panic" parameter.

[root@linux23 ~]# cat /proc/sys/kernel/panic
0
[root@linux23 ~]# sysctl -a | grep kernel.panic
kernel.panic = 0
[root@linux23 ~]#

To disable this and make the Linux OS reboot after a kernel panic, we have to set an integer N greater than zero to the paramter "kernel.panic", where "N" is the number of seconds to wait before a automatic reboot. For example , if you set N = 10 , then the system waits for 10 seconds before automatic reboot. To make this permanent, edit /etc/sysctl.conf and set it.

[root@linux23 ~]# echo "10" > /proc/sys/kernel/panic
0
[root@linux23 ~]# grep kernel.panic /etc/sysctl.conf
kernel.panic = 10
[root@linux23 ~]#

Netapp Ontap Management SDK

2009-01-21T04:06:00.001-08:00

Netapp provides an SDK ( Netapp API ) that contains resources necessary to develop third-party applications which monitor and manage Netapp Filers. The SDK contains libraries, code samples and bindings in C and Perl for the Netapp Ontap management API programming.

The Ontap SDK contains ..

SDK Core API library bindings in C and C++, Java and Perl.
SDK Core API and Data ONTAP API documentation, sample codes, developer tools, Design guides.
Manage ONTAP SDK Help.

For more detailed information read : http://communities.netapp.com/docs/DOC-1110
You can download the SDK from : http://communities.netapp.com/docs/DOC-1365

Solaris : Auditing File attributes

2009-01-20T19:51:00.001-08:00

Solaris has a software registry which maintains information of software packages installed. The registry is invaluable for auditing the system to determine what software has been changed, installed, removed, or patched. The software registry contains a database of installed files. This database is physically located in the file /var/sadm/install/contents . Each file, special file, and directory installed on the system has an entry in this database. If some attributes of files are changed after installation, "pkgchk" command can find it out and report it. A good command for auditing. Here is an example..

solaris98# pkgchk
ERROR: /etc/apache/magic
    file size <12965> expected <12441> actual
    file cksum <8026> expected <33401> actual
ERROR: /etc/apache/mime.types
    file size <14987> expected <9957> actual
    file cksum <46595> expected <27635> actual
ERROR: /etc/auto_master
    file size <113> expected <395> actual
    file cksum <9773> expected <34676> actual
ERROR: /etc/default/dhcpagent
    file size <3394> expected <2826> actual
    file cksum <26394> expected <43621> actual

Some fiiles are expected to change such as /etc/system - which gets edited by sysadmin very often. pkgchk has a -n option that will bypass checking these files. Though this is a tempting option to use for reducing the amount of output from an audit, it is good to know what got changed.

solaris98# pkgchk -l -p /etc/system
Pathname: /etc/system
Type: editted file
Expected mode: 0644
Expected owner: root
Expected group: sys
Referenced by the following packages:
SUNWcsr
Current status: installed

solaris98#

If you want to check what got changed on a filesystem , you can use find & pkgchk to know it. Check the example below.

solaris98# find /usr -mount -exec pkgchk -p {} \;
ERROR: /usr
permissions <0755> expected <0775> actual
WARNING: no information associated with pathname </usr/platform/TSBW>
WARNING: no information associated with pathname <8000>
WARNING: no information associated with pathname </usr/platform/TSBW>
WARNING: no information associated with pathname <Ultra-2i>
..

Reference : http://www.sun.com/blueprints/1299/repairing.pdf

Linux Tips : Useful links

2009-01-19T07:38:00.001-08:00

Came across the below links and it has lot of useful linux tips - neat and great. Checkout them

Netapp Storage Commands

2009-01-07T19:27:00.001-08:00

Here are some of the useful functions of "storage" command in Netapp.

1) To show all disks on the system : Use "storage show disk -T" to display all the disks attached to the filer, the disk serial number, vendor, model, disk firmware version and type of disk (SATA/ATA/FCAL)

# rsh filer12 storage show disk -T
DISK                  SHELF BAY SERIAL           VENDOR   MODEL      REV TYPE
--------------------- --------- ---------------- -------- ---------- ---- ------
0d.16                   1    0 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.17                   1    1 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.18                   1    2 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.19                   1    3 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.20                   1    4 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.21                   1    5 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
0d.22                   1    6 xxxxxxxxxxxxxxxx NETAPP   X276 NA07 FCAL
...

2) To see complete information of a particular disk : Use "storage show disk -a <disk-id>" to view complete information of a netapp disk. This command gives you the shelf, bay, serial number of disk, disk speed and many other.

# rsh filer12 storage show disk -a 0d.99
Disk:             0d.99
Shelf:            5
Bay:              13
Serial:           xxxxxxxxxxxxxxxxxxxx
Vendor:           NETAPP
Model:            X276
Rev:              NA07
RPM:              10000
WWN:              xxxxxxxxxxxxxxxxxxa
UID:              xxxxxxxxxxxxxxxxx:00000000:00000000:00000000:00000000
Downrev:          no
Pri Port:         B
Power-on Hours:   N/A
Blocks read:      0
Blocks written:   0
Time interval:    00:00:00
Glist count:      0
Scrub last done: 00:00:00
Scrub count:      0
LIP count:        0
Dynamically qualified: No
#

3) To list all storage adapters on the filer : Use "storage show adapter -a" command to display all the storage adapters (hba) on the filer.

# rsh filer12 storage show adapter -a

Slot:            0a
Description:     Fibre Channel Host Adapter 0a (Dual-channel, QLogic 2322 rev. 3)
Firmware Rev:    3.3.25
FC Node Name:    xxxxxxxxxxxxxxxxxxx
FC Packet Size: 2048
Link Data Rate: 2 Gbit
SRAM Parity:     Yes
External GBIC:   No
State:           Enabled
In Use:          No
Redundant:       Yes

Slot:            0b
Description:     Fibre Channel Host Adapter 0b (Dual-channel, QLogic 2322 rev. 3)
Firmware Rev:    3.3.25
FC Node Name:    xxxxxxxxxxxxxxxxxxx
..

4) To get shelf details of filer : Use "storage show shelf <shelf-id>" command to display the details of the shelf and its partner shelf.

# rsh filer12 storage show shelf 0c.shelf2
Shelf name:    0c.shelf2
Channel:       0c
Module:        A
Shelf id:      2
Shelf UID:     xxxxxxxxxxxxxxxxxxxxxxx
Term switch:   N/A
Shelf state:   ONLINE
Module state: OK

                               Loop Invalid Invalid Clock Insert Stall Util    LIP
Disk    Disk     Port            up      CRC     Word Delta   Count Count Percent Count
Id     Bay    State         Count    Count    Count
----------------------------------------------------------------------------------------
[IN ]          OK                0        0        0      8       0      0    71     0
[OUT ]          OK                0        0        0      0       0      0    52     0
[ 32]     0    OK                0        0        0     32       0      0     0     0
[ 33]     1    OK                0        0        0     32       0      0     2     0
[ 34]     2    OK                0        0        0     24       0      0     0     0
[ 35]     3    OK                0        0        0     24       0      0     1     0
[ 36]     4    OK                0        0        0      8       0      0     2     0
[ 37]     5    OK                0        0        0     24       0      0     4     0
...

More Netapp commands at : http://unixfoo.blogspot.com/search/label/netapp

Netapp Snapmirror Setup Guide

2009-01-06T23:46:00.000-08:00

Snapmirror is an licensed utility in Netapp to do data transfer across filers. Snapmirror works at Volume level or Qtree level. Snapmirror is mainly used for disaster recovery and replication.

Snapmirrror needs a source and destination filer. (When source and destination are the same filer, the snapmirror happens on local filer itself. This is when you have to replicate volumes inside a filer. If you need DR capabilities of a volume inside a filer, you have to try syncmirror ). This guides you quickly through the Snapmirror setup and commands.

1) Enable Snapmirror on source and destination filer

source-filer> options snapmirror.enable
snapmirror.enable            on
source-filer>
source-filer> options snapmirror.access
snapmirror.access            legacy
source-filer>

2) Snapmirror Access

Make sure destination filer has snapmirror access to the source filer. The snapmirror filer's name or IP address should be in /etc/snapmirror.allow. Use wrfile to add entries to /etc/snapmirror.allow.

source-filer> rdfile /etc/snapmirror.allow
destination-filer
destination-filer2
source-filer>

3) Initializing a Snapmirror relation

Volume snapmirror : Create a destination volume on destination netapp filer, of same size as source volume or greater size. For volume snapmirror, the destination volume should be in restricted mode. For example, let us consider we are snapmirroring a 100G volume - we create the destination volume and make it restricted.

destination-filer> vol create demo_destination aggr01 100G
destination-filer> vol restrict demo_destination

Snapmirror is always destination filer driven. So the snapmirror initialize has to be done on destination filer. The below command starts the baseline transfer.

destination-filer> snapmirror initialize -S source-filer:demo_source destination-filer:demo_destination
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
destination-filer>

Qtree Snapmirror : For qtree snapmirror, you should not create the destination qtree. The snapmirror command automatically creates the destination qtree. So just volume creation of required size is good enough.

destination-filer> snapmirror initialize -S source-filer:/vol/demo1/qtree destination-filer:/vol/demo1/qtree
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.

4) Monitoring the status : Snapmirror data transfer status can be monitored either from source or destination filer. Use "snapmirror status" to check the status.

destination-filer> snapmirror status
Snapmirror is on.
Source                          Destination                          State          Lag Status
source-filer:demo_source        destination-filer:demo_destination   Uninitialized -   Transferring (1690 MB done)
source-filer:/vol/demo1/qtree   destination-filer:/vol/demo1/qtree   Uninitialized -   Transferring (32 MB done)
destination-filer>

5) Snapmirror schedule : If you want to sync the data on a scheduled frequency, you can set that in destination filer's /etc/snapmirror.conf . The time settings are similar to Unix cron. You can also set a synchronous snapmirror schedule in /etc/snapmirror.conf

destination-filer> rdfile /etc/snapmirror.conf
source-filer:demo_source        destination-filer:demo_destination - 0 * * * # This syncs every hour
source-filer:/vol/demo1/qtree   destination-filer:/vol/demo1/qtree - 0 21 * * # This syncs every 9:00 pm
destination-filer>

6) Other Snapmirror commands

To break snapmirror relation - do snapmirror quiesce and snapmirror break.
To update snapmirror data - do snapmirror update
To resync a broken relation - do snapmirror resync.
To abort a relation - do snapmirror abort

Netapp Simulator

2009-01-06T21:47:00.001-08:00

Those who are newly learning Netapp can use Netapp Data OnTap Simulator to get comfortable with Netapp commands. This tool gives you the experience of administering and using a NetApp storage system with all the features of Data ONTAP. The Simulator can be downloaded from http://now.netapp.com/NOW/cgi-bin/simulator ( you need NOW access ). The simulator has fully functional license keys for all Netapp functionalities.

The simulator can be loaded onto a Red Hat or SuSE Linux box and looks and feels exactly like Data ONTAP. Almost anything you can do with Data ONTAP can be done with the simulator. Without purchasing new hardware or impacting your production environment, you can test functionality, export NFS and CIFS shares etc.

Documents related to Netapp Simulator :

Netapp volume commands

2009-01-06T05:23:00.001-08:00

How to find failed disks on a filer ? . "vol status -f" command gives you the failed disk on a filer.

# rsh filer12 vol status -f

Broken disks

RAID Disk       Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)    Phys (MB/blks)
---------       ------ ------------- ---- ---- ---- ----- --------------    --------------
failed          3a.33   3a    2   1   FC:A   - FCAL 10000 68000/139264000   68552/140395088
failed          4a.28   4a    1   12 FC:A   - FCAL 10000 68000/139264000   68552/140395088
#

How to find spare disks on a filer ? . "vol status -s" command gives you the spare disks on a filer.

# rsh filer12 vol status -s

Spare disks

RAID Disk       Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)    Phys (MB/blks)
---------       ------ ------------- ---- ---- ---- ----- --------------    --------------
Spare disks for block or zoned checksum traditional volumes or aggregates
spare           2a.45   2a    2   13 FC:A   - FCAL 10000 68000/139264000   68552/140395088
spare           4a.57   4a    3   9   FC:A   - FCAL 10000 68000/139264000   68552/140395088
#

To find out the disks in an aggregate : Use "aggr status -r <aggregate-name>" to list all the disks that are part of the aggregage. This command gives the plex, raid and disk information.

# rsh filer12 aggr status -r aggr0
Aggregate aggr0 (online, raid_dp) (block checksums)
Plex /aggr0/plex0 (online, normal, active)
    RAID group /aggr0/plex0/rg0 (normal)

      RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)    Phys (MB/blks)
      --------- ------ ------------- ---- ---- ---- ----- --------------    --------------
      dparity   4a.15   4a    2   13 FC:A   - FCAL 10000 68000/139264000   69536/142410400
      parity    4a.16   4a    4   12 FC:A   - FCAL 10000 68000/139264000   68552/140395088
      data      4a.22   4a    4   8   FC:A   - FCAL 10000 68000/139264000   68552/140395088
#

To find out the volumes on a filer : "vol status" command is used to find volume on a filer. It gives the volume names and its status (online/offline/restricted)

# rsh filer12 vol status
         Volume State      Status            Options
          vol10 online     raid_dp, flex     no_i2p=on
          vol11 online     raid_dp, flex     no_i2p=on
           root online     raid_dp, flex     root, no_i2p=on
          vol12 offline    raid_dp, flex     no_i2p=on
#

More Netapp commands at : http://unixfoo.blogspot.com/search/label/netapp

How to find the version history of a rpm in Linux?

2008-12-23T23:50:00.000-08:00

The rpm command gives you the version history & changelog of a package , when you query the package with --changelog option.

Here is an example :

[root@linux-cloud ~]# rpm -q --changelog gzip
* Thu May 29 2008 Karanbir Singh <> 1.3.5-10.el5.centos
- Roll in CentOS Branding

* Wed Jan 16 2008 Ivana Varekova <> 1.3.5-10.el5
- Resolves: #247269
gzip munges directory mtime with "-r"

* Fri Dec 15 2006 Ivana Varekova <> - 1.3.5-9.el5
- fix regression in unpack algorithm
Resolves: #219539

* Mon Oct 02 2006 Jesse Keating <> - 1.3.5-9
- rebuilt for unwind info generation, broken in gcc-4.1.1-21

* Thu Sep 21 2006 Ivana Varekova <> 1.3.5-8
- fix bug 204676 (patches by Tavis Ormandy)
- cve-2006-4334 - null dereference problem
- cve-2006-4335 - buffer overflow problem
- cve-2006-4336 - buffer underflow problem
- cve-2006-4338 - infinite loop problem
- cve-2006-4337 - buffer overflow problem

* Sat Jul 15 2006 Karsten Hopp <> 1.3.5-7
- buildrequire texinfo, otherwise gzip.info will be empty

* Thu Jul 13 2006 Jesse Keating <> - 1.3.5-6.2.2
- rebuild
...

File Synchronization tool : Unison

2008-12-22T08:51:00.000-08:00

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. Unison runs on both Windows and many flavors of Unix (Solaris, Linux, OS X, etc.) systems. Moreover, Unison works across platforms,allowing you to synchronize a Windows laptop with a Unix server.

How to mount a ISO file

2008-12-22T08:05:00.000-08:00

To mount a ISO file on Linux machine, you should mount it as "loop" back filesystem. The option to be used in mount command is "-o loop". Check out the below example.

[root@linux-cloud images]# mount -o loop boot.iso /mnt
[root@linux-cloud images]#
[root@linux-cloud images]# df /mnt
Filesystem 1K-blocks Used Available Use% Mounted on
/media/fedora/images/boot.iso
131708 131708 0 100% /mnt
[root@linux-cloud images]# ls /mnt
images isolinux

[root@linux-cloud images]# mount | grep /mnt
/media/fedora/images/boot.iso on /mnt type iso9660 (ro,loop=/dev/loop0)
[root@linux-cloud images]#

Linux : yum package manager

2008-12-22T07:28:00.001-08:00

"yum" (YellowDog Updater modified) is a easy to use package manager for Linux. It installs/updates packages and all its dependencies. I started using yum and it is real good one and here are some details about yum. Yum configuration is stored in /etc/yum.conf and its repository list is stored in /etc/yum.repos.d/ . Below is the sample yum.conf of Centos 5.

[root@apple ~]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
[root@apple ~]#

Yum will look for remote repository information on the files stored in /etc/yum.repos.d/ . A sample file here.

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

Yum commands:

Yum has wide range of commands to install, update, check-update, list packages etc. Here let us see some important ones.

1) yum check-update : This returns list of packages that requires update on your machine.q

[root@apple ~]# yum check-update
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* rpmforge: ftp-stud.fht-esslingen.de

agg.x86_64             2.5-1.el5.rf           rpmforge
cadaver.x86_64         0.22.5-1.el5.rf        rpmforge
firefox.x86_64         3.0.5-1.el5.centos     updates
firefox.i386           3.0.5-1.el5.centos     updates
iso-codes.noarch       1.0a-1.el5.rf          rpmforge
lftp.x86_64            3.7.6-1.el5.rf         rpmforge
libwpd.x86_64          0.8.14-1.el5.rf        rpmforge
mtr.x86_64             2:0.75-1.el5.rf        rpmforge
nmap.x86_64            2:4.20-1.el5.rf        rpmforge

2) yum list : List all packages installed on your machine as well as the packages available in your repository.

[root@apple ~]# yum list
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* rpmforge: fr2.rpmfind.net

Installed Packages
Deployment_Guide-en-US.noarch    5.2-11.el5.centos      installed
GConf2.i386                      2.14.0-9.el5           installed

3) yum update : This updates the whole machine or a package on the system. To update a package, pass argument to yum update [ ex : yum update spamassassin ]

4) yum install : To install a package, use this command. If you use "-y" option with yum update or yum install, that supresses all user input.

[root@apple ~]# yum install tomcat5
Loading "fastestmirror" plugin

Dependencies Resolved

=============================================================================
Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
tomcat5                 x86_64     5.5.el5_2.1      updates           358 k
Installing for dependencies:
ant                     x86_64     1.6.5-2jpp.2     base              2.3 M
axis                    x86_64     1.2.1-2jpp.6     base              3.6 M

5) yum info : This is used to list a description and summary information about installed & available packages

[root@apple ~]# yum info NetworkManager-gnome
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* rpmforge: fr2.rpmfind.net
Installed Packages
Name   : NetworkManager-gnome
Arch   : x86_64
Epoch : 1
Version: 0.6.4
Release: 8.el5
Size   : 415 k
Repo   : installed
Summary: GNOME applications for use with NetworkManager
Description:
This package contains GNOME utilities and applications for use with
NetworkManager, including a panel applet for wireless networks.