TechSutram(/=1)

Process monitoring on Unix/Linux/Solaris/HP-UX/AIX

2011-12-04T16:31:00.001+05:30

It’s often a requirement to monitor Unix processes in the field of performance monitoring. However, there is a catch. Different flavors of Unix have different process monitoring tools such as `top`. Of course `top` is not available on all platforms. Similarly HP-UX has something called ‘glance’. This is again very platform specific.

The only CLI that is supported for process monitoring across different flavors of Unix is ‘ps’. However, one of the most two parameters that we are interested in process monitoring is its SIZE and CPU usages. Both of these parameters can be retrieved using ‘ps’ CLI.

To obtain size of a process, we generally use PS parameters ‘vsz’. It’s the size of process in virtual memory. To obtain CPU usages of a process, we generally use PS parameter `cpu` which gives absolute CPU usage of a process. On some Unices, PS provides parameter called ‘pcpu’ which provides percentage of CPU usage by a process. There are still limitations of this approach as described above. Few of the unices do not provide ‘pcpu’ parameter and instead provide ‘cpu’ parameter e.g HP-UX.

However, it’s still better to rely on single UNIX provided `ps` CLI that can be used inside a any shell or Perl script. In order to use `ps` CLI we should be relying on UNIX95 Unix standard as defined in the Single UNIX Specification.

e.g. Following CLI prints “our_process” usages on HP-UX with columns viz. process name, its virtual memory size and its absolute CPU usages on single line. It is just a matter of changing cpu to pcpu and application of few filters on some of the Unix platforms as per our requirements. But it does work.

# export UNIX95=1; ps –eo args,vsz,cpu | grep “our_process”

It just a matter of directing output of above CLI to a file post application of filters

# cntr=1;while test $cntr;do export UNIX95=1; ps -eo args,vsz,sz,cpu |egrep -v "our_process" | awk '{print $1"\t"$2\”\t”$4}' ;done >> /tmp/our_process.txt &

Above while loop keep on dumping process virtual memory size and CPU usages of our_process in /tmp/our_process.txt file. Since output is in tabbed format, this output file can be easily opened in Microsoft Excel and graphs can be plotted.

Do you use any other methodology to monitor processes on Unix machines? If yes then do let everyone know in comments below.

CPU Info – Linux, Solaris, AIX, HP-UX

2011-09-30T22:36:00.001+05:30

Below are the CLIs to display CPU information on Linux, Solaris, AIX and HP-UX.

LINUX:

# cat /proc/cpuinfo

SOLARIS:

# psrinfo –v

# prtdiag

or

# prtdiag | grep SUNW

# prtconf

AIX:

# lscfg -l proc*

and

# lsattr -El <proc_num_from_above_cli>

HP-UX:

# machinfo

High Availability Lens 3

2011-09-30T21:19:00.001+05:30

In previous high availability lens 2, we saw general formula to express availability in terms of uptime and downtime. And also in terms of MTBF and MTTR.

MTTR is mean time to repair ,i.e. average time required to repair a failure. If we look at the formula for availability in terms of MTBF and MTTR, we can observe that we can make very interesting conclusion. As MTTR tends to zero, availability tends to 1 ,i.e. 100%

And this is where clustering softwares come into picture by helping MTTR reduce by automation. It should be also noted that MTTR is affected by lot of complexities such as application dependencies, underlying hardware being used etc.

Firefox script timeout and workaround

2011-09-27T20:39:00.001+05:30

In Firefox scripts time out frequently while accessing web pages which large data. By large data, I mean that scripts/web pages taking long time to crunch data on client-side. e.g. plotting JavaScript based graphs with more than 20K data points. In this case, data has to be downloaded at client side and graphs will be plot. (This is just an example. We can process these points on sever-side and send chart image to client side.)

However, Firefox will timeout in such cases and possibly hang. In order to workaround this problem, we can set few DOM preferences available in Firefox. Below are the steps that I used to resolve similar kind of problem with Firefox.

1. Navigate to Firefox preferences by typing “about:config “ in address bar

2. Type in “dom” and look for “dom.max_chrome_script_run_time” and “dom.max_script_run_time” and set this values to more than 30(seconds by default) e.g. 60 or 120 seconds.

More info @ DOM Preferences

3. Restart Firefox.

This workaround has worked for me. Are there are any other workarounds that you know? then please post it in the comments below.

Firefox sync settings are lost

2011-08-07T14:11:00.001+05:30

One of the most used features of Firefox on my computer is Firefox sync (previously Mozilla Weave).

However, once Firefox is upgraded from 3.x to 4.x (and now to 5.x), Mozilla Firefox stops syncing when Firefox is closed. The issue is default Privacy settings that shipped with Firefox and clears all history when Firefox is closed. This looks like a bug in Mozilla Firefox.

As a workaround to this issue, on Windows navigate to Tools –> Options –> Privacy Tab –> History and there set dropdown for “Firefox will:” to “Remember History”. For Linux, just navigate to Edit –> Preferences and follow same steps as above. These settings worked for me.

Hope it helps!

How to find Linux version ?

2011-07-30T17:55:00.001+05:30

Most of the times when we say a Linux version then we are referring to the specific Linux distributions…such as Red Hat, SuSe, Debian, Ubuntu etc.

Below are the CLIs that list their distribution specific Linux versions.

On Red Hat Linux Distribution:

# cat /etc/redhat-release

On Novell SuSe Linux Distribution:

# cat /etc/SuSE-release

On Debian Linux Distribution:

# cat /etc/debian_version

On Ubuntu Linux Distribution:

# lsb_release –a

# cat /etc/lsb-release

The contents of these files shouldn’t be changed or altered since different server softwares such as Oracle database refer to those files to decide their own compatibility. Sometimes /etc/issue or /etc/issue.net is also referred for their contents. However, it is not reliable since it is overwritten on some distributions during system boot.

To find exact Linux kernel version installed on the machine, following CLIs can be used…

# uname –a
Linux linux-bk6098 2.6.34.8-0.2-default #1 SMP 2011-04-06 18:11:26 +0200 i686 i686 i386 GNU/Linux

# cat /proc/version
Linux version 2.6.34.8-0.2-default (geeko@buildhost) (gcc version 4.5.0 20100604 [gcc-4_5-branch revision 160292] (SUSE Linux) ) #1 SMP 2011-04-06 18:11:26 +0200

If you know any Linux distribution that differs from above CLI for determining their version then do post your comments below.

HP-UX disk mapping between persistent and legacy DSF

2011-07-25T16:50:00.001+05:30

Below cli lists all mapping between legacy and agile DSF for storage attached to HP-UX server.

# ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
/dev/rdisk/disk1         /dev/rdsk/c0t0d0
/dev/rdisk/disk1_p1      /dev/rdsk/c0t0d0s1
/dev/rdisk/disk1_p2      /dev/rdsk/c0t0d0s2
/dev/rdisk/disk1_p3      /dev/rdsk/c0t0d0s3

HP-UX kernel : 32 bit or 64 bit ?

2011-07-25T14:23:00.001+05:30

Below is the CLI to identify kernel bits on HP-UX,

# getconf KERNLE_BITS
64 <---for 64 bit kernel or
32 <---for 32 bit kernel

Another alternate way to find out the same is...

# print_manifest | grep -i "OS mode"
OS mode: 64 bit

Identifying Processor support…

# getconf HW_CPU_SUPP_BITS
64

# getconf HW_32_64_CAPABLE
1

and it works! :)

Create single HP-UX tape .depot from multiple HP-UX depots

2011-07-13T18:00:00.001+05:30

I was looking for a way to create .depot file on HP-UX and CLI swpackage came to rescue.

Below is the CLI for creating single tape .depot file from multiple software depots.

Assumption: Your depot packages are in current directory

# /usr/sbin/swpackage -x target_type=tape -s `pwd` -d `pwd`/YourPackages.depot "YourPackages"

Is there any other way to create .depot package on HP-UX? Do let everyone know in comments below.

Perl Catalyst FormBuilder Plug-in and Jquery formwizard

2011-06-23T00:28:00.001+05:30

Recently I was working on popular Perl framework, Catalyst. One of the tasks was to generate form quickly and obviously used FormBuilder though it is no longer updated (I guess).

However, the traditional output of FormBuilder is not very nice as compared to newly designed forms so decided to integrate the same form (from FormBuilder) with jQuery Form Wizard. I have pretty lengthy form (around 20 fields) to fill.

Though it didn’t take me too long for Catalyst FormBuilder plug-in integration with jQuery Form Wizard what took most of my time is CSS.

Since I am not a good frontend designer, after lot of CSS tweaking I managed to generate nice form. And at the end, I am amazed to see such a nice wizard form with transition effects is generated using Perl and jQuery Form Wizard.

So instead of switching to any other language for designing website (backend), currently decided to stick to Perl. :)

Hopefully in next few posts, I will try to post sample code for this integration.

Shell script to translate uppercase text to lowercase

2011-05-29T19:37:00.001+05:30

While shell scripting on UNIX/LINUX/SOLARIS/AIX et cetera, it is often a requirement to translate uppercase characters to lowercase characters.

It can be a user input or contents of file that one might be processing using shell scripting.

Below one liner uses ‘tr’ CLI to translate uppercase characters to lowercase characters.

# strName=”TECHSUTRAM IN LOWERCASE"
# echo $strName| tr ‘[A-Z]’ ‘[a-z]’
techsutram in lowercase

Simple, isn’t it? However, there is a catch. The ‘tr’ CLI used for translation might not work on all UNIX flavors to convert all uppercase characters to lowercase.

The trick is to use slightly lengthy form of ‘tr’ to make translation almost portable to other UNIX flavors. So above code can be rewritten as,

# strName=”TECHSUTRAM IN LOWERCASE"
# echo $strName| tr ‘ABCDEFGHIJKLMNOPQRSTUVWXYZ’ ‘abcdefghijklmnopqrstuvwxyz’
techsutram in lowercase

Same mechanism works for translating lowercase characters to uppercase characters as below.

# strName=”techsutram in uppercase"
# echo $strName| tr ‘abcdefghijklmnopqrstuvwxyz’ ‘ABCDEFGHIJKLMNOPQRSTUVWXYZ’
TECHSUTRAM IN UPPERCASE

That’s it. This trick has worked for me.
Do you know any other common method that can be used to translate uppercase characters to lowercase characters? If YES then let everyone know in comments below.

You may also be interested in,
+ Converting .wav to .mp3 file format in Linux
+ MD5SUM for all files in directory
+ FTP file upload using shell script on LINUX/UNIX

Converting .wav to .mp3 file format in Linux

2011-05-11T18:42:00.001+05:30

I just had a requirement to convert a CD with .wav files to .mp3 format. The first thing which I did was to boot into OpenSuse 11.3 box and launch lame.

e.g. to convert Test.wav file to Test.mp3, we can use following syntax,

# lame –h –b 192 Test.wav Test.mp3

We can use above CLI to process set of files using following one liner assuming all .wav files are in current directory,

# for FILE in `ls *.wav`;do `which lame` –h –b 192 $FILE ${FILE//\.wav}.mp3;done

That's it!

You may also be interested in,
+ Shell script to translate uppercase text to lowercase
+ MD5SUM for all files in directory
+ FTP file upload using shell script on LINUX/UNIX

CPAN update all PERL modules

2011-05-08T12:42:00.001+05:30

Below is one liner that can be used to update all PERL modules,

#perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'

However, some modules eventually requires unstable version of PERL. Its just a matter of choice to go with unstable version of PERL.

I tried this CLI on OpenSuSe 11.3 and it worked well for me.

Product Testing Vs Project Testing

2011-04-10T12:58:00.001+05:30

Before we dive into differences of software Product Testing Vs Project Testing, it is essential to define the context that defines what software product and software project is.

A software Product can be a single software project or set of software projects owned by individual(s) or organization(s). Also, these software project(s) can be developed and tested individually or outsourced to different companies and finally integrated to work with each other to deliver end Product.

So the real difference is the scope of testing and requirements.

Software Product Testing	Software Project Testing
Testing is performed for your/organization's own product	Testing is performed for individual project from a client or different functional groups in same organization.
Market requirements influence the overall testing	Clients or different functional groups that drive requirements influence the overall testing
Test Cases are reviewed internal to the organization	Test Cases should be also reviewed by Client or different functional groups as well.
Product testing encompasses testing results of all individual software testing projects	Project testing is very specific to the software project under test.
Product testing will be continuous activity if features are being added into a product over different releases.	Since every project has its end date, software testing project also has its definite end date.

Do you feel any other subtle differences that worth noting in this post? If yes then let everyone know in comments below.

You may also be interested in,
+ Differences IPv4 Vs IPv6
+ Differences HTML Vs HTML5
+ Differences Routing Vs Routed protocol
+ Use Vs Require in PERL

MD5SUM for all files in directory

2011-04-04T22:33:00.001+05:30

Below is the CLI that can be used to recursively get md5sum for all files in a current directory.

#find . –type f –print0 | xargs –0 md5sum

DBD::mysql and Can't exec "mysql_config": No such file or directory at Makefile.PL error

2011-03-04T22:25:00.001+05:30

I was trying to install ‘DBD-mysql-4.018’ on OpenSuse 11.3 box and have following set of packages installed for MySQL.

-------------------------START----------------------->

linux:/home/Builder/DBD-mysql-4.018 # rpm -qa | grep -i mysql
php5-mysql-5.3.3-0.7.1.i586
libmysqlclient_r16-5.1.46-2.18.i586
mysql-community-server-client-5.1.46-2.18.i586
mysql-community-server-5.1.46-2.18.i586
libmysqlclient16-5.1.46-2.18.i586
libqt4-sql-mysql-4.6.3-1.3.i586
perl-DBD-mysql-4.014-1.10.i586

<------------------------END--------------------------

Interestingly, when I tried to run CLI ‘perl Makefile.PL’, it ran into following set issue…

------------------------START------------------------->

linux:/home/Builder/DBD-mysql-4.018 # perl Makefile.PL
Can't exec "mysql_config": No such file or directory at Makefile.PL line 82.

Cannot find the file 'mysql_config'! Your execution PATH doesn't seem
not contain the path to mysql_config. Resorting to guessed values!
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located

<------------------------END--------------------------

I tried to Google for this issue but couldn’t find exact steps. To resolve this issue, I have installed libmysqlclient-devel-5.1.46-2.18.i586.rpm package on my OpenSuse 11.3 box.

Now same rpm query for MySQL packages results in following output…

------------------------START----------------------->

linux:/home/mandar/Builder/DBD-mysql-4.018 # rpm -qa | grep -i mysql
=php5-mysql-5.3.3-0.7.1.i586
libmysqlcppconn1-1.0.5-6.2.i586
libmysqlclient_r16-5.1.46-2.18.i586
mysql-community-server-client-5.1.46-2.18.i586
mysql-community-server-5.1.46-2.18.i586
libmysqlcppconn-devel-1.0.5-6.2.i586
libmysqlclient-devel-5.1.46-2.18.i586
libmysqlclient16-5.1.46-2.18.i586
libqt4-sql-mysql-4.6.3-1.3.i586
perl-DBD-mysql-4.014-1.10.i586

<-----------------------END---------------------------

This has resolved issue with DBD::mysql module on OpenSuse 11.3

OpenSuse 11.3 and compiling Perl 5.12.3

2011-02-07T23:38:00.001+05:30

I was trying to compile Perl 5.12.3 on OpenSuse 11.3 for catalyst framework environment. However, during compilation of Perl source code, it exited at ODBM_File.xs (below is small snip from the session, not pasting whole compilation…)

----------------------START------------------------->

./miniperl -Ilib make_ext.pl lib/auto/ODBM_File/ODBM_File.so MAKE=make LIBPERL_A=libperl.a LINKTYPE=dynamic
Making ODBM_File (all)

Running Makefile.PL in ext/ODBM_File
../../miniperl Makefile.PL INSTALLDIRS=perl INSTALLMAN1DIR=none INSTALLMAN3DIR=none PERL_CORE=1 LIBPERL_A=libperl.a

LINKTYPE=dynamic
Processing hints file hints/linux.pl
Writing Makefile for ODBM_File
make[1]: Entering directory `/home/catalyst/perl-5.12.3/ext/ODBM_File'
make[1]: Leaving directory `/home/catalyst/perl-5.12.3/ext/ODBM_File'
Making all in ext/ODBM_File
make all PERL_CORE=1 LIBPERL_A=libperl.a LINKTYPE=dynamic
make[1]: Entering directory `/home/catalyst/perl-5.12.3/ext/ODBM_File'
cp ODBM_File.pm ../../lib/ODBM_File.pm
../../miniperl "-I../../lib" "-I../../lib" ../../lib/ExtUtils/xsubpp -noprototypes -typemap

../../lib/ExtUtils/typemap -typemap typemap ODBM_File.xs > ODBM_File.xsc && mv ODBM_File.xsc ODBM_File.c
cc -c -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

-O2 -DVERSION=\"1.07\" -DXS_VERSION=\"1.07\" -fPIC "-I../.." ODBM_File.c
ODBM_File.xs: In function ‘XS_ODBM_File_DESTROY’:
ODBM_File.xs:124:2: error: too few arguments to function ‘dbmclose’
/usr/include/dbm.h:62:12: note: declared here
make[1]: *** [ODBM_File.o] Error 1
make[1]: Leaving directory `/home/catalyst/perl-5.12.3/ext/ODBM_File'
Unsuccessful make(ext/ODBM_File): code=512 at make_ext.pl line 449.
make: *** [lib/auto/ODBM_File/ODBM_File.so] Error 25
catalyst@linux-bk68:~/perl-5.12.3> vi /usr/include/dbm.h

<------------------------END---------------------------

It was observed that ‘dbmclose’ call in ODBM_File.xs is dbmclose(void) to that specified in /usr/include/dbm.h dbmclose __P((DBM *)) on OpenSuse 11.3. Below is snip from session for ODM_FIle.xs and dbm.h file.

--------------------------START--------------------->

catalyst@linux-bk68:~/perl-5.12.3> grep -n dbmclose /home/catalyst/perl-5.12.3/ext/ODBM_File/ODBM_File.xs
15:int    dbmclose(void);
25: * DBM on at least Ultrix and HPUX call dbmclose() from dbminit(),
26: * resulting in duplicate free() because dbmclose() does *not*
28: * If some malloc/free calls have been done between dbmclose() and
35:#define dbmclose()    ((void) dbminit("/non/exist/ent"))
124:    dbmclose();
catalyst@linux-bk68:~/perl-5.12.3>

linux-bk68:/home/catalyst/perl-5.12.3 # grep dbmclose /usr/include/dbm.h
/*extern int dbmclose __P((DBM *));*/
extern int dbmclose __P((void));
linux-bk68:/home/catalyst/perl-5.12.3 #

<-------------------------END-------------------------

This has corrected the problem. However, keeping fingers crossed. Hope it doesn’t break my current system configuration.

Types of Test Automation Framework

2011-01-21T00:28:00.001+05:30

Just a quick post to list few categories of Test Automation Framework.

1. Data Driven Test Frameworks

2. Keyword Driven Test Frameworks

3. Test scripts’ Driven Test Frameworks

4. Test Library Driven Test Frameworks

5. Hybrid Test Frameworks

Do you know any other category to classify test automation frameworks?

High Availability Lens 2

2011-01-01T16:10:00.004+05:30

In our last post HA Lens 1, we have seen what is mean by High Availability. Availability can be measured in numbers provided these numbers are used consistently for behavior of given system. Availability is essentially a function of Reliability and it is just one of the factors that affects system availability. When we refer to high available system, we are referring to whole system that we are trying to highly available.
Availability can be expressed in % time in hours per week,month or year and measured using following formula,

                     (Operating Time) 
Availability = ------------------------------
               (Operating Time + Downtime)

Another common formula to measure availability is,

                         MTBF 
Availability = ------------------------------
                     (MTBF + MTTR) 
where, 
MTBF = Mean Time Between Failure
MTTR = Mean Time To Repair

High Availability Lens 1

2010-12-21T18:22:00.001+05:30

This is one of the series of posts that I have planned to write about high availability. This post exclusively tries to explain High Availability the way I understood.

The term Availability refers to the time required for system to respond to user request, i.e. the time period over which service is available often documented in Service Level Agreement (SLA). Since there could be different kinds of Outages that may cause services to become unavailable, there is need to manage those Downtimes (measured in time units, of course) caused by those outages. Minimizing these downtimes is collectively managed by designing Fault Tolerant systems and SLA.

So Highly Available systems are combination of SLA and fault tolerant systems essentially providing characteristics that totally/partially avoids downtimes or loss of service.

Agile Testing Video

2010-12-10T17:23:00.001+05:30

Agile testing video from Google Tech Talk.

Tutorials on IBM SAN Volume Controller and other key concepts

2010-12-01T12:54:00.001+05:30

In previous post, e-Learning module for IBM SAN Volume Controller, we referred to single e-Learning module for IBM SAN Volume Controller. However, it doesn’t cover other key concepts related to SAN Volumes.

Link below refers to all the Learning modules provided by IBM for SAN Volume controller and other key concepts.

e-Learning modules for the SAN Volume Controller other key concepts

e-Learning module for IBM SAN Volume Controller

2010-12-01T12:38:00.001+05:30

e-learning module for IBM SAN Volume Controller is good starting point for people who are new to storage systems and wanted to learn about what is storage virtualization, thin-provisioning and mirroring.

Below is link to access e-Learning module for IBM SAN Volume Controller.

Flash video for IBM SAN Volume Controller

Non-flash copy can be accessed at,

Non-flash copy script for IBM SAN Volume Controller

Data Center HA design guide from CISCO

2010-11-24T22:02:00.001+05:30

Just wanted to share data center high availability guide from CISCO. It is an excellent reference guide for HA Basics and data center design.

It can be accessed online in HTML format at http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/HA_Clusters/HA_Clusters.html

Download this book in PDF format.

Linux kernel exploit And privilege escalation

2010-10-23T17:56:00.000+05:30

A flaw in Reliable Datagram Sockets protocol (RDS) implementation in the Linux kernel can be exploited to gain root access. The kernel exploit released by http://www.vsecurity.com

I used OpenSuse 11.3 with latest kernel to try this exploit on my local machine.
And compiled, executed the exploit.

Indeed, it works great. Below is snapshot of this exploit in running condition on my local machine. I used 'id' CLI on Linux to verify privilege escalation.

TechSutram(*/*=1)