Kernel Dump

Quick and dirty Application Log collection script

2011-12-22T19:00:00.000-06:00

To troubleshoot a crash/memory problem for a certain employee time and attendance platform, I need to collect log-files just before bouncing the Jboss service. Since these files are big, I thought the best way would be to copy them to a temporary location, compress them, and then delete the temp files. The only external requirements are Robocopy (already installed on Server 2008) and 7za, the standalone 7zip CLI utility.
Here's what I came up with. (save as a .bat or .cmd file)

REM ## KRONOS Log Collection SCRIPT ##
REM ## Doug Woodgate - doug.woodgate@nospam.nospam
REM    ## Created: 12/21/2011
REM ## ChageLog - Changed date string and modified to collect Kronos logs,

REM Following Commands create the required robocopy JOB FILES
REM robocopy /NOSD /NODD *.log *.log.* /MAXAGE:7 /COPY:D /LEV:4 /NP /S /R:0 /SAVE:RCJ\Logs7d /QUIT
REM robocopy C:\inetpub\logs /NODD /SAVE:RCJ\IIS /QUIT
REM robocopy C:\Kronos /NODD /SAVE:RCJ\KRON /QUIT
REM robocopy C:\Kronos\jboss /NODD /SAVE:RCJ\JBOSS /QUIT
REM robocopy C:\Kronos\wfc /NODD /SAVE:RCJ\WFC /QUIT

SET PATH=%PATH%;%~dp0;C:\mlcTools\bin
set YYYYMMDD=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%
if "%TIME:~0,1%"==" " (
    set hhmmss=0%TIME:~1,1%%TIME:~3,2%%TIME:~6,2%
) else (
    set hhmmss=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
)
set now=%YYYYMMDD%-%hhmmss%

REM ### Editable options
Set dest=C:\temp\wfclogs-%now%
REM ### End of Editable options
mkdir %dest%

ECHO ON
robocopy /JOB:RCJ\IIS /JOB:RCJ\LOGS7D %dest%\IIS
robocopy /JOB:RCJ\JBOSS /JOB:RCJ\LOGS7D %dest%\JBOSS
robocopy /JOB:RCJ\WFC /JOB:RCJ\LOGS7D %dest%\WFC

7za.exe a -mx7 -m0=PPMd %dest%.7z %dest% -mmt
rmdir /S /Q %dest%

Monitor Jboss JMX values via SNMP in WhatsUp Gold

2011-12-09T19:12:00.000-06:00

After not finding much out there on the interwebs, I wrote a ActiveScript Performace Monitor(Jscript) to graph the Java heap size. It will grab the Committed or Used Cache, Eden, Survivor, OldGen or PermGen depending on what you name this monitor. 

JSCRIPT CODE 

var nSNMPPort = 16161; // change this value to the port your agent is running on  
var oSnmpRqst =  new ActiveXObject("CoreAsp.SnmpRqst");  
var nDeviceID = Context.GetProperty("DeviceID");  // Get the device ID  
var getOid = ".1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1";
var sName = Context.GetProperty("PerformanceMonitorTypeName");
if (sName.indexOf("Commit") > -1) getOid = getOid + ".12";
if (sName.indexOf("Used") > -1) getOid = getOid + ".11";
if (sName.indexOf("Cache") > -1) getOid = getOid + ".1";
if (sName.indexOf("Eden") > -1) getOid = getOid + ".2";
if (sName.indexOf("Survivor") > -1) getOid = getOid + ".3";
if (sName.indexOf("OldGen") > -1) getOid = getOid + ".4";
if (sName.indexOf("PermGen") > -1) getOid = getOid + ".5";

var oComResult = oSnmpRqst.Initialize(nDeviceID);  
if(oComResult.Failed) Context.SetResult(1, oComResult.GetErrorMsg);
var oComResult = oSnmpRqst.SetPort(nSNMPPort);
var oSnmpResult = oSnmpRqst.Get(getOid);  
if (oSnmpResult.Failed) Context.SetResult(1, "Epic FAIL sName = " + sName + "Oid=" + getOid + " Port=" + nSNMPPort + "Error=" + oSnmpResult.GetErrorMsg + oSnmpResult.GetPayload);      
else  Context.SetValue(oSnmpResult.GetValue);

Websense DC agent won't start.

2008-04-03T12:22:00.005-05:00

From polywogsys here is how to fix a "1067: Service terminated unexpectedly" error when trying to restart the Websense DC agent on Windows 2003

cd \path\to\websense\bin
XidDCAgent -s
XidDCAgent -c

Press enter after a little while and eventually the command prompt will respond. Also, you may notice that the service is now running after the last command. I restarted it to make sure it can start automatically on reboot.

Initially I thought that maybe someone changed the password of our websense user, so I reset it, reset the password of the service and within Websense, but that did not clear it up. I also noticed that DEP was killing the Database agent, so I turned that off. Eventually I found this post and it saved the day.

Hope this helps!

Doug

How to compile Nagios plugins for Solaris 10 Sparc

2008-01-31T23:48:00.000-06:00

Time to setup monitoring for our new Sun Fire T2000 servers running Solaris 10 and Oracle.... That being said, I began compiling the nagios plugins in the normal way, however, I discovered there a couple tweaks on Solaris 10.

Here's a way to get it done... BTW, check_disk does NOT work with ZFS zpools yet...

Log in as a normal user acount. Invoke bash shell.
bash

Un-tar the archive.
gunzip nagios-plugins-1.4.11.tar.gz tar -xvf nagios-plugins-1.4.11.tar
Set path as follows.
PATH=$PATH:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin

Mysql libraries are compiled using Sun's C compiler and have incompatible flags. Skip the mysql package in configure. If you absolutely need check_mysql, then consider using Sun Studio to compile the plugins.
./configure --without-mysql

Run gmake instead of make.
gmake

Su to root. set path, run gmake install
su
PATH=$PATH:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin
gmake install

Done. Now you can navigate to /usr/local/nagios/libexec to test them out. You should be able to tar them up and place them on any Solaris 10 sparc server and they should work. They will need to be re-compiled for Solaris 9 however.

Now here is what not to do...

-bash-3.00$ PATH=$PATH:/usr/sbin

-bash-3.00$ ./configure
checking for a BSD-compatible install... build-aux/install-sh -c
checking whether build environment is sane... yes
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... no
checking build system type... sparc-sun-solaris2.10
checking host system type... sparc-sun-solaris2.10
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.

-bash-3.00$ PATH=$PATH:/usr/sfw/bin
-bash-3.00$ ./configure
... Sucess (or so I think...)

-bash-3.00$ find / -name make 2>&1 | grep -v cannot
/usr/share/lib/make
/usr/ccs/bin/make
/usr/xpg4/bin/make

-bash-3.00$ /usr/xpg4/bin/make

make all-recursive
bash: make: command not found
*** Error code 127
make: Fatal error: Command failed for target `all'

*HINT*
http://forum.java.sun.com/thread.jspa?threadID=5096509&messageID=9331271
-bash-3.00$ PATH=$PATH:/usr/ccs/bin
-bash-3.00$ /usr/xpg4/bin/make
....
*** Error code 255
make: Fatal error: Command failed for target `libgnu.a'
Current working directory /export/home/datactr/nagios-plugins-1.4.11/gl
*** Error code 1
make: Fatal error: Command failed for target `all'
Current working directory /export/home/datactr/nagios-plugins-1.4.11/gl
*** Error code 1
The following command caused the error:
failcom='exit 1'; \
for f in x $MAKEFLAGS; do \
case $f in \
*=* | --[!k]*);; \
*k*) failcom='fail=yes';; \
esac; \
done; \
dot_seen=no; \
target=`echo all-recursive | sed s/-recursive//`; \
list='gl lib plugins plugins-scripts plugins-root po'; for subdir in $list; do \
echo "Making $target in $subdir"; \
if test "$subdir" = "."; then \
dot_seen=yes; \
local_target="$target-am"; \
else \
local_target="$target"; \
fi; \
(cd $subdir && make $local_target) \
|| eval $failcom; \
done; \
if test "$dot_seen" = "no"; then \
make "$target-am" || exit 1; \
fi; test -z "$fail"
make: Fatal error: Command failed for target `all-recursive'
Current working directory /export/home/datactr/nagios-plugins-1.4.11
*** Error code 1
make: Fatal error: Command failed for target `all'

Ah Ha... Re configure...
-bash-3.00$ ./configure
-bash-3.00$ make clean
-bash-3.00$ gmake ....
gcc: language strconst not recognized
gcc: check_mysql.c: linker input file unused because linking not done
mv: cannot access .deps/check_mysql-check_mysql.Tpo
gmake[2]: *** [check_mysql-check_mysql.o] Error 2
gmake[2]: Leaving directory `/export/home/datactr/nagios-plugins-1.4.11/plugins'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/export/home/datactr/nagios-plugins-1.4.11'
gmake: *** [all] Error 2

-bash-3.00$ ./configure --without-mysql
-bash-3.00$ gmake clean
-bash-3.00$ gmake
-bash-3.00$ su
# PATH=$PATH:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin
# gmake install

Installing Openfiler 2.2 on Dell Powervault 715N

2008-01-29T21:14:00.000-06:00

So, after re-organizing some stuff in our datacenter, we pulled an old Powervault 715n from the racks as it has been shut down for a while and just taking up space. Being somewhat adventurous, I decided to embark on the journey to get Openfiler installed on it. If you have a Powervault 715n, you probably already know it is a headless machine with two NICs and a serial console. Oh, and to make it trickier, no CD ROM drive...
Armed with a cisco roll (null modem) cable, a crossover cable, some patience and an extra machine running Vmware Server, I was able to get Openfiler to PXE boot and install via a serial port.

Here's how:

First, download the latest stable Openfiler 32bit ISO.
http://www.openfiler.com/download/
Either setup a PXE server or download a PXE server appliance. I found PXE Installer and Recovery Appliance to work well. (instructions below will cover the Appliance)
http://www.vmware.com/appliances/directory/371
Either transfer the ISO to the Vmware appliance or attach it to the cdrom of the vm.
Mount the ISO on the VM and share the mount point with NFS. (add it to the /etc/exports file)
/mnt *(rw,sync,no_root_squash)
Start nfs and portmap
/etc/init.d/portmap start
/etc/init.d/nfs start
Make a directory under /tftpboot called openfiler.
mkdir /tftpboot/openfiler
Copy initrd.img and vmlinuz from /images/pxeboot on the ISO to /tftpboot/openfiler
cp /mnt/images/pxeboot/initrd.img /tftpboot/openfiler
cp /mnt/images/pxeboot/vmlinuz /tftpboot/openfiler
Now edit the /tftpboot/pxelinux.cfg/default file and add the following to the top. If there is a "default whatever" line, comment it out...
serial 0 115200 0
default serial
Then add the following to the same file after the label 4 stanza
label 5
kernel openfiler/vmlinuz
append initrd=openfiler/initrd.img text ramdisk_size=8192 console=ttyS0,115200n8
Now bridge the PXE_Installer-Recovery VM to a nic off your corporate network!!! Connect the crossover cable to the NIC and connect the other end to the Powervault. Start dhcpd:
/etc/init.d/dhcpd start
Attach your serial cable to the PC that you will use to control the install. Fire up a serial port terminal and configure it to 115200,8n1 XON/XOFF.
Turn on the Powervault and ensure you are seeing the POST messages from boot. Press F2 When you see "Press F2 for Setup function. You may need to hit it a couple of times to get your point across...
Now press 3 for Reinstallation, the server will reboot and after some time, you should see the NICs searching for a DHCP address. If sucessfull, you should see a bunch of hard to read output in a column on the left of the screen and a prompt. Press 5 and enter.
Now you should see the standard output of linux boot processes kicking off. You should come up to a prompt asking for the install source. Select NFS.
Now, select the interface that is attached to the crossover cable and get a DHCP address.
On the next screen, type in the address of the NFS server "192.168.100.1" and the path, "/mnt" Select Next.

If everything is successful, you should be able to navigate through Anaconda and install Openfiler. It may take a couple of tries to get the system setup the way you want boot partition wise, but in the end, you should have a useable system. You then will be able to use the serial port to attach to the console of the device for configuration later.

Good Luck!!
-Doug

Swftools on Red Hat/Centos 5

2008-01-18T17:46:00.000-06:00

Since there was no RPM for RHEL 5 for swftools, I decided to compile one for a project at work.

Download the latest src rpm from http://dag.wieers.com/rpm/packages/swftools/

Install the src RPM on your redhat build machine. Also, install the rpmforg-release package as you will need to install another package to compile the code.

Now, install libjpeg-devel t1lib-devel freetype freetype-devel giflib and giflib-devel

NOTE: t1lib-devel is in DAG's repo

yum install libjpeg-devel t1lib-devel freetype freetype-devel giflib and giflib-devel

All of the dependencies should figure themselves out...

Now, build the package:

i386 (32-bit)

rpmbuild -bb --clean --define "rhel 5" -vv --target i386 --define "_smp_mflags -j1" --define "_rpmfilename %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm" --define "_initrddir %{_sysconfdir}/rc.d/init.d" --define "debug_package %nil" --define "dist el5" --define "disttag el5" --define "el5 1" --define "el5i 1" /usr/src/redhat/SPECS/swftools.spec

x86_64 (64-bit)

rpmbuild -bb --clean --define "rhel 5" -vv --target x86_64 --define "_smp_mflags -j1" --define "_rpmfilename %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm" --define "_initrddir %{_sysconfdir}/rc.d/init.d" --define "debug_package %nil" --define "dist el5" --define "disttag el5" --define "el5 1" --define "el5a 1" /usr/src/redhat/SPECS/swftools.spec

If all goes well, you should have your binary rpm in /usr/src/redhat/RPMS

Cya & Good Luck!!

Doug