IT on Demand

OMD (Check_MK) Alert Notification Integration with PagerDuty Done Right

2015-09-03T13:27:00.001-07:00

Yes, shit just hit the fan. What are you gonna do?

If you are thinking about integrating OMD or Check_MK alert notification with PagerDuty.com, you are in the right place. The official documentation from PagerDuty is not done right by the Flexible Notification feature provided with OMD (Open Monitoring Distribution) or Check_MK.

If you don’t know what Flexible Notification is or what OMD is about, I recommend you to check out my other blog post - The Best Open Source Monitoring Solution 2015.

Create Notification Service on PagerDuty

Step 1. Log in to PagerDuty as an admin user. Click Service under the Configuration menu option.

Step 2. Click the Add New Service button.

Step 3. Fill out the form as the arrows indicated in the following image.

Step 4. Congratulations, we are done with the PagerDuty part. Grab the Service API Key for you are going to need it later.

Install PagerDuty Notification Script

Step 1. SSH into the OMD server

Step 2. Install Perl dependencies

FOR RHEL, Fedora, CentOS, and other Redhat-based distributions:

yum install perl-libwww-perl perl-Crypt-SSLeay perl-Sys-Syslog

For Debian, Ubuntu, and other Debian-based distributions:

apt-get install libwww-perl libcrypt-ssleay-perl libsys-syslog-perl

Step 3. Download pagerduty_nagios.pl from github, copy it to /usr/local/bin and make it executable:

wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.pl 
cp pagerduty_nagios.pl /usr/local/bin
chmod +x /usr/local/bin/pagerduty_nagios.pl

Step 4. Create cron job to flush notification queue
First become the OMD/Check_MK site user in shell, then create a cron.d file under /omd/sites/<<Site Name>>/etc/cron.d/pagerduty with the following content:

#
# Flush PagerDuty notification queue
#

* * * * * /usr/local/bin/pagerduty_nagios.pl flush

Now enable the cron job as the OMD site user:

omd reload crontab

Since the cron job runs every minute, you can change back to the root user and check to see if the cron job has been triggered as expected.

[root@omd.server.com ~]# tail -f /var/log/cron
Sep  4 08:25:01 omd.server.com CROND[24090]: (omd_user) CMD (/usr/local/bin/pagerduty_nagios.pl flush)
Sep  4 08:26:01 omd.server.com CROND[27175]: (omd_user) CMD (/usr/local/bin/pagerduty_nagios.pl flush)
Sep  4 08:27:01 omd.server.com CROND[30195]: (omd_user) CMD (/usr/local/bin/pagerduty_nagios.pl flush)

Integrate with OMD Flexible Notification

Assume you are still connected to OMD server with SSH.

Step 1. Adding custom Flexible Notification script
copy and save the following script to location /omd/sites/{YOUR-SITE}/local/share/check_mk/notifications/pagerduty.sh

#!/bin/bash
# PagerDuty

PAGERDUTY="/usr/local/bin/pagerduty_nagios.pl"

# For Service notification
if [ "$NOTIFY_WHAT" = "SERVICE" ]; then
    echo "$PAGERDUTY enqueue -f pd_nagios_object=service -f CONTACTPAGER=\"$NOTIFY_PARAMETER_1\" -f NOTIFICATIONTYPE=\"$NOTIFY_NOTIFICATIONTYPE\" -f HOSTNAME=\"$NOTIFY_HOSTNAME\" -f SERVICEDESC=\"$NOTIFY_SERVICEDESC\" -f SERVICESTATE=\"$NOTIFY_SERVICESTATE\""
    $PAGERDUTY enqueue -f pd_nagios_object=service -f CONTACTPAGER="$NOTIFY_PARAMETER_1" -f NOTIFICATIONTYPE="$NOTIFY_NOTIFICATIONTYPE" -f HOSTNAME="$NOTIFY_HOSTNAME" -f SERVICEDESC="$NOTIFY_SERVICEDESC" -f SERVICESTATE="$NOTIFY_SERVICESTATE"

# For Host notification
else
    $PAGERDUTY enqueue -f pd_nagios_object=host -f CONTACTPAGER="$NOTIFY_PARAMETER_1" -f NOTIFICATIONTYPE="$NOTIFY_NOTIFICATIONTYPE" -f HOSTNAME="$NOTIFY_HOSTNAME" -f HOSTSTATE="$NOTIFY_HOSTSTATE"
fi

Step 2. Make it executable

chmod +x pagerduty.sh

Step 3. Log in to OMD or Check_MK web interface and configure Flexible Notification to use the NEW PagerDuty notification script.

Assuming you already know how to operate Flexible Notification. Select PagerDuty as the Notification Plugin and put the API key you acquired earlier when setting up PagerDuty service into the Plugin Arguments field as shown in the image.

Since pagerduty_nagios.pl was designed to work with Nagios, it doesn’t take flapping notifications. Make sure you uncheck those boxes.

You can create multiple PagerDuty services and pair them up with OMD/Check_MK Flexible Notification. Happy ending.

Testing and Troubleshooting

Test pagerduty_nagios.pl

To send a test notification directly with the pagerduty_nagios.pl, use the following example and swap out <<API Key>> and <<HOST Name>> with your own value.

Make sure you become the OMD site user first because once you run the command once, it’s going to create a directly in /tmp/pagerduty_nagios. If you run the command as root now, you will have permission issue later when OMD is trying to send notification to PagerDuty with a different user.

/usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f CONTACTPAGER="<<API Key>>" -f NOTIFICATIONTYPE="PROBLEM" -f HOSTNAME="<<HOST Name>>" -f SERVICEDESC="this is just a test" -f SERVICESTATE="CRIT"

You will be able to find output in syslog, and depend on the OS variation you use, location may vary.

If you get the following error message like I did:

perl: symbol lookup error: /omd/sites/<<site Name>>/lib/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Encode/Encode.so: undefined symbol: Perl_Istack_sp_ptr

Add the following lines to the beginning of the PagerDuty Perl Script located /usr/local/bin/pagerduty_nagios.pl

use lib '/usr/lib64/perl5/';
no lib '/omd/sites/monitor/lib/perl5/lib/perl5/x86_64-linux-thread-multi';

Test with Flexible Notification

Step 1.
You first need to enable debugging for notification from the web UI. Enable setting in Global Settings -> Notifications -> Debug notifications. The resulting log file is in the directory notify below Check_MK’s var directory. OMD users find the file in ~/var/check_mk/notify/notify.log. Remember switch it back after you are done debugging.

Step 2.
Now pick a Host for Service that you’ve configure it’s notification to use the PagerDuty plugin, Click the Hammer icon on the top and click on Critical button in the Various Commands section.

Step 3.
Now log in to the PagerDuty account and select Dashboard form the top menu. In a minute or two, you should see some thing as shown in the image. If not, you need to go back to the log files and figure out why.

If you do see your fake incident appear on the PagerDuty dashboard, CONGRATULATIONS .

Share with us, comment on what notification mechanism do you use? Do you build it in-house or use a popular 3rd party service?

Scale Selenium Grid in 5 Seconds with Zero Docker Experience

2015-07-17T15:01:00.001-07:00

The reason to use CoreOS as a Docker server is because CoreOS is an extremely light weight, stripped down Linux distribution containing none of the extras that are associated with Ubuntu and the like. And it was designed to run Docker and Docker clusters, so by using it we are buying the future.

Many cloud service provider already has CoreOS image to start with. If you are not going to be running the CoreOS server on bare metal or if you have other Docker server already installed, you can skip this step.

Install CoreOS on Bare Metal (Hardware)

Download the stable CoreOS ISO from here: Download Link

Then you can burn the ISO into a CD/DVD or a bootable USB disk and use it as the boot source to boot up your physical server.

Once the command line becomes available, create a cloud-config.yml file with the ssh-key you will be using to connect to the server from remote later. The content of the file should look like this:

#cloud-config

ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0g+ZTxC7weoIJLUafOgrm+h...

CoreOS allows you to declaratively customize various OS-level items, such as network configuration, user accounts, and systemd units. This document describes the full list of items we can configure. The coreos-cloudinit program uses these files as it configures the OS after startup or during runtime.

Unlike AWS, cloud-config file is run during EACH system boot. While it is inspired by the cloud-init project, there are tools isn’t used by CoreOS. Only the relevant subset of its configuration items is implemented in CoreOS cloud-config file. Please refer to the official CoreOS document for detial on the subject.
cloud-config Documentation Link

Now run the install command with the cloud-config file name as argument:

coreos-install -d /dev/sda -c cloud-config.yml

Once you complete the above steps, reboot the server and make sure you remove your boot disk or boot CD/DVD. The server will boot up in command line mode and display it’s IP address obtained from DHCP server. From this point on, you can only ssh into the server via the ssh-key you provided earlier.

Install and Setup Docker Compose

Compose is a tool for defining and running multi-container applications with Docker. With Compose, you define a multi-container application in a single file, then spin your application up in a single command which does everything that needs to be done to get it running.

Docker Compose is the key component here to make spinning up or tearing down an entire Selenium Grid Farm (1 hub + multiple browsers) in a single command. Yes, you read it right, just ONE command. Installing Compose is a bit tricky though because of how CoreOS is built.

Here is the video that showed me how to install docker-compose on CoreOS toward the end of the video. You can skip the video because it’s 46 minutes long. And I will show you the exact steps on how you can do the same next.

Install Docker Compose

docker-compose is just a precompiled binary file we can download from its Github Page. Check out the link and choose the latest version number to replace in the following code (run as the default core user):

mkdir ~/bin
curl -L https://github.com/docker/compose/releases/download/1.3.3/docker-compose-`uname -s`-`uname -m` > ~/bin/docker-compose
chmod +x ~/bin/docker-compose
echo "export PATH="$PATH:$HOME/bin"" >> ~/.bashrc
source ~/.bashrc

Again if you are not using CoreOS, please follow the official installation instruction and skip the above code block.

Now the docker-compose command is ready for use, simply run docker-compose in shell, you should see the following:

core@localhost ~ $ docker-compose
Define and run multi-container applications with Docker.

Usage:
  docker-compose [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

Options:
  -f, --file FILE           Specify an alternate compose file (default: docker-compose.yml)
  -p, --project-name NAME   Specify an alternate project name (default: directory name)
  --verbose                 Show more output
  -v, --version             Print version and exit

Commands:
  build              Build or rebuild services
  help               Get help on a command
...

Configure Docker Compose Application File

We’ll be using Docker images build by the official Selenium repository on Docker Hub, so you don’t need to build your own.

Under /home/core/ , create a file named docker-compose.yml with the following content:

hub:
  image: selenium/hub
  ports:
    - "4444:4444"
firefox:
  image: selenium/node-firefox
  links:
    - hub
chrome:
  image: selenium/node-chrome
  links:
    - hub

You can use other browser images in the Selenium repository in the docker-compose.yml file, and they should just work.

Manage Selenium Grid with Docker Compose

If you are running this for the first time, expect Docker to download all the necessary images in the beginning. Once all the images are downloaded into local repository, it will just take seconds to start the Selenium Grid in the future.

Start Selenium Grid

As user core, run:

docker-compose -f ~/docker-compose.yml up -d

By appending the -d argument at the end of the command, you are telling Compose to run the application in the background (daemon).

Sample output:

core@localhost /usr $ docker-compose -f ~/docker-compose.yml up -d
Recreating core_hub_1...
Recreating core_firefox_1...
Recreating core_chrome_1...

And you can verify it by running docker ps:

core@localhost /usr $ docker ps
CONTAINER ID        IMAGE                          COMMAND                CREATED              STATUS              PORTS                                                                         NAMES
a09d3ab302fa        selenium/node-chrome:latest    "/opt/bin/entry_poin   About a minute ago   Up About a minute                                                                                 core_chrome_1
298a612a387a        selenium/node-firefox:latest   "/opt/bin/entry_poin   About a minute ago   Up About a minute                                                                                 core_firefox_1
62136ab8fdb0        selenium/hub:latest            "/opt/bin/entry_poin   About a minute ago   Up About a minute   0.0.0.0:4444->4444/tcp                                                        core_hub_1

To see the real action in browser, open url http://your-coreos-IP:4444/grid/console

Scale Selenium Grid

Let’s say you need 5 Firefox browser and 5 Chrome browser in your Grid. Simply run:

docker-compose scale firefox=5 chrome=5

Sample output:

core@localhost ~ $ docker-compose scale firefox=5 chrome=5
Creating core_firefox_2...
Creating core_firefox_3...
Creating core_firefox_4...
Creating core_firefox_5...
Starting core_firefox_2...
Starting core_firefox_3...
Starting core_firefox_4...
Starting core_firefox_5...
Creating core_chrome_2...
Creating core_chrome_3...
Creating core_chrome_4...
Creating core_chrome_5...
Starting core_chrome_2...
Starting core_chrome_3...
Starting core_chrome_4...
Starting core_chrome_5...

And now your Selenium Hub management web page looks like this:
enter image description here

Tearing Down Selenium Grid

Demolishing the entire selenium grid is just as easy as starting them. Since it only takes seconds to start, have a clean slate for all the Selenium tests is not a dream any more. You want all the browser to start without previous cookies or settings, and all is possible now with Docker containers.

docker-compose -f ~/docker-compose.yml stop && docker-compose -f ~/docker-compose.yml rm -f

Sample output:

core@localhost ~ $ docker-compose -f ~/docker-compose.yml stop && docker-compose -f ~/docker-compose.yml rm -f
Stopping core_chrome_1...
Stopping core_firefox_1...
Stopping core_hub_1...
Going to remove core_chrome_1, core_firefox_1, core_hub_1
Removing core_hub_1...
Removing core_firefox_1...
Removing core_chrome_1...

Integrate with Jenkins

Now to have it integrated with Jenkins for automated testing. I will update this when I have this setup in my environment.

How to Clone a Live Production Linux Server with This Cool Technique

2015-05-08T01:10:00.001-07:00

Do you find yourself stuck with some old Ubuntu Server that runs some critical application on a Physical Hardware which is running out of resources. You don’t want to waste rack space by cloning it to another physical server. You are a smart guy, you know cloning it to a VM gives you all the benefit of modern day server management.

What if the original hard drive has couple hundred Gigs of disk space and the actual data is only a few to a few dozen Gigs? You don’t want all that unused disk space to be also cloned into the VM via dd command. Is there a way to clone the server to a smaller disk that lives on a KVM VM guest? What if you can’t risk shutting down the physical server because it is in production and is a single point of failure? Look no further, here’s how you can live clone a old running Linux server to a KVM virtual machine.

Prepare New KVM Guest Image

Step 1. - Log in to your KVM host server and create a qcow2 image file with enough disk space to clone the actual data from the physical server:

qemu-img create -f qcow2 new-vm.qcow2 100G

Step 2. - Let’s attach the qcow2 file to a local device on KVM host so you can partition and format it.

mkdir /mnt/new-vm
modprobe nbd max_part=8
qemu-nbd -c /dev/nbd0 new-vm.qcow2
cfdisk /dev/nbd0
mkfs.ext3 -v /dev/nbd0p1

cfsidk - Create partition. Tailor this to your environment, instruction HERE.
mkfs.ext3 - Format the disk to ext3 filesystem. You can also use mkfs.ext4 or other type of filesystem depend on your situation.

Step 3. - Mount the drive

partprobe /dev/nbd0
mount /dev/nbd0p1 /mnt/new-vm

Clone Data from Physical Server

Step 4. - Rsync (Copy) all the files from the physical server
Do a dry run first.

rsync -e 'ssh -p 22' -avxn root@physical.server.com:/ /mnt/new-vm/

Now do it for real if you don’t see any problem with the dry run.

rsync -e 'ssh -p 22' -avx root@physical.server.com:/ /mnt/new-vm/

Step 5. - Sync data for the last time
Make sure all cached data are written to the hard drive on the physical server by running:

sync;sync;sync

Go back to the KVM host and do final rsync again

rsync -e 'ssh -p 22' -avx root@physical.server.com:/ /mnt/new-vm/

Clean up

Step 6. - Update disk UUID in GRUB bootloader
On the KVM host:

blkid

You will get a list of mounted drives UUID. Copy the one that follows /dev/nbd0. Use that value and replace all the disk UUID in /mnt/new-vm/boot/grub/grub.cfg

Step 7. - Clean up /etc/fstab
Do the same for /mnt/new-vm/etc/fstab and replace with the new UUID or just remove all UUID in the fstab file.

Step 8. - Clean up network settings

Change IP address by editing /mnt/new-vm/etc/network/interfaces
Remove cached mac address from udev. /mnt/new-vm/etc/udev/rules.d/70-persistent-net.rules
Edit any application settings that might create conflict with your physical server once you stand the new VM up.

Step 9. - Un-mount the drive

umount /mnt/new-vm
qemu-nbd -d /dev/nbd0

Stand Up the New VM

virt-install \
--connect qemu:///system \
-n vm-name \
--os-type linux \
--vcpus=2 \
--ram 2048 \
--disk path=/path/to/new-vm.qcow2 \
--vnc \
--vnclisten=127.0.0.1 \
--noautoconsole \
--import

Change the vm-name, vcpus, ram, and disk location to your liking. If everything go smooth, you should see these success messages on screen:

Starting install...
Creating domain...                                                                             |    0 B     00:01
Domain creation completed. You can restart your domain by running:
  virsh --connect qemu:///system start new-vm

You can now try to ssh into the new VM using the new IP and see if things are working. To troubleshoot any issue, you will need to use VNC to connect to your VM and debug accordingly. Congratulations! You’ve now save your company from a legacy dying server to the new, shiny, scalable, replicable, and testable VM. Now this is money ~~~

Kamailio High Availability Done Right with Keepalived

2015-01-21T01:05:00.001-08:00

Keepalived is a Linux implementation of the VRRP (Virtual Router Redundancy Protocol) protocol to make IPs highly available - a so called VIP (Virtual IP).

Usually the VRRP protocol ensures that one of participating nodes is
master. The backup node(s) listens for multicast packets from a node
with a higher priority. If the backup node fails to receive VRRP
advertisements for a period longer than three times of the
advertisement timer, the backup node takes the master state and
assigns the configured IP(s) to itself. In case there are more than
one backup nodes with the same priority, the one with the highest IP
wins the election.

I have ditched Corosync + Pacemaker for simpler and easily manageable cluster via Keepalived. Any of you who have played with Pacemaker will understand what I mean. I assume you’ve already got your Kamailio server installed and configured.

Install Packages

I am using Ubuntu server for the installation.

apt-get install -y keepalived sipsak

sipsak here will be wrapped in a check script to ensure Kamailio is still accepting SIP quests. VRRP is good for detecting network failure, but we also need something in place to check on the service level.

Set Kamailio to listen on VIP

Allow Kamailio to bind or listen to the VIP even when the VIP does not exist on the backup node. You first need to append the following to /etc/sysctl.conf

# allow services to bind to the virtual ip even when this server is the passive machine
net.ipv4.ip_nonlocal_bind = 1

Enable the setting immediately:

sysctl -p

Change Kamailio’s config file /etc/Kamailio/Kamailio.cfg to force Kamailio to listen on the VIP:

listen=<<VIP>>:<<Port Num>>

Apply the above to both severs, and restart Kamailio:

service Kamailio restart

Configure Keepalived

Create /etc/keepalived/keepalived.conf with the following content on both nodes

!Configuration File for keepalived

vrrp_script check_sip {
    script       "/etc/keepalived/checksip.sh"
    interval 5   # check every 5 seconds
    fall 2       # require 2 failures for KO
    rise 4       # require 4 successes for OK
}

vrrp_instance SBC_1 {
    state BACKUP
    interface eth0
    virtual_router_id 56
#   priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass somepassword
    }
    virtual_ipaddress {
        123.123.123.123/24 brd 123.123.123.255 dev eth0 label eth0:0
    }
    track_script {
        check_sip
    }
    notify_master "/etc/keepalived/master-backup.sh MASTER"
    notify_backup "/etc/keepalived/master-backup.sh BACKUP"
    notify_fault "/etc/keepalived/master-backup.sh FAULT"
}

If you have multiple pair of Keepalived cluster running on the same subnet, make sure each pair’s vrrp_instance name and virtual_router_id number are different or else the failover will not work correctly.
Comment out the priority so that the VIP won’t flap when a failed node comes back online. In other words, make the VIP sticky to the node it’s currently running on.
Change the fall and rise interval under the vrrp_script check_sip according to your need. Once the script determined that Kamailio is down, Keepalived will failover the VIP to the backup server.
Notification parameters at the bottom of the configuration:
- notify_master - will be triggered on the server that is becoming the master server.
- notify_backup - will be triggered on the server that is degraded to be a backup server or starting up as a backup server.
- notify_faults - will be triggered when any of the vrrp check script reached it’s threshold and determines the service is no longer available, then the status will turn into faults and trigger the notify_faults script.

Apply the same configuration file on both the master and backup nodes.

Create Checking Script

There is no fencing mechanism available for Keepalived. If two participating nodes don’t see each other, both will have the master state and both will carry the same IP(s). When I was looking for a way to detect which one should stay the master or give up his master state, I discovered the Check Script mechanism.

A check script is a script written in the language of your choice which is executed regularly. This script needs to have a return value:

0 for “everything is fine”
1 (or other than 0) for “something went wrong”

This value is used by Keepalived to take action. Scripts are defined like this in /etc/keepalived.conf:

vrrp_script check_sip {
    script       "/etc/keepalived/checksip.sh"
    interval 5   # check every 5 seconds
    fall 2       # require 2 failures for KO
    rise 4       # require 4 successes for OK
}

As you can see in the example, it’s possible to specify the interval in seconds and also how many times the script needs to succeed or fail until any action is taken.

The script can check anything you want. Here are some ideas:

Is the daemon X running?
Is the interface X on the remote switch Y up?
Is the IP 8.8.8.8 pingable?
Is there enough disk space available to run my application?
$MYIDEA

Check out the Check Script Configuration Samples section on how to do some of those.

Create /etc/keepalived/checksip.sh and make sure it is executable.

#!/bin/bash

if ls /etc/keepalived/MASTER; then
    timeout 1 sipsak -s sip:s@10.10.10.10:5060
    exit
else
    exit 0
fi

This script will test SIP service on Kamailio’s private IP and make sure Kamailio is working on the Master node. I will explain how I use the /etc/keepalived/master-backup.sh to determine the master node in a little bit.

Notify Script

A notify script can be used to take other actions, not only removing or adding an IP to an interface. It can trigger any script you desire based on . And this is how it can be defined in the Keepalived configuration:

vrrp_instance MyVRRPInstance {
 [...]
 notify_master "/etc/keepalived/master-backup.sh MASTER"
 notify_backup "/etc/keepalived/master-backup.sh BACKUP"
 notify_fault "/etc/keepalived/master-backup.sh FAULT
}

notify_mater - is triggered when the node becomes a master node.
nofity_backup - is triggered when the node becomes a backup node.
notify_fault - is triggered when VRRP determine the network is at fault or when one of your check script has reached its fault threshold.

So with the above setting, we want to create the script in /etc/keepalived/master-backup.sh and make it executable:

#!/bin/bash

STATE=$1
NOW=$(date +"%D %T")
KEEPALIVED="/etc/keepalived"

case $STATE in
        "MASTER") touch $KEEPALIVED/MASTER
                  echo "$NOW Becoming MASTER" >> $KEEPALIVED/COUNTER
                  /etc/init.d/Kamailio start
                  exit 0
                  ;;
        "BACKUP") rm $KEEPALIVED/MASTER
                  echo "$NOW Becoming BACKUP" >> $KEEPALIVED/COUNTER
                  /etc/init.d/Kamailio stop || killall -9 Kamailio
                  exit 0
                  ;;
        "FAULT")  rm $KEEPALIVED/MASTER
                  echo "$NOW Becoming FAULT" >> $KEEPALIVED/COUNTER
                  /etc/init.d/Kamailio stop || killall -9 Kamailio
                  exit 0
                  ;;
        *)        echo "unknown state"
                  echo "$NOW Becoming UNKOWN" >> $KEEPALIVED/COUNTER
                  exit 1
                  ;;
esac

We rely on this script to start or stop Kamailio server. The reason I am not running active-active Kamailio HA is because is because if the backup Kamailio is running and sending probing to its load balancing targets, it will generate good amount of VIP errors continuous in the syslog file for the VIP it does not own yet.

I rather have a cleaner log file that I can use to troubleshoot later and sacrifice a few seconds waiting for the backup Kamailio to start.

Boot Sequence

As I mentioned, we don’t want Kamailio to start on boot on its own. We want Keepalived to manage the start and stop of Kamailio application. Here is how we disable Kamailio on boot:

update-rc.d -f Kamailio remove

The boot system on Ubuntu is currently messed up. Some application use Upstart and some application still use System V, there is no telling which upstart managed application will start first. But if you are using MySQL as the database for Kamailio, you will want to ensure MySQL is started before Kamailio. And because we are using Keepalived to control Kamailio, therefore we want MySQL to start before Keepalived when boot:

Disable Keepalived from starting via System V

update-rc.d -f keepalived remove

Create an Upstart script in /etc/init/keepalived.conf that will start Keepalived service after MySql server is started on boot:

# Keepalived Service

description     "Keepalived"

start on started mysql
pre-start script
  /etc/init.d/keepalived start
end script
post-stop script
  /etc/init.d/keepalived stop
end script

Restart Keepalived on both nodes when you are ready to :

service keepalived restart

Check Script Configuration Samples

Showcase different ways to check services:

vrrp_script chk_sshd {
       script "killall -0 sshd"        # cheaper than pidof
       interval 2                      # check every 2 seconds
       weight -4                       # default prio: -4 if KO
       fall 2                          # require 2 failures for KO
       rise 2                          # require 2 successes for OK
}

vrrp_script chk_haproxy {
       script "killall -0 haproxy"     # cheaper than pidof
       interval 2                      # check every 2 seconds
}

vrrp_script chk_http_port {
       script "</dev/tcp/127.0.0.1/80" # connects and exits
       interval 1                      # check every second
       weight -2                       # default prio: -2 if connect fails
}

vrrp_script chk_https_port {
       script "</dev/tcp/127.0.0.1/443"
       interval 1
       weight -2
}

vrrp_script chk_smtp_port {
       script "</dev/tcp/127.0.0.1/25"
       interval 1
       weight -2
}

How to incorporate the above checks into vrrp_instance configs:

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 100
    virtual_ipaddress {
        192.168.200.18/25
    }
    track_interface {
       eth1 weight 2   # prio = +2 if UP
       eth2 weight -2  # prio = -2 if DOWN
       eth3            # no weight, fault if down
    }
    track_script {
       chk_sshd                # use default weight from the script
       chk_haproxy weight 2    # +2 if process is present
       chk_http_port
       chk_https_port
       chk_smtp_port
    }
}

vrrp_instance VI_2 {
    interface eth1
    state MASTER
    virtual_router_id 52
    priority 100
    virtual_ipaddress {
        192.168.201.18/26
    }
    track_interface {
       eth0 weight 2   # prio = +2 if UP
       eth2 weight -2  # prio = -2 if DOWN
       eth3            # no weight, fault if down
    }
    track_script {
       chk_haproxy weight 2
       chk_http_port
       chk_https_port
       chk_smtp_port
    }
}

For every set of VIPs, use a new vrrp_instance profile with unique instance name and virtual_router_id number. track_interface is optional for checking network interfaces, it will mark FAULT state if any of the interface goes down.

Monitoring with OMD or Check_MK

Ah ha, I knew you would ask. This section will come out later.

TCP Failover?

I have done my share of research work to find a solution to safely migrate established TCP connection from master node to backup node without success. It will be so cool to have SIP TCP session to stay alive during the failover. Even so, I am keeping my notes here for people who would like to give it a stab.

All the following effort seemed like working when I check the TCP session after failover but when I actually trying to verify the session, it fails. If anyone of you know anything that works, please do share with us in the comment section.

conntrack commands

conntack man page
Make sure you have the proper kernel modules enabled before any of the command can produce results for you. Run lsmod in Linux shell and see if the output contains the following results.

nf_conntrack_ipv4      19716  0
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4

If you don’t have the above, try running modprobe ip_conntrack and retest the above.

Assume you have the above ready, you can use the following commands to validate connection session on the Linux server.

# List connection tacking or expectation table and display output in extended format
conntrack -L -o extended

# List connection tacking or expectation table
conntrack -L

# Display a real-time event log
conntack -E

Happy HA~

Special Thanks to These References:
https://tobrunet.ch/2013/07/keepalived-check-and-notify-scripts/

Fake it till You Make it ep. 1 - Learn New IT Skills Like a Maniac

2014-07-19T17:32:00.001-07:00

Remember the last time you need to pick up a piece of new technology or programming language among the other hundred thousand things you need to take care of? Well, it wasn’t pleasant was it. When I started learning about Linux, I wanted to be fast and know lots of command on top of my head. Years goes by, I realize my dream of wanting to be a COOL Sys Admin is slipping away because… until I found a solution toward the end of 2012.

The technique I am about to reveal not only help IT people but anyone who needs to deal with typing, emailing, documenting, and customer support all the time. If my technique is able to save you an average of 5 minutes a day, that is about 30 HOURS SAVED every year. That time saved can be used to learn one extra skill or free for whatever you want to do. Without further ado, check out my presentation below:

Sys Admin on Steroids

Let me know if you guys need some video demos to show how well it works in the comments area.

CPU Load Spiked with No Apparent Reason - Check RAID Battery

2014-05-14T23:08:00.001-07:00

Scenario

If you got here in an emergency to bring back your server health, please skip to the Ugent Care section at the bottom of this page.

You are here because you can’t figure out what on earth is eating up your CPU load. I mean, you’ve checked almost everything that can cause high CPU load, things like:

Busy application
Disk I/O bound
Slow database queries
System using swap instead or RAM
Dependency on a remote mount point being slow

When none of the above seems to be the cause of your CPU load problem, you might want to check your RAID battery status or BBU (Battery Backup Unit) relearn cycle. The only kind of RAID controller covered here is LSI which you would find in most Dell servers.

Root Cause

This is often seen on RAID controllers in Dell servers, specifically LSI controllers. The default behavior of the LSI RAID controller is to set to periodically go through a so-called ‘relearn cycle’ for its battery backup unit (BBU). During the process, it discharges, then re-charges and calibrates itself to find the current charge level. In this time frame, the RAID write cache policy is set to WriteThrough instead of WriteBack where WriteThrough is accessing the physical disk directly and WriteBack uses the RAID cache as the write buffer.

Without the cache buffering, disk I/O is drastically slowed down and will cause CPU to wait much longer for any application requests. During busy hours, it will appears as a symptom of high CPU load with no visible reasons. There are a few things we can do to prevent this happening again in the future, but lets begin with the right tool and commands to use for troubleshooting.

MegaCli and BBU

MegaCli is the command line tool for managing LSI RAID controllers on Dell, and a few other servers. We want to first verify if your server is using LSI RAID controllers.

Check RAID Controller Brand

lspci | grep -i raid

Sample Output:

01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)

Check Storage Device Detail

lshw -class storage

Sample Output:

  *-storage
       description: RAID bus controller
       product: MegaRAID SAS 1078
       vendor: LSI Logic / Symbios Logic
       physical id: 0
       bus info: pci@0000:01:00.0
       version: 04
       width: 64 bits
       clock: 33MHz
       capabilities: storage pciexpress msi msix pm vpd bus_master cap_list rom
       configuration: driver=megaraid_sas latency=0
       resources: irq:16 memory:fc480000-fc4bffff ioport:ec00(size=256) memory:fc440000-fc47ffff memory:fc300000-fc307fff

Install MegaCli Tool

Download the MegaCli package from LSI website. Click Here to Find the right package. I would just get the Latest MegaCli for Linux package for Linux servers.

Unzip the downloaded file and install it with package manager that suits you. I am using Ubuntu here as an example:

unzip MegaCli_Linux.zip
dpkg -i megacli_8.07.08-1_all.deb

Linux Kernel >= 3.0

If your server is running on Kernel that is newer or equal to version 3.0, you want to prefix your MegaCli command like the following as a fix.

setarch x86_64 --uname-2.6 MegaCli

Check RAID and Battery Backup Unit Health

Check Enclosure Info

MegaCli -EncInfo -aALL -NoLog

Sample output:

    Number of enclosures on adapter 0 -- 1

    Enclosure 0:
    Device ID                     : 32
    Number of Slots               : 6
    Number of Power Supplies      : 0
    Number of Fans                : 0
    Number of Temperature Sensors : 0
    Number of Alarms              : 0
    Number of SIM Modules         : 0
    Number of Physical Drives     : 4
    Status                        : Normal
    Position                      : Unavailable
    Connector Name                : Unavailable
    Partner Device Id             : 65535

Show physical drive information of all connected drives

MegaCli -PDList -aALL -NoLog

Sample output:

Adapter #0

Enclosure Device ID: 32
Slot Number: 0
Device Id: 0
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 238.474 GB [0x1dcf32b0 Sectors]
Non Coerced Size: 237.974 GB [0x1dbf32b0 Sectors]
Coerced Size: 237.875 GB [0x1dbc0000 Sectors]
Firmware state: Online
SAS Address(0): 0x4433221107000000
Connected Port Number: 1(path0)
Inquiry Data: S1ATNSAF156143F     Samsung SSD 840 PRO Series              DXM05B0Q
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Solid State Device
------------------truncated-----------------------

Show virtual drive information

MegaCli -LDInfo -Lall -aALL -NoLog

Sample output:

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:475.599 GB
State: Optimal
Stripe Size: 64 KB
Number Of Drives per span:2
Span Depth:2
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Encryption Type: None

Make sure the Current Cache Policy is using WriteBack when battery is healthy. If you see WriteThrough instead, you need to check into the BBU health and if it is running a relearn of battery status.

References (extracted from Dell OpenManage doc):

Write Policy: The write policies specify whether the controller sends a write-request completion signal as soon as the data is in the cache or after it has been written to disk.

Write-Back. When using write-back caching, the controller sends a write-request completion signal as soon as the data is in the controller cache but has not yet been written to disk. Write-back caching may provide improved performance since subsequent read requests can more quickly retrieve data from the controller cache than they could from the disk. Write-back caching also entails a data security risk, however, since a system failure could prevent the data from being written to disk even though the controller has sent a write-request completion signal. In this case, data may be lost. Other applications may also experience problems when taking actions that assume the data is available on the disk.
Write-Through. When using write-through caching, the controller sends a write-request completion signal only after the data is written to the disk. Write-through caching provides better data security than write-back caching, since the system assumes the data is available only after it has been safely written to the disk.

Read Policy: The read policies indicate whether or not the controller should read sequential sectors of the logical drive when seeking data.

Read-Ahead. When using read-ahead policy, the controller reads sequential sectors of the logical drive when seeking data. Read-ahead policy may improve system performance if the data is actually written to sequential sectors of the logical drive.
No-Read-Ahead. Selecting no-read-ahead policy indicates that the controller should not use read-ahead policy.
Adaptive Read-Ahead. When using adaptive read-ahead policy, the controller initiates read-ahead only if the two most recent read requests accessed sequential sectors of the disk. If subsequent read requests access random sectors of the disk, the controller reverts to no-read-ahead policy. The controller continues to evaluate whether read requests are accessing sequential sectors of the disk, and can initiate read-ahead if necessary.

Cache Policy: The Direct I/O and Cache I/O cache policies apply to reads on a specific virtual disk. These settings do not affect the read-ahead policy. The cache policies are as follows:

Cache I/O. Specifies that all reads are buffered in cache memory.
Direct I/O. Specifies that reads are not buffered in cache memory. When using direct I/O, data is transferred to the controller cache and the host system simultaneously during a read request. If a subsequent read request requires data from the same data block, it can be read directly from the controller cache. The direct I/O setting does not override the cache policy settings. Direct I/O is also the default setting.

Show battery information

MegaCli -AdpBbuCmd -GetBbuStatus -aALL -NoLog

Sample output:

BBU status for Adapter: 0

BatteryType: BBU
Voltage: 4023 mV
Current: 0 mA
Temperature: 20 C

BBU Firmware Status:

  Charging Status              : None
  Voltage                      : OK
  Temperature                  : OK
  Learn Cycle Requested        : No
  Learn Cycle Active           : No
  Learn Cycle Status           : OK
  Learn Cycle Timeout          : No
  I2c Errors Detected          : No
  Battery Pack Missing         : No
  Battery Replacement required : No
  Remaining Capacity Low       : No
  Periodic Learn Required      : No

Battery state:

GasGuageStatus:
  Fully Discharged        : No
  Fully Charged           : No
  Discharging             : Yes
  Initialized             : Yes
  Remaining Time Alarm    : No
  Remaining Capacity Alarm: No
  Discharge Terminated    : No
  Over Temperature        : No
  Charging Terminated     : No
  Over Charged            : No

Relative State of Charge: 83 %
Charger Status: Complete
Remaining Capacity: 699 mAh
Full Charge Capacity: 840 mAh
isSOHGood: Yes

The last section of the output would give you a summary of how much of the charge is still remaining in the battery and its capacity.

Show battery capacity information

MegaCli -AdpBbuCmd -GetBbuCapacityInfo -aALL -NoLog

Sample output:

BBU Capacity Info for Adapter: 0

Relative State of Charge: 57 %
Absolute State of charge: 48 %
Remaining Capacity: 921 mAh
Full Charge Capacity: 1619 mAh
Run time to empty: 65535 Min
Average time to empty: 65535 Min
Average Time to full: 65535 Min
Cycle Count: 11
Max Error: 100 %
Remaining Capacity Alarm: 190 mAh
Remaining Time Alarm: 10 Min

Get event log since last reboot

MegaCli -AdpEventLog -GetSinceReboot -f events.log -aALL
cat events.log

Sample output:

seqNum: 0x00000594
Time: Mon Jan 13 07:28:33 2014

Code: 0x0000009b
Class: 0
Locale: 0x08
Event Description: Battery relearn pending: Battery is under charge
Event Data:
===========
None


seqNum: 0x00000595
Time: Mon Jan 13 08:34:38 2014

Code: 0x00000093
Class: 0
Locale: 0x08
Event Description: Battery started charging
Event Data:
===========
None
-----------------------truncated-----------------------

Use the event log to help understand when did BBU status has changed.

Here is a list of other MegaCli command for reference.

Long Term Solution

Disable BBU Auto-Learn Mode

Show battery learn property:

MegaCli -AdpBbuCmd -GetBbuProperties -aALL

Sample output:

BBU Properties for Adapter: 0

Auto Learn Period: 7776000 Sec
Next Learn time: 455912927 Sec
Learn Delay Interval:0 Hours
Auto-Learn Mode: Enabled

From the above output, we can tell that the auto battery relearn is enabled. And the scheduled learn time is in 455912927 seconds in relation to UTC 2000-01-01. If you want to find out exactly when the scheduled learn time is going to happen, run the following:

date -d "UTC 2000-01-01 455912927 secs"

The result would be Thu Jun 12 11:28:47 PDT 2014. This is just to give you an idea of when the relearn is going to take place, but our purpose here is to disable the automation entirely.

Disable Auto-Learn Mode:

echo "autoLearnMode=1" > tmp.txt
MegaCli -AdpBbuCmd -SetBbuProperties -f tmp.txt -aALL

You can run the show battery learn property command again to verify if you’ve successfully disable the auto learn.

Forced Scheduled BBU Relearn Cycle

Obviously, one would want to perform this action during less busy hours to avoid heavy disk I/O:

MegaCli -AdpBbuCmd -BbuLearn -aALL

Either put the above in a cron job that runs every 90 days or do scheduled manual maintanence every 90 days. 90 days is the default BBU auto relearn cycle, but I guess you can change the interval accordingly toward your needs.

Urgent Care

Do a quick check on RAID cache policy:

MegaCli -LDInfo -Lall -aALL -NoLog

If you found the Current cache Policy to start with WriteThrough and end with No Write Cache if Bad BBU. If the WriteThrough mode is an artifact of BBU in relearn process, but you need it to be in WriteBack mode.

You need to first allow write cache if bad BBU, then change the write cache policy to WriteBack:

MegaCli -LDSetProp CachedBadBBU -Lall -aALL
MegaCli -LDSetProp WB -Lall -aALL

Restore original setting once the situation is under control:

MegaCli -LDSetProp NoCachedBadBBU -Lall -aALL

You also want to consider implementing the long term solution.

HTTP Response Code Metrics with Check_MK (OMD)

2014-04-14T00:36:00.001-07:00

You Thought You are Ready

Let’s say your company provide web services or API services that rely on REST. I mean, who doesn’t these days? You’ve got OMD monitoring basic server health and maybe even web server(apache) health. Time to kickback and have a good night sleep right? If there is anything, the monitoring system shall alert us.

Not so fast!! Monitoring server health is the least you have to do to provide 99.999% of service uptime. What if people are able to reach your web server but keep getting 404 Page Not Found or 503 Service Unavailable? Or if there is an DOS attack on web service that happen so fast that brought your server to its knee and you have no idea what hit you?

You Need Metrics that Screams

One thing I like about OMD(Check_MK) is that it is able to collect metrics into graphical charts via PNP4Nagios. Having historical graph that keeps track of all the different HTTP response codes will give you the ability to QUICKLY diagnose the root causes of your web server issues. Inspired by the Logstash project, I have wrote a custom script that will generate graphical chart similar to the following:

Not only will OMD collect HTTP response code counts, you can even tweak the custom script I have open sourced on Github to add thresholds to the number of each HTTP response and have the monitoring server send alerts to you.

How to Use the Script

You can find check_http_code_counter script on Github or download it directly from HERE. The script was written in Bash to collect Apache web server HTTP response code from the access log file. Just move the script to /usr/lib/check_mk_agent/local/check_http_code_counter. Make sure to make the script executable.

chmod +x check_http_code_counter

When the script gets executed, you should get output similar to this:

0 Check_HTTP_Code_Counter 200=102|206=3|301=0|304=7|400=0|401=0|403=2|404=0|416=0|500=0|501=0|503=0|504=0 Just collecting status code counts

Once you have the OMD server start collecting data from it, you should have graph that looks like the following over time.

You can modify the beginning of the code for the response codes you would like collect or change the location of your Apache access log. Here is the excerpt of the script:

#!/bin/bash

STATUSCODES="200 206 301 304 400 401 403 404 416 500 501 503 504"
ACCESSLOG="/var/log/apache2/access_log"

Let me know your thoughts and leave comments below.

How to Setup OMD in 1 Hour

2014-02-16T03:37:00.001-08:00

What is OMD

OMD bundles Nagios together with many important add-ons and can easily be installed on every major Linux distribution. We provide pre-built packages for all enterprise Linux distributions and also for some other, such as Ubuntu.

Want to learn more about what OMD is about and what it can do for you? Check out my other post - Best Monitoring Solution - OMD (Nagios + Check_MK)

Install OMD Server on Ubuntu 12.04

Setup OMD Repository

Run the following on the Ubuntu server command line.

Stable

gpg --keyserver keys.gnupg.net --recv-keys F8C1CA08A57B9ED7
gpg --armor --export F8C1CA08A57B9ED7 | apt-key add -
echo 'deb http://labs.consol.de/repo/stable/ubuntu precise main' >> /etc/apt/sources.list.d/omd-stable.list
apt-get update

Testing-Release

gpg --keyserver keys.gnupg.net --recv-keys F8C1CA08A57B9ED7
gpg --armor --export F8C1CA08A57B9ED7 | apt-key add -
echo 'deb http://labs.consol.de/repo/testing/ubuntu precise main' >> /etc/apt/sources.list.d/omd-testing.list
apt-get update

Install OMD

apt-get install omd

And then just follow the instructions on the screen.

Setup Check_MK Agent on the Monitoring Target

You want to install the Check_MK Linux agent on the server you want to monitor - a remote host. The Check_Mk agent version on the repository is not up to date, so we will just download it from the website.

Install Check_MK Agent

~~Go to the check_mk download page.~~ The Check_MK folks have removed the easy to install Ubuntu/CentOS packages since they also need to make a living. You can get those (and other cool features) by subscribing to their supported service. But for those of us who are just trying it out or staying on the open source version. You can find the agent by appending /agents at the end of your OMD server URL like so:

http://OMD-URL/SITE-NAME/check_mk/agents/

This is what you will see, and check_mk_agent.linux is what you want to use:

You want to save the check_mk_agent.linux file under /usr/bin/check_mk_agent on the target server. Make it executable by running chmod +x check_mk_agent

Setup xinetd Connection

First, make sure you have xinetd service installed on your target host. It is used to open a tcp port 6556 for the OMD server to connect to and execute the Check_MK agent this way. If xinetd is not yet installed, install it now:

apt-get install xinetd

~~By default, the xinetd connection configuration will be setup by the package itself and reloaded.~~ Since the package is no longer available, We want to download the file from the same /agents page called xinetd.conf and save it under /etc/xinetd.d/check_mk. Secure the connection to allow only the OMD server to connect to the xinetd port. Edit /etc/xinetd.d/check_mk, uncomment the following line and add the OMD server’s IP besides 127.0.0.1

only_from      = 127.0.0.1 xx.xx.xx.xx

Now reload xinetd service

service xinetd reload

Patch for MegaCli Check (Optional)

You want to do this if you are running hardware RAID with LSI raid controller on your servers, otherwise you can skip this step.

MegaCli command currently doesn’t work with Linux kernel newer than 2.6 without a fix. Create a bash script using the following example but change the path of the original MegaCli command relative to your environment. Save the script under /usr/local/bin/ and name it MegaCli. Make it executable and it will work.

#!/bin/bash

setarch x86_64 --uname-2.6 /usr/local/sbin/MegaCli $@

change the path above to locate your MegaCli command accordingly.

Setup Firewall (Optional)

You can also skip this step if you don’t care about security. The following iptable rules will allow only the OMD server to contact this target host through tcp port 6556 and ping test. And this is assuming you have everything else locked down in the iptable rules, or else this is useless.

# Monitoring Agent communication
/sbin/iptables -A INPUT -p tcp -s 1.1.1.1 --dport 6556 -j ACCEPT
/sbin/iptables -A INPUT -p icmp -s 1.1.1.1 --icmp-type 8 -j ACCEPT
/sbin/iptables -A OUTPUT -p icmp -d 1.1.1.1 --icmp-type 0 -j ACCEPT

Change the 1.1.1.1 IP address to your OMD server IP address

Start Monitoring Target Hosts

Now that you have successfully installed and enabled the Check_MK agent on a target host. Let’s start monitoring this little pain in the butt without further ado.

Log in to Check_MK Multisite interface with admin privilege. Under the WATO - Configuration section on the left hand side panel. Click on Host & Folders, then on the right hand side of the page, click the New Host button. You will see the following:

Enter the host name or IP address for the Hostname field. If you entered FQDN instead of IP address, make sure it is able to translate by DNS server to an IP address that is pointing to the host you want to monitor. By default, the Agent Type for monitoring any new host is using Check_MK Agent (Server), so all you need to do is just enter Hostname information and click the Save & go to Services button at the end.

Your OMD server is now connecting to the check_mk agent on the target host. In a few seconds, you will see a list of available services to be monitored on the browser. Click on Activate missing button and you will see on the top of the page it tells you that you’ve made 1 change to the system.

You want to click on that button that tells you how many changes you have and then followed up by clicking Activate Changes! button. Now you should see the host and its services are now being monitored by the OMD server. Give it a few more minutes to populate some monitoring data to give you some visuals in the PNP4Nagios graphs.

Look how beautiful it is with all the basic health information you need in nice descriptive format and dynamic historical performance graph at your finger tips. This is just the tip of the iceberg of what OMD can do for you. I will be showing you the more advanced stuff from here on.

Now you know how to setup agents and add target hosts to be monitored, we are going to move on and find out how can we have the OMD system send notifications when things go south on any of your servers or services.

Add Custom Monitoring Script

This is a bigger topic which will be covered in future post. In short, there are 4 types of checks that sits on the target host instead of the OMD server.

Setup Notification

Notifications are important if your company is providing services with 99.999% high availability. You want your system engineers to jump on the issues the moment they happen. The goal is to have incidents resolved before customer even notice it. So having a great notification methodology is vital for establishing a 24 on call framework. OMD(Check_MK) has just the right tools to help you acheive that.

Flexible Notification from Check_MK

Source Document
Require Check_MK version > 1.2.1i2

Each user can have multiple notification methods with separate filters. That way you can for example send always emails, but SMS only for certain services or in certain time periods - without creating multiple contacts per user which is a huge shortcoming in traditional Nagios!

For each notification the Nagios core will simply pass them to the Check_MK flexible notification feature while providing all necessary information about the alert. Then Check_MK will process those notification however you want via the configuration in the GUI (WATO).

Preparing Notifications in General

Setting up the notifications requires some care. If you miss one of the following steps then the most probable case is that simply nothing will happen. You need to do the following:

Create a contact group. WATO users: this is done with the module Contact Groups.
Important: put some hosts/services into that contact group. WATO: Host & Service Parameters -> Grouping -> Assignment of hosts/services to contact groups.
Create a user and put him into that contact group. Enable notifications for that user.
Enter an email address. WATO: Users & Contacts
Make sure that your monitoring server is correctly setup so that it can send emails. Test this with:
```
root@linux# echo "Mailbody" | mail -s "Testsubject" test@example.com
```
Restart your core with cmk -R in Linux command line or activating changes in WATO.

Testing and Debugging

When dealing with notifications it’s important to know how to test and debug your setup. A very easy way to send test-notifications is to pick a random service that is contained in the upper contact group, open the Commands dialog and select Custom Notification:

Check nagios log
~/var/log/nagios.log
Check Check_MK notification log
~/var/check_mk/notify/notify.log

WATO users find this setting in Global Settings -> Notifications -> Debug notifications. The resulting log file is in the directory notify below Check_MK’s var directory. OMD users find the file in ~/var/check_mk/notify/notify.log. Isn’t it great to have fixed paths in OMD?

Send Notification through Email to SMS Gateway

Some countries’ mobile carriers support sending Email to their SMS gateway which will convert that Email message to a SMS(text) message to the destination mobile phone number. For example if you have a 5555555555 T-Mobile phone number in the USA, you can send SMS message to that number with Email using the following Email address:

5555555555@tmomail.net

Having notifications being sent to mobile devices are extremely helpful if you are looking to setup some kind of 24/7 on call monitoring. I wrote a custom notification script that will let you send notification through Email to SMS gateway with the bare minimum information you need from an alert.

Download the most updated script email-sms-custom.sh from Github.

Save my script into this directory on your OMD server and make it executable:
/omd/site/YOURSITE/local/share/check_mk/notifications/email-sms-custom.sh

Check_MK will consider every executable file in this directory as a notification script. You can use any programming language you want. If you use a scripting language (Shell, Perl, Python, etc.) then you can use a comment in the second line for giving the script a name. This name will appear in the UI for you to choose in a drop down list of different notification methods. I have named the script on the 2nd line to be SMS via Email to SMS Gateway.

Here is a list of all the Email to SMS Gateway provider in the world. Use them and test them at your own risk. All SMS message fees still applies.

OMD Maintenance

OMD Upgrade

http://www.mail-archive.com/checkmk-en@lists.mathias-kettner.de/msg07525.html

http://deranfangvomende.wordpress.com/2011/06/07/omd-update-0-42-to-0-49/
- OMD with lastest Check_MK version
- Download Demo Version

OMD Backup/Migrate

Install the same version of OMD package on your new hardware
Stop omd on the source server
Copy /omd/sites/ to your new target server.
Mind the permission! Create the same “Site” user on the target server
Start OMD on your target server

High Resolution Graphing for PNP4Nagios for One Year

When you have longer period of graphing data on PNP4Nagios, zooming into year graph down to the day seems to have averaged out greatly. You can’t rely on the graph to dig up any historic event metrics this way.

To address the issue, edit rra.cfg in your pnp4nagios config. Documentation is on rrdtool and pnp4nagios websites.
http://serverfault.com/questions/534186/how-to-change-the-amount-of-data-stored-in-pnp4nagiosomd
http://apfelboymchen.net/gnu/rrd/create/

Best Monitoring Solution 2015 - OMD (Open Monitoring Distribution)

2013-11-28T04:29:00.001-08:00

Why is OMD the Best Open Source Monitoring Solution over Zabbix, Zenoss or Sensu

I spend about 3 weeks vetting through 20+ open source monitoring solutions and at the end of the process, the choices had boiled down to few major ones - OMD (best combination of open source plug-ins put together for Nagios), Zabbix, Sensu, and Zenoss.

The main components of OMD are Check_MK, PNP4Nagios, Nagvis, and of course Nagios. Among these projects, Check_MK is the core of OMD that makes Nagios easy to configure, easy to scale, and mashed together all the other popular Nagios plug-ins into one unified user interface. Thus the following comparisons are done using Check_MK as the keyword, but I will also cover how other plug-ins makes OMD project stand out from the competitions.

Update: 2015.08.29
I have the opportunity to build a new monitoring platform from scratch again, so I wanted to make sure if there is any better solution than OMD. I have gone though a series of new comers and the Monitorama conferences (2014 ~ 2015). OMD still is the only one that ships with everything you need to do the monitoring job right. Some of the new tools focus on API and scalability (which OMD has both) but doesn’t really give you a easy to use or the lack of GUI to manage your infrastructure with less command line or coding capable people.

Trend

Check_MK vs Zabbix vs Zenoss Core Trend

A quick Google Trend search will tell you that Check_MK is up and rising. Together with the Nagios’s community size, you can certainly find custom monitoring plug-ins created by community members and save yourself time from reinventing the wheel.

Project Health

Before you pick any open source tool for enterprise projects, you want to make sure that their code is not stale and the community is vibrant for the years to come. Active community and frequent code updates ensure your questions get answered and fast bug fixes. Free service from Ohloh will give you an overview of those aspects on open source projects. The following comparison charts are created from Ohloh.

Number of Code Commits Made by Each Project

Check_MK is a clear winner in this chart. It tells you that Check_MK is constantly making more improvement than the other 2 projects.

Number of Contributor of Each Project

In this chart, Check_MK’s contributor is increasing and will soon surpass Zabbix. And don’t forget it is standing on giant’s shoulder, the largest monitoring community - Nagios.

User Reviews

Don’t just listen to me. Here is one of the blog post that talks about why moving away from Zabbix to Check_MK.

Moving From Zabbix to Check_MK

Architecture Design Advantage

OMD

What is OMD:: OMD is a combination of best practices on how Nagios should be setup and integrated. It has incorporated all of the most popular 3rd party Nagios plug-ins in single easy to maintain, easy to install, and easy to upgrade package. Once you have your Linux server running, installing and have your OMD monitoring suite running only takes about 10 minutes with one command.

Administrators can really save time on not having to compile Nagios, or other plug-ins, trying to integrate and mess with configurations between plug-ins and Nagios. It really is a no-brainer to setup and start with.

Why use OMD instead of other flavors of Nagios combos, e.g. ?: Founded July, 2010 by a group of well known Nagios community members and Nagios addon developers; e.g. NagVis, Check_MK, PNP4Nagios, and others

Check_MK

What is Check_MK

Check_MK is an extension to the Nagios monitoring system that allows creating rule-based configuration using Python and offloading work from the Nagios core to make it scale better, allowing more systems to be monitored from a single Nagios server.

There are 2 significant modules that Check_MK uses to improve Nagios performance. One is called Livestatus and the other is called Livecheck.

Livestatus

Before Livestatus ☹

Monitoring results are stores to a single file status.dat. It becomes a bottleneck on CPU and IO for larger installation.
status file status is not realtime, default is to update every 10 seconds.
NDOUtils utilize databases for monitoring results (MySQL or PostgreSQL), but still have some severe shortcomings.
NDOUtils has complex setup.
NDOUtils needs a databases to be administered, a rapidly growing one.
NDOUtils eats up significant portion of your CPU resources just to keep the database up to date.
Some similar projects that still uses NDOUtils:
- Centreon
- Opsview
Regular housekeeping of the database can hang your Nagios for minutes or even an hour once a day.

After Livestatus ☺

Livestatus also uses Nagios Event Broker API like NDO, but it does not actively write out data. Instead, it opens a socket by which data can be retrieved on demand.
Livestatus imposes no measurable burden on CPU at all.
Livestatus produces zero disk IO when querying status data.
No configuration is needed. No database is needed. No administration is necessary.
Livestatus scales well to large installation even beyond 50,000 services.
Livestatus give you access to Nagios-specific data that is not available to any other methods.

Livecheck

Before Nagios 4.0, Even a perfectly tuned system rarely manages to execute more then a few thousand checks per minute.

What make things worse: while your system is getting larger, the maximum check rate is even getting worse. The more hosts and services your system manages, the less checks per second it will be able to perform. Why?

Existing Problems of Nagios (before Nagios 4.0) ☹

Each new check creates a new fork
The new process prepare everything needed to execute the check plug-in, then fork the second time when ready
Forking is costly even for highly optimized Linux kernel
The forking of Nagios core (before v.4.0) does not scale on multiple CPUs (single thread process).
you can well run into a situation where your powerful 16-CPU server is limited to 100 Checks per second while most of its CPU cores are idle most of the time.

How does Livecheck solve those bottlenecks ☺

It uses a number of helper processes. The core communicate with each helper through a Unix socket (that does not appear in file system).
Only a small helper program is forked instead of the complete Nagios monitoring core.
The helper forks distribute over all available CPUs instead of single CPU.
The total process VM size of Livecheck is about 100KB only!
Inline implementation of check_icmp (PING tests). To give you an idea of how much improvement this has done, here is a benchmark example using dual core 2800 MHz CPU:
- Before inline check_icmp: 300 ICMP checks per second.
- After inline check_icmp: 2600 ICMP checks per second. The checks generated an ICMP traffic of 45Mb/s.

Nagios Monitoring Core working with the best plug-ins (Check_MK, NagVis, PNP4Nagios and etc)

Multisite - An Advance Web Interface for Nagios

Multisite is part of the Check_MK project as a better web UI alternative for Nagios.

A new and innovative GUI for viewing Nagios status information and controlling your monitoring system. It is based on MK Livestatus and aims at replacing the Nagios web GUI (also known as “the CGIs”). Multisite supports distributed monitoring in a very efficient way.

Zero Configuration Files with WATO

This is one of the most brilliant solutions from Check_MK project to tackle the notorious Nagios configuration disaster. Although Nagios is a flexible and powerful monitoring system, having to mess with its multi-level and confusing configuration files scares many people away. Now, there are many web interface plug-ins that try to take a stab at the issue, but WATO is by far the best that simplify the complexity of Nagios configuration while staying very flexible and more flexible by sitting on top of Check_MK.

WATO is a web based administration tool for Check_MK. It allows you to manage your hosts and services to be monitored and perfectly supports Check_MK’s mechanism of inventory to autodetect services to be checked on a host. WATO allows to move a substantial part of the daily workload from the monitoring administrator to his colleagues.

Monitoring Agent for both Linux and MS Windows

Responsive UI for Mobile Client

Powerful Search Function

Visual Meters with Perf-O-Meter

NOC with Dashboards (Thanks to PNP4Nagios & Nagvis)

PNP4Nagios

Nagvis

NagVis is a visualization addon for the well known network managment system Nagios. NagVis can be used to visualize Nagios Data, e.g. to display IT processes like a mail system or a network infrastructure.

Check_MK (OMD) API for Automated Provisioning

Automation is build into Multisite (Check_MK UI). You can make web service request against the API to automate adding new host, enabling new service checks, or embed any of the host/service check web pages into any other websites.

This feature makes it very easy to integrate with configuration mangement tools like Puppet or Ansible for automatically adding new servers(hosts) and services to the monitoring system.

24/7 NOC with Flexible Notification

With Check_MK abstracting the original Nagio’s notification scheme, it has become possible to send notifications of any hosts or services to any number of people at any time.

You can even create custom script to send the notification in some creative ways like having the notification be ☎called via a VoIP server to your cell phone and read you the alert message or have the alert be sent to your ✐instant messenger.

If you are currently using PagerDuty as your notification service provider, you can check out my other post on OMD (Check_MK) Alert Notification Integration with PagerDuty Done Right.

Custom Icons

This is really one of the hidden gem of Check_MK (OMD) feature that will dramatically reduce the MTTR (Mean Time to Repair). I will be writing another post about the things I have implemented to make the NOC team more efficient at troubleshooting and putting out fires.

http://mathias-kettner.de/checkmk_devel_multisite_icons.html

Management and Maintenance

Scaling with Distributed Monitoring

Distributed WATO allows you to manage several monitoring sites through a logically centralized WATO.

1200 Check_MK installations
- Centralized status of all 1200 stores per minute
- Using NagVis to show all 1200 stores’ status on the map using the Geomap function
- All stores’ overall status is aggregated through the use of the Business Intelligence function

Backup of Changes

Automatic Check_MK configuration backup on every change you make
Easy restoration with the Thunder icon

Upgrade OMD

OMD upgrade DEMO Version
OMD copy a site

Business Intelligence

Available from version 1.2.3

Predictive Monitoring

Smart threshold that detects anomaly from daily operation
Set warning level based on prediction

Available from check_mk version 1.2.3

Monitor Cronjobs

Before

5 0 * * * root /usr/local/bin/backup >/dev/null

After

5 0 * * * root mk-job nightly-backup /usr/local/bin/backup >/dev/null

Available from check_mk version 1.2.3

Dive in to the OMD World

I will be sharing how I install OMD, optimized web interface (Multisite), utilized passive checks, implemented 24/7 on call plan, and integrated with automated business processes. I will add link here once they become available.

How to Setup OMD in 1 Hour

What monitoring tool do you use and how do you think it’s doing the job for you? (leave comments below )

Automated Web Testing with Perl and WebDriver

2013-08-17T20:22:00.001-07:00

What is Automated Web Testing you may ask, and how is it done? If you search this topic on Google, you may be well flooded by confusing and overwhelming information from different websites and blog post. But lucky for you, I have done the homework and consolidated those into an easy to understand presentation for you:

Selenium

Selenium automates browsers. That’s it. What you do with that power is entirely up to you. Primarily it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

Here we are using it to automate acceptance test for our web based control panel. We will be triggering those tests periodically and monitor the results. If anything customer facing stops working, we’ll have the first hand and quickly react to issues before they spread or damages company reputation.

Many documents or blogs you find online regarding Selenium are out dated and are only covering the old Selenium. I will be focusing on Selenium 2 with the WebDriver standard which is adopted by major browsers.

Selenium IDE or Selenium Builder

Both Selenium IDE and Selenium Builder are implemented as a Firefox extensions. They allow you to record, edit, and debug your automated test scripts. Selenium Builder was created in the attempt to replace Selenium IDE in the long run and is still in the beta stage as I am writing. Having used both, I prefer to use Selenium IDE because Selenium is crashing a lot.

Selenium IDE

It currently supports both Selenium 1 and 2 (WebDriver) format. It’s able to convert your tests into scripts in the programming languages you use. Here we use Perl and we’ll need to install a plugin provided by the Selenium::Remote::Driver project in order for Selenium IDE to convert test scripts to Perl.

Selenium Grid 2

Don’t be intimidated by the term “Grid”; It simply means a cluster of Selenium servers. The Wiki has full instructions on how to use and configure Selenium Grid 2. Here I am just going to cover what I use on my project. The Grid is consisted with a hub and multiple nodes. The selenium-server-standalone package includes the Hub, WebDriver, and legacy RC needed to run the grid. Ant is not required anymore. You can download the selenium-server-standalone-*.jar from here. This walk-through assumes that you have already installed Java run time on your testing servers.

Grid Hub

The Selenium Hub is the central point that will receive all the test requests from your testing script and distribute them the the appropriate Selenium nodes.

Open a command prompt and navigate to the directory where you copied the selenium-server-standalone file. Type the following command:

java -jar selenium-server-standalone-2.33.0.jar -role hub

The hub will automatically start-up using port 4444 by default. To change the default port, you can add the optional parameter -port when you run the command. You can view the status of the hub by opening a browser window and navigating to: http://localhost:4444/grid/console

Grid Node

Regardless on whether you want to run a grid with new WebDriver functionality, or a grid with Selenium 1 RC functionality, or both at the same time, you use the same selenium-server-standalone jar file to start the nodes.

java -jar selenium-server-standalone-2.33.0.jar -role node  -hub http://localhost:4444/grid/register

Note: The port defaults to 5555 if not specified whenever the “-role” option is provided and is not hub.

java -jar selenium-server-standalone-2.33.0.jar -role node

For backwards compatibility “wd” and “rc” roles are still a valid subset of the “node” role. But those roles limit the types of remote connections to their corresponding API, while “node” allows both Selenium RC and Selenium WebDriver remote connections.

WebDriver

WebDriver is the new Selenium 2. It takes a different approach to solve the same problem as Selenium 1. Rather than being a JavaScript application running within the browser, it uses whichever mechanism is most appropriate to control the browser. For Firefox, this means that WebDriver is implemented as an extension. For IE, WebDriver makes use of IE’s Automation controls.

By changing the mechanism used to control the browser, we can circumvent the restrictions placed on the browser by the JavaScript security model. In those cases where automation through the browser isn’t enough, WebDriver can make use of facilities offered by the Operating System. For example, on Windows we simulate typing at the OS level, which means we are more closely modeling how the user interacts with the browser, and that we can type into “file” input elements.

Run Tests

My Environment

OS: Mac OSX
Selenium Nodes:
1. Ubuntu 12.04
2. Mac OSX
Selenium Server Version: WebDriver running on Mac OSX
Test Scripting Language: Perl

For Selenium (WebDriver) nodes, you will need to use the RemoteWebDriver and the DesiredCapabilities object to define which browser, version and platform you wish to use. Create the target browser capabilities you want to run the tests against. Examples will be under the “Perl” section.

Writing Test Script with Perl

Majority of the examples I found online regarding the use cases of Selenium 2 (WebDriver) are done either in Java or Ruby. Java is out of the question because:

No heavy IDEs like Eclipse or Netbeans running on my already suffocating Mac OSX
The need to compile testing scripts seems like a waste of time
We are a Perl shop at work, I will get much support using the language people are comfortable with

Decision has been made. Let’s take a look at what Perl modules we will need to make it work.

Selenium::Remote::Driver
Test::More

Selenium::Remote::Driver

Perl Client for Selenium Remote Driver, it is used to connect to Selenium server (Hub/Node) remotely and send command to the WebDriver API. Install this module with CPAN from your Linux or OS X terminal:

cpan -i Selenium::Remote::Driver Test::More

In your Perl script, load this module at the beginning of the script with the other ones that you would normally use like so:

#!/usr/bin/perl
use strict;
use warnings;
use Test::More;
use Selenium::Remote::Driver;

Here is the full documentation on how to initiate and use Selenium::Remote::Driver module. Use the following in your Perl script to connect to a Selenium hub or node by specifying the server’s IP address, the browser type and/or version you want to test with:

my $driver = Selenium::Remote::Driver->new(
    'remote_server_addr' => 'localhost',
    'browser_name' => 'phantomjs',
    'port' => '5555',
);

Test::More

Yet another framework for writing test scripts. It will generate TAP test results which I can integrate into Jenkins for CI (continues integration) or Nagios for monitoring test results.
Documentation

Browsers

List of supported browser names that you can use in the Perl script.

iphone
firefox
internet explorer
htmlunit
iphone
chrome
phantomjs

Firefox

Supported by default

Chrome

Require you to install ChromeDriver. To use it in conjunction with Selenium server, start your Selenium server with the following command:

java -Dwebdriver.chrome.driver=chromedriver -jar selenium-server-standalone-2.33.0.jar

chromedriver is the actual ChromeDriver executable you’ve just downloaded.

PhantomJS

Safari

Internet Explorer

Example

Search Google Image for me

Tools for Other Programing Language

Javascript

Nightwatch.js