Krystianek's blog

Linux: UDP buffer monitoring utility (udpstat)

2013-01-15T12:51:00.001-08:00

Udp statistics tool

The goals

The goal for this utility was to:

1. Present /proc/net/udp in human readable form:
- timestamp
- local address - from hex to dot notated (in case of IPv6 in hex) + port
- remote address - from hex to dot notated (in case of IPv6 in hex) + port
- rx, tx queue - in bytes, kbytes, mbytes (constrolled by parameter)
- drops

2. Constant monitoring with configurable interval polling:
- statistics presented every n seconds
- monitor interval controlled by command line parameter
- number of polls controlled by command line interface

3. Print Per pid statistics - present udp statistics per pid (/proc//net/udp)

4. Print kernel configuration parameters: max size + utilization of the buffer (rx queue / rx max size & tx queue / tx max size)

Where it can be downloaded from?

It has become a part of the yapcg project. The latest version available here: http://code.google.com/p/yapcg/source/browse/measurements/perftools/udpstat should meet all the goals . You can give it a try - what you will need is ruby 1.9.3 with optparse, ostruct gems (should be available in standard installation, at least these was the case on my Ubuntu 12.10 box)

The syntax of the utility is:

$ udpstat
Usage: udpstat [options]

Specific options:
-6, --ipv6 Enable IPv6 stat parsing
-K, --kilobytes Present values in kilobytes
-M, --megabytes Present values in megabytes
-p, --ppid pid PID of the process to be monitored

Common options:
-h, --help Show this message
-v, --version Show version

- interval in which statistics are to be reported
- the number of statistics to be shown

@2013 Krystian Brachmanski, Wroclaw

As previously mentioned it has been verified on Ubuntu 12.10 64-bit OS, running 3.5.0-21-generic kernel and ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux].

How to use it?

I always say that I learn the most on examples therefore I recommend to check another article on my blog to see how you can truoubleshoot a simulated udp buffer problem in the application.

Ubuntu 12.10: UDP buffer monitoring (kernel 3.5.0-21)

2013-01-14T13:34:00.003-08:00

Some time ago I was involved in analysis of a networking problem, the symptom of which were client-side timeouts (no answer received). The protocol was standardized, based on UDP.

Let's first analyze the flow of information from network adapter towards the application. There is an excellent document provided by RedHat as part of their OS documentation - it can be found here. I recommend reading chapter 8 about networking. In general the flow looks as presented on the pic below:

Figure 1 Packet reception diagram (from RedHat Performance Tuning Guide)

Hardware/interrupt buffers

These are internal HW buffers inside the network interface card (NIC). If you see hw buffer overruns on your system the only thing that you will know is that the transfer rate is too high for this interface. The document previously mentioned gives indication how to monitor it using ethtool and what you could do to minimize the impact.

You can monitor this using tools getting stats from the NIC like ifconfig. Below you can find an example from my Ubuntu box (overruns indicate the number of times the HW buffer overrun happened):

$ ifconfig eth0
eth0      Link encap:Ethernet HWaddr b4:99:ba:e8:d3:6c
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
          Interrupt:20 Memory:d7500000-d7520000

UDP socket buffers

There is a lot explained what can be tuned in the document mentioned above. I will only try to explain how it can happen that the buffer queue will grow. In order to do that I will prepare a sample Java application (a modification of UDP echo client/server app). It consists of the server application and client one.

Test application

The client will be sending every 10 milliseconds an UDP datagram containing 64 bytes long string (16 times the word "test").

The server on the other hand will be one-thread application reading from socket. The reading thread after retrieving information from the socket will be sleeping for 1 second - 1s is set to show the effect in shorter timeframe but please notice any value above 10 ms (sleep on the client side) will cause this behaviour. In general such problem might happen in real life scenario if the same thread that is reading from socket is doing some processing.

Code for udp echo client:
package org.krystianek.udp.test.echo;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.SocketException;
import java.util.logging.Logger;

public class EchoClient {

          public static void main(String[] args) throws Exception {
                    String hostname = "localhost";
                    InetAddress ia = InetAddress.getByName(hostname);
                    SenderThread sender = new SenderThread(ia, 9999);
                    sender.start();
                  }

}

class SenderThread extends Thread {
          private InetAddress server;
          private DatagramSocket socket;
          private boolean stopped = false;
          private int port;

          public SenderThread(InetAddress address, int port) throws SocketException {
            this.server = address;
            this.port = port;
            this.socket = new DatagramSocket();
            this.socket.connect(server, port);
          }

          public void halt() {
            this.stopped = true;
          }

          public DatagramSocket getSocket() {
            return this.socket;
          }

          public void run() {

            try {
              String theLine ="testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttest";
              while (true) {
                byte[] data = theLine.getBytes();
                DatagramPacket output = new DatagramPacket(data, data.length, server, port);

                System.out.println("Sending : " + output.getData().length + " bytes ");
                socket.send(output);
                Thread.yield();
                Thread.sleep(100);
              }
            }
            catch (Exception ex) {
              System.err.println(ex);
            }
          }
        }

Code for udp echo server:

package org.krystianek.udp.test.echo;

import java.net.DatagramSocket;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.DatagramChannel;

/**
* @author krychu
*
*/
public class EchoServer {

        public static void main(String[] args) throws Exception {
                DatagramChannel channel = DatagramChannel.open();
                DatagramSocket socket = channel.socket();
                SocketAddress address = new InetSocketAddress(9999);
                socket.bind(address);
                ByteBuffer buffer = ByteBuffer.allocateDirect(65507);
                while (true) {
                        SocketAddress client = channel.receive(buffer);
                        try {
                           Thread.sleep(1000);
                        } catch (Exception ex) {
                           ex.printStackTrace();
                        }
                        buffer.flip();
//                      System.out.println("Server received : " + buffer);
//                      channel.send(buffer, client);
//                      buffer.clear();
                }
        }

}

For the test the classes have been packaged into echo-0.0.1-SNAPSHOT.jar using maven build system.

Monitoring the system

The next step was to identify how to monitor the buffer utilization. There are different ways - in my article I will use the statistics available in the proc filesystem - quite good description can be found in the man page:

$ man proc
...

/proc/net/udp

Holds a dump of the UDP socket table. Much of the information is not of use apart from debugging. The "sl" value is the kernel hash slot for the socket, the "local_address" is the local address and port number pair. The "rem_address" is the remote address and port number pair (if connected). "St" is the internal status of the socket. The "tx_queue" and "rx_queue" are the outgoing and incoming data queue in terms of kernel memory usage. The "tr", "tm->when", and "rexmits" fields are not used by UDP. The "uid" field holds the effective UID of the creator of the socket. The format is:

sl  local_address rem_address   st tx_queue rx_queue tr rexmits  tm->when uid
1: 01642C89:0201 0C642C89:03FF 01 00000000:00000001 01:000071BA 00000000 0
1: 00000000:0801 00000000:0000 0A 00000000:00000000 00:00000000 6F000100 0
1: 00000000:0201 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0

...

There are two files available one for IPv4 (/proc/net/udp) and IPv6 (/proc/net/udp6) - the example output can be found below

$ cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
564: 00000000:0801 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 13276 2 0000000000000000 0
601: 00000000:E026 00000000:0000 07 00000000:00000000 00:00000000 00000000 118 0 12564 2 0000000000000000 0
...
1833: 00000000:9CF6 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 13808 2 0000000000000000 0
1884: 00000000:ED29 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 1145241 2 0000000000000000 0
$

$ cat /proc/net/udp6
sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
2: 00000000000000000000000000000000:ADCF 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 13287 2 0000000000000000 0
37: 00000000000000000000000000000000:9DF2 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000 104 0 10786 2 0000000000000000 0
...

1820: 00000000000000000000000000000000:14E9 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000 104 0 10784 2 0000000000000000 0 2024: 00000000000000000000000000000000:B5B5 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 13796 2 0000000000000000 0

In addition I developed a simple ruby script for parsing the udp stats.
The full parameter set for the utility:

$ udpstat
Usage: udpstat [options]

Specific options:
-6, --ipv6 Enable IPv6 stat parsing
-K, --kilobytes Present values in kilobytes
-M, --megabytes Present values in megabytes
-p, --ppid pid PID of the process to be monitored

Common options:
-h, --help Show this message
-v, --version Show version

- interval in which statistics are to be reported
- the number of statistics to be shown

@2013 Krystian Brachmanski, Wroclaw

The example dump can be found below:

$ ./measurements/perftools/udpstat -6 1 2
Timestamp Local IP:port Remote IP:port rx[B] rbuf[B] tx[B] tbuf[B] drops
1353956875 00000000000000000000000000000000:44495 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:40434 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:24105 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:36499 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:51186 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:2049 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:111 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000001000000:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 000080FE00000000FFD724029C55BDFE:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:45593 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:612 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:5353 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956875 00000000000000000000000000000000:46517 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:44495 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:40434 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:24105 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:36499 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:51186 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:2049 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:111 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000001000000:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 000080FE00000000FFD724029C55BDFE:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:123 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:45593 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:612 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:5353 00000000000000000000000000000000:0 0 1048576 0 1048576 0
1353956876 00000000000000000000000000000000:46517 00000000000000000000000000000000:0 0 1048576 0 1048576 0

You can find the utility in google code as part of the yapcg tool: http://code.google.com/p/yapcg/source/browse/measurements/perftools/udpstat

Testing the UDP buffers

Configuration of the test application (actually it is in the code):
- udp port: 9999
- IPv6 socket listening
- system settings:
     net.core.rmem_default = 1048576
     net.core.rmem_max = 1048576
1. Starting the Echo UDP server in the background:
$ java -cp ./target/echo-0.0.1-SNAPSHOT.jar org.krystianek.udp.test.echo.EchoServer &

2. Start the udp6 monitor the ipv6 udp sockets in the background and present the statistics in kilobytes:
$ ./udpstat -6 -KB 1 100000 > udpstat.out 2>&1 &
...

3. Starting the client:
$ java -cp ./target/echo-0.0.1-SNAPSHOT.jar org.krystianek.udp.test.echo.EchoClient

4. Analysis of the results:

- if we take a look at the generated output:
$ cat udpstat.out
   Timestamp       Local IP:port        Remote IP:port     rx[KB] rbuf[KB]    tx[KB] tbuf[KB]   drops
...
1358196897 00000000000000000000000000000000:9999 00000000000000000000000000000000:0            0     1024        0     1024       0
...
1358196898 00000000000000000000000000000000:9999 00000000000000000000000000000000:0            8     1024        0     1024       0
...
1358196902 00000000000000000000000000000000:9999 00000000000000000000000000000000:0           38     1024        0     1024       0
...
1358196906 00000000000000000000000000000000:9999 00000000000000000000000000000000:0           68     1024        0     1024       0
...
1358196909 00000000000000000000000000000000:9999 00000000000000000000000000000000:0           90     1024        0     1024       0
...
1358196910 00000000000000000000000000000000:9999 00000000000000000000000000000000:0           97     1024        0     1024       0
...
1358196917 00000000000000000000000000000000:9999 00000000000000000000000000000000:0          149     1024        0     1024       0
...
1358196928 00000000000000000000000000000000:9999 00000000000000000000000000000000:0          230     1024        0     1024       0
...
1358197036 00000000000000000000000000000000:9999 00000000000000000000000000000000:0         1024     1024        0     1024       9 1358197037 00000000000000000000000000000000:9999 00000000000000000000000000000000:0         1024     1024        0     1024       8 1358197038 00000000000000000000000000000000:9999 00000000000000000000000000000000:0         1024     1024        0     1024       7 1358197039 00000000000000000000000000000000:9999 00000000000000000000000000000000:0         1024     1024        0     1024       7
...

As expected the receive socket buffer starts to increase as soon as the UDP echo client starts since the transmission rate of the client is much higher than the reception rate of the server (Server receiver once per second, while client transmitting 100 times 64 bytes per second). As soon as the receive socket buffer utilization level reaches the maximum defined in system kernel parameters one can see packets drops happening (marked with red colour).

Everything seems to be working as expected...

Conclusions

When designing an application that is processing high volumes of data always ensure that the thread reading from the socket is not doing any additional, unnecessary processing since it may cause your socket buffers utilization go high and eventually would cause packet drop. If the additional processing/decoding is necessary then still an option with a thread pool might solve the buffer issue.

Anyway in my opinion the general rule of thumb should be that it is always beneficial to make the socket reading layer as lightweight as possible. With a single threaded approach one could think of creating a queue to which the reader thread would put the payload and a pool of worker threads would read from the queue and continue with the processing.

Ubuntu: port-scan detection tools (psad, snort)

2013-01-12T04:54:00.002-08:00

Do you know any port scan detection tools under Linux?

Today I needed one but did not have much experience or should I rather say overview on what is available. After some googling I found some tools and have chosen two of them to verify more in depth: psad and snort.

Psad

1. Installation

The psad utility is present in standard ubuntu repository and can be installed via apt-get utility:

#apt-get install psad
...

2. Configuration

2.1 Psad configuration

The details about parameters can be found here: http://cipherdyne.org/psad/docs/config.html
Generally in my case the email was pointing to root@localhost; HOME_NET and EXTERNAL_NET were pointing to any.

Iptables log file location:
IPT_SYSLOG_FILE /var/log/kern.log;

2.2 Iptables configuration

Turn on iptables logging:

# iptables -A INPUT -j LOG
# iptables -A FORWARD -j LOG

3. Startup

Ensure that the service is restarted after changing the configuration (of course from root account):

# service psad restart

* Stopping the psadwatchd process
* Stopping the kmsgsd process
* Stopping the psad process
* Stopping Port Scan Attack Detector psad [ OK ]
* Starting Port Scan Attack Detector psad [ OK ]

4. Test

For doing the test the nmap utility has been used by me (from localhost - my local IP address is 192.168.1.106).

NMap scan:

# nmap -v -p1-65535 192.168.1.106

Starting Nmap 6.00 ( http://nmap.org ) at 2013-01-11 07:52 CET
Initiating Parallel DNS resolution of 1 host. at 07:52
Completed Parallel DNS resolution of 1 host. at 07:52, 0.33s elapsed
Initiating SYN Stealth Scan at 07:52
Scanning 192.168.1.106 [65535 ports]

...

After few minutes the information about the scan appeared in the psad statistics (before that nothing was shown):

# psad -S

[+] psadwatchd (pid: 12733) %CPU: 0.0 %MEM: 0.0
Running since: Fri Jan 11 07:29:55 2013

[+] psad (pid: 12731) %CPU: 3.0 %MEM: 1.2
Running since: Fri Jan 11 07:29:55 2013
Command line arguments: [none specified]
Alert email address(es): root@localhost

[+] Version: psad v2.2

[+] Top 50 signature matches:
"BACKDOOR DoomJuice file upload attempt" (tcp), Count: 60, Unique sources: 1, Sid: 2375
"P2P BitTorrent communication attempt" (tcp), Count: 9, Unique sources: 1, Sid: 2181
"BACKDOOR NetSphere Connection attempt" (tcp), Count: 3, Unique sources: 1, Sid: 100044
"POLICY HP JetDirect LCD communication attempt" (tcp), Count: 3, Unique sources: 1, Sid: 510
"DOS DB2 dos communication attempt" (tcp), Count: 2, Unique sources: 1, Sid: 1641
"POLICY vncviewer Java applet communication attempt" (tcp), Count: 2, Unique sources: 1, Sid: 1846
"MISC Ghostsurf communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100203
"MISC HP Web JetAdmin communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100084
"MISC Microsoft PPTP communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100082
"P2P Napster Server Login communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 565
"MISC Xtramail communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 1636
"BACKDOOR Subseven connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100207
"BACKDOOR Doly 2.0 Connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 119
"MISC VNC communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100202
"MISC MS Terminal Server communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100077
"BACKDOOR RUX the Tick connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100063
"BACKDOOR HackAttack 1.20 Connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 141
"RPC portmap listing TCP 32771" (tcp), Count: 1, Unique sources: 1, Sid: 599
"MISC Microsoft SQL Server communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100205
"BACKDOOR NetBus Pro 2.0 Connection Cttempt" (tcp), Count: 1, Unique sources: 1, Sid: 100029
"MISC Radmin Default install options attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100204
"DDOS Trin00 Attacker to Master connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100007
"BACKDOOR QAZ Worm Client Login access" (tcp), Count: 1, Unique sources: 1, Sid: 108
"DDOS mstream client to handler" (tcp), Count: 1, Unique sources: 1, Sid: 249
"BACKDOOR WinCrash 1.0 communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 163
"BACKDOOR GirlFriend Connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 145
"PSAD-CUSTOM Kuang2 virus communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100206
"BACKDOOR netbus Connection Cttempt" (tcp), Count: 1, Unique sources: 1, Sid: 100028
"MISC PCAnywhere communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100073
"MISC Alcatel PABX 4400 connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 1819
"BACKDOOR Remote PC Access connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 2124
"BACKDOOR BackConstruction 2.1 connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 152
"MISC xfs communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 1987
"DOS arkiea backup communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 282
"BACKDOOR PhaseZero Server Active on Network" (tcp), Count: 1, Unique sources: 1, Sid: 208
"SNMP AgentX/tcp request" (tcp), Count: 1, Unique sources: 1, Sid: 1421
"MISC Insecure TIMBUKTU communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 505
"FTP Yak! FTP server communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100100
"P2P eDonkey transfer attempt" (tcp), Count: 1, Unique sources: 1, Sid: 2586
"BACKDOOR Infector.1.x Connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100040
"BACKDOOR Asylum 0.1 connection request" (tcp), Count: 1, Unique sources: 1, Sid: 100064
"P2P napster communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100090
"BACKDOOR NetMetro File List connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 159
"P2P Napster Client Data communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 564
"P2P Napster Client Data communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 562
"P2P Napster Client Data communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 563
"DOS Real Audio Server communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 100112
"P2P Fastrack kazaa/morpheus communication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 1383
"BACKDOOR DonaldDick 1.53 connection attempt" (tcp), Count: 1, Unique sources: 1, Sid: 153
"POLICY HP JetDirect LCD commnication attempt" (tcp), Count: 1, Unique sources: 1, Sid: 568

[+] Top 25 attackers:
192.168.1.106 DL: 5, Packets: 56653, Sig count: 126, (local IP)
127.0.0.1 DL: 1, Packets: 13, Sig count: 0, (local IP)
193.41.112.18 DL: 1, Packets: 6, Sig count: 0

[+] Top 20 scanned ports:
tcp 51708 1 packets
tcp 58684 1 packets
tcp 57573 1 packets
tcp 19544 1 packets
tcp 44919 1 packets
tcp 8563 1 packets
tcp 58017 1 packets
tcp 9323 1 packets
tcp 28336 1 packets
tcp 5006 1 packets
tcp 28801 1 packets
tcp 63406 1 packets
tcp 55548 1 packets
tcp 38756 1 packets
tcp 31922 1 packets
tcp 62374 1 packets
tcp 55430 1 packets
tcp 3509 1 packets
tcp 10713 1 packets
tcp 36235 1 packets

udp 111 12 packets
udp 53 7 packets
udp 17500 5 packets
udp 5353 5 packets
udp 50685 1 packets
udp 30969 1 packets
udp 22220 1 packets
udp 63507 1 packets
udp 44743 1 packets
udp 50374 1 packets
udp 41192 1 packets
udp 22675 1 packets
udp 123 1 packets
udp 60162 1 packets
udp 26877 1 packets
udp 33571 1 packets
udp 52566 1 packets
udp 60029 1 packets
udp 37490 1 packets

[+] iptables log prefix counters:
"[UFW BLOCK]": 11

Total packet counters: tcp: 56646 udp: 44

[+] IP Status Detail:

SRC: 192.168.1.106, DL: 5, Dsts: 4, Pkts: 56653, Unique sigs: 26, Email alerts: 2, Local IP

DST: 255.255.255.255
Scanned ports: UDP 17500, Pkts: 1, Chain: INPUT, Intf: wlan0
DST: 192.168.1.106, Local IP
Scanned ports: TCP 1-65535, Pkts: 56646, Chain: INPUT, Intf: lo
Signature match: "BACKDOOR Remote PC Access connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 34012, SYN, Sid: 2124
Signature match: "MISC Ghostsurf communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 7212, SYN, Sid: 100203
Signature match: "MISC xfs communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 7100, SYN, Sid: 1987
Signature match: "BACKDOOR NetSphere Connection attempt"
TCP, Chain: INPUT, Count: 2, DP: 30101, SYN, Sid: 100044
Signature match: "DOS arkiea backup communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 617, SYN, Sid: 282
Signature match: "SNMP AgentX/tcp request"
TCP, Chain: INPUT, Count: 1, DP: 705, SYN, Sid: 1421
Signature match: "P2P BitTorrent communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 6889, SYN, Sid: 2181
Signature match: "BACKDOOR Subseven connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 27374, SYN, Sid: 100207
Signature match: "P2P eDonkey transfer attempt"
TCP, Chain: INPUT, Count: 1, DP: 4242, SYN, Sid: 2586
Signature match: "BACKDOOR Infector.1.x Connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 146, SYN, Sid: 100040
Signature match: "POLICY vncviewer Java applet communication attempt"
TCP, Chain: INPUT, Count: 2, DP: 5801, SYN, Sid: 1846
Signature match: "BACKDOOR HackAttack 1.20 Connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 31785, SYN, Sid: 141
Signature match: "DOS DB2 dos communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 6790, SYN, Sid: 1641
Signature match: "MISC Microsoft SQL Server communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 1433, SYN, Sid: 100205
Signature match: "P2P Napster Client Data communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 7777, SYN, Sid: 562
Signature match: "P2P Napster Client Data communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 6666, SYN, Sid: 563
Signature match: "DDOS Trin00 Attacker to Master connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 27665, SYN, Sid: 100007
Signature match: "DDOS mstream client to handler"
TCP, Chain: INPUT, Count: 1, DP: 15104, SYN, Sid: 249
Signature match: "BACKDOOR DonaldDick 1.53 connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 23476, SYN, Sid: 153
Signature match: "BACKDOOR WinCrash 1.0 communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 5714, SYN, Sid: 163
Signature match: "POLICY HP JetDirect LCD communication attempt"
TCP, Chain: INPUT, Count: 1, DP: 9001, SYN, Sid: 510
Signature match: "BACKDOOR Doly 1.5 Connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 1015, SYN, Sid: 1985
Signature match: "BACKDOOR DoomJuice file upload attempt"
TCP, Chain: INPUT, Count: 25, DP: 3198, SYN, Sid: 2375
Signature match: "MISC Alcatel PABX 4400 connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 2533, SYN, Sid: 1819
Signature match: "BACKDOOR - Dagger_1.4.0 Connection attempt"
TCP, Chain: INPUT, Count: 1, DP: 2589, SYN, Sid: 100038
Signature match: "DDOS mstream client to handler"
TCP, Chain: INPUT, Count: 1, DP: 12754, SYN, Sid: 247
DST: 192.168.1.255, Local IP
Scanned ports: UDP 111-17500, Pkts: 4, Chain: INPUT, Intf: wlan0
DST: 224.0.0.251
Scanned ports: UDP 5353, Pkts: 2, Chain: INPUT, Intf: wlan0

...

Total scan sources: 4
Total scan destinations: 6

[+] These results are available in: /var/log/psad/status.out

And in the syslog you should have the following lines (rsyslog in backward compatibility mode):

Jan 11 07:53:54 krystianek psad: scan detected: 192.168.1.106 -> 192.168.1.255 udp: [111-17500] udp pkts: 3 DL: 5
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "BACKDOOR Remote PC Access connection attempt" (sid: 2124) tcp port: 34012
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "MISC xfs communication attempt" (sid: 1987) tcp port: 7100
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "MISC Ghostsurf communication attempt" (sid: 100203) tcp port: 7212
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "BACKDOOR NetSphere Connection attempt" (sid: 100044) tcp port: 30101
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "DOS arkiea backup communication attempt" (sid: 282) tcp port: 617
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "SNMP AgentX/tcp request" (sid: 1421) tcp port: 705
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "P2P BitTorrent communication attempt" (sid: 2181) tcp port: 6889
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "BACKDOOR Subseven connection attempt" (sid: 100207) tcp port: 27374
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "P2P eDonkey transfer attempt" (sid: 2586) tcp port: 4242
Jan 11 07:53:54 krystianek psad: src: 192.168.1.106 signature match: "POLICY vncviewer Java applet communication attempt" (sid: 1846) tcp port: 5801
Jan 11 07:53:54 krystianek psad: scan detected: 192.168.1.106 -> 192.168.1.106 tcp: [6-65532] flags: SYN tcp pkts: 16889 DL: 5

Snort

Snort is much more than port scanner detection tool. It is an Network Intrusion Detection System (NIDS) allowing to . Actually the port scan detector (sfportscan) is only a module in it

1. Installation

The package is available in the standard repo (installation can be done via apt-get as below):

# apt-get install snort
...

2. Configuration

2.1 Main configuration file: snort.conf

What you need to do is to uncomment the sfportscan preprocessor in the main configuration file (snort.conf) as below:

# cat /etc/snort/snort.conf
...

# Portscan detection. For more information, see README.sfportscan
preprocessor sfportscan: proto { all } memcap { 10000000 } sense_level { low } logfile { /var/log/snort/portscan.log }

...

In my case I added the location of the log file (marked with red) to /var/log/snort/portscan.log. The sense_level is set to low, it is possible to set higher level. However you can expect plenty of messages concerning for example access to your samba shares (this happened in my case).

2.2 Additional configurations: snort.debian.conf

You can configure it after installation of the snort package or also afterwards:
# dpkg-reconfigure snort

In my case I wanted snort to monitor two interfaces, send stats via email once per day - the content of the configuration file can be found below:

# cat /etc/snort/snort.debian.conf
...

DEBIAN_SNORT_STARTUP="boot"
DEBIAN_SNORT_HOME_NET="192.168.0.0/16 10.0.0.0/8"
DEBIAN_SNORT_OPTIONS=""
DEBIAN_SNORT_INTERFACE="eth0 wlan0"
DEBIAN_SNORT_SEND_STATS="true"
DEBIAN_SNORT_STATS_RCPT="root@localhost"
DEBIAN_SNORT_STATS_THRESHOLD="1"

3. Startup

The startup of snort is controlled via init start scripts - ensure that after changing the configuration the service is restarted:

# service snort restart
* Stopping Network Intrusion Detection System snort [ OK ]
* Starting Network Intrusion Detection System snort [ OK ]

Now the snort daemon should be up and running with the new configuration.

4. Test

In this case I executed the scanning from a remote host (WinXP) using nmap. The IP address of the remote host was 192.168.1.103 and the local one being scanned 192.168.1.106 (as in previous test run). The nmap parameters were the same as in case of psad, scanning was initiated from GUI.

snort portscan log file (lines appeared almost real-time):

...

Time: 01/12-12:26:15.168523

event_ref: 0

192.168.1.103 -> 192.168.1.106 (portscan) TCP Portscan

Priority Count: 8

Connection Count: 10

IP Count: 1

Scanner IP Range: 192.168.1.103:192.168.1.103

Port/Proto Count: 10

Port/Proto Range: 21:5900

...

syslog (auth.log):

...
Jan 12 12:26:16 krystianek snort[1225]: [1:1421:11] SNMP AgentX/tcp request [Classification: Attempted Information Leak] [Priority: 2] {TCP} 192.168.1.103:49221 -> 192.168.1.106:705
...
Jan 12 12:26:29 krystianek snort[1225]: [1:1420:11] SNMP trap tcp [Classification: Attempted Information Leak] [Priority: 2] {TCP} 192.168.1.103:49221 -> 192.168.1.106:162
...
Jan 12 12:26:51 krystianek snort[1225]: [1:249:8] DDOS mstream client to handler [Classification: Attempted Denial of Service] [Priority: 2] {TCP} 192.168.1.103:49221 -> 192.168.1.106:15104
...

The scan has been detected really fast.

Summary

Both utilities (psad and snort) can be used to detect a port scanning activity. However they use different methods for achieving this goal. It seems that psad relies mostly on parsing iptables logs in a text format, while snort does that in binary form.

In my opinion the biggest disadvantage of psad utility is that it requires iptables logging enabled, which might cause flood of messages going into the logs (if misconfigured or due to excessive ammount of traffic to be logged). In case of snort there is no need to enable any logging it does the monitoring of the packtes by itself (in binary form). The danger of flooding the logs, leading to any service impact, is minimal.

As far as automatic actions after discovering a scan are concerned psad can be configured to automatically block the IP address (from which the scan has been detected) using iptables and tcpwrapper. For details please refer the documentation I have not used this option. For snort I could not easily find anything, for sure there are additional utilities that could help to do this like snortsam: http://www.snortsam.net/ or sagan log (auth.log) parser: http://sagan.quadrantsec.com/.

Ubuntu 12.10: Rip audio to wav/mp3 directly from bin/cue image

2012-12-01T11:51:00.002-08:00

Today I wanted to rip audio tracks from the image of a CD stored in cue format into a bunch of encoded audio files. Instead of burning the CD first and then ripping it into the desired format I decided to check if this can be done directly from the image file. Unfortunately I did not find a way to do it in one step but the described below two-step approach is not that bad at all.

Ripping audio from bin/cue into wav files

In order to do that you need to install bchunk utility (available in the standard repo) as in the example below:

$ sudo apt-get install bchunk
[sudo] password for krychu:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
daemon
Use 'apt-get autoremove' to remove it.
The following NEW packages will be installed:
bchunk
0 upgraded, 1 newly installed, 0 to remove and 24 not upgraded.
Need to get 13.1 kB of archives.
After this operation, 59.4 kB of additional disk space will be used.
Get:1 http://ubuntu.task.gda.pl/ubuntu/ quantal/universe bchunk amd64 1.2.0-12 [13.1 kB]
Fetched 13.1 kB in 8s (1,468 B/s)
Selecting previously unselected package bchunk.
(Reading database ... 548740 files and directories currently installed.)
Unpacking bchunk (from .../bchunk_1.2.0-12_amd64.deb) ...
Processing triggers for man-db ...
Setting up bchunk (1.2.0-12) ...

Afterwards you are ready to rip the audio files from the bin/cue. Please check the command below for reference:

$ bchunk -w Audio\ Disc.bin Audio\ Disc.cue audio
binchunker for Unix, version 1.2.0 by Heikki Hannikainen
Created with the kind help of Bob Marietta ,
partly based on his Pascal (Delphi) implementation.
Support for MODE2/2352 ISO tracks thanks to input from
Godmar Back , Colas Nahaboo
and Matthew Green .
Released under the GNU GPL, version 2 or later (at your option).

Reading the CUE file:

Track 1: AUDIO 01 00:00:00
Track 2: AUDIO 01 03:13:72
Track 3: AUDIO 01 07:05:72
Track 4: AUDIO 01 10:41:25
Track 5: AUDIO 01 14:47:30
Track 6: AUDIO 01 17:57:65
Track 7: AUDIO 01 20:56:54
Track 8: AUDIO 01 24:29:40
Track 9: AUDIO 01 28:07:22
Track 10: AUDIO 01 31:07:67
Track 11: AUDIO 01 33:39:47
Track 12: AUDIO 01 37:18:62

Writing tracks:

1: audio01.wav 32/32 MB [********************] 100 %
2: audio02.wav 39/39 MB [********************] 100 %
3: audio03.wav 36/36 MB [********************] 100 %
4: audio04.wav 41/41 MB [********************] 100 %
5: audio05.wav 32/32 MB [********************] 100 %
6: audio06.wav 30/30 MB [********************] 100 %
7: audio07.wav 35/35 MB [********************] 100 %
8: audio08.wav 36/36 MB [********************] 100 %
9: audio09.wav 30/30 MB [********************] 100 %
10: audio10.wav 25/25 MB [********************] 100 %
11: audio11.wav 36/36 MB [********************] 100 %
12: audio12.wav 18/18 MB [********************] 100 %
$

Ok, now you have the audio tracks in the wav file format. in the next section you can find how to encode it into your preferred format (e.g. mp3).

Converting wav files to mp3

There are few utilities allowing to encode the wav file into mp3 available on linux. I used soundconverter, simple GUI tool, downloadable from the repo.

$ sudo apt-get install soundconverter
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
daemon
Use 'apt-get autoremove' to remove it.
The following packages will be upgraded:
soundconverter
1 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
1 not fully installed or removed.
Need to get 0 B/138 kB of archives.
After this operation, 1,029 kB of additional disk space will be used.
(Reading database ... 548746 files and directories currently installed.)
Preparing to replace soundconverter 2.0.1-1 (using .../soundconverter_2.0.1-1_all.deb) ...
Unpacking replacement soundconverter ...
Processing triggers for bamfdaemon ...
Rebuilding /usr/share/applications/bamf.index...
Processing triggers for desktop-file-utils ...
Processing triggers for gnome-menus ...
Processing triggers for hicolor-icon-theme ...
Processing triggers for man-db ...
Setting up soundconverter (2.0.1-1) ...
$

Afterwards you can start it from command line (soundconverter) or main menu and encode the wav files into desired format (mp3, ogg etc.) as presented on the figure below:

Figure 1 Soundconverter GUI - converting wav files into mp3

Ubuntu 12.10: Recovering removed files on FAT32 usb stick

2012-11-20T13:46:00.001-08:00

Have you ever removed a file from your disk and later on regret it... I am pretty sure you did, recently it happened to one of my colleagues. The lost or should I say deleted file was a document (.docx) on a USB stick.

Unfortunately none of the free windows based utilities was able to do the job. We decided to give the linux based open source utilities a try. On the net I found two utilities (originating from one foremost) that could be used for that purpose (actually there were more but these looked the most promising):

1) scalpel

2) foremost

Both are available in the Ubuntu repos, below one can find some guidelines on installing, configuring the utilities. Before we start - few things about my environment

1) /dev/sdb1 - usb stick device

2) /tmp/recovery - the directory where the recovered files will be put into

Please keep in mind if you have automount feature turned on (which mounts automatically the device from which you would like to recover files) please ensure that you unmount it (in my opinion it is safer to do so).

Scalpel

Scalpel was the first utility that I decided to give a try. First you need to install the utility using the apt-get as follows:

# apt-get install scalpel

# dpkg -l | grep scalpel

ii scalpel 1.60-1build1 amd64 A Frugal, High Performance File Carver

Scalpel utility comes with a configuration file located at /etc/scalpel/scalpel.conf, the default configuration coming with the package has all file extensions commented out (not configured). Fortunately for most typical file types it has already samples so all you need is to edit and uncomment the lines for the file types you want to recover (in my case .doc):

# vim /etc/scalpel/scalpel.conf

...

doc y 10000000 \xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1\x00\x00 \xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1\x00\x00 NEXT

doc y 10000000 \xd0\xcf\x11\xe0\xa1\xb1

...

Now all you need is to launch the scalpel utility as follows:

# scalpel /dev/sdb1 -o /tmp/recovery/scalpel

Afterwards you should go to the output directory where you should be able to find the recovered files (in my case recovered files were placed in two folders doc-0-0 and doc-1-0):

# ls -l

-rw-r--r-- 1 root root 760 Nov 20 13:03 audit.txt

drwxr-xr-x 2 root root 4096 Nov 20 13:05 doc-0-0

drwxr-xr-x 2 root root 4096 Nov 20 13:06 doc-1-0

Unfortunately it recovered only .doc files not even a single .docx ! I tried to find a proper definition for .docx that could be placed in scalpel.conf file - I found one example on the net but it did not work in my case, I guess the definition was not correct.

Summarizing the scalpel tool recovered the document files (.doc) very fast (2 GB usb stick) but I was not able to make it working for .docx.

That is why I checked the second tool...

Foremost

As mentioned before the foremost utility is available in the standard Ubuntu repos so all you need is to install it using apt-get as shown below:

# apt-get install foremost

# dpkg -l | grep foremost

ii foremost 1.5.7-2 amd64 forensic program to recover lost files

The foremost utility has a set of predefined file formats (types) to be recovered but there is no configuration file like in scalpel. However it also makes the utility to be simpler for a first time user.

Below you can find the command I tried - with file types doc, ole and zip.

# foremost -t doc,ole,zip -i /dev/sdb1 -o /tmp/recovery/foremost

It took a bit longer than in case of scalpel but... After checking the output directory;

# ls -l

-rw-r--r-- 1 root root 2405 Nov 20 13:25 audit.txt

drwxr-xr-- 2 root root 4096 Nov 20 13:26 docx

drwxr-xr-- 2 root root 4096 Nov 20 13:24 ole

drwxr-xr-- 2 root root 4096 Nov 20 13:24 zip

Voila - in the docx directory I could find the docx files !

Summary

Summarizing - both tools are able to recover permanently deleted files from the filesystem (in my case FAT32). Scalpel seems to have more advanced configuration capabilities that might make it usable in advanced tasks (custom format files etc.), however one needs to perfectly know how to properly define the type, which i was not able to do for the .docx format. On the other hand if you are looking for something simple, easy and the data to files to be recovered fit into the list of standard formats (jpg, gif, png, bmp, avi, exe, mpg, wav, riff, wmv, mov, pdf, ole, doc, zip, rar, htm, cpp) then foremost is the perfect utility for your needs.

KVM windows guest on Ubuntu 12.10 - no sound

2012-11-15T08:42:00.001-08:00

Recently I faced a problem that I could play sound from the KVM windows based guest machine (even though the virtual sound card AC'97 was properly detected and installed). I went through several article recommending to change the type of the card, configure qemu.conf to grant access to the audio. Unfortunately none of these really helped. Then after doing all the previously described things I tried disabling the apparmor and... Now it works...

Below I enclose more detailed procedure on what was needed in my case to make the audio from the guest kvm machine working.

1. Change the type of the card (might not be necessary). In my case the es1370 was chosen

2. Configure qemu.conf - please ensure that the following entries are present in the qemu.conf configuration file. The username and groupname shall be adapted to your system configuration

# vim /etc/libvirt/qemu.conf
...

#
vnc_allow_host_audio = 1

...

# The user ID for QEMU processes run by the system instance.

user = "krychu"

# The group ID for QEMU processes run by the system instance.

group = "krychu"

...

3. Disable apparmor. Disabling armor really made the sound working on my box. Actually I completely removed the apparmor from my system (I think it is not necessary, you might try only disabling it).

First I stopped the service:
# service apparmor stop

Next I removed it from the startup scripts
# update-rc.d -f apparmor remove

Since I do not use apparmor at the moment I eventually decided to remove it completely (if it happens that you installed apparmor-utils as well you should remove it as well)
# apt-get remove apparmor apparmor-utils

Reboot...

Ubuntu 12.04 LTS "Precise Pangolin"

2012-05-04T23:26:00.001-07:00

After long time of inactivity I am hopefully back ;) I am not sure if all of you have noticed but the new 12.04 LTS Ubuntu version has been recently released (26-th of April 2012) - if you wantto find out what is new please check http://www.ubuntu.com/ubuntu/whats-new.

I have updated yesterday my HP EliteBook 8540W laptop from 64-bit Ubuntu 11.10 (Oneiric Ocelot), no major problems encountered so far - only minor related to ruby gems installed using the ruby gem engine (but some of them were resolved by reinstalling the gem).

Forwarding all root mails to local user and receiving them using Thunderbird

2012-05-04T23:14:00.003-07:00

Yesterday I checked my root local mailbox on my laptop running Ubuntu 12.04 LTS and found out 1500 hanging mails regarding problems with anacron daemon, That is quite a lot. It seems that I forgot to check the mailbox for quite long time so I decided to make it more visible to a standard user that I am using on my laptop and use the Thunderbird mail client (that I am daily using) to get them. You will need to replace the username with your username.

1. Create mailbox

If you want to create a new one:
root@localhost# touch /var/spool/mail/

If you want to create one with root:

root@localhost# cp /var/spool/mail/root /var/spool/mail/

Next create proper access permissions:
root@localhost# chown :mail /var/spool/mail/
root@localhost# chmod 640 /var/spool/mail/

2. Forward mails from root account to local users

You need to add the following line to the .forward file in the root home directory (if it does not exists you need to create it e.g. using vim utility):

root@localhost# cat /root/.forward
@localhost

Now the mails coming to root user should be forwarded to the user indicated via

3. Configure Thunderbird

On my Ubuntu 12.04 LTS I am using Thunderbird version 11.0.1.

In order to manage your local mailbox account you need to configure a 'Unix Mailspool (Movemail)' account. You will find it when creating an other account (File -> New -> Other Accounts...):

Figure 1 The Account wizard for creating the Movemail account

Then just follow next steps (acoount name, mail etc.). As soon as you finish, you should have your local mailbox visible in the Thunderbird main window.

Now I hope to get more visibility especially with respect to anacron jobs running on my laptop.

JMeter DB benchmarks: Embedded Apache Derby vs. Network Apache Derby

2011-12-03T02:45:00.001-08:00

The Apache Derby database as you probably know from my previous posts has two modes of operation:

embedded - in which the datbase itself is started and owned by the process starting it (the only way to connect to it is via the owning process, moreover there is no other process that can open the database from outside world)
network server - the database itself is started as a standalone process to which one can connect via TCP/IP (JDBC, ij or any other Derby client). It allows multiconnection, multiuser mode of working. This mode is very similar to what other database are offering.

This article will cover the description howto start and benchmark the existing Derby database in the network server mode.

Network server Derby

It is not that difficult actually. What you need is the binary that you can download from the ApacheDerby web site. Next there is a great tutorial on how to startup Derby in network mode: Apache Derby tutorial. One remark - in my case (Derby 10.8.2.2) it was important to set the DERBY_HOME variable (not the DERBY_INSTALL). That is mainly because the : setNetworkServerCP script was expecting this one rather than DERBY_INSTALL.

#!/bin/sh

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License. You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing,

# software distributed under the License is distributed on an

# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

# KIND, either express or implied. See the License for the

# specific language governing permissions and limitations

# under the License.

if [ -z "$DERBY_HOME" ]; then

echo "Error: DERBY_HOME is not set. Please set the DERBY_HOME environment variable"

echo "to the location of your Derby installation."

return 1

export CLASSPATH="${DERBY_HOME}/lib/derbynet.jar:${DERBY_HOME}/lib/derbytools.jar:${CLASSPATH}"

In short you need to invoke the following commands:

$ export DERBY_HOME=`/home/krychu/mediate/experiments/derby/binaries/db-derby-10.8.2.2-bin`
$ cd bin
$ . ./setNetworkClientCP
$ ./startNetworkServer ∓
[1] 12629
$ Sat Dec 03 13:21:59 CET 2011 : Security manager installed using the Basic Server security policy.
Sat Dec 03 13:22:00 CET 2011 : Apache Derby Network Server - 10.8.2.2 - (1181258) started and ready to accept connections on port 1527

Migrating existing DB

For a totally new installation you would usually create a completely new db, create the corresponding data model and do the provisioning but for the existing db (when you change only the mode of working or you just upgrade to a newer version) you might want to use/migrate the existing Apache Derby database to the network server mode.

In my exercise I wanted to migrate the DB from Apache Derby version 10.3.3.0 (so far use only in embedded mode) to 10.8.2.2 . In order to do that you need only to copy the database directory to $DERBY_HOME/bin one:

# cp -Rf /home/krychu/mediate/experiments/derby/mzdb $DERBY_HOME/bin

JMeter enhancements

Now you need to add the JDBC driver for connecting to the Derby network server. Again you only need to copy a couple of jars to the JMeter lib directory:

#cp $DERBY_HOME/lib/* $JMETER_HOME/lib/

Configure the JDBC connection

Make sure that the Derby network server is running. By default it should open the 1527 port for incoming connections. Good approach is to make a simple connection test using the ij as in the example below:

$ $DERBY_HOME/bin/ij

ij version 10.8

ij> connect 'jdbc:derby://localhost:1527/mzdb;create=false;user=mzadmin;password=mz';

ij>

If everything is ok it is time to start the JMeter and define the proper JDBC URL. I took the testcase defined in the previous article and changed only the embedded mode into network mode by disabling the database configuration for embedded mode and adding a new one with the parameters as below:

JDBC connection configuration

As I mentioned the test is organized as in previous article:

read access - single, simple search by indexed field
write access - single write operation into one table
backup - in the background the backup procedure is invoked (total size of db: 285 MB)

All three groups are running in parallel.

Apache Derby benchmark - JMeter testcase

Now everything is ready for the first benchmark. Do not forget to start the agents if you want to also monitor the CPU, memory, network and disk activity.

Benchmark results

Below one can find the tables that summarizes the results obtained on HP EliteBook 8540W machine running Ubuntu 11.04 Natty. As you see the results are similar, slightly better for network server but I would still say that in the range of errors. As expected the network server consumes much more network bandwidth (also a proof that the communication goes over loopback interface), moreover it will increase with the number of transactions.
As described previously the test has been invoked when all types of accesses triggered by separate thread groups. The backup procedure has only effect on the number of writes (as expected) in my opinion since both are I/O consuming. During the tests when the backup was initiated the write rate was decreased by a factor of ~3 but they were still processed (not a single one suspended or rejected). As far as reads are concerned there was no difference measured.

Category	Apache Derby embedded	Apache Derby - network server
Read - single row during backup (Tps)	27 000	27 000
Read - single row (Tps)	27 000	27 000
Write - single row during backup (Tps)	22	23
Write - single row (Tps)	65	70
Network bandwidth - lo during backup (KB/s)	2	15
Network bandwidth - lo (KB/s)	2	42
Backup time (s)	35	32

Summary

As far as performance is concerned it does not make that big difference if Apache Derby is running in embedded mode or in standalone (network server running on the same host - communication via localhost) - at least on my small machine (HP EliteBook 8540W). It might be that the differences will be more visible when larger amount of data are being transferred over the network - using queries returning multiple rows. However taking into account other aspects like: allowing simultaneous access by multiple clients, better serviceability, easier administration, configuration and flexibility in deployment (it might run on the same server or on remote one) - I would choose the network server mode.

Using JMeter for DB (Apache Derby) benchmarking

2011-11-19T13:48:00.001-08:00

Recently I had to do check or I was for support in a case where the Apache Derby was used in embedded mode. The questions were related to performance, demiensioning and also some non-functional aspects like backup and restore. Going through documentation is one thing but being able to verify some performance figures is another. This article will describe how I used the JMeter to setup environment measuring reads, writes and doing the backup in the background (to check the impact on processed transactions) for the Apache Derby working in the embedded mode (the external one is out of scope of this article).

Obtaining JMeter

Although the JMeter is available in the repos for most popular distribtions (e.g. Ubuntu), I recommend to download it from the webpage (newer version, more features etc.): http://jmeter.apache.org/ - the installation is about unzipping an archive (you can use e.g. unzip).

Instaling JMeter

After unpacking you will need to add the jdbc driver to the classpath - in my case it basically meant that I needed to copy the derby.jar into the lib subdirectory of jmeter. If you already have a derby database on which you want to do benchmark (my case) it would be better to find the Derby version matching the one which was used to create the database. As I mentioned before in this article the embedded mode for derby will be used.

Install additional chart/monitoring JMeter plugins

There is a set of nice, additional plugins available on google code: http://code.google.com/p/jmeter-plugins/ that I would recommend to install. Not only they provide nicer charting listeners but also monitoring capabilities (each system to be monitor will need to have a special agent started). The installation procedure is described in the README and on the webpage.
What I did is after downloading I unpacked the archive to a subdirectory under JMETER_HOME. Next I copied the JMeterPlugins.jar to the lib/ext subdirectory. That's it, the new plugins should be available now.

Building the testcase

Ok, now you are ready to prepare the testcase.

1. Start the JMeter:
# $JMETER_HOME/bin/jmeter.sh

2. Add the database (ApacheDerby) connection definition (JDBC Connection Definition) as follows:

The database URL in case of embedded derby shall contain the path to the database location on the filesystem.
The JDBC Driver class is the driver class - here a specific one for Derby.
In addition one needs to provide username and password for connecting to the database.

3. Add thread pools and samplers

I added three different thread pools: one for testing the reads, one for testing the writes and one (single threaded) for backup. As you see all are using the same JDBC connection definition - defined on the global level

4. Add JDBC request samplers

Inside each thread pool you should define the JDBC request sampler, responsible for generating the appropriate traffic type. Below you can find example used by me for read, write and starting the backup:

The JDBC request sampler for read access

The JDBC request sampler for write access

The JDBC request sampler for initiating the backup

5. Add listeners for summary, CPU, memory, IO statistics.

For getting the summary you should add to each thread group the so called 'Aggregate report' listener. It will provide statistics like: number of samples, average response time, q90, median, min, max, error rate and throughput.

Example aggregate report listener for read access

In addition on the top level you should add the collectors for CPU, memory and disk, network - these are available after installing the additional google code plugin in the Listener section. The important thing is you need to start the agent on the monitored node as follows:

$ $JMETER_PLUGIN/serverAgent/startAgent.sh
JMeterPlugins Agent version 1.4
No port specified, the default value is used: 4444
------ File Systems init: ------
File System detected: /dev/mapper/krystianek-root
File System detected: /dev/sda1
--------------------------------
--- Network Interfaces init: ---
Network interface detected: lo
Network interface detected: wlan0
Network interface detected: virbr1
Network interface detected: virbr2
Network interface detected: virbr0
Network interface detected: tun0
Network interface detected: eth0
--------------------------------
Waiting for incoming connections...
Client id=0 connected!
Client id=1 connected!
Client id=2 connected!
Client id=3 connected!
Client id=3 disconnected!
Client id=0 disconnected!
Client id=1 disconnected!
Client id=2 disconnected!

And the statistics collector shall be configured to connect to that agent:

Example listener for collecting disk, network statistics

Example listener for collecting CPU statistics

I also recommend to put the Transaction per second listener from google code to each thread pool section, which will provide nice charts for per second statistics.

Example lstener providing graphical represenation of throughput per second

Now you just have to start the execution of the test and do the measurements;)

Resizing root filesystem: KVM, LVM and EXT4

2011-11-12T04:32:00.001-08:00

Ever faced a problem that you run out of the root filesystem space? I recently faced that problem on my KVM CentOS 6.0 box (a lot of experiments done on it) and wanted to solve it without reinstallation of the system. It seems to be possible without losing any data and not that difficult on all levels:

KVM virtual disk size
LVM volume group size
EXT4 filesystem size

Below you can find the instruction on how to do that:

1. Resize the KVM virtio disk image. First one need to resize the virtual disk for the KVM. In order to do that you shoudl create a new add on raw disk and append it to the existing disk image. I recommend to first move the original disk for the backup purposes. The steps to be done are shown below:

root@krystianek:/var/lib/libvirt/images# qemu-img create -f raw addon.raw 4G
Formatting 'addon.raw', fmt=raw size=4294967296

root@krystianek:/var/lib/libvirt/images# mv centos6.img centos6.img.backup
root@krystianek:/var/lib/libvirt/images# cat centos6.img.backup addon.raw > centos6
centos6.img.backup centos6-oracle-1.img centos6-oracle-2.img centos6-oracle.img
root@krystianek:/var/lib/libvirt/images# cat centos6.img.backup addon.raw > centos6.img

2. Boot from LiveCD. It is required since you will be modyfing the root filesystem's volume group if you plan to modify another volume group/filesystem it may not be required. I used the CentOS LiveCD in order to do that.

3. Create a new partition. Create a new partition from the free disk space (4G). In my case I created a /dev/vda3 partition using the CentOS graphical disk utility.

4. Extend the LVM volume group. Add the newly /dev/vda3 device to the volume group

[root@livecd ~]# vgdisplay
--- Volume group ---
VG Name               vg_centos6hosta
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No 3
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               1
Max PV                0
Cur PV                1
Act PV                1
VG Size               7.51 GiB
PE Size               4.00 MiB
Total PE              1922
Alloc PE / Size       1922 / 7.51 GiB
Free PE / Size       0 / 0
VG UUID               DIeDAo-5Y1O-GEC0-JCb6-keUy-8Yc1-MhRBdN

[root@livecd ~]# vgextend vg_centos6hosta /dev/vda3
No physical volume label read from /dev/vda3
Physical volume "/dev/vda3" successfully created
Volume group "vg_centos6hosta" successfully extended
[root@livecd ~]# vgdisplay
--- Volume group ---
VG Name               vg_centos6hosta
System ID
Format                lvm2
Metadata Areas        2
Metadata Sequence No 4
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               1
Max PV                0
Cur PV                2
Act PV                2
VG Size               11.50 GiB
PE Size               4.00 MiB
Total PE              2945
Alloc PE / Size       1922 / 7.51 GiB
Free PE / Size       1023 / 4.00 GiB
VG UUID               DIeDAo-5Y1O-GEC0-JCb6-keUy-8Yc1-MhRBdN

[root@livecd ~]# lvresize -L+3995M /dev/mapper/vg_centos6hosta-lv_root
Rounding up size to full physical extent 3.90 GiB
Extending logical volume lv_root to 8.50 GiB
Logical volume lv_root successfully resized
[root@livecd ~]#

5. Resize the filesystem residing on the LVM volume. Last but not least you have to resize the filesystem residing on the LVM volume group. For ext4 you can do it as follows (where vg_centos6hosta-lv_root is the root fs volume group):

[root@livecd ~]# e2fsck -f /dev/mapper/vg_centos6hosta-lv_root
e2fsck 1.41.12 (17-May-2010)
_CentOS-6.0-x86_: recovering journal
Clearing orphaned inode 131145 (uid=27, gid=27, mode=0100600, size=0)
Clearing orphaned inode 131144 (uid=27, gid=27, mode=0100600, size=0)
Clearing orphaned inode 131140 (uid=27, gid=27, mode=0100600, size=0)
Clearing orphaned inode 131136 (uid=27, gid=27, mode=0100600, size=0)
Clearing orphaned inode 131130 (uid=27, gid=27, mode=0100600, size=0)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

_CentOS-6.0-x86_: ***** FILE SYSTEM WAS MODIFIED *****
_CentOS-6.0-x86_: 139003/303104 files (0.1% non-contiguous), 1195597/1206272 blocks
[root@livecd ~]# resize2fs /dev/mapper/vg_centos6hosta-lv_root
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/mapper/vg_centos6hosta-lv_root to 2229248 (4k) blocks.
The filesystem on /dev/mapper/vg_centos6hosta-lv_root is now 2229248 blocks lo

CentOS: Creating rpm packages (ceph-0.38)

2011-11-11T02:29:00.001-08:00

This article describes step by step how to create own packages on the example of the ceph package (0.38)

1. Create the directory structure - the easiest way is to install the rpmdevtools and use it to create the structure:

[root@localhost /]# yum install rpmdevtools

2. Download the sources as a tar balls and put it to the SOURCES directory:

[root@localhost /]# cd ~/rpmbuild/SOURCES
[root@localhost rpmbuild]# wget http://ceph.newdream.net/download/ceph-0.38.tar.gz

3. Prepare and customize the spec. If you want to pass some options to the configure utility just customize the configure options and specify the directories to be included in the rpm package (all marked with red color below). Check also if the source tar archive name matches the one placed in the SOURCES folder.

[root@localhost rpmbuild]# vim ./SPECS/ceph.spec
Name: ceph
Version: 0.38
Release: 1%{?dist}
Summary: Krystianek's package

Group: HA-experiments
License: GPL
URL: http://http://ceph.newdream.net/
Source0: ceph-0.38.tar.gz
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)

%description
CEPH utilities

%prep
%setup -q

%build
%configure --without-tcmalloc
make %{?_smp_mflags}

%install
rm -rf $RPM_BUILD_ROOT
make install DESTDIR=$RPM_BUILD_ROOT

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%doc

/etc/
/var/
/usr/

%changelog

4. Create the rpm. Go to the directory structure root and invoke the rpmbuild to create the rpm

[root@localhost rpmbuild]# cd ~/rpmbuild
[root@localhost rpmbuild]# rpmbuild -ba SPECS/ceph.spec

You might need to set the mask for RPATH to ignore the errors regarding the standard paths:

[root@localhost rpmbuild]# QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba SPECS/ceph.spec

5. Install the previously create rpm.

[root@localhost rpmbuild]# rpm -i ./RPMS/x86_64/ceph-0.38-1.el6.x86_64.rpm

Remarks: For cuilding the ceph you need additional development packages for tcmalloc (part of google performance, can be downloaded from http://code.google.com/p/google-perftools/) and libatomic (available in the standard CentOS repo as libatomic_ops-devel)

CentOS 6.0: Updating the kernel to 3.0.8

2011-11-05T11:32:00.000-07:00

I wanted to continue the work on the clustered filesystems and found out that linux kernel since 2.6.37 provides additional cluster filesystem ceph (http://ceph.newdream.net/) to the previosuly available gfs (gfs2), ocfs2. The description sounds promising so why not to give it a try.
However in my virtualized environment I had only CentOS'es (one running the latest 6.0) with much older kernel version (2.6.32). The standard repositories does not contain any updates and the only way to run a newer version is to download it from kernel.org and compile it by yourself. The instruction below describes the steps that I did in order to have the kernel 3.0.8 on the CentOS 6.0.

1. Prepare disk space for kernel sources, compilation output and produced rpm packages (about 5GB needed for creating the binary package, header and sources).
2. Download the sources from the kernel.org and unpack them. You can do it on the fly using the following command:

[root@localhost tmp]# wget -O - http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.8.tar.bz2 | bzip2 -d - | tar -xvf -

3. Now copy your existing configuration file and adapt it to your needs as described in step 4 (I used the menuconfig and loaded the file):

[root@localhost linux-3.0.8]# cp /boot/config-2.6.32-71.29.1.el6.x86_64 .
[root@localhost linux-3.0.8]# make menuconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/lxdialog/checklist.o
HOSTCC scripts/kconfig/lxdialog/inputbox.o
HOSTCC scripts/kconfig/lxdialog/menubox.o
HOSTCC scripts/kconfig/lxdialog/textbox.o
HOSTCC scripts/kconfig/lxdialog/util.o
HOSTCC scripts/kconfig/lxdialog/yesno.o
HOSTCC scripts/kconfig/mconf.o
SHIPPED scripts/kconfig/zconf.tab.c
SHIPPED scripts/kconfig/lex.zconf.c
SHIPPED scripts/kconfig/zconf.hash.c
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/mconf
scripts/kconfig/mconf Kconfig
#
# configuration written to config-3.0.4
#

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

[root@localhost linux-3.0.8]#

4. Apply the changes. I have added support for ceph, ocfs2 and added the module to dump the current kernel config to the /proc filesystem.

5. Now you are ready to compile and prepare the rpms containing the new kernel. I used the target to build only binary packages (you can also create the sources package). The rpms will be created in user's home directory: $HOME/rpmbuild/:

[root@localhost linux-3.0.8]# make help | grep rpm
rpm-pkg - Build both source and binary RPM kernel packages
binrpm-pkg - Build only the binary kernel package

[root@localhost linux-3.0.8]# time make binrpm-pkg

/bin/sh /mnt/tmp/linux-3.0.8/scripts/package/mkspec prebuilt > /mnt/tmp/linux-3.0.8/binkernel.spec

make KBUILD_SRC=

CHK include/linux/version.h

UPD include/linux/version.h

CHK include/generated/utsrelease.h

UPD include/generated/utsrelease.h

CC kernel/bounds.s

GEN include/generated/bounds.h

CC arch/x86/kernel/asm-offsets.s

GEN include/generated/asm-offsets.h

CALL scripts/checksyscalls.sh

HOSTCC scripts/genksyms/genksyms.o

SHIPPED scripts/genksyms/lex.c

SHIPPED scripts/genksyms/parse.h

SHIPPED scripts/genksyms/keywords.c

HOSTCC scripts/genksyms/lex.o

...

Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1

Processing files: kernel-headers-3.0.8-1.x86_64

Provides: kernel-headers = 3.0.8

Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1

Obsoletes: kernel-headers

Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/kernel-3.0.8-1.x86_64

Wrote: /root/rpmbuild/RPMS/x86_64/kernel-3.0.8-1.x86_64.rpm

Wrote: /root/rpmbuild/RPMS/x86_64/kernel-headers-3.0.8-1.x86_64.rpm

Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.VjVenh

+ umask 022

+ cd /mnt/tmp/linux-3.0.8

+ rm -rf /root/rpmbuild/BUILDROOT/kernel-3.0.8-1.x86_64

+ exit 0

real 85m20.491s

user 59m40.551s

sys 12m32.836s

[root@localhost linux-3.0.8]# ls -lh /root/rpmbuild/RPMS/x86_64/

-rw-r--r-- 1 root root 262M Nov 5 19:13 kernel-3.0.8-1.x86_64.rpm

-rw-r--r-- 1 root root 786K Nov 5 19:13 kernel-headers-3.0.8-1.x86_64.rpm

6. Install the new kernel rpm package:

[root@localhost linux-3.0.8]# rpm -i /root/rpmbuild/RPMS/x86_64/kernel-3.0.8-1.x86_64.rpm
[root@localhost linux-3.0.8]#

7. Do the postconfiguration to boot the new kernel - the easier way is to do it using the 7.1 step and only check if the files described in manual procedure are created/configured properly.

7.1 Automatically using the new-kernel-pkg utility:

[root@localhost linux-3.0.8]# new-kernel-pkg --mkinitrd --install --make-default --depmod 3.0.8

7.2 Manually (when previous used it can be skipped - everything already done by the new-kernel-pkg utility)

7.2.1 Create the initrd image:

[root@localhost linux-3.0.8]# mkinitrd -f /boot/initramfs-3.0.8.x86_64.img 3.0.8

7.2.2 Adapt the grub menu so that the new kernel becomes an option, below an example from my system:

[root@localhost boot]# vim /boot/grub/menu.lst
...

title CentOS Linux (3.0.8.x86_64)
root (hd0,0)
kernel /vmlinuz-3.0.8 ro root=/dev/mapper/vg_centos6hosta-lv_root rd_LVM_LV=vg_centos6hosta/lv_root rd_LVM_LV=vg_centos6hosta/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
initrd /initramfs-3.0.8.x86_64.img

8. Reboot the system and start the new kernel. Afterwards you should see:

[krychu@localhost ~]$ uname -a
Linux localhost.localdomain 3.0.8 #1 SMP Sat Nov 5 18:56:32 CET 2011 x86_64 x86_64 x86_64 GNU/Linux
[krychu@localhost ~]$

Please notice: when running on a virtual system with VirtIO disks (recognized by the guest system as vda devices) please check if the virtio_blk module is enabled (otherwise you will not be able to mount the root filsystem).

Ubuntu: SSL/TLS handshake - determine version

2011-11-04T14:36:00.000-07:00

Recently I had to figure out what is the SSL and TLS version supported by some framework for which the only information I had was that it supports encryption. After looking into the TLS RFC (http://www.ietf.org/rfc/rfc2246.txt) and information about SSL I figured out that one might get those information by sniffing the handshake procedure.
Having the plan I decided to first check it locally on my host. As a first step I configured the apache2 on my Ubuntu 11.04 box as described: https://help.ubuntu.com/11.04/serverguide/C/httpd.html to support encryption. Next step was to use openssl (plus wireshark to be confident;)) to get the protocol version used in the handshake procedure.

SSL 3.0

Figure 1 SSL 3.0 handshake

Now let's have a test connecting using the openssl to the local apache2 server supporting SSL 3.0.

# openssl s_client -ssl3 -connect 127.0.0.1:443
CONNECTED(00000003)
depth=0 /CN=krystianek
verify error:num=18:self signed certificate
verify return:1
depth=0 /CN=krystianek
verify return:1
---
Certificate chain
0 s:/CN=krystianek
   i:/CN=krystianek
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICpjCCAY4CCQCXhncRWkNj1DANBgkqhkiG9w0BAQUFADAVMRMwEQYDVQQDEwpr
cnlzdGlhbmVrMB4XDTExMTEwMjE2MzgyOVoXDTIxMTAzMDE2MzgyOVowFTETMBEG
A1UEAxMKa3J5c3RpYW5lazCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALU90KhdAKy4dBTGO8GLIpHLvO5L9DNZjLURgVoY2H8r1piYfvZzQ6sgNIrQzZjg
vGDTg2bUT7g4UQFJouSVGGRy8qk5afipoUFS+CxQEoWHkIQoRLdDSEtLxpYlQvk3
ID2qTzLGBeNm+YBSsvzg2e4sWBykKh1ePJ/TFxSdnW9+UZWbwytFlA6qNPVWA2Ks
lX4+N5xHIL1afd1+qGKUR85xJBcXXUbbZM7Fe+neGHmMqxuUXOmrU7uHE7igcHq7
UUXByIertkS33cgpV7QbWr9w+ip8ggKAUVkY/03NpaYfk40L06sef7jyWhMXFO8i
3TbnwDH0m/I3qxZqQZifYbECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAJ4BIrQ/U
KeJWsWQJsOAfJlpQ2giUiEYLDZyP8gnmhNCYnEdCBl9ltXxPIe1RIFK/O5dnAgV/
GLPYREW+jZ4tyip4cwWMIgixtcpyWyESJlH4vF4lKj6dP5C0xP5cyfbfvR9rlNxk
eejBsUv24FK9xCPN2V8IdkS+CNneGW06yAf+EQP9mF5weNXkbE7tX+yBwKuR8BgE
AUZ1qohBPz5OZqSV61HikVUf3RmqapnIo4lejilj/1uVhU8QJ1FxwJu2UyAJaiPj
qRSVvW4H7LSkz326DaKzoFFUylhAZPImEPN0LHv1ee0yzbWAzN8iEM6bzdNYJfJ2
fGQu5yAue2vxHQ==
-----END CERTIFICATE-----
subject=/CN=krystianek
issuer=/CN=krystianek
---
No client certificate CA names sent
---
SSL handshake has read 1413 bytes and written 319 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 6C8A75631A4964A27410DF69CFD267C8C1EE6363A6FDED270BD0671B0DFAE99F
    Session-ID-ctx:
    Master-Key: FDA851ACE2E2320690D55C7766A51718FCD6B2A89ED6887A4368583D8EB2FF2D4559D2408135D60A3401711B9C5FC7A9
    Key-Arg   : None
    Compression: 1 (zlib compression)
    Start Time: 1320299309
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
GET /

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

closed
You have new mail in /var/mail/root
#

Ok, it works. In my case there was no client certificate sent towards the SSL server.

TLS 1.0

Figure 2 TLS 1.0 handshake

And now let's try to connect again using the openssl to the apache2 server but this time using the tls1 protocol.

# openssl s_client -tls1 -connect 127.0.0.1:443
CONNECTED(00000003)
depth=0 /CN=krystianek
verify error:num=18:self signed certificate
verify return:1
depth=0 /CN=krystianek
verify return:1
---
Certificate chain
0 s:/CN=krystianek
   i:/CN=krystianek
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICpjCCAY4CCQCXhncRWkNj1DANBgkqhkiG9w0BAQUFADAVMRMwEQYDVQQDEwpr
cnlzdGlhbmVrMB4XDTExMTEwMjE2MzgyOVoXDTIxMTAzMDE2MzgyOVowFTETMBEG
A1UEAxMKa3J5c3RpYW5lazCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALU90KhdAKy4dBTGO8GLIpHLvO5L9DNZjLURgVoY2H8r1piYfvZzQ6sgNIrQzZjg
vGDTg2bUT7g4UQFJouSVGGRy8qk5afipoUFS+CxQEoWHkIQoRLdDSEtLxpYlQvk3
ID2qTzLGBeNm+YBSsvzg2e4sWBykKh1ePJ/TFxSdnW9+UZWbwytFlA6qNPVWA2Ks
lX4+N5xHIL1afd1+qGKUR85xJBcXXUbbZM7Fe+neGHmMqxuUXOmrU7uHE7igcHq7
UUXByIertkS33cgpV7QbWr9w+ip8ggKAUVkY/03NpaYfk40L06sef7jyWhMXFO8i
3TbnwDH0m/I3qxZqQZifYbECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAJ4BIrQ/U
KeJWsWQJsOAfJlpQ2giUiEYLDZyP8gnmhNCYnEdCBl9ltXxPIe1RIFK/O5dnAgV/
GLPYREW+jZ4tyip4cwWMIgixtcpyWyESJlH4vF4lKj6dP5C0xP5cyfbfvR9rlNxk
eejBsUv24FK9xCPN2V8IdkS+CNneGW06yAf+EQP9mF5weNXkbE7tX+yBwKuR8BgE
AUZ1qohBPz5OZqSV61HikVUf3RmqapnIo4lejilj/1uVhU8QJ1FxwJu2UyAJaiPj
qRSVvW4H7LSkz326DaKzoFFUylhAZPImEPN0LHv1ee0yzbWAzN8iEM6bzdNYJfJ2
fGQu5yAue2vxHQ==
-----END CERTIFICATE-----
subject=/CN=krystianek
issuer=/CN=krystianek
---
No client certificate CA names sent
---
SSL handshake has read 1560 bytes and written 293 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 94FA2FB475C3FA47EDE2373C610F3F3C02CB2714F7344EC433B6B8ACEDC0AC43
    Session-ID-ctx:
    Master-Key: 958861A6AC4D901FA6263C0DA92E81F9430AEB201F82032D562813309957593E9E4259F8548AFBE2CB5A7145026135F5
    Key-Arg   : None
    TLS session ticket:
    0000 - 9a c9 2b 9c e9 54 7e e4-05 de 32 40 38 0a 6d b9   ..+..T~...2@8.m.
    0010 - 6d 21 2c 2e c8 ba 6e b7-de 37 72 0f 5b 5c 69 a8   m!,...n..7r.[\i.
    0020 - da 55 4a f6 73 31 59 4c-c4 3e 37 7b 9c 87 47 97   .UJ.s1YL.>7{..G.
    0030 - 03 f4 c9 62 45 95 a9 ab-11 31 ab de bf c9 5d b4   ...bE....1....].
    0040 - 50 75 ec 6b 54 c4 05 c8-bf 44 d3 14 41 d9 ea e9   Pu.kT....D..A...
    0050 - 0a 57 c8 d1 89 4f 3b 20-c6 0b 1e f6 f4 19 af 8e   .W...O; ........
    0060 - ca f7 18 28 1c 7b c7 9f-d4 03 c1 3f bc 47 be a0   ...(.{.....?.G..
    0070 - e0 74 0c c5 57 d6 16 4b-b3 a4 f5 c7 b7 10 7e 11   .t..W..K......~.
    0080 - 03 6a 5b e6 06 aa d7 75-40 d1 fe b0 62 ae e9 aa   .j[....u@...b...
    0090 - bc 0e 2d 59 06 97 99 78-ac 69 3e 8e c4 7f 34 e8   ..-Y...x.i>...4.
    00a0 - 84 89 c3 01 13 1b 01 b2-49 21 62 b8 4b e5 93 ea   ........I!b.K...
    00b0 - 88 09 bb d2 27 d2 ab cd-b6 94 67 0e a5 9b 7c fc   ....'.....g...|.

    Compression: 1 (zlib compression)
    Start Time: 1320299341
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
GET /

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

closed
#

GTP prime library for Java (3GPP 32.295)

2011-10-31T13:54:00.000-07:00

Recently I needed to have a client application for a GTP' external system. I did some research on the web but unfortunately did not find anything usable for me (found one java library on kenai.com but the implementation is in very early phase, the other was open-cgf written in Erlang for which I found two obstacles the language + crash dumps when it was starting up on my Ubuntu Natty 64-bit box). Having some development background from the past and having a look on the 3GPP 32.295 specification I decided that it is not a big amount of effort to implement it. I think this is a good opportunity to learn the protocol and additionally check the JBoss Netty framework - which seems to be a perfect match for such a solution. Indeed writing a client/server application using JBoss Netty is pretty straightforward (I will try to provide more details in another article) - it mainly about writing proper encoder/decoder and handlers.

The outcome can be found in the google code under the link: http://code.google.com/p/gtpprime/.

Linux: Creating encrypted partition using cryptsetup

2011-10-16T11:38:00.000-07:00

In order to protect sensitive data in case your storage is lost or stolen, one usually encrypts the drive using trusted/safe encryption algorithms. Under Linux one can use e.g. cryptsetup - available in standard Ubuntu repository (I guess for other distros it should be also available in standard repos). Since it works on the block level, one can mix unencrypted partitions with encrypted ones. It also used in most of the distros (e.g. Ubuntu alternate installer) as a tooling providing disk encryption.

Creating/mounting encrypted partition

First of all I did not want to experiment with my HDD and decided to use the USB stick for this experiment. One needs to partition the disk - I created two partitions sdb1 and sdb2 using the fdisk. The partition table has been presented below:

root@krystianek:~# fdisk -l /dev/sdb

Disk /dev/sdb: 4156 MB, 4156555264 bytes
255 heads, 63 sectors/track, 505 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e56eb

   Device Boot      Start         End      Blocks   Id System
/dev/sdb1               1         132     1060258+ 83 Linux
/dev/sdb2             133         505     2996122+ 83 Linux
Now you need to format the partition as shown below (I used the 256-bit key length for encryption). Please type the uppercase 'YES' as prompted below. Also you will have to provide the encryption password.

root@krystianek:~# cryptsetup --verbose --key-size 256 luksFormat /dev/sdb2

WARNING!
========
This will overwrite data on /dev/sdb2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.

That's it now you can open the encrypted partition. As a parameter you will provide the alias to which the device will be mapped as presented below:

root@krystianek:~# cryptsetup --verbose luksOpen /dev/sdb2 usbencrypted
Enter passphrase for /dev/sdb2:
Key slot 0 unlocked.
Command successful.

root@krystianek:~# ls -l /dev/mapper/usbencrypted 
lrwxrwxrwx 1 root root 7 2011-10-16 20:06 /dev/mapper/usbencrypted -> ../dm-3

root@krystianek:~# cryptsetup status usbencrypted
/dev/mapper/usbencrypted is active:
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  device:  /dev/sdb2
  offset:  2056 sectors
  size:    5990189 sectors
  mode:    read/write 

Next you can create the filesystem - I created the ext4 and mounted it under /mnt/hd1:

root@krystianek:~# mkfs.ext4 /dev/mapper/usbencrypted 
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
187312 inodes, 748773 blocks
37438 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=767557632
23 block groups
32768 blocks per group, 32768 fragments per group
8144 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@krystianek:~# mount /dev/mapper/usbencrypted /mnt/hd1/
root@krystianek:~# ls -l /mnt/hd1/
total 16
drwx------ 2 root root 16384 2011-10-16 20:12 lost+found

Now you can start copying files to that filesystem.

Unmounting/closing the encrypted partition

In order unmount and close the encrypted partition one needs to:

root@krystianek:~# umount /mnt/hd1/

root@krystianek:~# cryptsetup --verbose luksClose usbencrypted
Command successful.

Next time when you attach the usb stick you should be automatically prompted for password.

Linux: Sharing disks via iSCSI on Ubuntu 11.04

2011-10-16T00:56:00.000-07:00

Recently I wanted to test a clustering solution that was based on a shared storage and I was looking for a solution that could work in my virtualized environment. One requirement was that the shared disk had to be visible as device (not a mounted NFS share). The choice went to iSCSI. There is quite interesting open source solution for providing NAS functionality (http://www.freenas.org/) but since it is based on FreeBSD 8.2 and my native system in Linux I would have to run in as another virtual machine - might be too much for my box (one FreeNAS virtual machine + two virtual machines hosting cluster - Figure 1). Therefore I searched for something that could be configured natively on my Ubuntu 11.04 box. What I found and decided to configure was the iscsitarget daemon - below you can the step by step instruction how to do that.

Figure 1 iSCSI client-server architecture

iSCSI Server configuration - Ubuntu

First of all you need to install the iscsitarget software - it is available in the standard Ubuntu repo - as below:

# apt-get install iscsitarget

Reading package lists... Done

Building dependency tree      

Reading state information... Done

Suggested packages:

  iscsitarget-source iscsitarget-dkms

Recommended packages:

  iscsitarget-module

The following NEW packages will be installed:

  iscsitarget

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 78.6 kB of archives.

After this operation, 291 kB of additional disk space will be used.

Get:1 http://us.archive.ubuntu.com/ubuntu/ natty/universe iscsitarget amd64 1.4.20.2-1ubuntu1 [78.6 kB]

Fetched 78.6 kB in 0s (128 kB/s)    

Selecting previously deselected package iscsitarget.

(Reading database ... 315227 files and directories currently installed.)

Unpacking iscsitarget (from .../iscsitarget_1.4.20.2-1ubuntu1_amd64.deb) ...

Processing triggers for ureadahead ...

ureadahead will be reprofiled on next reboot

Processing triggers for man-db ...

Setting up iscsitarget (1.4.20.2-1ubuntu1) ...

 * iscsitarget not enabled in "/etc/default/iscsitarget", not starting...

Define the LUNs in the configuration file as follows (red colour marks the location of the LUNs):

# vim /etc/iet/ietd.conf 

...

Target ubuntu.mediate:storage.sys1
        Lun 0 Path=/luns/storagelun0,Type=fileio,ScsiId=lun0,ScsiSN=lun0
        Lun 1 Path=/luns/storagelun1,Type=fileio,ScsiID=lun1,ScsiSN=lun

Create the files that will represent the LUNs. One can also use devices (e.g. USB stick) as placeholders for LUNs but for me files were just perfect - easy to move and control.

# cd /luns/

krychu@krystianek:/luns$ sudo dd if=/dev/zero of=storagelun0 count=0 obs=1 seek=10G

0+0 records in

0+0 records out

0 bytes (0 B) copied, 9.01e-06 s, 0.0 kB/s

krychu@krystianek:/luns$ ls -latr

total 8

drwxr-xr-x 26 root root        4096 2011-10-15 10:39 ..

-rw-r--r--  1 root root 10737418240 2011-10-15 10:40 storagelun0

drwxr-xr-x  2 root root        4096 2011-10-15 10:40 .

krychu@krystianek:/luns$ ls -lh

total 0

-rw-r--r-- 1 root root 10G 2011-10-15 10:40 storagelun0

krychu@krystianek:/luns$ sudo dd if=/dev/zero of=storagelun1 count=0 obs=1 seek=1G

0+0 records in

0+0 records out

0 bytes (0 B) copied, 1.285e-05 s, 0.0 kB/s

Next enable the iscsitarget in the default configuration file - modify the /etc/default/iscsitarget file's content so that it matches the one below:

# cat /etc/default/iscsitarget
ISCSITARGET_ENABLE=true

Start the iscsitarget service:

# service iscsitarget start

 * Starting iSCSI enterprise target service                              [ OK ] 

                                                                         [ OK ]

Ok, that's it - the iSCSI should be configured to publish two LUNs (storagelun0 and storagelun1) from the /luns directory. Next step is to configure the client machines.

iSCSI Client- CentOS 6.0 on kvm

First of all check if the required software is installed: iscsi-initiator-utils (in my case it was). If it is not then install it from the standard repository as follows:

[root@localhost ~]# yum install iscsi-initiator-utils

...

Start and enable the iscsi and multipathd service during boot of the system:

[root@localhost ~]# service iscsi start

[root@localhost ~]# chkconfig --list iscsi

iscsi              0:off    1:off    2:on    3:on    4:on    5:on    6:off

[root@localhost ~]# chkconfig --list multipathd

multipathd         0:off    1:off    2:off    3:off    4:off    5:off    6:off

[root@localhost ~]# chkconfig --add multipathd

[root@localhost ~]# chkconfig --list multipathd

multipathd         0:off    1:off    2:off    3:off    4:off    5:off    6:off

[root@localhost ~]# chkconfig multipathd on

[root@localhost ~]# service multipathd start

Starting multipathd daemon:                                [  OK  ]

Now you can perform the discovery of the available iSCSI targets:

[root@localhost ~]# iscsiadm -m discovery -t st -p 192.168.122.1:3260

192.168.122.1:3260,1 ubuntu.mediate:storage.sys1

192.168.1.133:3260,1 ubuntu.mediate:storage.sys1

192.168.100.1:3260,1 ubuntu.mediate:storage.sys1

192.168.101.1:3260,1 ubuntu.mediate:storage.sys1

Next connect to the target. There two options you either specify the target (name, IP) and in that case the tooling will login to only this target or you leave it unspecified and you will be connected to all targets. In this manual I will use the first approach.

[root@localhost ~]# iscsiadm -m node -l -T ubuntu.mediate:storage.sys1 -p 192.168.122.1:3260

Logging in to [iface: default, target: ubuntu.mediate:storage.sys1, portal: 192.168.122.1,3260]

Login to [iface: default, target: ubuntu.mediate:storage.sys1, portal: 192.168.122.1,3260] successful.

Ok, now let's get to the multipath configuration. First of all copy the example configuration file to the /etc directory and restart the multipathd daemon:

[root@localhost ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.synthetic /etc/ 

[root@localhost ~]# multipath -v2

Oct 15 11:01:38 | /lib/udev/scsi_id exitted with 1

Oct 15 11:01:38 | /lib/udev/scsi_id exitted with 1

[root@localhost ~]# multipath -ll

149455400000000006c756e31000000000000000000000000 dm-3 IET,VIRTUAL-DISK

size=1.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:1 sdb 8:16  active ready  running

149455400000000006c756e30000000000000000000000000 dm-2 IET,VIRTUAL-DISK

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:0 sda 8:0   active ready  running

[root@localhost ~]# ls -l /dev/mapper/total 0

lrwxrwxrwx 1 root root      7 Oct 15 11:01 149455400000000006c756e30000000000000000000000000 -> ../dm-2

lrwxrwxrwx 1 root root      7 Oct 15 11:01 149455400000000006c756e31000000000000000000000000 -> ../dm-3

crw-rw---- 1 root root 10, 58 Oct 15 09:52 control

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_root -> ../dm-0

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_swap -> ../dm-1

Ok, now let us configure the multipath daemon so that the device is always available as an alias (e.g. ha-mediate under /dev/mapper/ha-mediate). Add the following section to the /etc/multipath.conf file and restart the multipathd daemon:

[root@localhost ~]# cat /etc/multipath.conf
##
## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
##
multipaths {
    multipath {
        wwid            149455400000000006c756e30000000000000000000000000
        alias            ha-mediate
        path_grouping_policy    multibus
        path_selector        "round-robin 0"
        failback        manual
        rr_weight        priorities
        no_path_retry        5
        rr_min_io        100
    }
}
[root@localhost ~]# service multipathd restart

Stopping multipathd daemon:                                [  OK  ]

Starting multipathd daemon:                                [  OK  ]

[root@localhost ~]# multipath -ll

149455400000000006c756e31000000000000000000000000 dm-3 IET,VIRTUAL-DISK

size=1.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:1 sdb 8:16  active ready  running

ha-mediate (149455400000000006c756e30000000000000000000000000) dm-2 IET,VIRTUAL-DISK

size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  `- 6:0:0:0 sda 8:0   active ready  running

[root@localhost ~]# ls -l /dev/mapper/

total 0

lrwxrwxrwx 1 root root      7 Oct 15 11:05 149455400000000006c756e31000000000000000000000000 -> ../dm-3

crw-rw---- 1 root root 10, 58 Oct 15 09:52 control

lrwxrwxrwx 1 root root      7 Oct 15 11:05 ha-mediate -> ../dm-2

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_root -> ../dm-0

lrwxrwxrwx 1 root root      7 Oct 15 09:52 vg_centos6hosta-lv_swap -> ../dm-1

As you see now the device is available under it's alias - in my case ha-mediate. So that's it - now you can create a filesystem on a device, mount it and start using it ;)

If you mount the iSCSI target only from one server you can create a cluster unaware filesystem like etx3, ext4 (example below). However for granting access from multiple servers a cluster-aware filesystem has to be created.

[root@localhost ~]# mkfs.ext4 /dev/mapper/ha-mediate 

mke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

655360 inodes, 2621440 blocks

131072 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=2684354560

80 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks: 

    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                            

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

[root@localhost ~]# mkdir /mnt/tmp

[root@localhost ~]# mount /dev/mapper/ha-mediate /mnt/tmp/

[root@localhost ~]# ls -l /mnt/tmp/

total 16

drwx------ 2 root root 16384 Oct 15 11:06 lost+found

Linux: cronjob redundancy and failover with rcron & keepalived)

2011-10-12T15:00:00.000-07:00

The solution described in this article is about introducing additional HA features like redundancy and failover to the periodic jobs executed via cron. It means that a periodic job is spread over multiple nodes but only one is executing it. In case of failure of this node another one can take over this task and execute it.
Actually what is interesting this kind of functionality might be provided by the by the combination of the keepalived and rcron (deployment figure presented below). How will it work? The rcron is an enhanced version of cron that first checks the state of the node in a file and only if the node is the master node the job is executed. The state of the node is on the other hand controlled by the keepalived daemon namely only the master node has the active state for the rcron job. In case of master node failure the active state is transferred to the backup node with the highest priority.

Figure 1 Failover and reduncancy using rcronjob and keepalived

That was theory let's check it in practice.

Getting the rcron binary

The rcron is not available in the standard CentOS repo and has to be built from the source. The best way is to install subversion and checkout the latest source tree from the googlecode. Rcron in order to compile requires two additional packages: byacc and flex. You can download the packages from the standard repository. Below one can find the commands to be used to install dependencies and build rcron from source code:

# yum install subversion byacc flex
# svn co http://rcron.googlecode.com/svn/trunk rcron
# cd rcron/
# ./configure
# make

# ./install-sh

# make install

Please repeat the operation on all nodes on which you would like to have the redundant cronjob configured.

Configure keepalived for redundancy and failover

Now as we have the rcron binary available let's get to the keepalived. For the initial configuration of the keepalived please check the previous articles on my blog. For the redundant cron job an additional vrrp instance is to be configured with a special handling of the state for rcron (/var/run/rcron/state file).

Host A's keepalived configuration file (/etc/keepalived/keepalived.conf) should be:

vrrp_instance CRON_1 {

state MASTER

interface eth0

virtual_router_id 31

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

notify_backup "/bin/echo passive > /var/run/rcron/state"

notify_master "/bin/echo active > /var/run/rcron/state"

notify_fault "/bin/echo passive > /var/run/rcron/state"

}

Host B's configuration file (/etc/keepalived/keepalived.conf) should be:

vrrp_instance CRON_1 {
state BACKUP
interface eth0
virtual_router_id 31
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state"
notify_master "/bin/echo active > /var/run/rcron/state"
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

After startup of keepalived daemon the state on:

HostA is active:
[root@centOS-hostA ~]# cat /var/run/rcron/state

active

HostB is passive:

[root@centOS-hostB ~]# cat /var/run/rcron/state
passive

Configuring rcron daemon

The configuration of the rcron is based on the examples.
The main configuration file for HostA:

[root@centOS-hostA log]# cat /etc/rcron/rcron.conf
# An arbitrary name
cluster_name = rcron_redundantjobs
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
default_state = active
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities (see nice(1)).
nice_level = 19

The main configuration file for HostB:

[root@centOS-hostB]$ cat /etc/rcron/rcron.conf
# An arbitrary name
cluster_name = rcron_redundantjobs
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
default_state = passive
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities (see nice(1)).
nice_level = 19

The cronjob configuration for both nodes - the rcron job shall be executed in rcron environment as follows (the job is only example, syntax as in cron, this job is configured to be executed every minute - in the output file we will check if the job has been executed or not):

[root@centOS-hostA log]# crontab -l
* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output
[root@centOS-hostA log]#

And that is it.

Limitations

Keepalived does not have any hook/notification that would allow to update the status file when the daemon is exiting. It means that there is an additional piece of software needed that would monitor the status of keepalived and update the rcron file if it shuts down. The perfect candidate for that purpose would be the upstart init daemon but unfortunately it is not part of CentOS 5.7 (as soon as I upgrade to CentOS 6 I will try it out)
RCron is usually not part of any standard repository and have to be compiled from the source. Also there is no support for it which makes it difficult for commercial purposes.

Conlusions

RCron together with keepalived could provide the redundancy and failover for the periodic jobs but there are certain limitation described above that have to be taken into account.

Ubuntu: encrypted private folder for user

2011-10-01T14:46:00.000-07:00

Have you ever wanted to protect your sensitive data on your Linux workstation so that nobody (even the root user) would be able to access it without providing the proper password? Recently I had this problem but fortunately it is not that difficult to achieve on Ubuntu box.
I hope that you find the instructions below useful as these were the commands that I used to create/mount and use an encrypted folder. First of all you need to install the ecryptfs-utils package - it is in official repos:

krychu@krystianek:~$ sudo apt-get install ecryptfs-utils 
[sudo] password for krychu: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
...

Next you need to setup the encrypted private folder as follows:

krychu@krystianek:~$ ecryptfs-setup-private 
Enter your login passphrase: 
Enter your mount passphrase [leave blank to generate one]: 
Enter your mount passphrase (again): 

************************************************************************
YOU SHOULD RECORD YOUR MOUNT PASSPHRASE AND STORE IT IN A SAFE LOCATION.
  ecryptfs-unwrap-passphrase ~/.ecryptfs/wrapped-passphrase
THIS WILL BE REQUIRED IF YOU NEED TO RECOVER YOUR DATA AT A LATER TIME.
************************************************************************

/sbin/restorecon
/sbin/restorecon

Done configuring.

Testing mount/write/umount/read...
Inserted auth tok with sig [cbb481a8ce68b47a] into the user session keyring
Inserted auth tok with sig [d38d63afc3777fe9] into the user session keyring
Inserted auth tok with sig [cbb481a8ce68b47a] into the user session keyring
Inserted auth tok with sig [d38d63afc3777fe9] into the user session keyring
Testing succeeded.

Logout, and log back in to begin using your encrypted directory.          

Now by default the encrypted private folder will be automatically mounted during next user logon (if you want to change that behavior) but there is also a possibility to explicitly mount/umount it - below you can find commands/instructions to achieve that:

krychu@krystianek:~$ ecryptfs-mount-private 
Enter your login passphrase:
Inserted auth tok with sig [cbb481a8ce68b47a] into the user session keyring
krychu@krystianek:~$ mount
...

/home/krychu/.Private on /home/krychu/Private type ecryptfs (ecryptfs_check_dev_ruid,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs,ecryptfs_sig=cbb481a8ce68b47a,ecryptfs_fnek_sig=d38d63afc3777fe9)
krychu@krystianek:~$ ecryptfs-umount-private 

Disable auto-mount for encrypted private folder

The auto mounting of the encrypted Private folder is controlled by the presence of the empty auto-mount file in the $HOME/.ecryptfs/ directory. If you would like to remove it then just follow the instructions (remove it from this folder):

krychu@krystianek:~$ cd .ecryptfs/
krychu@krystianek:~/.ecryptfs$ ls
auto-mount auto-umount Private.mnt Private.sig wrapped-passphrase
krychu@krystianek:~/.ecryptfs$ rm auto-mount auto-umount
krychu@krystianek:~/.ecryptfs$

After the next login the encrypted Private folder will not be automatically mounted. The user will have to explicitly call the ecryptfs-mount-private command in order to mount it.

Linux: Configuring cluster VIP (Virtual IP) with keepalived: Part 3

2011-09-11T12:15:00.000-07:00

The first part of this article is focusing on the configuration of keepalived, the second part is focusing on two test scenarios - node failure and recovery. In this part I would like to discuss the option that was added recently to keepalived namely monitoring the network interface status (meaning if the VIP interface is down there should a failover happen) and show which enhancements are required in order to enable it.
Let me first remind you the environment configuration- Figure 1.

Figure 1 Network deployment of the test environment for
the keepalived's VIP failover

Case C - VIP network interface status monitoring - interface down

What we would like to have is the failover to the BACKUP keepalived node in case something happens to the network interface to which the VIP address is assigned. This feature is supported by the keepalived 1.2.2 but required an enhancement of the configuration file:

[root@centOS-hostA ~]# cat /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    track_interface {
        eth1
    }
    virtual_ipaddress {
        192.168.122.50/24 brd 192.168.122.255 dev eth1 label eth1:0
    }
}

and the same for the BACKUP node (HostB):

[root@centOS-hostB ~]# cat /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 101
    track_interface {
        eth1
    }
    virtual_ipaddress {
        192.168.122.50/24 brd 192.168.122.255 dev eth1 label eth1:0
    }
}

Where the eth1 is the interface to which the VIP address is assigned to, eth0 stands for the interconnect.
Test methodology:

We constantly ping the VIP address from the client station
We simulate a network interface error (we shut it down)
We check ping output if there was a smooth failover
We check the ARP table which MAC is used
We sniff the interface to get the information what caused the client's machine ARP table update
Check the HostA's log files

Let's first do a snapshot of the client's station before issuing any additional steps - the entry in the ARP maps the VIP address to the HostA's MAC:

krychu@krystianek:~$ arp -na
...

? (192.168.122.50) at 52:54:00:03:ba:f2 [ether] on virbr0
...

krychu@krystianek:~$

Now the results. After issuing the first two steps (the interface failure was simulated by ifconfig eth1 down) the keepalived on hostA has changed its status to FAILED and as expected the VIP address has been failed over. However this time one can notice that some ICMP echo replies are lost:

krychu@krystianek:~$ ping 192.168.122.50
PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.
...

bytes from 192.168.122.50: icmp_req=5 ttl=64 time=0.411 ms
bytes from 192.168.122.50: icmp_req=6 ttl=64 time=0.376 ms
bytes from 192.168.122.50: icmp_req=7 ttl=64 time=0.387 ms
bytes from 192.168.122.50: icmp_req=8 ttl=64 time=0.411 ms
bytes from 192.168.122.50: icmp_req=13 ttl=64 time=0.532 ms
bytes from 192.168.122.50: icmp_req=14 ttl=64 time=0.324 ms
bytes from 192.168.122.50: icmp_req=15 ttl=64 time=0.444 ms
... 

^C
--- 192.168.122.50 ping statistics ---
22 packets transmitted, 18 received, 18% packet loss, time 20997ms
rtt min/avg/max/mdev = 0.295/0.408/0.683/0.088 ms
krychu@krystianek:~$ arp -na
...

? (192.168.122.50) at 52:54:00:56:c3:f4 [ether] on virbr0
...

The information about transition can be found in the logs:
Sep 11 19:09:45 centOS-hostA Keepalived: Starting VRRP child process, pid=3411
Sep 11 19:09:45 centOS-hostA Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Sep 11 19:09:45 centOS-hostA Keepalived_vrrp: Configuration is using : 63057 Bytes
Sep 11 19:09:45 centOS-hostA Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Sep 11 19:09:45 centOS-hostA Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Sep 11 19:09:46 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 19:09:52 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: Kernel is reporting: interface eth1 DOWN
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state

Case D - VIP network interface status monitoring - interface up

Now let's simulate that the network interface on HostA is brought back (by executing the ifconfig eth1 up).

As one can see there is nothing to be noticed in the ICMP echo replies, also client's station's ARP table entry remained unchanged.

krychu@krystianek:~$ ping 192.168.122.50
PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.
64 bytes from 192.168.122.50: icmp_req=1 ttl=64 time=3.57 ms
64 bytes from 192.168.122.50: icmp_req=2 ttl=64 time=0.446 ms
64 bytes from 192.168.122.50: icmp_req=3 ttl=64 time=0.387 ms
64 bytes from 192.168.122.50: icmp_req=4 ttl=64 time=0.374 ms
64 bytes from 192.168.122.50: icmp_req=5 ttl=64 time=0.395 ms
64 bytes from 192.168.122.50: icmp_req=6 ttl=64 time=0.368 ms
64 bytes from 192.168.122.50: icmp_req=7 ttl=64 time=0.571 ms
64 bytes from 192.168.122.50: icmp_req=8 ttl=64 time=0.288 ms
64 bytes from 192.168.122.50: icmp_req=9 ttl=64 time=0.380 ms
64 bytes from 192.168.122.50: icmp_req=10 ttl=64 time=0.371 ms
64 bytes from 192.168.122.50: icmp_req=11 ttl=64 time=0.353 ms
64 bytes from 192.168.122.50: icmp_req=12 ttl=64 time=0.359 ms
64 bytes from 192.168.122.50: icmp_req=13 ttl=64 time=0.387 ms
64 bytes from 192.168.122.50: icmp_req=14 ttl=64 time=0.367 ms
64 bytes from 192.168.122.50: icmp_req=15 ttl=64 time=0.366 ms
^C
--- 192.168.122.50 ping statistics ---
15 packets transmitted, 15 received, 0% packet loss, time 14000ms
rtt min/avg/max/mdev = 0.288/0.599/3.579/0.798 ms
krychu@krystianek:~$ arp -na
... 

? (192.168.122.50) at 52:54:00:56:c3:f4 [ether] on virbr0
... 

krychu@krystianek:~$

So what has happened? The answer can be found in the logs:

Sep 11 19:09:46 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 19:09:52 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: Kernel is reporting: interface eth1 DOWN
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state
Sep 11 21:00:50 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

The HostA's keepalived has changed the state from FAULTY to BACKUP and HostB remained the MASTER. It loks strange and not consistent with the node failure/recovery transitions but let's try to shutdown the HostB now and see if HostA will transistion into MASTER.

Sep 11 19:09:46 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 11 19:09:47 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 19:09:52 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: Kernel is reporting: interface eth1 DOWN
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 11 20:55:37 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state
Sep 11 21:00:50 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 11 21:06:15 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 11 21:06:16 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 11 21:06:16 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 11 21:06:16 centOS-hostA Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

As expected - after shutdown of HostB, HostA took over the role of MASTER and send the gratuitous ARP to update client station's ARP table. So the HA is provided but it still looks a bit inconsistent compared to the other scenarios

Summary

Although I did not check all scenarios related to the network problems - it seems that keepalived provides good support for monitoring the network interface. In the doc/samples/(...)track_interface file one can also find examples how to monitor multiple network interfaces.

Linux: Configuring cluster VIP (Virtual IP) with keepalived: Part 2

2011-09-11T03:16:00.000-07:00

The first part of this article is focusing on the instructions/guide how to setup keepalived with VIP address for a two node configuration. I have enhanced the picture from the previous article with the ARP configuration since the ARPs will be used throughout this part to describe the mechanisms for VIP failover.

Figure 1 Deployment diagram of the keepalived for a two node configuration

In this part there will be two test scenarios described:

The master goes down (Host A)
The master starts up again (Host A)

Other cases involving Backup node (Host B) are not relevant to be described in detail since in these cases there is no VIP failover done.

Case A The master goes down (Host A)

Let's first check how the envrionment looks like from the client station perspective. Both nodes (HostA and HostB) are up and running with the keepalived as described in part 1 or this article. The client station is able to ping the VIP address:

krychu@krystianek:~$ ping 192.168.122.50

PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.

64 bytes from 192.168.122.50: icmp_req=1 ttl=64 time=0.394 ms

64 bytes from 192.168.122.50: icmp_req=2 ttl=64 time=0.353 ms

64 bytes from 192.168.122.50: icmp_req=3 ttl=64 time=0.270 ms

64 bytes from 192.168.122.50: icmp_req=4 ttl=64 time=0.814 ms

--- 192.168.122.50 ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 2998ms

rtt min/avg/max/mdev = 0.270/0.457/0.814/0.212 ms

By looking into the arp table:

krychu@krystianek:~$ arp -an

...

? (192.168.122.50) at 52:54:00:03:ba:f2 [ether] on virbr0? (192.168.122.179) at on virbr0

...

we see that the Host A is the one available under the VIP (192.168.122.50) address, which is as we expected it to be since this is the master.

Now let's simulate a crash of the master node as follows:

We constantly ping the VIP address from the client station
We simulate a crash of the keepalived process on the master node (Host A), e.g. kill -9
We check ping output if there was a smooth failover
We check the ARP table which MAC is used
We sniff the interface to get the information what caused the client's machine ARP table update

Ok, so what are the results? First of all the node crash does have effect on ICMP packets being dropped - there is a small increase in the response time to about 20.1 ms (please take a look below).

krychu@krystianek:~$ ping 192.168.122.50

PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.

...

64 bytes from 192.168.122.50: icmp_req=37 ttl=64 time=0.336 ms

64 bytes from 192.168.122.50: icmp_req=38 ttl=64 time=0.398 ms

64 bytes from 192.168.122.50: icmp_req=39 ttl=64 time=0.273 ms

64 bytes from 192.168.122.50: icmp_req=40 ttl=64 time=0.298 ms

64 bytes from 192.168.122.50: icmp_req=41 ttl=64 time=20.1 ms

64 bytes from 192.168.122.50: icmp_req=42 ttl=64 time=0.320 ms

64 bytes from 192.168.122.50: icmp_req=43 ttl=64 time=0.379 ms

64 bytes from 192.168.122.50: icmp_req=44 ttl=64 time=0.314 ms

...

--- 192.168.122.50 ping statistics ---

67 packets transmitted, 67 received, 0% packet loss, time 65997ms

rtt min/avg/max/mdev = 0.198/0.742/20.167/2.472 ms

The proof that the failover has happened can be found in the ARP table - which now has the assignment for the VIP address to the HostB's MAC (previousle Backup keepalived server).

krychu@krystianek:~$ arp -an

...

? (192.168.122.50) at 52:54:00:56:c3:f4 [ether] on virbr0

...

In the wireshark we see that the backup node (HostB) has sent the Gratuitous ARP message to a broadcast MAC announcing that he is the one owning now the VIP address. Afterwards the client station has updated it's ARP cache table.

If we check the keepalived logs on the HostB one can see that after the crash of the HostA has been detected the keepalived is transitioning from the BACKUP to MASTER state:

Sep 11 09:50:49 centOS-hostB Keepalived_vrrp: Using LinkWatch kernel netlink reflector...

Sep 11 09:50:49 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Sep 11 09:50:49 centOS-hostB Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]

Sep 11 10:55:04 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep 11 10:55:05 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Sep 11 10:55:05 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Sep 11 10:55:05 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Sep 11 10:55:10 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Sep 11 10:55:45 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Sep 11 10:55:45 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Sep 11 10:55:45 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

Sep 11 10:58:35 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep 11 10:58:36 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Sep 11 10:58:36 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Sep 11 10:58:36 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Sep 11 10:58:41 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Sep 11 11:32:40 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Sep 11 11:32:40 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Sep 11 11:32:40 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

Sep 11 11:39:54 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep 11 11:39:55 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Sep 11 11:39:55 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Sep 11 11:39:55 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Sep 11 11:40:00 centOS-hostB Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.122.50

Case B The master is back (Host A)

Now let's focus on the situation when the Host A is brought back after a crash. Just as a reminder the situation is that after a crash HostB took over the Master role, owns the VIP address and the client station has an association to HostB's MAC in its ARP table.

The methodology is similar as in Case A:

We constantly ping the VIP address from the client station
We start the keepalived on the HostA
We check ping output if there was a smooth failover
We check the ARP table which MAC is used
We sniff the interface to get the information what caused the client's machine ARP table update

After issuing step 1 and 2 we see again an increase in the response time in the ICMP traffic but much smaller than in case of the crash.

krychu@krystianek:~$ ping 192.168.122.50

PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.

...

64 bytes from 192.168.122.50: icmp_req=23 ttl=64 time=0.191 ms

64 bytes from 192.168.122.50: icmp_req=24 ttl=64 time=0.346 ms

64 bytes from 192.168.122.50: icmp_req=25 ttl=64 time=0.331 ms

64 bytes from 192.168.122.50: icmp_req=26 ttl=64 time=0.291 ms

64 bytes from 192.168.122.50: icmp_req=27 ttl=64 time=2.53 ms

64 bytes from 192.168.122.50: icmp_req=28 ttl=64 time=0.315 ms

64 bytes from 192.168.122.50: icmp_req=29 ttl=64 time=0.338 ms

...

However it is still not a proof that a failover actually has happened. Let's take a look at the client station's ARP table:

krychu@krystianek:~$ arp -an

...

? (192.168.122.50) at 52:54:00:03:ba:f2 [ether] on virbr0

...

We see that the original mapping (presented in case A), which was pointing to HostB's MAC now points to HostA's. Which indicates the VIP address is now owned by the HostA again. In the wireshark we again see the Gratuitous ARP message to a broadcast MAC announcing sent by the HostA. On this basis the client station has updated it's ARP cache table.

In the logs on HostB one can find the information that after startup of HostA keepalived on HostB is transitioning to BACKUP state since it has received an announcement with a higher priority (review the configuration files from part 1)