BSD Based

Firewall for VMware vSphere virtual machines [10]

jos — Wed, 28 Apr 2010 22:02:31 GMT

We have been using VMware vSphere in an environment where VM admins are not to be trusted as they are only customers who purchase resources from us. It might happen that they use their virtual machine and the assigned public IPv4 address to spoof network packets, share illegal materials over various P2P networks or attack other VM guests locally.
VMware offers no solutions for the above problems, the assumption is that ESX hosts are supposed to be inside of a protected corporate environment. Networking is done via virtual switches which only implement little network security.

What we need to prevent:

MAC, IP spoofing
Illegal P2P traffic
Illegal SMTP traffic (spam relays, etc)

What we already have:

Forged transmit protection: VMware knows the MAC addresses associated to the network interfaces of the VM, so it can drop packets with forged source addresses. (Doesn’t protect against IP spoofing)

What we did:
We created a small virtual machine where we installed OpenWrt, a small yet efficient Linux based firewall/router solution for embedded systems.

The VM parameters were:

CPU: 500mhz

RAM: 32MB

HDD: 64MB

Network interfaces: 2

The actual image only took 10MBs which already included every tool we needed.
Then we connected the interfaces between the VM and the outside network, and created a software bridge.

Benchmarking:
We ran tests on an ESX server (4.0) with a quad core E5420 Xeon CPU.
Initiating a P2P network traffic from the protected VM resulted in 100% CPU pull for the bridge VM, the network troughput couldn’t go any higher than 3.5MB/s. Increasing the CPU limit for the VM resulted in linear growth of network troughput. With 1Ghz we had 7MB/s, and without CPU limitation the CPU usage went up to 1.6Ghz maximizing our 100Mbit/s uplink. Turning on and off the firewall with many Layer2,3,7 rules didn’t seem to affect the throughput performance. At least we knew that the bottleneck was the CPU and the bridge.

The results were somewhat disappointing so we started to dig further.
Our first guess was that the 32bit Bridge VM didn’t use the HW Virtualization Technology in the CPU. To have VMware use the HW method we had to port openwrt to x86_64 CPU. The task was not easy and trivial but we managed to create an image. For our disappointment the results were nearly the same if not worse. Checking through pubications and papers we found that engineers at VMware made SW virtualization better than Intel-VT. URL
In almost every of their benchmarks the software method was faster. Funny thing for Intel.

Bottom line:
We implemented our ideas into the Bridging Firewall VM, changed the port groups of the VMs to be protected and now they are spam, warez and dos-attack free.

Do you need a production ready application-level firewall in your vSphere environment similar to this one? Contact Us!

FreeBSD binary package repository howto [9]

jos — Tue, 23 Mar 2010 22:10:41 GMT

For a long time we have been trying to come up with a way to have a binary package repository which is based on heterogeneous package sources. (pkg_add/ports)
After playing with “portupgrade” and “make package” which often failed due to various reasons, it became obvious this method wont be beneficial in long-term.
This howto explains how to create the exact package replica of a jail full of installed packages.

First create a jail and build/install packages. I have created a build jail only for this purpose.

Use the script posted below to backup currently installed packages and setup a package tree similar to “make package” used at ports build. (Modify the first line according to your needs)



PACKAGEDIR=/packages

mkdir -p $PACKAGEDIR/All

PACKAGELIST=`find /var/db/pkg -type d | awk -F/ {‘print $NF’} | tail -n +2`

for i in $PACKAGELIST; do
    cd $PACKAGEDIR/All
    if [ ! -r $i.tbz ]; then
    echo “Packaging $i”
    pkg_create -b $i
    else
    echo “Packaging $i skipped”
    fi

done

	for i in $PACKAGELIST; do
    cd $PACKAGEDIR/All
    ORIGIN=`grep “@comment ORIGIN” /var/db/pkg/$i/+CONTENTS | awk -F: {‘print $2’}`
    ORIGIN_CAT=`echo $ORIGIN | awk -F/ {‘print $1’}`
    ORIGIN_PORTNAME=`echo $ORIGIN | awk -F/ {‘print $2’}`

    echo “Linking $ORIGIN_CAT/$ORIGIN_PORTNAME”

    [ -f $PACKAGEDIR/$ORIGIN_CAT/$i.tbz ] && rm $PACKAGEDIR/$ORIGIN_CAT/$i.tbz
    [ -f $PACKAGEDIR/Latest/$ORIGIN_PORTNAME.tbz ] && rm $PACKAGEDIR/Latest/$ORIGIN_PORTNAME.tbz
    FINDNAME=`echo $i |  sed ‘s/\(.\)\-./\1/’`
    OLD_TARGETS=`ls 1 | grep -e “^$FINDNAME[-]*\.tbz” | grep -v $i`

    [ ! -d $PACKAGEDIR/$ORIGIN_CAT ] && mkdir -p $PACKAGEDIR/$ORIGIN_CAT
    [ ! -d $PACKAGEDIR/Latest ] && mkdir -p $PACKAGEDIR/Latest

    for x in $OLD_TARGETS; do
        TEMP_ORIGIN_NAME=`tar -xjOf $x “+CONTENTS” | grep “@comment ORIGIN” | awk -F: {‘print $2’} | awk -F/ {‘print $2’}`
        if [ $ORIGIN_PORTNAME == $TEMP_ORIGIN_NAME ]; then
            echo “Deleting unneeded target: $x”
            [ -r $PACKAGEDIR/$ORIGIN_CAT/$x ] && rm $PACKAGEDIR/$ORIGIN_CAT/$x;
            [ -r $PACKAGEDIR/All/$x ] && rm $PACKAGEDIR/All/$x;
        fi
    done

    cd $PACKAGEDIR/$ORIGIN_CAT && ln -s ../All/$i.tbz $i.tbz
    cd $PACKAGEDIR/Latest && ln -s ../All/$i.tbz $ORIGIN_PORTNAME.tbz
done

Download

Explanation:
The first loop backs up all the installed packages on the system to $PACKAGEDIR/All. Then by going through the packages again the script recreates the symlinks used in FreeBSD packaging. It also checks for package files with different versions (but same origin) and deletes those which are not installed.
Example #1:
A common roll-back scenario

mysql-server-5.1.43.tbz [installed]
mysql-server-5.1.45.tbz
To be deleted: mysql-server-5.1.45.tbz

Example #2:
Packages with different versions that can co-exist on the same system

ruby-1.8.7.248,1 [installed]
ruby-1.9.1.376_1,1 [installed]
To be deleted: none

The script checks for package origins and by determining that lang/ruby18 is different from lang/ruby19 both packages remain untouched and included to the repository.

Known bugs: Does not check for deleted packages. It’s better to recreate the whole repository after a big portupgrade anyway.

Put the files onto a webserver using the following directory scheme:
pub/FreeBSD/ports/<arch>/packages-<version>-release
Example:
pub/FreeBSD/ports/i386/packages-7.2-release/Latest/bash.tbz

Use this as a source with pkg_add (or pkg_replace):
export PACKAGEROOT=http://example.com
Using pkg_add -r bash will fetch from the following URL:
http://example.com/pub/FreeBSD/ports/i386/packages-7.2-release/Latest/bash.tbz

FreeBSD 7 jail source address selection [97]

jos — Fri, 05 Mar 2010 10:35:04 GMT

I was browsing through the changesets of the upcoming 7.3 release and stumbled upon a nice addition which would have come handy a few months ago.

According to the default behaviour, when jails are trying to reach a host through one of the external interfaces, the source address used for communication is taken from the interface which has route to the destination.
In other words, if you have a single external IP address, and a couple of jails on the local interface, the source address for outbound connections will be the address on the external interface.
This is especially annoying if you’re trying to deploy per jail firewall rules for outgoing connections: you can’t use the jail ip address as source address for matching packets.
Until now there we had three different approaches to overcome this problem:

Multiple routing tables
The idea is that every process in a certain jail should use a routing table which has no default route, hereby restricting every outgoing connection. Note that incoming connections are assigned to the default (0) routing table, so you would still be able to access your services from the outside.

Add the following kernel configuration option and rebuild the kernel. The 2 is the number of FIB (Forward Information Base, synonym for a routing table here). The maximum value is 16.

options ROUTETABLES=2

This number can be modified on boot time. To do so, add the following to /boot/loader.conf and reboot the system:

net.fibs=6

Set a loader tunable net.my_fibnum if needed. This means the default number of routing tables. If not specified, 0 will be used. Set a loader tunable net.add_addr_allfibs if needed. This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.

Then add jail_name_fib=n to your rc.conf or ezjail configuration to assign the jail process to a certain routing table.

Jail match support
Find a firewall with jail match support. As we didn’t want to use anything else but pf, we looked into the source and found that an actual support for matching jails could be added easily. We even got to the stage where we had beta patches, but due to the fact that we needed a stable solution in our production environment we dropped the idea. Not to mention that it would have never been merged into pf, as OpenBSD has no jails and FreeBSD has no pf fork. (They treat pf as a port, so feature additions have to go through OpenBSD first. Uh good luck with that.)

Uid match support
Luckily pf has uid/gid match support, and by having all our jails use globally unique uids it became obvious that this method was the best so far.
pass out on $ext_if inet proto tcp from ($ext_if) to any port 25 user > 1000 keep state
The above pf rule blocks outgoing smtp access for each process that has a UID larger than 1000.

Disabling source address selection for jails
And finally the new addition of FreeBSD 7.3 allows you to disable the default source address selection method, always assigning the primary address of the jail to each outgoing packet. This can be done via the following sysctl flags:
security.jail.ip4_saddrsel=0 for IPv4
security.jail.ip6_saddrsel=0 for IPv6
Note that this is global and not a per jail setting.
Changeset: r202924

ld-elf.so Local DoS Vulnerability or not [15]

jos — Mon, 22 Feb 2010 20:20:10 GMT

We have found an interesting feature in the FreeBSD run-time link editor (rtld), which links dynamic executables with their needed libraries at run time.

The ld-elf.so.1 utility itself is loaded by the kernel together with any dynamically-linked program that is to be executed. The kernel transfers control to the dynamic linker. After the dynamic linker has finished loading, relocating, and initializing the program and its required shared objects, it transfers control to the entry point of the program.

It also has an executable flag, so let’s try to execute it.

Results:
FreeBSD 6.3.x:
$ /libexec/ld-elf.so.1
bash: /libexec/ld-elf.so.1: cannot execute binary file
$

FreeBSD 7.x:
$ /libexec/ld-elf.so.1
(no return)

Turns out the ld-elf.so keeps loading itself over and over, maxing out a cpu core while doing so. I had to enforce a cputime limit in login.conf so funny users won’t be able to profit from their discovery.
A fix isn’t likely as it looks like this is just one of those things you shouldn’t do

Ruby/Quota [5]

alex — Mon, 07 Dec 2009 10:31:39 GMT

When looking to access quota under Ruby, we had to look for extensions and the first search revealed the Ruby/Quota project, however it was unmaintained. Someone sent in a patch for NetBSD and support for newer Linux versions, but that was not applied either. We sent an email to the owner, but till now no response received.

So we took the CVS, imported it into git (using git cvsimport), applied the patches, applied own patches, added a gemspec, extended the manual and uploaded it to Github. Tested under FreeBSD 7.2, Mac OS X 10.5.8 and Linux 2.6.24.

You can get it here.

FreeBSD 8 VIMAGE + epair howto [29]

jos — Sun, 06 Dec 2009 00:56:18 GMT

The following text is about to show you how to use the new feature of FreeBSD 8: VIMAGE in a multi-jail environment.

Compile VIMAGE support into your kernel
Add the “option VIMAGE” to your kernel config and make sure to remove the SCTP support. Lack of SCTP support is one of the reasons VIMAGE is still considered to be experimental.

If you don’t know how to build your own custom kernel image, follow the detailed instructions of the corresponding FreeBSD Handbook chapter .

Reboot with your new kernel

First let’s create a pair of epair interfaces then quickly start two VIMAGE jails. I’m using the same fs root to make it simple, but you should create your jails as you always do, you can even use ezjail to it. The only difference is the “vnet” jailparam which is passed as a command line argument to the jail binary.
If you use rc.conf you could try adding the “vnet” parameter to your jail__flags variable for automatic startup.

test# ifconfig epair create
epair0a
test# jail -c vnet name=tibi1 host.hostname=tibi1 path=/ persist
test# jls
   JID  IP Address      Hostname                      Path
     1  -               tibi1                         /
test# jail -c vnet name=tibi2 host.hostname=tibi2 path=/ persist
test# jls
   JID  IP Address      Hostname                      Path
     1  -               tibi1                         /
     2  -               tibi2                         /

So we have two instances and an epair device. Let’s see the interface list on the host.

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:04:0a
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0b

Both sides of the pair is in the host system. Put one end into one of your jails with the ifconfig vnet command and verify the results by running ifconfig inside your jail.

test# ifconfig epair0b vnet 1
test# jexec 1 ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0b

OK, we have a layer 2 connection. Let’s add some IPs and run a ping test

test# jexec 1 ifconfig epair0b 192.168.11.2
test# ifconfig epair0a 192.168.11.1
test# ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.576 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.081 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/0.328/0.576/0.247 ms

It works!

Let’s do the same with your other jail

test# ifconfig epair1b vnet 2
test# jexec 2 ifconfig epair1b 192.168.11.3

Oh wait, these are completely different set of epair interfaces, you can’t use the same IP subnet on them. In order to mash them together on the host side, you have to make a bridge.

test# ifconfig bridge create
bridge0
test# ifconfig bridge0 addm epair0a addm epair1a up
test#

The commands above will create a new bridge interface, and add the host side of both epair interfaces to the bridge.
You can see it with ifconfig as well:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:04:0a
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0a
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:4b:75:2d:2b:9b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 14183
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 14183

Let’s put the host IP we set for epair0a earlier on the bridge interface instead and bring UP the host side of epair1. (Note: If you assign an IP to an interface, its state should automatically change to UP)

test# ifconfig epair0a -alias
test# ifconfig bridge0 192.168.11.1
test# ifconfig epair1a up
test# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:4b:75:2d:2b:9b
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 14183
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 14183

Running ping tests from the second jail, you can now ping your host and your other jail(s) too.

test# jexec 2 ping 192.168.11.1
PING 192.168.11.1 (192.168.11.1): 56 data bytes
64 bytes from 192.168.11.1: icmp_seq=0 ttl=64 time=0.193 ms
^C
--- 192.168.11.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.193/0.193/0.193/0.000 ms
test# jexec 2 ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.410 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.089 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.089/0.249/0.410/0.160 ms

Remember, now that you have separate networking stacks for each of your jails, the choice of topology is yours.

OpenSSH chroot and shell ambiguity [10]

alex — Thu, 03 Dec 2009 07:30:17 GMT

OpenSSH will fail in a scenario where the server is configured with chroot and a shell used by a user is not available outside, just inside the chroot.

The reason behind this is that ssh checks whether the given shell is a file and is executable, but this check doesn’t takes the chroot path into account. This feature was introduced by this patch about a year ago.

We have filed a bugreport to OpenSSH.

FreeBSD LD_PRELOAD security fault [1]

alex — Tue, 01 Dec 2009 22:01:05 GMT

This issue was disclosed yesterday sharing a local root exploit for FreeBSD systems. This blog post investigates and explains the issue, also the links and comments are worth reading. There are two types of exploits out there, one requires a setuid root binary to work, the other will use setuid(2).

An initial fix has been committed into the CVS today.

PureFTPD virtual chroot [9]

alex — Mon, 30 Nov 2009 13:51:58 GMT

PureFTPD has a feature called “virtual chroot”, where it will mimic a chroot by its own means, but without using the chroot() system call.

An excerpt from the PureFTPD FAQ:

– The ‘virtual chroot’ implementation. With that feature, users can
follow all symbolic links, even when they don’t point inside the jail. This
is very handy to set up directories shared by multiple users. Binary
packages are compiled with virtual chroot by default.

To enable the virtual chroot feature when you are compiling the server, use
the —with-virtualchroot with ./configure . If you want a restricted chroot,
don’t include —with-virtualchroot.

Please note that the FTP server will never let people create new symbolic
links. Symbolic links have to be already there to be followed. Or if your
users can create symbolic links through Perl or PHP scripts, your hosting
platform is really badly configured. People can install any web file
browser, they don’t need FTP to look at your system files. Recompile PHP
without POSIX functions and run all Perl scripts chrooted.

This feature is turned on by default in the FreeBSD ports.

Jos sent in a comment to the ports maintainer of pureftpd noting the above problem and today they have made this selectable when compiling the package.

Freshports commit message and direct link to CVS.

Grow FreeBSD UFS filesystem on VmWare HDDs [22]

jos — Mon, 30 Nov 2009 06:31:56 GMT

In this artice I’m going to show you how to expand your UFS filesystem under FreeBSD that runs in a Vmware Virtual Machine.

FreeBSD uses slices instead of traditional PC partitions. To sum it up shortly, it means that your whole disk contains only a single traditional partition with the partition type ‘freebsd(165)’. Inside this partition you will have slices. In a typical FreeBSD installation you have seperate slices for /, /usr, /var, /tmp and swap. In most cases the last slice on the partition is the /usr, and hopefully this is the one we have to extend, because in this case the only thing needed is to add some space to the end of the drive and extend the last slice. Sounds easy? Don’t think so!

The main steps are:

Increase the actual (virtual) hdd size
Extend the partition to cover the whole disk
Extend the size of the last slice to cover the whole partition
Extend the actual UFS filesystem on the newly modified slice

All the details:
First, forget livecds like gparted-live, knoppix, other partition hacking tools because they will not work.

Stop your VM and Edit its settings

Increase HDD size and click OK

Boot into single user mode (select 4 in the FreeBSD boot menu)

check your new disk via dmesg, you will see the increased block number

but your partition is still at the previous size (fdisk -s)

Let’s change the partition size using fdisk -u

In my example my partition starts at 63 and has the size of 18860247 blocks
What you would normally do is to adjust the new size to totalblocks-start. In this case fdisk will complain that the ending block is not on cylinder boundry. Fdisk will fix the value you entered but if you want to do the math yourself here is how it should be done correctly:
cylinder_size = 16065 [blocks]
new size = trunc(totalblocks/cylinder_size)*cylinder_size-startblock
in my case
trunc(25165824/16065)*16065-63 = 25157727

Next step is to extend the slice size. Slices are edited with the bsdlabel utility. You might need to mount a couple of more partitions to access your editor.
EDITOR=<yourfav> bsdlabel -e will bring up the slice table in your favorite editor. Now you need to change two things.

First the line where it says do not edit: change it to your new partition size.
To get the new size of the last slice you need to do the following calculations:
new_size_of_last_slice=partition_size-offset_of_last_slice

The last step is to extend the FS itself on the new slice. Make sure you don’t have the slice in question mounted. Then run growfs /dev/. In my case it was /dev/da0s1f

To check your results run fsck, mount -a, df -h
If you’re happy, you can reboot into multiuser mode now