Since around versions 5.1.40 and 5.4.3 CMake structure has changed in MySQL and the former instructions on how to build it with SphinxSE (aka Sphinx storage engine) are outdated. Compiling has become easier and requires fewer steps, for which I will provide a tutorial below. But first check whether you installed these requirements:

1. Microsoft Visual Studio – I will use version 2008 Professional.
   For you to compare, my cl.exe gets found in C:/Program Files (x86)/Microsoft Visual Studio 9.0/VC/bin/amd64/cl.exe
2. Bison – mine is 2.1
   On the command line, bison —help should be successfull.
3. CMake
   Again, a call to cmake should not result in a “file not found” error but invoke cmake.
4. A text file editor.
   Notepad++ will do the job, Windows’ notepad not due to line breaks getting ignored.

Then we can actually compile MySQL. For the example’s sake I will use MySQL 5.4.3 and Sphinx 0.9.9-rc2:

1. Download and extract MySQL sources somewhere, say in E:\mysql-5.4.3-beta.
   Do the same with Sphinx Search sources.
2. Copy sphinx storage engine folder sphinx-0.9.9-rc2\mysqlse to mysql-*\storage\sphinx
   Yes, “mysqlse” gets renamed to “sphinx” in that process, so that the latter destination folder contains CMakeLists.txt.
3. Open mysql-*\storage\sphinx\CMakeLists.txt and
   1. Remove this line:

      ADD_LIBRARY(sphinx ha_sphinx.cc)

   2. In place of that, add these lines:

      INCLUDE("${PROJECT_SOURCE_DIR}/storage/mysql_storage_engine.cmake")
      MYSQL_STORAGE_ENGINE(SPHINX)

4. Launch the shell (hit Win+R and type in cmd.exe) and change into E:\mysql-*
5. This command will generate a win\configure.data.
   You could alter that line, but make sure  WITH_SPHINX_STORAGE_ENGINE is in the options, of course.

   win\configure WITH_PARTITION_STORAGE_ENGINE WITH_ARCHIVE_STORAGE_ENGINE WITH_BLACKHOLE_STORAGE_ENGINE WITH_SPHINX_STORAGE_ENGINE WITH_FEDERATED_STORAGE_ENGINE WITH_CSV_STORAGE_ENGINE WITH_INNOBASE_STORAGE_ENGINE __NT__ CYBOZU MYSQL_SERVER_SUFFIX=-Sphinx

6. By following command you will run CMake, which will create project files for Visual Studio.
   There are several win\build-*.bat batch files, so make sure you call the one which matches your Visual Studio version. For 2008 it is build-v9.bat, for a 64bit MySQL it is build-v9_x64.bat (you can change this in VS, too):

   win\build-v9_x64

7. Open ALL_BUILD.vcproj in Visual Studio (e.g., by double click).
8. Go to “build » configuration manager”, select “Release” in the upper left dropdown box and in the list below, check “build” of “ALL_BUILD”. See the screenshot for everything (Win32 can be x64):
   
9. Run “Build » Build Solution”, wait.
10. mysql-*\sql\Release will contain executables, the remaining files can be found as usual.

When at compiling I prefer to alter default settings of SphinxSE, e.g. increase max_matches or m_iMaxMatches as it is called in the source code. You can find the settings in mysql-*/storage/sphinx/ha_sphinx.cc.

See also:

• MySQL with Sphinx storage engine – for the link to my pre-compiled binaries
• MySQL native function implementation of FNV hash (and others) – for the patch to MySQL to integrate hash functions FNV1A_64, FNV1_64 and MURMURHASH2

When talking about Linux, most people really mean that-Linux-Kernel-with-userland-tools. And there are a lot of flavours of that as distrowatch shows. As being asked a lot what and why I use [it], here is the answer: Gentoo. And that’s why:

I don’t use Gentoo because it gives me some percentile of more speed – taking compile times into the math that tailoring won’t pay off.
I do use it, because:

• It runs on every systems architecture I use or am to use. And that’s some: x86, amd64, arm, ppc; on way too many different processors
• It can be tailored to utilize or optionally utilize various command-sets (MMX, 3Dnow, SSE, SSE4, …) so it is possible e.g. to run a copy of one machine’s hdd as other’s system. Even dsitributed cross-compiling is no problem.
• Like BSD’s portage, there is no searching for a suitable RPM, DEB, LZM, PET, PISI or whatever in the right combination for the arch, but with Ebuilds there are simply instructions on how the system shall obtain, compile and install the software run by a toolset called Portage.
  • Portage can install those packages as well.
  • You can instantly see what goes where and what patches are applied prior compilation.
  • Files don’t get overwritten.
  • Configuration files can be merged with new ones.
  • Ebuilds could contain even that non-tuned configure; make; make install run and you get software management for free.
  • For every package you could choose by what compiler it is created (gcc, icc to name two) and with what optimizations.
• Gentoo can handle those packages as well.
  • You can even create your own “binhosts“ and spare you compiling something more than once.
    Yes, that’s right: You don’t need to compile everything, you can use packages like in Fedora, Suse and friends.
  • ICC from RPM works.
• Unlike DEBs or RPMs you have the freedom to say with which options something has to be installed.
  Apache2 with threads? A compatible PHP for that? Without MySQL but SQLite and sablotron, XML and curl? You named it, Gentoo’s Portage will figure out the dependencies for you and you get your feature combination.
• Gentoo leaves you the choice whether you want Linux as kernel or the BSD suite, and software management is the same on both.
• Due to the mighty Ebuilds (and my sys-meta branch in OSSDL overlay) I can clone and install a new machine by three commands, and it will always be the same installation no matter on which system I issued them.
  • An EC2 instance is created within minutes with exactly the tools you run on other machines.
  • Restoring a failed server is quite easy: emerge configuration files, emerge the system (by sys-meta), emerge the last backup. Voila!

Of course, with custom scripts (hints are welcome!) some of these points would be possible on any other distribution, but I found it nowhere easier and fitting the distributions culture than in Gentoo or its younger sister, Sabayon.

G’s fascination is that it stays out of your way and does not limit you, does what you want the way you want it.

Last Thursday we’ve had a conference about innovativations in IT and systems design here in Hannover. And as its main organizer I held a rather long workshop and speech of about three hours about database (mis)use and which and for what purpose a certain DBMS or storage would fit. In case you missed it, here are the slides (in German). It is about:

• row-based vs. column-oriented vs. index-only DBMS and storages
• key/value-stores, where SQL futile and where not
• MySQL, MonetDB, BerkeleyDB, …
• Tokyo Cabinet
• Redis
• Memcached

Feedback was good, and most probably we will see and hear more about this. Next time I am going to add distributed systems to that and elaborate more on Valdemort, Hypertable (and MySQL cluster…) and friends;what they do wrong, what’s around in academia and my own breed, Project Orkan – distributed K/V storages and group communications in general.

And, I’d like to give out some tech-demos along with Gentoo and all necessary tools to compile in a virtual machine, on DVD.

If possible, I’d like the demos be able to compile under MacOS, too, therefore allow me to ask you to try out compiling and running this one: Download RDFimport. Please drop me a line if make demo does not work, and if so, what to accomplish for it.

• merged the Linux kernel tree from Linus Torvalds
• with the Orion sources from Marvell (i.e. patches for SheevaPlug and OpenRD)
• added the Gentoo patches (distribution independent)
• and those for USB displays and SDHC cards
• and eliminated some harmless, but annoying warnings on ARM.

As I found the configurations of the other kernels for the SheevaPlug on the net a bit unfortunate, you can download my latest kernel flavour here:
sheevaplug-linux-2.6.32-rc5-mark.

Please find the merged Linux kernel sources at git.ossdl.de, along with the various configurations I run them with.
But, do me a favour and fetch the actual linux kernel from git.kernel.org and only the differences from my site – bandwidth.

Binhost: If you run Gentoo you can use my binhost for the SheevaPlug. I recommend you pull portage configurations from git.ossdl.de, the “arm” (or “wais-sheeva”) branch and/or set this in your /etc/portage/make.conf and it will spare you long compile runs:

FEATURES="ccache parallel-fetch userfetch userpriv getbinpkg"
PORTAGE_BINHOST="http://binhost.ossdl.de/armv5tel-softfloat-linux-gnueabi/"

I run a Gentoo overlay for all systems. You might consider using it, too. It contains Nginx with patches for the ARM (and MIPS, btw), PHP-FPM and Cyrus-IMAPd.
See http://git.ossdl.de/?p=ossdl-overlay.git;a=summary.

In case you didn’t know, today is German Unification Day and no store has opened. Great.
And yes, I have backups. And spare parts, but no additional PSU.

This is how I installed Gentoo on the SheevaPlug:

First I’ve made some preparations to decrease overall installation time:

• Setup a cross-compiling suite for the SheevaPlug.
  You can compile the Linux kernel parallel to installing GNU/Gentoo.
• Get an USB drive or SD flash card.
  Although the Linux kernel will be written to flash we will install Gentoo on that external device.
  Of course you could install the latter on flash, too. Just stop following the instructions when it says “eix-sync” or “emerge –sync” and use /dev/mtdblock2 (jffs2 filesystem, though) where it says /dev/sda2.
  Applies analogously to SD card installation.
• Make sure you can network boot the SheevaPlug.
  You could boot the pre-installed Debian and even skip compiling a new kernel, though.

Step 0: boot an installation system

Either boot the pre-installed Linux or network boot the SheevaPlug. I have done the latter.

Make sure you have access to the internet and /dev/null is writable by all:

# if eth0 doesn't work, try eth1
ifconfig eth0 192.168.1.13 up
# check whether the nameserver is correct
nano /etc/resolv.conf
# provide the IP of your router
route add default gw 192.168.1.1
chmod a+rw /dev/null

Step 1: prepare the destination medium

x means press Enter, leave out what is behind the octothorpe (#):

fdisk /dev/sda
p
d
n 1 x +2GB   # /dev/sda1, SWAP
n 2 x +8GB   # /dev/sda2, ROOT
n 3 x +300MB # /dev/sda3, LOG
n 4 x x      # /dev/sda4, PORTAGE
t 1 85       # SWAP
a 2          # BOOT
w

mkswap /dev/sda1
swapon /dev/sda1

mkfs.ext3 -L ROOT /dev/sda2
mkfs.ext3 -L LOG /dev/sda3
mkfs.ext2 -j -N 2000000 -L PORTAGE /dev/sda4

Step 2: extract a Gentoo tarball

You can find Gentoo tarballs with a basic system (called stage 3) at the Gentoo mirrors’ page, descending into releases/arm/current-stage3/armv5tel-softfloat-linux-gnueabi/ e.g. here or there. Select a tarball, copy its link.

mount /dev/sda2 /mnt/gentoo
cd /mnt/gentoo
wget -O - http://...the-link.../stage3-armv5tel-20090726.tar.bz2 | tar -xjp

This will extract the tarball immediately, i.e. as it gets downloaded without storing it on the plug.

Step 3: chroot into Gentoo

Create folders for the remaining partitions (sda1 is swap, sda2 is root!) and insert their list into /etc/fstab:

mount -t proc none /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev
mount /dev/sda3 /mnt/gentoo/var/log
mkdir -p /mnt/gentoo/usr/portage
mount /dev/sda4 /mnt/gentoo/usr/portage
mkdir -p /mnt/gentoo/var/tmp
mount --bind /tmp /mnt/gentoo/var/tmp
mount --bind /tmp /mnt/gentoo/tmp

cp -L /etc/resolv.conf /mnt/gentoo/etc/

chroot /mnt/gentoo /bin/bash
env-update
source /etc/profile
export PS1="(chroot) $PS1"

nano /etc/fstab

You can see my fstab here.
After that we will select profile, the languages our system will run and chose suitable compile settings for the architecture:

cd /etc
ln -snf ../usr/portage/profiles/default/linux/arm/2008.0 /etc/make.profile
nano locale.gen

ln -s portage/make.conf
nano make.conf

Here you can download my locale.gen and make.conf. Just remember to remove feature distcc because we have not installed it yet.

Step 4: kernel and preparations for first boot

You should have already compiled an kernel by now. See this post for configuration file and how to do it. Of course, you could use the pre-installed kernel, but I have not tested whether that will work.
Compile and copy the Linux kernel as described to your SheevaPlug.

Unless you like the serial console, have SSH be run on system startup. Don’t forget to set a root password:

passwd
rc-update add sshd default
nano /etc/conf.d/local.start

nano -w /etc/inittab
# (Uncomment the line and change 9600 with 115200)
s0:12345:respawn:/sbin/agetty 115200 ttyS0 vt100
# (remove all conventional TTYs)

nano -w /etc/conf.d/hostname
cp /usr/share/zoneinfo/Europe/Berlin localtime

Step 5: fetch ebuilds and packages

If you installed Gentoo on the system flash and if you have no dedicated device for portage, skip this step. You will have to install a DHCP client by hand or configure static IP. Else:

emerge --oneshot eix
eix-update
eix -cI
emerge udhcp
nano /etc/conf.d/net
# ...
cd /etc/init.d
ln -s net.lo net.eth1

See my net file. Remember to replace 192.168.1.2 by your nameserver in case it didn’t already get provided by DHCP.
On some SheevaPlugs – even of the same revision – eth0 is eth1, therefore I make that additional softlink.

I tend to grab the portage overlays from my workstation. But you can also skip this:

mkdir -p /usr/local/overlays
cd /usr/local/overlays
rsync -azuv -e ssh 192.168.1.6://usr/local/overlays/* /usr/local/overlays/

Step 6: exit chroot environment and restart the SheevaPlug

exit
cd /
umount /mnt/gentoo/usr/portage
umount /mnt/gentoo/var/log
umount /mnt/gentoo/var/tmp
umount /mnt/gentoo/tmp
umount /mnt/gentoo/proc
umount /mnt/gentoo/dev
umount /mnt/gentoo
exit

Step 7: set boot parameters

You’re almost done. The last thing is to tell the SheevaPlug to boot Linux and the latter to use the correct root device. Therefore reset (or shutdown -r now) your plug and get into the u-boot shell:

setenv bootcmd usb start\;nand read 0x01000000 0x00100000 0x00400000\;setenv bootargs $(console) root=/dev/sda2 rootdelay=10\; bootm 0x01000000
saveenv
reset

Keep yor fingers crossed. Linux kernel should boot into GNU/Gentoo on /dev/sda2 (or your SD card, if you changed /dev/sda; or the vintage Debian if you changed it into /dev/mtdblock2).

Now you could install distcc for distributed cross-compiling and customize your Gentoo Linux!

First of all, please see a former blog of mine how to configure TFTPd32. Most noticeably how to do so and still leaving your DHCP configuration (e.g. that of your router’s) in place.

Copy uImage_kwplug (mirror copy) or any other kernel uImage to TFTPd’s /boot directory, as well as sheevaplug.initramfs (mirror copy).

Then,

1. Plug in the serial/USB converter cable.
2. Reset your SheevaPlug.
3. Run PuTTY, select COM5 at 115200 connection speed.
4. Hit any key when the countdown appears:

   USB 0: host mode
   PEX 0: interface detected no Link.
   Net:   egiga0 [PRIME], egiga1
   Hit any key to stop autoboot:  1

   (If it doesn’t, reset the SheevaPlug with cables attached.)

Now you’re in the Das U-Boot bootloader shell. For the fun of it we will update that bootloader by a version with SD-card support. Download it here (mirror) to TFTPd32’s root folder:

setenv ipaddr 192.168.1.13 (replace with the IP you want the plug to have)
setenv serverip 192.168.1.2 (replace with the TFTP server IP)
bubt u-boot.bin
reset

You can leave out the bubt command if you want skip updating.

After that procedure and a last reset enter this to restore settings for Linux:

setenv ipaddr 192.168.1.13
setenv serverip 192.168.1.2
setenv arcNumber 2097
setenv mainlineLinux yes
saveenv
reset

Now the network booting:

tftpboot 0x8000000 boot/sheevaplug.initramfs
tftpboot 0x2000000 boot/uImage_kwplug
setenv bootargs console=ttyS0,115200 root=/dev/ram rw
bootm 0x2000000 0x8000000

Linux kernel will load and you will find yourself at another shell. This time it will be the usual shell, with /dev folder, mkfs.ext2 and mount commands. Have fun experimenting!

Compiling on the SheevaPlug can take long but is not neccessary if you have already your desktop pc running (which will be referred to as “workstation”). Unfortunately it will most probably not being run by an ARM processor, therefore you have cross-compile for the plug.

In this post I describe how to install a recent GCC version for cross-compiling, how to compile the latest kernel for the plug and how to enable distributed cross-compiling. By the latter you invoke compiling commands on the SheevaPlug and they will be executed by another machine transparently.

My workstation runs Gentoo Linux and listens on 192.168.1.6; the SheevaPlug runs Gentoo, too. By command emerge packages are installed, therefore if you used Debian you have to replace that call by apt-get and find out appropriate package names.

Install crossdev and setup a cross-compiling suite for the armv5tel-softfloat-linux-gnueabi architecture, which powers the SheevaPlug:

emerge portage-utils crossdev dev-embedded/u-boot-tools
crossdev armv5tel-softfloat-linux-gnueabi

This will take long as the latest GCC, binutils and the such will be compiled from scratch for the SheevaPlug.

1. After having done that fetch kernel sources and review the default configuration for the plug shipped with it:

   emerge --oneshot >=sys-kernel/gentoo-sources-2.6.31
   # or >=sys-kernel/vanilla-sources-2.6.31
   cd /usr/src/linux
   make ARCH=arm kirkwood_defconfig menuconfig

   Starting from Linux 2.6.31 all patches for the SheevaPlug are included in the main tree, therefore you can virtually use every kernel flavour since that.

2. In a hurry? You can download my kernel configuration file here.
3. After configuration you will have to compile the kernel into an uImage:

   make -j4 ARCH=arm CROSS_COMPILE=armv5tel-softfloat-linux-gnueabi- uImage

4. If you enabled dynamic module loading, you will have to compile modules as well. I did it, so here are the commands:

   make -j4 ARCH=arm CROSS_COMPILE=armv5tel-softfloat-linux-gnueabi- modules
   make -j4 ARCH=arm CROSS_COMPILE=armv5tel-softfloat-linux-gnueabi- INSTALL_MOD_PATH=.. modules_install

   It is very important to set INSTALL_MOD_PATH, or modules will be installed at your workstation. We want them in a separate folder for they will be copied to where they belong – the SheevaPlug.

The kernel should be compiled by now. Therefore log into your SheevaPlug and copy kernel and modules to it:

scp 192.168.1.6://usr/src/linux-2.6.31-gentoo/arch/arm/boot/uImage /boot/uImage-2.6.31-gentoo-mark
rsync -azuv -e ssh 192.168.1.6://usr/src/lib/modules /lib/
cat /boot/uImage-2.6.31-gentoo-mark > /dev/mtdblock1

The latter command will write the new kernel into the plug’s flash. You could leave that command out and have the kernel loaded from SD card or external USB drive, but it supports more USB devices (such as hubs) than your plug’s booting system. Consequently loading it from flash means less trouble in future.

Reboot your plug, smile. Cross-compiling has not been that hard. For your convenience, here’s my compiled kernel and modules for download. It includes iptables and samba support.

Setting up distributed cross-compiling is even easier. Do this on your workstation:

emerge distcc
nano /etc/conf.d/distccd
# add allow=192.168.1.0/24
nano /etc/distcc/hosts
# add localhost, don't add the sheevaplug! no work shall be distributed to it
/etc/init.d/distccd start
rc-update add distccd default

That’s all for the workstation. We will make sure armv5tel-softfloat-linux-gnueabi-gcc and the such will be selected automatically.

1. Repeat the above step on the SheevaPlug:

   emerge distcc
   # last time we will compile something on the plug alone
   nano /etc/conf.d/distccd
   # same subnet, same allow string
   nano /etc/distcc/hosts
   # 192.168.1.6/4,lzo -- don't specify localhost here! everything shall be delegated
   /etc/init.d/distccd start
   rc-update add distccd default

   But wait, that’s not all or calls of gcc will be executed by the wrong compiler on your workstation.

2. We have redirect that calls by a wrapper script (copied from this manual):

   cd /usr/lib/distcc/bin
   nano /usr/lib/distcc/bin/arch-wrapper
       #!/bin/bash
       exec /usr/lib/distcc/bin/armv5tel-softfloat-linux-gnueabi-g${0:$[-2]} "$@"

3. Now come the symlinks:

   chmod a+x /usr/lib/distcc/bin/arch-wrapper
   ln -sf arch-wrapper c++
   ln -sf arch-wrapper cc
   ln -sf arch-wrapper gcc
   ln -sf arch-wrapper g++

Voila! Compiling will happen at your workstation, even if you invoked emerge on the plug.
Just make sure workstation and SheevaPlug use the same GCC version.

*Update: patch to support libatomic-ops for increased performance and no segfaults *

Last Friday, after about a week of order processing and five days in shipping, my SheevaPlug has arrived in Germany. It is revision 1.2 and came pre-installed with Kernel 2.6.30-rc5 and Debian – which surprised me as many sites made me expect Ubuntu.

Nevertheless, I have already installed Gentoo on it preserving that Debian as rescue system. But before having done that I have installed Nginx on the SheevaPlug. And this is how:

1. Boot the SheevaPlug as usual, and log in as root (default password is nosoup4u).
2. Better set the current date and time correctly:

   ntpdate 130.75.1.32

3. Mount /tmp into RAM. No need to download archives onto the flash and increase wear:

   nano /etc/fstab
   # shm /tmp tmpfs defaults,noatime,nosuid 0 0
   # ctrl+x, y
   mount /tmp

4. Check whether your nameservers are setup correctly. The pre-installed DHCP client didn’t set this for me, so I had to do that by hand:

   nano /etc/resolv.conf
   # nameserver 192.168.1.1 -- most probably your router's IP
   # ctrl+x, y

After these prerequisities you can finally compile Nginx:

1. Install the bare minimum for compiling Nginx:

   apt-get install \
       gcc openssl libssl-dev make automake libpcre3 libpcre3-dev \
       libatomic-ops-dev

2. Get and extract Nginx:

   cd /tmp
   wget http://sysoev.ru/nginx/nginx-0.8.20.tar.gz -O - \
   | tar -xz
   cd nginx*

3. Download the patch to support atomic operations on ARM: nginx-libatomic patch

   patch -p1 --input=nginx-libatomic.patch

4. Compile Nginx as follows for all paths and the users be set correctly:

   ./configure \
       --pid-path=/var/run/nginx.pid \
       --conf-path=/etc/nginx/nginx.conf \
       --lock-path=/dev/shm \
       --prefix=/usr --user=www-data --group=www-data \
       --http-client-body-temp-path=/tmp \
       --http-proxy-temp-path=/tmp \
       --http-fastcgi-temp-path=/tmp \
       --with-md5-asm --with-sha1-asm \
       --http-log-path=/var/log/nginx/access.log \
       --error-log-path=/var/log/nginx/error.log \
       --with-http_ssl_module --with-http_realip_module \
       --with-libatomic \
       && make \
       && make install

You can get an init script here. Remember to strip the /local from paths as nginx will be in /usr/sbin/nginx.

I have run ab -c 5 -n 5000 from another machine to see how many requests the SheevaPlug can serve. It is about 280 1350 requests per second, with an average serving time of 3.5ms0.8ms. That’s sufficiently enough for a personal home server.

latency compared

The problem with RewriteRules and redirects is, that although you save bandwidth because files will be served from CDN, your server will still have to face all requests.

And Every one will result in another request – this time to CDN. In terms of latency you add yours to CDNs and even lose keep-alive transfers. I.e., the browser won’t be able to combine several requests into a single HTTP connection.

The solution is to rewrite all the links at your blog or application so that they point directly to CDN omitting your server.