services:
    twig.extension.debug:
        class: Twig_Extensions_Extension_Debug
            tags:
                - { name: twig.extension }

Afterwards clear the cache and use the debug extension:

{% debug my_var %}

Sources

• http://groups.google.com/group/symfony2/browse_thread/thread/40df2ca0b22b0166

First of all, you need to install HTC Sync and the included USB drivers to get your device working normally with your computer (e.g. showing up as disk drive). Then install the Android SDK and the USB drivers package (from SDK manager) and add the following lines to the file extras\google\usb_driver\android_winusb.inf inside the [Google.NTamd64] section (before [USB_Install] around line 65):

;HTC Desire
%CompositeAdbInterface%     = USB_Install, USB\VID_0BB4&PID_0C87&MI_01

Use this driver for the “My HTC” device. It should install, show up as “Android Composite ADB Interface” afterwards and work with adb (do an adb devices to check).

Sources

• frenetic rapport: HTC Desire Windows 7/Vista 64bit ADB Driver

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel:( \[ *[[:digit:]]+\.[[:digit:]]+\])? Shorewall:net2([a-z0-9]*):(REJECT|DROP)

Test if the regex works for your config:

sed -e 's/[[:space:]]*$//' /var/log/syslog | egrep '^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel:( \[ *[[:digit:]]+\.[[:digit:]]+\])? Shorewall:net2([a-z0-9]*):(REJECT|DROP)'

virt-install --connect qemu:///system --arch=x86_64 -n win2k8 -r 1024 --vcpus=2 \
--disk pool=vmstore,size=50,bus=virtio,cache=none -c /path/to/win2k8.iso --vnc \
--noautoconsole --os-type windows --os-variant win2k8 --network network=subnet,model=e1000 \
--disk path=/path/to/virtio-win-1.1.16.iso,device=cdrom,perms=ro

When the guest is running, shut it down and edit the os section of the XML file to look like this (otherwise Windows setup won’t let you install on the disk):

virsh destroy win2k8
virsh edit win2k8

<os>
  ...
  <boot dev='cdrom'/>
  <boot dev='hd'/>
</os>

virsh start win2k8

Connect to the VNC console and start the installation process. When you reach the form to select a disk device you won’t see any devices available. Click on “Load drivers” at the bottom left and load the drivers from E:\viostor\wlh\{amd64|x86}. After the drivers are installed, you’ll see a disk device and can continue with the installation.

Sources

• Installing Win Server 2008 R2 as a KVM Guest with VirtIO
• Installing Windows 7 as a KVM virtualized guest on Debian 6.0 Squeeze

1. I don’t use the sieve part anymore. Instead, I use the ManageSieve server provided by Dovecot. It integrates with Dovecot’s authentication system and you can use all kinds of clients supporting the managesieve protocol, e.g. Thunderbird or Roundcube.
2. The ISPMail database structure changed since I wrote that tutorial which still relied on the DB structure for the Debian Etch structure. I updated the script to reflect both environments (take a look at the config file). However, it does not rely on any DB views anymore.
3. The script (only the Fetchmail part) is now hosted on GitHub as I rarely use SVN anymore and the SVN may go offline in the near future. If anybody is interested in the Sieve part, just drop me a line.

So, step one: in a virtual machine/spare server/local pc/whatever install a MySQL server and replace its data directory:

$ aptitude install mysql-server
$ /etc/init.d/mysql stop
$ mv /var/lib/mysql /var/lib/mysql.orig
$ cp -pr /tmp/backup/mysql /var/lib
$ chown -R mysql.mysql /var/lib/mysql

I also checked that file permissions match the normal permissions on Debian MySQL installations. Should be like this:

root@host:/var/lib/mysql# ls -al
drwx------  5 mysql mysql     4096 Mar  1 18:20 .
drwxr-xr-x 33 root  root      4096 Mar  1 18:20 ..
-rw-r--r--  1 root  root         0 Mar  1 18:07 debian-5.1.flag
-rw-rw----  1 mysql mysql 27262976 Mar  1 18:21 ibdata1
-rw-rw----  1 mysql mysql  5242880 Mar  1 18:21 ib_logfile0
-rw-rw----  1 mysql mysql  5242880 Mar  1 18:21 ib_logfile1
drwx------  2 mysql mysql     4096 Mar  1 18:20 database1
drwx------  2 mysql mysql     4096 Mar  1 18:21 database2
drwx------  2 mysql root      4096 Mar  1 18:08 mysql
-rw-------  1 root  root         6 Mar  1 18:08 mysql_upgrade_info

root@host:/var/lib/mysql# ls -al database1
drwx------ 2 mysql mysql 4096 Mar  1 18:20 .
drwx------ 5 mysql mysql 4096 Mar  1 18:20 ..
-rw-rw---- 1 mysql mysql   65 Mar  1 18:20 db.opt
-rw-rw---- 1 mysql mysql 8668 Mar  1 18:20 table1.frm
-rw-rw---- 1 mysql mysql  879 Mar  1 18:20 table2.frm
-rw-rw---- 1 mysql mysql 1520 Mar  1 18:20 table3.frm

Now you can try to start the server and look if your databases are readable:

$ /etc/init.d/mysql start
$ mysql -uroot -p -e "show databases;"
Enter password:
+--------------------+
| Database           |
+--------------------+
| information_schema |
| database1          |
| database1          |
| mysql              |
+--------------------+

Alright, if this works, simply dump your needed databases with mysqldump, transfer them to your server and import them normally.

$ mysqldump -uroot -p database1 > /tmp/database1.sql
$ scp /tmp/database1.sql user@server:/tmp

On the server:

$ mysql -uroot -p -e "create database database1;"
$ mysql -uroot -p database1 < /tmp/database1.sql

And don't forget to restore the temporary MySQL server to normal operation in case you need it later.

$ /etc/init.d/mysql stop
$ rm -rf /var/lib/mysql
$ mv /var/lib/mysql.orig /var/lib/mysql
$ /etc/init.d/mysql start

First of all, create the XML definition for the storage pool in /etc/libvirt/storage/vg1.xml. This is the minimal configuration needed, libvirt will extend it with things like UUID when you define it.

<pool type='logical'>
  <name>vg1</name>
  <target>
    <path>/dev/vg1</path>
  </target>
</pool>

Now you can tell libvirt about the new storage pool and let it start automatically.

$ virsh pool-define /etc/libvirt/storage/vg1.xml
$ virsh pool-start vg1
$ virsh pool-autostart vg1
$ virsh pool-info vg1

Creating virtual machines inside that storage pool is easy as pie:

$ virt-install -d --hvm --vnc --name=vm01 \
    --ram 512 --disk pool=vg1,size=10,bus=virtio,cache=none \
    --network network=default,model=virtio \
    --location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64/ \
    --os-type=linux --os-variant=debiansqueeze

Cheers!

// user node
$node = eZContentObjectTreeNode::fetch($nodeID);

// user object
$dataMap = $node->dataMap();

// change attribute
$firstname = $dataMap['firstname'];
$firstname->setAttribute('data_text', 'Foo');
$firstname->sync();

// user account attribute
$user = $dataMap['user_account'];
$userData = $user->content();

// user settings
$userSetting = eZUserSetting::fetch($user->ContentObjectID);

// get account status
$isEnabled = $userSetting->attribute('is_enabled');

// enable user account
$userSetting->setAttribute('is_enabled', 1);
$userSetting->store();

Or if you just want to access the user account object or the user setting directly:

// user account object
$user = eZUser::fetch($objectID);

// user settings
$userSetting = eZUserSetting::fetch($objectID);

#!/bin/bash
BN=paper
pdflatex -interaction=nonstopmode $BN.tex
bibtex $BN
pdflatex -interaction=nonstopmode $BN.tex
bibtex $BN
pdflatex -interaction=nonstopmode $BN.tex
makeindex -s $BN.ist -t $BN.glg -o $BN.gls $BN.glo
pdflatex -interaction=nonstopmode $BN.tex
pdflatex -interaction=nonstopmode $BN.tex
rm -rf $BN.aux
rm -rf $BN.lof
rm -rf $BN.lot
rm -rf $BN.out
rm -rf $BN.toc
rm -rf $BN.bbl
rm -rf $BN.blg
rm -rf $BN.brf
rm -rf $BN.idx
rm -rf $BN.glo
rm -rf $BN.ist
rm -rf $BN.glg
rm -rf $BN.gls
rm -rf texput.log

This is OK on my workstation, but running a build on my notebook using a small 1.4 GHz single core processor can take up to a minute which is definitely too long. So I looked for solutions how to move the build process to a central server. As I was already using Git for source control on the project, I tried setting up a remote repository on the server which triggered a build using a post-receive script. This basically worked fine, but I wanted to go a step further. I had a look at CI servers and gave Hudson a try as it seems to have a lot of features while being quite easy to set up.

The result is the following: Hudson is polling the Git repository (can be remote or local, in my case it’s a self-hosted remote gitosis installation, but could be github too), starting a new build on changes and publishing the resulting PDF if successful. Hudson is accessible over https using an Apache2 server as frontend to a Tomcat installation.

Ready? Let’s go.

Installing the toolset

First of all, install some basic tools (including git).

aptitude install nano curl wget git-core

We need Java to run Hudson, but Sun’s Java has moved to the partner repository which is not enabled by default. Add the repository and install the Java JDK:

echo "deb http://archive.canonical.com/ubuntu  lucid partner" >> /etc/apt/sources.list
aptitude update
aptitude install sun-java6-jdk

To build LaTeX papers, we need (of course) a TeX distribution. Texlive should provide you with all you need. I simply installed the full distribution as I didn’t want to hassle with missing packages. Be aware that texlive-full is quite heavy:

Need to get 1035MB of archives. After unpacking 1862MB will be used.

If you are not lazy and are limited on disk space, just install the texlive-* packages you need.

aptitude install texlive-full

Install the web server stack:

aptitude install apache2 tomcat6 libapache2-mod-jk libapache2-mod-gnutls ssl-cert

We’ll use ant build files to build our projects, so install ant:

aptitude install ant

Configure Apache and mod_jk

First of all, activate needed apache modules:

a2enmod rewrite jk gnutls

Enable name-based virtual hosts on SSL. Edit /etc/apache2/ports.conf and add a NameVirtualHost *:443 in the mod_gnutls section:

...
<IfModule mod_gnutls.c>
    NameVirtualHost *:443
    Listen 443
</IfModule>

Disable and remove the default apache config:

a2dissite default
rm /etc/apache2/sites-available/default
rm /etc/apache2/sites-available/default-ssl

Create a new virtual host file in /etc/apache2/sites-available/hudson. Mine looks like this (you’ll probably want to adjust the SSL certificates or omit the SSL part by removing the *:80 vhost, changing the *:443 vhost to *:80 and removing the gnutls lines):

<VirtualHost *:80>
        ServerName hudson.example.com

        RewriteEngine On
        RewriteRule ^(.*)$ https://hudson.example.com$1  [R,L]
</VirtualHost>
<VirtualHost *:443>
        ServerName hudson.example.com

        GnuTLSEnable on
        GnuTLSCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
        GnuTLSKeyFile         /etc/ssl/private/ssl-cert-snakeoil.key
        GnuTLSPriorities NORMAL

        DocumentRoot /var/lib/tomcat6/webapps/ROOT/
        <Directory "/var/lib/tomcat6/webapps/ROOT/">
                Options Indexes FollowSymlinks +Includes
                AllowOverride All
        </Directory>

        JkMount /* hudson

        ErrorLog /var/log/apache2/hudson.error.log
        CustomLog /var/log/apache2/hudson.access.log combined
        LogLevel error
</VirtualHost>

Create the file /etc/apache2/mods-available/jk.conf with the following content:

JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel info

Disable and re-enable the jk module to let apache use jk.conf

a2dismod jk
a2enmod jk

Create the file /etc/apache2/workers.properties:

workers.tomcat_home=/usr/share/tomcat6
workers.java_home=/usr/lib/jvm/java-6-sun
ps=/
worker.list=hudson

worker.hudson.port=8009
worker.hudson.host=localhost
worker.hudson.type=ajp13
worker.hudson.lbfactor=1

Enable the virtual host and restart apache. When you try to connect to your server through your browser you should get redirected to the SSL vhost. The site should say 503 Service Temporarily Unavailable as tomcat is not configured yet.

a2ensite hudson
/etc/init.d/apache2 restart

Configure Tomcat

Edit /etc/tomcat6/server.xml and enable the AJP connector on port 8009 by uncommenting the following line (somewhere around line 93):

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Bind the direct access to tomcat on port 8080 to the local interface. Edit line 71 and add the address attribute:

    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" address="127.0.0.1" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443" />

Restart tomcat and connect to your apache virtual host defined before. You should see tomcat’s “It works!” page.

Deploy hudson

As hudson is the only application running on this tomcat instance, I deployed it as root application.

/etc/init.d/tomcat6 stop
rm -rf /var/lib/tomcat6/webapps/ROOT
wget http://hudson-ci.org/latest/hudson.war -O /var/lib/tomcat6/webapps/ROOT.war

Hudson needs the env variable HUDSON_HOME to run. Edit /etc/tomcat6/context.xml and add the following line inside the Context element:

<Environment name="HUDSON_HOME" value="/var/lib/tomcat6/webapps/ROOT" type="java.lang.String" override="false" />

Now start tomcat and reload the page. You should see a running hudson instance.

Configure hudson

Go to Manage/Configuration and set all options you need (e.g. email settings). If you are running this on a public server, make sure you set some kind of hudson security (I use the internal hudson user database with projectmatrix security). In addition set the environment variable HOME with /usr/share/tomcat6 as value (this is needed for git to work properly).

Next, go to the plugin management site and install the plugins you need. I installed “Hudson GIT plugin” and “Green balls”. After installation restart hudson.

Hudson and Git

To make hudson work with git, we need to configure git for the tomcat user. First, create a .gitconfig file in /usr/share/tomcat6 (tomcat6′s homedir):

[user]
        name = Hudson
        email = hudson@example.com

If you want to use repositories only accessible via SSH, tomcat needs a SSH keypair. Create a keypair with an empty passphrase:

cd /usr/share/tomcat6
mkdir .ssh
ssh-keygen -t rsa -f .ssh/id_rsa
chown tomcat6.tomcat6 .ssh -R

Hudson git cloning will fail if the server you are connecting to is not in tomcat’s known_hosts file. For each server you want to connect to, clone a random repository. This will add the host and avoid later problems. Of course, tomcat has to have access to that server you are cloning from (you need set up the public key we created before on the server – e.g. add it to your public keys on github). For example:

cd /tmp
sudo -u tomcat6 git clone git@example.com:testrepo.git
# answer with yes
rm -rf testrepo

Create a new job

Hudson provides different methods to specify how to build a job – I’m using ant build files. My git repository layout is basically the following:

.gitignore
src/paper.tex

I created a sample project on github to test the build process. You can find it at http://github.com/maff/sampletex.

To reflect the build process listed above in the shell script, I created the following buildfile and saved it as /var/lib/tomcat6/webapps/ROOT/build-latex.xml. The file expects a parameter basename, which is the base name of the tex source to be compiled. In my case, it’s set to paper (see job configuration below). Of course, you’ll have to adapt the build file if you need a different build process

<?xml version="1.0" encoding="UTF-8"?>
<project name="latex" default="build" basedir=".">
  <property environment="env"/>
  <property name="workspace" value="${env.WORKSPACE}"/>

  <target name="build">
    <antcall target="clean" />
    <antcall target="cleanoutput" />
    <antcall target="compile" />
    <antcall target="clean" />
  </target>

  <target name="pdflatex">
    <exec executable="pdflatex" dir="${workspace}/src">
      <arg line="-interaction=nonstopmode ${basename}.tex" />
    </exec>
  </target>

  <target name="bibtex">
    <exec executable="bibtex" dir="${workspace}/src">
      <arg line="${basename}" />
    </exec>
  </target>

  <target name="makeindex">
    <exec executable="makeindex" dir="${workspace}/src">
      <arg line="-s ${basename}.ist -t ${basename}.glg -o ${basename}.gls ${basename}.glo" />
    </exec>
  </target>

  <target name="compile">
    <antcall target="pdflatex" />
    <antcall target="bibtex" />
    <antcall target="pdflatex" />
    <antcall target="bibtex" />
    <antcall target="pdflatex" />
    <antcall target="makeindex" />
    <antcall target="pdflatex" />
    <antcall target="pdflatex" />
  </target>

  <target name="clean">
    <delete>
      <fileset dir="${workspace}/src" includes="${basename}.aux, ${basename}.lof, ${basename}.lot, ${basename}.out, ${basename}.toc, ${basename}.bbl, ${basename}.blg, ${basename}.brf, ${basename}.idx, ${basename}.glo, ${basename}.ist, ${basename}.glg, ${basename}.gls, texput.log" />
    </delete>
  </target>

  <target name="cleanoutput">
    <delete>
      <fileset dir="${workspace}/src" includes="${basename}.pdf, ${basename}.log" />
    </delete>
  </target>
</project>

Switch to the web interface and create a new job. Choose “Free style” software project and click next. Using the default settings, I changed the following (labels may differ a bit as my interface is in german, but you’ll find what you need):

Source-Code-Management

Choose Git and set the repo-URL (can be local or remote, e.g. /tmp/project.git, git://example.com/project.git or git@example.com:project.git)

Build trigger

Select “poll SCM” and set the value to a valid cron-style line. * * * * * would poll the repository every minute and build the project if it finds changes.

Build process

Add an ant task, click on advanced and specify ../../../build-latex.xml as buildfile. Additionally enter basename=paper in the properties input (specifies the file basename mentioned before).

Post-build

I only chose “archive artifacts” and set the pattern to src/paper.pdf, src/paper.log. This depends on your environment and your needs.

Build!

Alright, time to test our setup. Select the newly created job and click build in the left menu. If all goes well, the build will succeed and you can access the built PDF directly from the web interface. Take a look at the job’s console output to identify potential errors. Happy building :)

Zutaten (reicht für ca. 16-20 kleine Quadrate)

• 125g Zartbitterschokolade (mindestens 70% Kakao)
• 100g Vollmilchschokolade
• 75g Weiße Schokolade
• 125g Butter
• 125g Zucker
• 125g Mehl
• 125ml Sahne
• 60g Kakaopulver
• 2 Eier
• 1 Schuss Rum
• Nach Belieben Nüsse o.Ä.

Backofen auf 200°C Ober-/Unterhitze vorheizen. Weiße und Vollmilchschokolade hacken oder fein raspeln. Zartbitterschokolade über dem Wasserbad schmelzen, dann die Butter hinzufügen. Eier mit dem Zucker schaumig schlagen, dann langsam Mehl und Kakao einrühren. Sahne und geschmolzene Schokolade hinzufügen und zu einer gleichmäßigen Masse verarbeiten. Zum Schluss die geraspelten Schokoladestückchen untermischen. Nach Belieben mit Rum oder Nüssen (z.B. Pinienkerne oder Cashews) verfeinern.

Eine kleine Backform ausfetten und die Masse ca. 3 cm hoch in die Form füllen. Ab in den Ofen und ca. 20-25 min backen, bis sich oben eine knusprige Schicht gebildet hat, die Brownies innen aber noch schön feucht sind (zum Testen einfach mit einem trockenen Zahnstocher hineinstechen – es sollte noch etwas daran kleben bleiben). Aus dem Ofen nehmen, aufschneiden und lauwarm oder ausgekühlt servieren (z.B. mit Vanilleeis, Minze und Sauerkirschen).

#!/bin/sh
while read f
do
  echo $f
  echo ""
  curl -IL --silent $f
  echo "------------------"
  echo ""
done < $1

Usage:

$ ./checklinks.sh links.txt > stats.txt

Mangochutney

Zutaten

• 1 Mango (500 – 600g)
• 2-3 Schalotten oder ½ Zwiebel
• 1 rote Paprika
• 1 Limette
• Chili
• Salz
• Zucker
• Safran
• Paprikapulver
• 1-2 Knoblauchzehen
• schwarzer Pfeffer, grob gemahlen
• etwas Grün von einer Frühlingszwiebel
• Butter

Butter in einem Topf zergehen lassen. Schalotten/Zwiebel und Paprika sehr fein schneiden und langsam in der Butter weich dünsten. Mit etwas Zucker bestäuben und leicht ankaramellisieren lassen. Mango schälen, vom Stein befreien und in kleine Würfel schneiden. Die Knoblauchzehen schälen, anquetschen, mit der Mango in den Topf geben und langsam andünsten. Abrieb und Saft von der Limette hinzugeben und mit ca. ½ Liter Wasser aufgießen. Paprikapulver und ½ TL Safranfäden hinzufügen und mit Salz, Pfeffer, Zucker und Chili abschmecken. Langsam für 30-45 Minuten köcheln lassen, damit das Ganze eindickt und die Mangos weich werden. Kurz vor Ende das feingeschnittene Grün einer Frühlingszwiebel hinzufügen. Mit einer Gabel die Mangostücke leicht zerquetschen.

Chutney durch ein feines Sieb drücken und den Saft aufbehalten.

Chicken Curry

Zutaten

• ~500g Hühnerfilet
• 1 rote Paprika
• 2 Frühlingszwiebeln
• 2 Karotten
• 2 Lorbeerblätter
• ein paar Zuckerschoten
• Kokosmilch
• Sahne
• Currypulver
• 1 Reihe Schokolade

Hühnerfilet in Stücke schneiden und in dem Chutneysaft 2-3 Stunden marinieren. Paprika in Streifen schneiden, scharf anbraten, nach ein paar Minuten das Fleisch hinzugeben und ebenfalls kurz scharf anbraten. Currypulver hinzugeben, Hitze reduzieren und in feine Scheiben/Streifen geschnittene Karotten, Frühlingszwiebeln und Zuckerschoten hinzufügen. Ein paar Minuten dünsten lassen, 2-3 EL Chutney hinzufügen und mit etwas Kokosmilch und Sahne zum köcheln bringen. Lorbeerblätter hinzugeben und mit Salz, Pfeffer und fein geraspelter Schokolade abschmecken. Ca. 20min köcheln lassen, bis das Gemüse bissfest ist. Mit Reis servieren.

Can not write log, openpty() failed (/dev/pts not mounted?)

Additionally, after setting up SSH, I got the following error when logging in with SSH:

Server refused to allocate pty

Solution: install udev, reboot the VM and you’re good to go.

I clearly wanted to go with Spamassassin (SA), as I made good experiences with it in the past and it’s more or less the standard spamfilter on linux based mailservers. The most common solutions to integrate SA into a Postfix based mailserver are the following:

• Using amavisd-new
• Using Postfixs content_filter

I don’t really like both of them. Amavis is quite heavy for the pure spam filtering purpose and the content filter checks both ingoing and outgoing mails by default which is obviously not in my interest. Amavis avoids checking outgoing mail just by checking if the sender domain is managed by the same system, but spammers can bypass this quite easily by faking the sender’s address to be the same as the recipient’s one (which is done quite often). There’s a discussion about this on the ISPMail page, so head there for more information. All this can be improved by using multiple Postfix instances and different ports (e.g. using 587/submission for authenticated clients and 25/smtp for normal SMTP traffic), but I want my mailserver to be as interoperable as possible without the need of any special setups on the client side.

So I was looking for another solution. I read some tutorials where people used procmail in user scripts to pass incoming mail to spamc before delivering it to the mailbox. I like this approach as the MTA isn’t involved into the spam filtering process, outgoing mail isn’t touched and you don’t need any complicated setups on the MTA side. All alias and transport definitions work fine and the final mail is checked right before being delivered to the user’s inbox.

First I thought about Sieve, which is already running through Dovecot’s Sieve implementation until I noticed that Sieve is not able to call any external programs (correct me if I’m wrong). Then I had a look at spamc and Postfix’ master.cf. spamc is capable to pipe its output to another program and in the ISPMail setup, Postfix passes the mail directly to Dovecot’s deliver, so why not just let Spamassassin check the mail right before it’s getting passed to Dovecot? I gave it a try and seems to work fine. I still need some automation in training SA databases (might follow in a later post), but the plain SA checking is working reliably and mails can easily be filtered with Sieve afterwards.

So much for the backstory, let’s get our hands dirty. Note: I’m running Debian Lenny.

Installing and configuring Spamassassin

First of all, let’s install our magic little helpers.

$ aptitude install spamassassin pyzor razor

DCC

Additionally, I want to use DCC, which is not in the Lenny repositories. First, let’s create a user for DCC.

$ groupadd dcc
$ useradd -g dcc -s /bin/false -d /var/dcc dcc

Then download and build it manually (you might need some additional Debian packages like build-essential). Just ignore the sendmail warnings during configure. The DCC version may vary.

$ mkdir ~/build
$ cd ~/build
$ wget http://www.dcc-servers.net/dcc/source/dcc-dccproc.tar.Z
$ tar xzvf dcc-dccproc.tar.Z
$ cd dcc-dccproc-1.3.116
$ ./configure --with-uid=dcc
$ make
$ make install
$ chown -R dcc.dcc /var/dcc
$ ln -s /var/dcc/libexec/dccifd /usr/local/bin/dccifd

Configuring Spamassassin

We make use of spamd to let SA run as daemon. To do this, we need a user for SA.

$ groupadd spamd
$ useradd -g spamd -s /bin/false -d /var/lib/spamassassin spamd

Then, edit /etc/default/spamassassin to look like the following listing (changed ENABLED to 1, added SAHOME and edited OPTIONS). The virtual-config-dir allows us to have separate user preferences and bayes databases for each virtual user. An improvement would be to save this data directly to the virtual user’s “home” directory in /var/vmail, but for now I got it like this. The bayes database for user@example.org would therefore be stored in /var/lib/spamassassin/users/example.org/user/.

[...]

# Spamassassin home
SAHOME="/var/lib/spamassassin"

# Change to one to enable spamd
ENABLED=1

# Options
# See man spamd for possible options. The -d option is automatically added.

# SpamAssassin uses a preforking model, so be careful! You need to
# make sure --max-children is not set to anything higher than 5,
# unless you know what you're doing.

OPTIONS="--create-prefs -x --max-children 3 --username spamd --helper-home-dir ${SAHOME} -s ${SAHOME}/spamd.log --virtual-config-dir=${SAHOME}/users/%d/%l"

[...]

The home directory:

$ mkdir -p /var/lib/spamassassin/users
$ chown spamd.spamd /var/lib/spamassassin -R

Now, let’s do some configuring. You can find all relevant config files in /etc/spamassassin. First of all, local.cf:

[...]

#   Save spam messages as a message/rfc822 MIME attachment instead of
#   modifying the original message (0: off, 2: use text/plain instead)
#
report_safe 0

[...]

use_dcc 1
dcc_path /usr/local/bin/dccproc

use_pyzor 1
pyzor_path /usr/bin/pyzor

use_razor2 1
razor_config /etc/razor/razor-agent.conf

Afterwards, edit v310.pre and check that the DCC, Razor and Pyzor plugins are enabled (DCC is disabled by default).

You can check your Spamassassin configuration with lint (use the -D flag for output):

$ spamassassin --lint

And you can update SA’s rules with sa-update:

$ sa-update --no-gpg

Now we are ready to start the SA daemon (will be automatically started at boot time):

$ /etc/init.d/spamassassin start

Configure Postfix

This step is quite easy. First, edit /etc/postfix/master.cf, copy the existing dovecot transport and edit it to look as follows (you can change the name ;)):

dovecot-spamass   unix  -       n       n       -       -       pipe
    flags=DRhu user=vmail:vmail argv=/usr/bin/spamc -u ${recipient} -e /usr/lib/dovecot/deliver -d ${recipient}

This transport will pass the mail to spamc which will pass it to deliver after checking.

Now just change the transport in /etc/postfix/main.cf. I used the second transport to be flexible. In case anything should go wrong with Spamassassin you just need to change the transport in main.cf and you get your mails without the SA step.

virtual_transport = dovecot-spamass

That’s all you need on the Postfix side. Just restart it to use the new transport.

$ /etc/init.d/postfix restart

Finishing touches

Now let’s do some testing. Send yourself a mail and take a look at the mail headers. There should be quite a lot of new mail headers which get injected by Spamassassin (you can configure the headers in Spamassassin’s configuration). A normal non-spam mail could look like this:

X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.example.com
X-Spam-Level: *
X-Spam-Status: No, score=1.2 required=5.0 tests=ALL_TRUSTED,AWL,
	TVD_SPACE_RATIO autolearn=no version=3.2.5

Now cross your fingers and hope that a friendly spammer sends you new crap or just use the GTUBE test (include that line in your mail). SA should recognize it as junk and give detailed information about it in its headers:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.example.com
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=1000.0 required=5.0 tests=ALL_TRUSTED,AWL,GTUBE
	autolearn=no version=3.2.5
X-Spam-Report: =?ISO-8859-1?Q?
	* 1000 GTUBE BODY: Test zur Pr=fcfung von Anti-Spam-Software

Other example for a real spam mail:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.example.com
X-Spam-Level: *******************
X-Spam-Status: Yes, score=19.6 required=5.0 tests=BAD_ENC_HEADER,DCC_CHECK,
	DIGEST_MULTIPLE,HELO_LOCALHOST,HTML_MESSAGE,PYZOR_CHECK,
	RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RDNS_NONE,
	URIBL_RHS_DOB,URIBL_SBL,URIBL_WS_SURBL autolearn=spam version=3.2.5
X-Spam-Report: =?ISO-8859-1?Q?
	*  0.9 URIBL_RHS_DOB Contains an URI of a new domain (Day Old Bread)
	*      [URIs: spammersdomain.com]
	*  2.1 URIBL_WS_SURBL Enth=e4lt URL in WS-Liste (www.surbl.org)
	*      [URIs: spammersdomain.com]
	*  4.5 HELO_LOCALHOST HELO_LOCALHOST
	*  2.9 BAD_ENC_HEADER Message has bad MIME encoding in the header
	*  0.0 HTML_MESSAGE BODY: Nachricht enth=e4lt HTML
	*  1.5 RAZOR2_CF_RANGE_E8_51_100 Razor2 gives engine 8 confidence level
	*      above 50%
	*      [cf:  58]
	*  0.5 RAZOR2_CHECK Gelistet im "Razor2"-System (http://razor.sf.net/)
	*  0.5 RAZOR2_CF_RANGE_51_100 Razor2 Spam-Bewertung liegt zwischen 51 und
	*      100
	*      [cf:  58]
	*  2.8 PYZOR_CHECK Gelistet im Pyzor-System (http://pyzor.sf.net/)
	*  1.4 DCC_CHECK Gelistet im DCC-System
	*      (http://rhyolite.com/anti-spam/dcc/)
	*  2.5 URIBL_SBL Enth=e4lt URL in SBL-Liste (http://www.spamhaus.org/sbl/)
	*      [URIs: spammersdomain.com]
	*  0.0 DIGEST_MULTIPLE Mehrere Internettests (Razor, DCC, Pyzor, etc.)
	*      treffen zu
	*  0.1 RDNS_NONE Delivered to trusted network by a host with no rDNS?=

If you are wondering about the autolearn flag in the mail headers: SA triggers autolearning only starting from a certain treshold. For more information see Spamassassin Wiki/AutolearningNotWorking.

Filtering

Using SA’s mail headers, you can easily filter your mails. A simple Sieve filter could look like this:

require "fileinto";

if header :contains "X-Spam-Flag" ["YES"] {
  fileinto "Spam";
  stop;
}

If you are using Thunderbird, you can even configure it to filter mail using SA’s headers (in your account settings), however this shifts the filtering a bit too much on the client side for my taste.

Per-user preferences

As we use SA’s virtual-config-dir, we have a separate bayes db for each virtual user and can additionally specify user preferences (like blacklists) on a per-user basis. To make use of this, create a file called user_prefs in the virtual user directory for SA and add individual rules there.

Example: add a blacklist entry for all mails which are recieved for joe@example.com (assuming joe has received any mail before and spamd created the needed directories):

$ echo "blacklist_from jane@example.com" > /var/lib/spamassassin/users/example.com/joe/user_prefs

You can try adding yourself to your blacklist and sending yourself a mail. If all is working fine, the mail should be classified as spam. Example:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.example.com
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=101.4 required=5.0 tests=ALL_TRUSTED,AWL,
	TVD_SPACE_RATIO,USER_IN_BLACKLIST autolearn=no version=3.2.5
X-Spam-Report: =?ISO-8859-1?Q?
	*  100 USER_IN_BLACKLIST From: address is in the user's black-list
[...]

Conclusion

This setup seems to me quite straightforward and lightweight, but it still needs some improvements like automated bayes database training (take a look at the first source) or moving user’s preferences to the database. However, the stock Spamassassin installation does its work quite reliably and is working fine at the moment.

Sources

• http://hints.jeb.be/2008/12/17/per-virtual-user-sa-learn-training/
• http://www.dovecot.org/list/dovecot/2008-June/031175.html
• http://www.howtoforge.com/virtual-users-domains-postfix-courier-mysql-squirrelmail-debian-lenny-p3
• http://www.debianhelp.co.uk/spam.htm
• http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html

However, the described setup has a few issues which bugged me:

1. CruiseControl runs from the shellscript as root, posts all output to the console and is not automatically started at boot time.
2. CruiseControl runs on port 8080, but I wanted to manage access to the webinterface through the apache which is already running on the box
3. There’s no authentication – everybody can access my CI server, see the build details and start new builds through the webinterface.

I solved these issues with 2 steps.

Init script for CruiseControl

I wrote a simple init script which allows me to control CC. I’m not really into shellscripting and just hacked around on the apache init script until I got a working solution, so if you have any suggestions how to improve this script please let me know. The script implements the following functions: start, stop, restart, status. Save it to /etc/init.d/cruisecontrol and make it executable (chmod +x /etc/init.d/cruisecontrol).

#!/bin/sh
### BEGIN INIT INFO
# Provides:          cruisecontrol
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start/stop cruisecontrol CI server
### END INIT INFO
#
# cruisecontrol         This init.d script is used to start cruisecontrol.
#                       It basically just calls cruisecontrol.sh.

# ENV="env -i LANG=C PATH=/usr/local/bin:/usr/bin:/bin"

CCON_PATH=/opt/cruisecontrol
CCON_USER=www-data
PIDFILE=$CCON_PATH/cc.pid
LOGFILE=/var/log/cruisecontrol.log

. /lib/lsb/init-functions

ccon_is_running() {
    if [ -f $PIDFILE ]; then
       PID=`cat $PIDFILE`
       PID_COUNT=`ps aux | grep -E $PID | grep -v grep | wc -l`
       if [ $PID_COUNT = 1 ]; then
         return 1
       fi
    fi

    return 0
}

ccon_start() {
        ccon_is_running
        rc=$?
        if [ $rc -eq 1 ]; then
                log_failure_msg "CruiseControl is already running"
        else
                log_daemon_msg "Starting CI server" "CruiseControl"
                cd $CCON_PATH
                sudo -E -H -u $CCON_USER ./cruisecontrol.sh >> $LOGFILE 2>&1
                log_end_msg $?
        fi
}

ccon_stop() {
        ccon_is_running
        rc=$?
        if [ $rc -eq 1 ]; then
                log_daemon_msg "Stopping CI server" "CruiseControl"
                PID=`cat $PIDFILE`

                retval=0
                i=0
                while $(kill "$PID" 2> /dev/null);  do
                        if [ $i = '60' ]; then
                                echo ""
                                log_failure_msg "CruiseControl is taking too long to shutdown"
                                retval=1
                                break
                        else
                                if [ $i = '0' ]; then
                                        echo -n " ... waiting "
                                else
                                        echo -n "."
                                fi
                                i=$(($i+1))
                                sleep 1
                      fi
                 done

                log_end_msg $retval
        else
                log_failure_msg "CruiseControl is not running"
        fi
}

ccon_status() {
        ccon_is_running
        rc=$?
        if [ $rc -eq 1 ]; then
                PID=`cat $PIDFILE`
                log_success_msg "CruiseControl is running (pid $PID)."
        else
                log_failure_msg "CruiseControl is not running."
        fi

}

case $1 in
        start)
                ccon_start
        ;;
        stop)
                ccon_stop
        ;;
        restart)
                ccon_stop
                ccon_start
        ;;
        status)
                ccon_status
        ;;
        *)
                log_success_msg "Usage: /etc/init.d/cruisecontrol {start|stop|restart|status}"
                exit 1
        ;;
esac

Then change the ownership of the cruisecontrol installation to the user you specified in the init script (in my case www-data):

$ chown -R www-data.www-data /opt/cruisecontrol-bin-2.8.2/

You can now control CC just by using the init script.

$ /etc/init.d/cruisecontrol status
 * CruiseControl is not running.
$ /etc/init.d/cruisecontrol start
 * Starting CI server CruiseControl                        [ OK ]
$ /etc/init.d/cruisecontrol status
 * CruiseControl is running (pid 13721).
$ /etc/init.d/cruisecontrol stop
 * Stopping CI server CruiseControl
 ... waiting ..                                            [ OK ]
$ /etc/init.d/cruisecontrol status
 * CruiseControl is not running.

Additionally you can add it to the default runlevels to start it automatically on system boot.

$ update-rc.d cruisecontrol defaults

Access the webinterface via Apache

I wanted to use my existing apache installation to serve the webinterface. Doing this you can make the webinterface accessible through port 80, control authentication through apache’s various auth modules and eventually even use SSL to encrypt the traffic. The first thing I did was to bind the CruiseControl’s webserver (Jetty) to the local IP. Open /opt/cruisecontrol/etc/jetty.xml and find the following line:

<Set name="host"><SystemProperty name="jetty.host" /></Set>

Add the default attribute to that line and afterwards restart CC.

<Set name="host"><SystemProperty name="jetty.host" default="127.0.0.1" /></Set>

Now try to access CruiseControl through the external IP, you should not get a connection. But if you try to open a connection from the same host, you should get a response:

$ curl http://127.0.0.1:8080/cruisecontrol/

To serve this instance via apache you need 3 apache modules (install them via apt if you don’t have them installed):

$ a2enmod proxy proxy_http ext_filter

Create a new virtual host config file in /etc/apache2/sites-available/. Modify ServerName and ExtFilterDefine to match your environment.

<VirtualHost *:80>
        ServerName cruisecontrol.example.org

        ProxyRequests Off

        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

        ProxyPass / http://127.0.0.1:8080/
        ProxyPassReverse / http://127.0.0.1:8080/

        ExtFilterDefine fixurls mode=output intype=text/html cmd="/bin/sed s%http://127.0.0.1:8080%http://cruisecontrol.example.org%g"
        SetOutputFilter fixurls
</VirtualHost>

As a last step, enable the virtual host and restart apache (I named the config file cruisecontrol):

$ a2ensite cruisecontrol
$ /etc/init.d/apache2 restart

Now you should have access to your phpUnderControl installation via http://cruisecontrol.example.org/cruisecontrol. As a last step, we add basic apache authentication (replace with your preferred apache auth method). Edit the vhost config file and modify as follows:

        <Proxy *>
                Order deny,allow
                Allow from all

                AuthName        "phpUnderControl"
                AuthType        Basic
                AuthUserFile    /etc/apache2/cruisecontrol.htpasswd
                require         valid-user
        </Proxy>

As a last step, you have to create the .htpasswd file, which contains the user map.

$ htpasswd -c /etc/apache2/cruisecontrol.htpasswd myuser

Add additional users with the following command (same command but without the -c switch):

$ htpasswd /etc/apache2/cruisecontrol.htpasswd otheruser

Finally, restart apache and you are done:

$ /etc/init.d/apache2 restart

Sources

• PHP Staging environment for continuous integration part 2
• Making phpUnderControl (CruiseControl) accessible via a Proxy

Da ich hier weder ein Floppy-Laufwerk (glaub das wär noch in irgendeinem Schrank aufzutreiben gewesen), funktionierende Disketten (die schon schwieriger) oder Nero installiert hatte, habe ich mich nach einer Alternativlösung umgesehen. Abhilfe schafft hier wieder mal Linux und mkisofs, mit dem sich das Floppy-Image einfach in ein CD-ISO umwandeln lässt. Diese Lösung setzt natürlich voraus, dass man in dem zu installierenden PC 2 optische Laufwerke hat, eines für die Windows7-DVD und eines für die Bootloader-CD. Also flugs eine Ubuntu-VM gebootet und darin das gujin-ISO gebastelt. Dazu lädt man sich in der VM zuerst die standard-Version von gujin herunter und entpackt das Archiv. In dem Archiv befindet sich eine Datei full.img.gz, die man wiederum entpackt. Die darin befindliche Datei floppy.144 schiebt man in einen Arbeitsordner (bei mir z.B. /tmp/floppyimage) und öffnet anschließend ein Terminal. Darin macht man dann ca. folgendes:

$ cd /tmp/floppyimage
$ mkisofs -pad -b floppy.144 -R -o /tmp/gujin.iso floppy.144

Und schon liegt in dem Ordner eine Datei gujin.iso, die sich auf CD brennen und booten lässt. Anschließend legt man die Bootloader-CD in das eine, die Win7-DVD in das andere Laufwerk und bootet von der soeben gebrannten CD. Nach kurzer Erkennungszeit lassen sich in gujin die verschiedenen zur Verfügung stehenden Bootoptionen auswählen. Wählt man hier das Laufwerk mit der Win7-DVD, bootet die DVD ohne Probleme und man kann sich daran machen, (wieder mal) ein neues System auf die Platte zu packen. Ist zumindest bei mir so, auf der doch recht betagten Hardware ist der Installer durchgelaufen, während ich diesen Post hier verfasst habe ;)

Zuerst die heruntergeladenen Dateien ausführen, wodurch die Installer entpackt werden. Im meinem Fall waren das die 2 Dateien 9-7_vista64_win7_64_dd.exe (display driver) und 9-7_vista32-64_ccc_lang2.exe (control center). Danach befinden sich in C:\ATI die entsprechenden Setup-Dateien in Unterordnern, die gleich benannt sind wie die heruntergeladenen Dateien. Um jetzt den display driver zu installieren sind folgende Schritte nötig:

1. Im Startmenü nach Zubehör, Rechtsklick auf Eingabeaufforderung und dann “Als Administrator ausführen”. Dadurch öffnet sich eine CMD mit administrativen Rechten
2. cd C:\ATI
3. cd 9-7_vista64_win7_64_dd (je nach Treiberversion, Sprache und Betriebssystem)
4. cd Bin64 (bei 64-Bit Betriebssystemen, bei 32-Bit Systemen heißt der Ordner wahrscheinlich Bin)
5. ATISetup.exe -install -output screen

Analog funktioniert das auch mit dem CCC und dem Avivo Video Converter (Hydravision hab ich nicht getestet, da ichs nicht brauche).

You can find tutorials on ftplicity/duplicity here (Note: they use the original c’t version of ftplicity):

• http://www.weareroot.de/2007/08/17/ftplicity-inkrementelle-backups-sicher-ablegen/ (German)
• http://www.howtoforge.com/ftp-backups-with-duplicity-ftplicity-debian-etch

Basically you can use this setup for any kind of LVM snapshot based system, but I’m focusing on backing up Xen VMs here. I assume you got your LVM and Xen system up and running so far. I did this on a Debian Lenny system, but it should be similar on other distros. I did all steps as root.

Setup and Goal

My setup is the following: I got 1 volume group (VG) which contains all logical volumes (LV) used for virtual machines. Every virtual machine has 2 LVs, vm01-disk and vm01-swap, where vm01 is the name of the VM. Of course I want to backup only the *-disk LVs.

Goal: automatically create a LVM snapshot, mount it and back up the VM to the remote FTP server. Afterwards unmount the snapshot and remove it.

LVM snapshots

To automatically create LVM snapshots, I wrote a simple bash script which does all the needed steps. As my bash mojo isn’t too advanced, I got some ideas from here ;). You can download the script at Github.

Disclaimer: I do neither issue any guarantee that the script will work for you nor do I take any responsibility for potential data loss caused by this script. So please use with care.

The script has some options to configure:

LVMPATH

Path to your volume group (VG), e.g. /dev/lvmstore.

LVMEXTENSION

Extension which will be added to LV names, e.g. if you specify vm01 as LV name and LVMEXTENSION is set to -disk, the snapshot will be created from /dev/lvmstore/vm01-disk (using the LVMPATH example from before).

MOUNTPATH

Path where snapshots will be mounted to.

SNAPSHOTSIZE

Size of the snapshot.

IDENTIFIER

An identifier, which will be used to create the snapshot name (useful to distinguish automatic backups from others). Using the examples from the other options, the snapshot will be named as follows: vm01-disk-snapshot-backupscript (with IDENTIFIER set to backupscript)

Edit the options according to your needs, make the script executable and drop it somewhere on your system. Example:

$ nano lvmsnapshot.sh
[ ... edit options ...]
$ chmod 700 lvmsnapshot.sh
$ mv lvmsnapshot.sh /usr/sbin/lvmsnapshot

Assuming all options are set correctly for your system, you can use the script as follows. Create and mount a snapshot of virtual machine vm01, located at /dev/lvmstore/vm01-disk:

$ lvmsnapshot create vm01
Checking if /dev/lvmstore/vm01-disk is mounted...No
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript...
  Logical volume "vm01-disk-snapshot-backupscript" created
...successful

Mounting LVM snapshot for backup...
  Creating mount directory at /mnt/lvm/vm01...OK
...successful

Umount and remove the snapshot:

$ lvmsnapshot remove vm01
Checking if /dev/lvmstore/vm01-disk is mounted...Yes
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Unmounting LVM snapshot after backup...
  Deleting mount directory at /mnt/lvm/vm01...OK
...successful

Deleting LVM snapshot vm01-disk-snapshot-backupscript
  Logical volume "vm01-disk-snapshot-backupscript" successfully removed
...successful

If this works, we are able to automatically create snapshots and mount them. So we can pass to the next step: setting up ftplicity and configuring it to use the script to get access to our VM data.

Install ftplicity/duplicity

First of all, install duplicity:

$ aptitude install duplicity ncftp

Get ftplicity from the SourceForge project site and make it executable:

$ wget http://downloads.sourceforge.net/sourceforge/ftplicity/ftplicity_1.4.2.tgz
$ tar xvzf ftplicity_1.4.2.tgz
$ cp ftplicity_1.4.2/ftplicity /usr/sbin/ftplicity
$ chmod 700 /usr/sbin/ftplicity
$ rm -rf ftplicity_1.4.2 ftplicity_1.4.2.tgz

Afterwards, create /etc/ftplicity. This way, ftplicity profiles will be stored in /etc instead of root‘s home directory.

mkdir /etc/ftplicity

Calling ftplicity should now work and tell you to specify a valid profile.

Create a GPG key

In order to be able to encrypt your backups, you have to create a GPG key. The tutorials mentioned at the beginning explain this in detail, so here’s only the short version. Open a second shell and run the following command (this generates some “randomness” on your system, which will be useful to create a secure key). Kill the command with CTRL+C when you are done with key generation.

while /bin/true; do cat /var/log/syslog > ~/temp.txt; sleep 1; done;

On your other shell, create your GPG key. Be sure to use a secure passphrase and to copy/write down the key ID which is displayed at the end of the generation process (we’ll need it for ftplicity). Also, make sure to backup the key to a secure location outside your server. As all your backups will be encrypted, they will be worthless if your server crashes and you lose the key.

gpg --gen-key

Default options should be fine. This will create your key in ~/.gnupg/.

Set up ftplicity profiles

Now it’s time to set up a ftplicity profile for a virtual machine. The virtual machine is called vm01 and is running a webserver stack using Apache/MySQL. Call ftplicity to initialize the profile vm01:

$ ftplicity vm01 create

Warning:

The profile's folder
'/etc/ftplicity/vm01'
permissions were not safe (drwxr-xr-x). Secured them to 700.

Congratulations. You just created the profile 'vm01'.
The initial config file has been created as
'/etc/ftplicity/vm01/conf'.
For ftplicity to work you have to insert details on
the gpg key to use and the ftp server for the backup
in this config file.

IMPORTANT:
  Copy the _whole_ profile folder after the first backup to a safe place.
  It contains everything needed to restore your backups. You will need
  it if you have to restore the backup from another system (e.g. after a
  system crash). Keep access to these files restricted as they contain
  _all_ informations (gpg data, ftp data) to access and modify your backups.

  Repeat this step after all configuration changes. Some configuration
  options are crucial for restoration.

Ftplicity creates a config file for your profile in /etc/ftplicity/vm01/conf, which you will have to edit to match your system. I changed the following options:

# gpg key data
GPG_KEY='keyid'
GPG_PW='keypassphrase'

# ...

# credentials & server address of the ftp server (URL-Format)
TARGET='ftp://username@ftpserver/vm01'
TARGET_PW='ftppassword'

# base directory to backup
SOURCE='/mnt/lvm/vm01'

# ...

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
MAX_FULLBKP_AGE=6D
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

Make sure the directory on the FTP server exists. If you get any Python problems when running the backups, try to create an empty file in your backup directory on the FTP server (there is or was a Python bug which caused problems with empty directories when connecting to FTP).

Next, we have to tell ftplicity to use the snapshot-script to get access to vm01‘s data. Fortunately, ftplicity supports pre and post scripts, which are executed before and after a backup job. Create these scripts and make them executable:

$ cd /etc/ftplicity/vm01
$ touch pre
$ touch post
$ chmod 700 pre
$ chmod 700 post

pre

#!/bin/bash
/usr/sbin/lvmsnapshot create vm01

post

#!/bin/bash
/usr/sbin/lvmsnapshot remove vm01

As a last step, create a file /etc/ftplicity/vm01/exclude which tells ftplicity which files to exclude from the backup. Note that I excluded /var/lib/mysql too, as backing up the MySQL data directory could lead to inconsistent results. To solve this, I use automysqlbackup inside VMs which dumps SQL data regularly. These dumps are included in my FTP backup. I’ll write more on automysqlbackup in a later tutorial.

/mnt/lvm/vm01/var/run/**
/mnt/lvm/vm01/var/tmp/**
/mnt/lvm/vm01/tmp/**
/mnt/lvm/vm01/dev/**
/mnt/lvm/vm01/sys/**
/mnt/lvm/vm01/proc/**
/mnt/lvm/vm01/floppy/**
/mnt/lvm/vm01/cdrom/**
/mnt/lvm/vm01/var/lib/mysql/**

Run ftplicity

OK, all is set up and should be working now. So let’s try using ftplicity. You can see all ftplicity commands by calling ftplicity usage.

Check status of our profile:

$ ftplicity vm01 status
Start ftplicity v1.4.2, time is 07/08/09 20:09:47.
Using profile '/etc/ftplicity/vm01'.
Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg)
Test - Encryption with key xxxxxxxx (OK)
Test - Decryption with key xxxxxxxx (OK)
Test - Compare Original w/ Decryption (OK)
Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK)

--- Start running command STATUS (20:09:47.943) ---
Running duplicity - OK
Output: NcFTP version is 3.2.1
Last full backup date: none
Connecting with backend: ftpBackend
Archive dir: None

Found 0 backup chains without signatures.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
--- Finished (20:09:48.206) - Runtime 00:00:00.263 ---

OK, profile seems to work. Let’s try to make a backup.

ftplicity vm01 backup
Start ftplicity v1.4.2, time is 07/08/09 20:14:57.
Using profile '/etc/ftplicity/vm01'.
Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg)
Test - Encryption with key xxxxxxxx (OK)
Test - Decryption with key xxxxxxxx (OK)
Test - Compare Original w/ Decryption (OK)
Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK)

--- Start running command PRE (20:14:58.152) ---
Running '/etc/ftplicity/vm01/pre' - OK
Output: Checking if /dev/lvmstore/vm01-disk is mounted...No
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript...
  Logical volume "vm01-disk-snapshot-backupscript" created
...successful

Mounting LVM snapshot for backup...
  Creating mount directory at /mnt/lvm/vm01...OK
...successful
--- Finished (20:14:58.490) - Runtime 00:00:00.338 ---

--- Start running command BKP (20:14:58.498) ---
Running duplicity - OK
Output: NcFTP version is 3.2.1
Reading globbing filelist /etc/ftplicity/vm01/exclude
Last full backup date: none
Last full backup is too old, forcing full backup
--------------[ Backup Statistics ]--------------
StartTime 1247076898.77 (Wed Jul  8 20:14:58 2009)
EndTime 1247076977.92 (Wed Jul  8 20:16:17 2009)
ElapsedTime 79.15 (1 minute 19.15 seconds)
SourceFiles 12398
SourceFileSize 276924030 (264 MB)
NewFiles 12398
NewFileSize 276924030 (264 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 12398
RawDeltaSize 130935522 (125 MB)
TotalDestinationSizeChange 101194951 (96.5 MB)
Errors 0
-------------------------------------------------
--- Finished (20:16:19.864) - Runtime 00:01:21.365 ---

--- Start running command POST (20:16:19.872) ---
Running '/etc/ftplicity/vm01/post' - OK
Output: Checking if /dev/lvmstore/vm01-disk is mounted...Yes
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Unmounting LVM snapshot after backup...
  Deleting mount directory at /mnt/lvm/vm01...OK
...successful

Deleting LVM snapshot vm01-disk-snapshot-backupscript
  Logical volume "vm01-disk-snapshot-backupscript" successfully removed
...successful
--- Finished (20:16:20.718) - Runtime 00:00:00.846 ---

Perfect. The LVM snapshot is created and mounted, backuped and afterwards all gets cleaned up. As this was the first backup, ftplicity created a full backup. Following backups will be incremental until the full backup is too old (see ftplicity config for details).

Restore

Ftplicity allows simple restoring of your backups. Some examples:

Restore the complete last state to /tmp/vm01restore:

$ ftplicity vm01 restore /tmp/vm01restore

Restore /etc/passwd from the last backup state to /tmp/vm01restore/etc/passwd:

$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd

The same as before, but take the state from four days before:

$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd 4D

Perfect.

Set up cronjobs

To run a backup automatically every night, I set up cronjobs to do this. Example:

0 1 * * * /usr/sbin/ftplicity vm01 cleanup --force ; /usr/sbin/ftplicity vm01 backup
0 2 * * 1 /usr/sbin/ftplicity vm01 purge-full --force ; /usr/sbin/ftplicity vm01 purge --force

This will run ftplicity on vm01 every night at 1:00 and purge old backups every monday night at 2:00.

Conlusion

The above setup gives you a secure and handy solution to run automated backups on LVM snapshots. Ftplicity is quite simple to use and gives you many possibilities to restore your files. To add more VMs, just create a new profile (or copy the existing one) and adjust the config files to match the new VM paths. Any suggestions are welcome :)

GRANT SELECT, SHOW VIEW, LOCK TABLES ON *.* TO 'backup'@'localhost';

$ mkdir -p extract/DEBIAN
$ dpkg-deb -x package.deb extract/
$ dpkg-deb -e package.deb extract/DEBIAN
[...do something, e.g. edit the control file...]
$ mkdir build
$ dpkg-deb -b extract/ build/

• -x extracts the package contents
• -e extracts the control files
• -b builds the new package

Done.

Zutaten:

• 125 ml Pure Cofain 699
• 200 ml Sprite
• 1 Spritzer Zitronensaft
• ½ TL Rohrzucker

Alle 3-5h ein Gläschen sollte für “3 Tage wach” reichen. Achtung bei Sprite und Zucker: die Mischung schäumt dadurch ziemlich hoch auf, also am besten in einem ausreichend dimensionerten Glas anmischen (siehe Foto).

Im Gegensatz zu Pure Cofain existieren zu diesem Drink weder wissenschaftliche Studien noch Feldtests, die die Verträglichkeit dieses Drinks garantieren. Ich übernehme daher keine Haftung für etwaige, durch den Konsum des Drinks entstandene, gesundheitliche Schäden ;). Mehr Infos zu Gefahren, Wirkung und allem weiteren gibt es auf der Webseite des Herstellers.

Viel Spaß ;)

Um zum Beispiel immer die aktuelle Woche des Studiengangs WI07 als Kalender in meinem Thunderbird anzeigen zu können, habe ich dort folgende URL als iCal-Quelle angegeben:

http://raumbelegung.stud.ailoo.net/week/show/class/wi07-vz/format/ical

Vorschläge sind wie üblich willkommen :)

I’m working with the default application which is generated when creating a new ASP.NET MVC project in Visual Studio.

FlashMessenger

To store all the messages, I created a class called FlashMessenger which is responsible to store the messages. I created a new folder LibrarySession and saved the class as FlashMessenger.cs. The class allows saving of messages to 3 namespaces (success, notice, error) by default – to add more namespaces just edit the enum.

using System;
using System.Collections;
using System.Collections.Generic;

namespace MyProject.Library.Session
{
    public class FlashMessenger
    {
        public enum MessageType
        {
            Success,
            Notice,
            Error
        };

        private Dictionary<MessageType, Queue<string>> _messages;
        public Dictionary<MessageType, Queue<string>> Messages
        {
            get { return _messages; }
            private set { _messages = value; }
        }

        public FlashMessenger()
        {
            Messages = new Dictionary<MessageType, Queue<string>>();
            foreach (MessageType type in Enum.GetValues(typeof(MessageType)))
            {
                Messages[type] = new Queue<string>();
            }
        }
    }
}

BaseController

The next element is the BaseController which initializes the FlashMessenger and implements methods to easily create new messages. All my controllers inherit from this controller, so I can use messaging everywhere. The [Messaging] attribute on the class level ensures all actions use the MessagingAttribute we’ll create in the next step. I created the controller as BaseController.cs in LibraryControllers.

using System;
using System.Web;
using System.Web.Mvc;
using MyProject.Library.Session;

namespace MyProject.Library.Controllers
{
    [Messaging]
    public class BaseController : Controller
    {
        protected override void Initialize(System.Web.Routing.RequestContext requestContext)
        {
 	    base.Initialize(requestContext);
            this.InitializeFlashMessenger();
        }        

        # region FlashMessenger
        public FlashMessenger Messenger
        {
            get { return (FlashMessenger)Session["FlashMessenger"]; }
            private set { }
        }

        protected void InitializeFlashMessenger()
        {
            if (Messenger == null)
                Session["FlashMessenger"] = new FlashMessenger();
        }

        public void FlashMessage(string message)
        {
            FlashMessage(message, FlashMessenger.MessageType.Success);
        }

        public void FlashMessage(string message, FlashMessenger.MessageType type)
        {
            Messenger.Messages[type].Enqueue(message);
        }
        # endregion
    }
}

This allows us to use the FlashMessaging system in every controller which inherits from BaseController. For example:

public ActionResult Foo()
{
    FlashMessage("You have been redirected to Bar from Foo", FlashMessenger.MessageType.Notice);
    return RedirectToAction("Bar", "Home");
}

MessagingAttribute

Our messaging system is set up, but we still need to display our messages. To achieve this, I created an action filter called MessagingAttribute which injects the messages into the ViewData object. The output are simple unordered lists which get the MessageType as class name. As the attribute is set up on the class level of BaseController, the filter gets executed on every action and messages are available in every view. I saved the filter as MessagingAttribute.cs in LibraryControllersFilters.

using System;
using System.Collections.Generic;
using System.Text;
using System.Linq;
using System.Web;
using System.Web.Mvc;
using MyProject.Library.Session;

namespace MyProject.Library.Controllers
{
    public class MessagingAttribute : ActionFilterAttribute, IActionFilter
    {
        void IActionFilter.OnActionExecuting(ActionExecutingContext filterContext)
        {
            FlashMessenger Messenger = (FlashMessenger) filterContext.HttpContext.Session["FlashMessenger"];
            StringBuilder output = new StringBuilder();

            foreach (KeyValuePair<FlashMessenger.MessageType, Queue<string>> kvp in Messenger.Messages)
            {
                if (kvp.Value.Count > 0)
                {
                    output.AppendLine("<ul class="" + kvp.Key.ToString() + "">");

                    while(kvp.Value.Count > 0)
                    {
                        output.AppendLine("<li>" + kvp.Value.Dequeue() + "</li>");
                    }

                    output.AppendLine("</ul>n");
                }
            }

            filterContext.Controller.ViewData["FlashMessenger"] = output;
        }
    }
}

View and CSS

To display the messages in your application, edit the following. In your Site.master, display the output from the action filter:

<div id="main">
    <div id="messages">
        <%= ViewData["FlashMessenger"] %>
    </div>            

    <asp:ContentPlaceHolder ID="MainContent" runat="server" />

    <div id="footer">
    </div>
</div>

In your Site.css, add some styling for your messages:

#messages {
    margin-bottom: 2em;
}

#messages ul {
    list-style: none;
    margin: 0;
    padding: 0;
}

#messages ul li {
    display: block;
    margin: .75em 0;
    padding: 10px;
    border: 2px solid #ddd;
    background-color: #eee;
    color: #222;
}

#messages ul.Notice li {
    background: #FFF6BF;
    color: #817134;
    border-color: #FFD324;
}

#messages ul.Success li {
    background: #E6EFC2;
    color: #529214;
    border-color: #C6D880;
}

#messages ul.Error li {
    background: #FBE3E4;
    color: #D12F19;
    border-color: #FBC2C4;
}

Example usage

As an example, we modify the AccountController to show a nice confirmation message on login and logout.

First of all, import the Library.Session namespace and let the controller inherit from BaseController.

@@ -8,11 +8,13 @@ using System.Web.Mvc;
 using System.Web.Security;
 using System.Web.UI;
 using MyProject.Models;
+using MyProject.Library.Controllers;
+using MyProject.Library.Session;

 namespace MyProject.Controllers
 {
     [HandleError]
-    public class AccountController
+    public class AccountController : BaseController
     {

         // This constructor is used by the MVC framework to instantiate the controller using

Add a flash message to your logon action:

@@ -63,6 +63,7 @@ namespace MyProject.Controllers
             }

             FormsAuth.SignIn(userName, rememberMe);
+            FlashMessage("Logged on successfully", FlashMessenger.MessageType.Success);^M

             if (!String.IsNullOrEmpty(returnUrl))
             {

And to your logoff action:

@@ -78,6 +78,7 @@ namespace MyProject.Controllers
         public ActionResult LogOff()
         {
             FormsAuth.SignOut();
+            FlashMessage("Logged off successfully", FlashMessenger.MessageType.Success);^M
             return RedirectToAction("Index", "Home");
         }

Conclusion

So far the system is finished and allows easy flashing of messages between page requests. However, I’m not completeley happy with the solution of having to use a base controller to initialize and use the messaging system. If anyone knows a better solution, please let me know.