nme.pl/en

Multiple arguments in Django template filters

By default, it is not currently possible to pass multiple arguments to Django template filter - documentation states: "Custom filters are just Python functions that take one or two arguments".

Here I describe solution that allows passing more arguments.

Lets say we have key and current query_string in our template context. We loop in paginator and alter query_string current key value with page. The key is to group arguments in an array. We use following custom filter:

@register.filter
def keys (first,second):
    if isinstance(first,list):
        return first+[second]
    else:
        return [first,second]

following filter allows us to:

{{ "1"|keys:"2"|keys:"3" }}

which will return in our template:

[u'1', u'2', u'3']

Returning to described query_string altering problem - we use alter_query filter:

@register.filter
def alter_query (keys, query_string):
    from django.http import QueryDict
    query_dict = QueryDict(query_string, mutable=True)
    query_dict[keys[0]] = keys[1]
    return query_dict.urlencode()

inside pagination template, we use following code:

<a href="?{{ key|keys:page|alter_query:query_string }}">{{ page }}</a>

Doesn't look pretty, but works perfectly.

hv - kvm virtualization swiss knife (requires kvm-admin)

Kvm is currently one of best choices for virtualization under GNU Linux operating systems. It is fast and stable. No wonder there are many frontend utilities that handles it.

RedHat Enterprise Linux and Ubuntu Server Linux are both suggesting to relay on libvirt, but whenever I try to use that I always finish with "It is not good enough yet", "new features come, old bugs still exist" etc. As I suppose - it will never be good enough. The reason is the libvirt layer which completly separates system administrator from kvm console.

In opposition to libvirt - we can use kvm and qemu tools, but sooner or later we will start to look after another toolkit to make life easier. I found mine - kvmtools - it is slick, lightweight, written in Python language and offers possibility to perform most tasks using single command. It is really great, but... I don't like kvm-admin command syntax. That is why I created hv - simple bash script that makes kvm-admin more friendly - especially - for (former) virsh users.

Mercurial repository for hv util is available here.
You can download it using the following command: hg clone http://nme.pl/pub/repos/hv

hv

Usage:

	hv (list|running)
	hv (boot|bootp|reboot|kill) domain-name (bootp = boot domain-name, but keep it in paused state)
	hv (vnc|vncinfo) domain-name
	hv serial domain-name
	hv monitor domain-name qemu-monitor-command
	hv (pause|cont|status) domain-name
	hv edit domain-name
	hv snapshots domain-name
	hv (savevm|loadvm|delvm|bootvm) domain-name snapshot-name
	hv shutdown domain-name
	hv verify
	hv example-domain

	Commands might be shortened, eg. hv li

What does hv offer?

List of all commands with examples, like mentioned in hv usage above - it is possible to shorten commands, as long as they are unambiguous. hv offers as much syntatic sweetness as I could figure out ;)

More verbose help:

`hv list`	list all available virtual machines.
`hv running`	list running virtual machines.
`hv boot domain-name`	start given virtual machine.
`hv bootp domain-name`	start given virtual machine in a paused state.
`hv reboot domain-name`	warm reboot virtual machine.
`hv kill domain-name`	power off virtual machine.
`hv vnc domain-name`	connect vncviewer with virtual machine (and keep it turning on when it disconnects - in case of resolution change etc, ^C breakable).
`hv vncinfo domain-name`	return vnc listen ip:port.
`hv serial domain-name`	connect to serial tty of virtual machine.
`hv monitor domain-name qemu-monitor-command`	kvm monitor interface - if no monitor-command is given, issuing help command will be suggested.
`hv pause domain-name`	freeze virtual machine.
`hv cont domain-name`	continue feezed virtual machine.
`hv status domain-name`	virtual machine status.
`hv edit domain-name`	same as vim /etc/kvm/domain/name.
`hv snapshots domain-name`	list current virtual-machine snapshots.
`hv savevm domain-name snapshot-name`	create snapshot for virtual-machine
`hv loadvm domain-name snapshot-name`	apply (load) snapshot for virtual-machine
`hv delvm domain-name snapshot-name`	delete snapshot
`hv bootvm domain-name snapshot-name`	boot domain and apply (load) snapshot
`hv shutdown domain-name`	gracefully shutdown virtual machine (requires acpid / acpi support on virtual-machine).
`hv verify`	simplifies manual check of all virtual machines configuration - disks, mac addresses, tap interfaces and vnc ports.
`hv example-domain`	bootstrap configuration of virtual machine, eg. hv example >/etc/kvm/domain/new-vm

Feedback?

As usual - every opinion and suggestion is appreciated!

Upgrading Postgres 8.4 to 9.0 - Ubuntu / GNU Debian step by step

If your safe-upgrade just upgraded your postgresql to 9.0, your databases probably does not work and there is no postgresql-8.4 in your system. It is an unfortunate behavior, but we can simply migrate our data to new Postgres, here is how:

First, install your old postgres 8.4:

sudo -i
aptitude install postgresql-8.4

Stop all services using postgres databases and make backup of all databases and roles:

su - postgres
pg_dumpall >migration
exit

Make copy of your data (just to be sure you have backup):

cp ~postgres/migration /root/

Purge old Postgres 8.4 and install Postgres 9.0:

aptitude purge postgresql-8.4
aptitude install postgresql-9.0

Get all of your data back to Postgres:

su - postgres
psql <migration
exit

Turn your services back online, check if everything works again - if it does, remove migration data from postgres user and make a bz2 compressed archive containing all databases and roles:

rm ~postgres/migration
bzip2 migration
mv migration.bz2 `date +%Y-%m-%d`-migration.bz2

Keep that backup for a while until you will be sure that is safe to delete it.

Real-time graph using Javascript

Post below describes solution which requires Raphaël JS to draw graph using SVG.
I've also made a cleaner version - html5 canvas real-time graph - enjoy!

This idea was wandering through my mind for quite some time now, and finally I got a project I'm working on where it will be perfect to fit. That is why I decided to try to put this idea to the test. Suprisingly, it works :)

I was never a fan of Flash technology which would solve real-time graphing very fast so I had to pick Javascript libraries that will do the work. One of them is jQuery, because I like to have clean code. The other is Raphaël JS - Javascript library which aims allowing web developers to draw using SVG.

So, here is the effect of "step" scrolling:

See a working example.

Full sources are published and dual licensed under MIT and GPL. The above example is called step.html.

As an alternative, there is also "smooth" scrolling example included, but unfortunately - it makes web browser to consume 100% time of one CPU core. It might not be a problem on a desktop computer, but some older laptops will most likely "feel" the issue. If you know how to solve the problem - please let me know.

The actual source code consists of three parts:

animatedGraph jQuery plugin

// animatedGraph jQuery plugin, requires jQuery and Raphael libraries
(function($) {
    $.fn.extend({
        animatedGraph: function(width, height, steps, max, single, many, ms) {
            var path = function(array, step, left) {
                var scale = height/max,
                    path = '',          
                    x=-left,            
                    y=0;                
                for (var i=0;i<array.length;i++) {
                    if (i) {            
                        path += "S" + [(x += step), (y = height - array[i]*scale), x, y];
                    } else {            
                        path += "M" + [-10, (y = height - array[i]*scale)]; 
                        path += "C" + [0, (y = height - array[i]*scale)]; 
                    }                   
                }               
                return path+'L'+(array.length*step+10)+','+(height+10)+' '+(-10)+','+(height+10)+'+z';
            };          
            return this.each(function() {
                var self = this;
                self.step = width/(steps-1);
                $(self)         
                    .css('width',width+'px')
                    .css('height',height+'px');
                self.r = Raphael(self, width, height); 
                self.data = many(steps);
                self.path = self.r.path(path(self.data, self.step, 0))
                    .attr({             
                        stroke: '#66f',         
                        fill: '#ddf',           
                        'stroke-width': 2,      
                        'clip-rect': ''+[0,0,width-1,height]
                    });                 
                setInterval(function() {
                    self.data.shift();  
                    self.data.push(single());
                    self.path.animate({path:path(self.data, self.step, 0)});
                }, ms);         
            });         
        }       
    }); 
})(jQuery);

Pseudo-random source data generator

// rotating across pseudo-random values array
var Rotator = function() {
    var self = this;
    self.data = [ 
        16,23,41,38,45,31,29,22,27,22,25,29,32,45,88,30,
        26,22,45,23,36,48,99,22,32,26,22,27,5,0,0,3,15
    ];  
    self.index = 0;
    self.single = function() {
        self.index += 1;
        if (self.index > self.data.length) {
            self.index = 1;
        }       
        return self.data[self.index-1];
    };  
    self.many = function(length) {
        var results = [];
        for (var i=0;i<length;i++) {
            results.push(self.single());
        }       
        return results;
    };  
};

The main function

$(document).ready(function() {

    var self = this;

    $('#animatedgraph-demo')
        .replaceWith(function() {
            return $('<div id="animatedgraph-demo"></div>');
        });     

    var rotator = new Rotator();

    $('#animatedgraph-demo')
        .animatedGraph(200, 50, 20, 100, rotator.single, rotator.many, 1000);

});

Django application development

Most developers work on their startups far away from the daylight - they keep it in complete secret till the end. I've decided to do something new - I have started a project which is available to preview during the whole development. It is available at urevs.com. I also share all the project info on dedicated blog: blog.urevs.com.

This project consists currently of authentication framework and a lot of backend solutions. Site allows users to login using OpenID's, Google, Twitter, Facebook or Yahoo Accounts. Full details are available here - social bootstrap engine for Django.

I'd be very glad to hear from You, about every suggestion You have.

urevs.com has been dropped. I've merged its functionality into this blog site but it is not currently available to public.

Ubuntu: which package a file belongs to?

Some of us dont know this command, so:

nme@werewolf ~ $ dpkg -S /usr/bin/ffmpeg 
ffmpeg: /usr/bin/ffmpeg
nme@werewolf ~ $

nme@werewolf ~ $ dpkg -S `which gcc`
gcc: /usr/bin/gcc
nme@werewolf ~ $

nme@werewolf ~ $ dpkg -S *bash 
xz-utils: /usr/share/doc/xz-utils/extra/7z2lzma/7z2lzma.bash
bash: /usr/share/menu/bash
util-linux: /usr/share/doc/util-linux/examples/getopt-test.bash
util-linux: /usr/share/doc/util-linux/examples/getopt-parse.bash
bash: /bin/bash
apparmor: /etc/apparmor.d/abstractions/bash
bash: /bin/rbash
bash-completion, bash: /usr/share/doc/bash
global: /usr/bin/globash
nme@werewolf ~ $

You know smarter way? Add Your comment!

IPv4 CIDR to netmask in Python

I needed small function to validate IPv4 netmasks, havent found one, so I wrote my own. I've decided to use IPv4 CIDR notation to get corresponding netmasks. Here is the code:

def ipv4_cidr_to_netmask(bits):

    """ Convert CIDR bits to netmask """

    netmask = ''
    for i in range(4):
        if i:
            netmask += '.'
        if bits >= 8:
            netmask += '%d' % (2**8-1)
            bits -= 8
        else:
            netmask += '%d' % (256-2**(8-bits))
            bits = 0
    return netmask

Example usage is presented below. They also show how lambda mappings are useful in regular, daily use.

List all possible netmasks:

for netmask in map(lambda x: ipv4_cidr_to_netmask(x), range(0,33)):
    print netmask

0.0.0.0
128.0.0.0
192.0.0.0
224.0.0.0
240.0.0.0
248.0.0.0
252.0.0.0
254.0.0.0
255.0.0.0
255.128.0.0
255.192.0.0
255.224.0.0
255.240.0.0
255.248.0.0
255.252.0.0
255.254.0.0
255.255.0.0
255.255.128.0
255.255.192.0
255.255.224.0
255.255.240.0
255.255.248.0
255.255.252.0
255.255.254.0
255.255.255.0
255.255.255.128
255.255.255.192
255.255.255.224
255.255.255.240
255.255.255.248
255.255.255.252
255.255.255.254
255.255.255.255

Get netmask for /24:

print ipv4_cidr_to_netmask(24)

255.255.255.0

Is 255.255.254.0 valid IPv4 netmask?

print "255.255.254.0" in map(lambda x: ipv4_cidr_to_netmask(x), range(0,33))

True

Reverse search - get CIDR for 255.255.192.0 netmask

print map(lambda x: ipv4_cidr_to_netmask(x), range(0,33)).index('255.255.254.0')

23

As usual - enjoy :)

Simple Wordpress plugin debugging

I'd like to present simplest way I know to debug Wordpress plugins. It is a method I was using developing plugin to cloak URL's described earlier.

Below You can find the most minimalistic example plugin code possible - all it does is log REQUEST_URI to our debug file located at wp-content/plugins/debug. Row by row. Dirty and simple - and the most important - it does not require us to switch our Wordpress to debug mode.

Plugin consists of three parts - plugin description header, debugging function and hook to main function which is executed right before page is rendered.

<?php
/*
Plugin Name: test
Plugin URI:
Description:
Version:
Author:
Author URI:
*/

function debug($msg)
{
        $fp = fopen('wp-content/plugins/debug', 'a');
        fwrite($fp, $msg."\n");
        fclose($fp);
}

function main()
{
        debug($_SERVER['REQUEST_URI']);
}

add_action('init', 'main');

?>

Of course the most important part of the code is debug function which stores results we need to verify to our debug file.

jQuery-UI themeselect widget published

I've recently published jQuery-UI themeselect widget allowing users to add friendly looking, themable dropdown to instant theme switch.

In oppose to standard jQuery-UI themeswitcher, this one looks exacly like the rest of the theme.

My ui-themeselect has been published on New BSD License making it freely available and the code is hosted at Google Code. If You are familiar with Mercurial, you can clone the repository using command below:

hg clone https://ui-themeselect.googlecode.com/hg/ ui-themeselect

I have ripped the code out from my own Django / Google App Engine applications development Javascript bootstrap, because I think that someone else might consider it useful.

As an example how can it look inside jQuery-UI themed interface, take a look at this screenshot taken from one of my apps:

Enjoy! :)

Wordpress plugin to redirect or cloak URL's

After few unsuccessful searches for free Wordpress plugin that would allow me to cloak URL's on my blog - I decided to write my own. It was not really hard. I'm publishing it on GPLv3 license.

Plugin is really simple - full code consists of 75 lines of code. It does not contain database support, hits counting nor wp-admin interface. If some of You will like to extend it including those features - please do it and inform me - I will be first betatester :)

Plugin assumes that You already accomplished redirecting URL's to Wordpress subsystem that You would like to be handled by Wordpress in a different way.

There are three possible kinds of URL threating:

Permanent redirect - typical HTTP 301 redirect
Temporary redirect - typical HTTP 302 redirect
Local cloak - it works only on Your blog web page (thats why it is called local) - it will hide real destination URL leaving it cloaked

How does it work?

The core of the plugin source is pasted below. Maybe some of You will consider it as interesting :) The most unfortunate part of the plugin is need to use /wp-admin/plugins.php Edit plugin feature to manually change URL's. Those are listed inside $redirect variable.

define('PERMANENT_REDIRECT',301);
define('TEMPORARY_REDIRECT',302);
define('LOCAL_CLOAK',1);

$redirects = array(
  # source match         TYPE                           destination url
  # /u
  "^/u$" =>              array(TEMPORARY_REDIRECT,      "/u/"),
  "^/u/(.*)$" =>         array(LOCAL_CLOAK,             "/category/u/$1"),
}

function url_cloaker()
{
    global $redirects;
    $src_url = $_SERVER['REQUEST_URI'];
    foreach ($redirects as $src => $dst) {
        $src = str_replace('/','\/',$src);
        if (preg_match('/'.$src.'/',$src_url)) {
            #__d('Matched: '.$src);
            $dst_url = preg_replace('/'.$src.'/', $dst[1], $src_url);
            switch ($dst[0]) {
                case PERMANENT_REDIRECT:
                    wp_redirect($dst_url, PERMANENT_REDIRECT);
                    exit;
                case TEMPORARY_REDIRECT:
                    wp_redirect($dst_url, TEMPORARY_REDIRECT);
                    exit;
                case LOCAL_CLOAK:
                    $_SERVER['REQUEST_URI'] = $dst_url;
                    break;
            }
        }
    }
}

add_action('init', 'url_cloaker');

Plugin use actions hooking framework inside Wordpress to make sure it is fired before page is shown.

The plugin is available here for download.

How to add IMQ patch to Ubuntu 10.04 Lucid Lynx

Those of you, who are having problems adding IMQ support to recent Ubuntu/Debian release, might consider this blog entry useful :)

I will not discuss what IMQ is and wheater is it good or bad - I will describe how to prepare kernel, iptables packages and how to deploy them to our Debian/Ubuntu system including taking care of making them immune to accidential system upgrade packages replacement.

Kernel package with IMQ support

Before we start - we need few packages that will might be required:

sudo aptitude install fakeroot build-essential kernel-package ncurses-dev

Now we are ready to start: first, we need our current kernel with distro patches. For desktop instance I would use -generic instead of -server:

apt-get source linux-image-2.6.32-22-server Now, we download imq patch, apply it and compile kernel:

wget http://linuximq.net/patchs/linux-2.6.32-imq-test2.diff

cd linux-2.6.32/

patch -p1 < ../linux-2.6.32-imq-test2.diff

fakeroot time make-kpkg --initrd --append_to_version=imq linux-image

During the compilation process, make-kpkg script will discover unanswered features that we need to check as modules:

  "IMQ" target support (NETFILTER_XT_TARGET_IMQ) [N/m/?] (NEW) m

  IMQ (intermediate queueing device) support (IMQ) [M/y/?] (NEW) M

    IMQ behavior (PRE/POSTROUTING)

      1. IMQ AA (IMQ_BEHAVIOR_AA) (NEW)

    > 2. IMQ AB (IMQ_BEHAVIOR_AB) (NEW)

      3. IMQ BA (IMQ_BEHAVIOR_BA) (NEW)

      4. IMQ BB (IMQ_BEHAVIOR_BB) (NEW)

    choice[1-4?]: 2

    Number of IMQ devices (IMQ_NUM_DEVS) [16] (NEW)

Finally... in parent directory we will find:

linux-image-2.6.32.11+drm33.2imq_2.6.32.11+drm33.2imq-10.00.Custom_i386.deb

If everything went fine and our package is present - we can clean up compiled object files that will not be required anymore, recovering few GB of hard disk space:

./debian/rules clean

in case You forget to generate initrd file, it always might be generated by hand:

sudo -i

cd /boot/

mkinitramfs-kpkg -o initrd.img-2.6.32.11+drm33.2imq 2.6.32.11+drm33.2imq

update-grub

Iptables package with IMQ support

First, the patch - there is no iptables-1.4.4 imq patch available on linuximq.net I'm afraid. I have used iptables-1.4.6-imq patch and fixed it to compile with iptables-1.4.4.

Original patch can be found here.

Changes:

diff -Naurw iptables-1.4.6-imq.diff iptables-1.4.4-imq.diff
--- iptables-1.4.6-imq.diff	2010-01-27 11:53:22.000000000 +0100
+++ iptables-1.4.4-imq.diff	2010-05-08 13:18:21.000000000 +0200
@@ -43,7 +43,7 @@
 +
 +	switch(c) {
 +	case '1':
-+		if (xtables_check_inverse(optarg, &invert, NULL, 0, argv))
++		if (xtables_check_inverse(optarg, &invert, 0, argv))
 +			xtables_error(PARAMETER_PROBLEM,
 +				   "Unexpected `!' after --todev");
 +		mr->todev=atoi(optarg);

Ready to use patch can be downloaded from here using command below:

wget http://nme.pl/pub/patches/iptables-1.4.4-imq.diff

Ok, now since we got patch ready, we can download iptables sources and compile our deb package:

apt-get source iptables

cd iptables-1.4.4

~~cp ../iptables-1.4.4-imq.diff debian/patch/1009-iptables-1.4.4-imq.diff echo "1009-iptables-1.4.4-imq.diff" >>debian/patch/series~~

patch -p0 < ../iptables-1.4.4-imq.diff

Following will be required:

sudo aptitude install debhelper quilt autoconf automake linuxdoc-tools libtool

dpkg-buildpackage -rfakeroot -uc -b

In case You are recompiling for some reason, one of the distro patches might fail - in this case edit debian/patches/series using Your favourite editor and comment out the following patch:

0902-docs-version-reference.diff    ->    #0902-docs-version-reference.diff

Operation above might be archived by in place edition of debian/patch/series using command below:

sed -i 's/^0902/#0902/' debian/patch/series

When compilation ends, You should get two packages in parent directory: iptables and iptables-dev.

Installation & freezing our changes

Now we can install our packages:

dpkg -i *.deb

It will install following packages:

iptables_1.4.4-2ubuntu2_i386.deb
linux-image-2.6.32.11+drm33.2imq_2.6.32.11+drm33.2imq-10.00.Custom_i386.deb
iptables-dev_1.4.4-2ubuntu2_i386.deb

You might also consider holding packages to be sure that they will not be replaced during standard regular-basis upgrade:

aptitude hold linux-image iptables iptables-dev

IMQ development, status and replacement discussion

It is not true that recent IMQ patches are not stable as I have read on some web pages. Since Jussi joined the IMQ team, problems I have had with 2.6.18-24 kernels have gone to past.

Kernel 2.6.28.9 with iptables 1.4.0 works perfectly stable taking care of huge loads of network traffic. I think that current patch described above will work the same (im making before-production tests currently and it seems to work fine).

On the other hand - IFB - which is meant as replacement for IMQ - as for me - it does not offer the same functionality for bridge environment I need... Of course - I might be mistaken. Thats why - if anyone of You have replaced IMQ with IFB for bridge devices with ingress and egress traffic shaping - I'm very interested in the solution.