Bruno Carlin's Blog

Locking buzhug

2012-02-07T17:00:00+01:00

I have recently decided to work with Buzhug on a project. As far as I can tell, it has proven efficient, fast, easy to use and to maintain. However, I ran into a few gotchas.

Simple solutions are often the best

I came to use Buzhug for the following requirements:

I needed a single table
I did not want to add additional dependencies to the project
The size of the table will average 5K entries (without having more than 10k entries in peaks)

And an additional (personal) one:

I did not want to bother with SQL. Really not. no way!

That left me one option: pure-python embedded database.

After having considered a few libraries, I have been seduced by the way Buzhug interface is close to manipulating python objects. And the benchmarks seemed to show that it is performant enough for this project.

After a quick prototyping (1 day), the choice was done.

Then came a few weeks of development and the first stress tests...

And the real world came back fast

A few times a day, the application backed by this database is intensely used:

It can be run up to 50 times simultaneously in separate python process
Each run makes a read and a write/delete operation

This causes a race condition on the files used to store data, and concurent writes corrupts database.

Using buzhug.TS_Base instead of buzhug.Base did not solve anything, as the problem is not thread, but processes. What I need is a system-wide cross-process lock.

Here is the answer

First step was to find how to implement a cross-process, system-wide lock. As it only has to work on Linux, the Lock class given by Chris from Vmfarms fits perfectly. Here is a version slightly modified to make it a context manager :

import fcntl

class PsLock:
    """
    Taken from:
    http://blog.vmfarms.com/2011/03/cross-process-locking-and.html
    """
    def __init__(self, filename):
        self.filename = filename
        self.handle = open(filename, 'w')

    # Bitwise OR fcntl.LOCK_NB if you need a non-blocking lock 
    def acquire(self):
        fcntl.flock(self.handle, fcntl.LOCK_EX)

    def release(self):
        fcntl.flock(self.handle, fcntl.LOCK_UN)

    def __del__(self):
        self.handle.close()

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is None:
            pass
        self.release()

    def __enter__(self):
        self.acquire()

The second step is to define a new class that inheritates from buzhug.Base that uses PsLock (inspired by TS_Base):

import buzhug

_lock = PsLock("/tmp/buzhug.lck")

class PS_Base(buzhug.Base):

    def create(self,*args,**kw):
        with _lock:
            res = buzhug.Base.create(self,*args,**kw)
        return res

    def open(self,*args,**kw):
        with _lock:
            res = buzhug.Base.open(self,*args,**kw)
        return res

    def close(self,*args,**kw):
        with _lock:
            res = buzhug.Base.close(self,*args,**kw)
        return res

    def destroy(self,*args,**kw):
        with _lock:
            res = buzhug.Base.destroy(self,*args,**kw)
        return res

    def set_default(self,*args,**kw):
        with _lock:
            res = buzhug.Base.set_default(self,*args,**kw)
        return res

    def insert(self,*args,**kw):
        with _lock:
            res = buzhug.Base.insert(self,*args,**kw)
        return res

    def update(self,*args,**kw):
        with _lock:
            res = buzhug.Base.update(self,*args,**kw)
        return res

    def delete(self,*args,**kw):
        with _lock:
            res = buzhug.Base.delete(self,*args,**kw)
        return res

    def cleanup(self,*args,**kw):
        with _lock:
            res = buzhug.Base.cleanup(self,*args,**kw)
        return res

    def commit(self,*args,**kw):
        with _lock:
            res = buzhug.Base.commit(self,*args,**kw)
        return res

    def add_field(self,*args,**kw):
        with _lock:
            res = buzhug.Base.add_field(self,*args,**kw)
        return res

    def drop_field(self,*args,**kw):
        with _lock:
            res = buzhug.Base.drop_field(self,*args,**kw)
        return res

Now I just use

database = PS_Base( ... )

And all the errors have vanished.

Aptana Studio/Eclipse and Xulrunner

2011-12-16T17:00:00+01:00

Since a few months, I encountered an annoying error in Aptana Studio and Eclipse 3.7 (the autonomous packages, not the packages from the repositories) whenever I tried to do a git or hg action.

I could live without until now, but today, it was really bothering me.

The error is:

Unhandled event loop exception
No more handles [Unknown Mozilla path (MOZILLA_FIVE_HOME not set)]

The log file showed the following backtrace:

!ENTRY org.eclipse.ui 4 0 2011-12-16 17:17:30.825
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTError: No more handles [Unknown Mozilla path (MOZILLA_FIVE_HOME not set)]
    at org.eclipse.swt.SWT.error(SWT.java:4109)
    at org.eclipse.swt.browser.Mozilla.initMozilla(Mozilla.java:1739)
    at org.eclipse.swt.browser.Mozilla.create(Mozilla.java:656)
    at org.eclipse.swt.browser.Browser.<init>(Browser.java:119)
    at com.aptana.git.ui.internal.actions.CommitDialog.createDiffArea(CommitDialog.java:237)
    at com.aptana.git.ui.internal.actions.CommitDialog.createDialogArea(CommitDialog.java:158)

    [...]

    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:620)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1384)

To make it short, after having read a lot of posts about this problem, it seemed it was enough to give the path to Xulrunner to Aptana.

on my Arch Linux, it was

export MOZILLA_FIVE_HOME=/usr/lib/xulrunner-8.0

Trying to start Aptana Studio, I had a new error. It simply stated

XPCOM error -2147467261

The solution is that Aptana Studio cannot work with the version of Xulrunner in Arch LinuX repositories because it is too recent.

To solve this problem, I had to install xulrunner 1.9.2 from AUR:

yaourt -S xulrunner192

The PKGBUILD was broken this morning and ended in a 404 Error when fetching sources. If you have the same problem, here is an updated PKGBUILD

Finally, I put

-Dorg.eclipse.swt.browser.XULRunnerPath=/usr/lib/xulrunner-1.9.2

at the end of the AptanaStudio3.ini file in the Aptana Studio folder. For the package in the Arch Linux repositories, this file is /usr/share/aptana/AptanaStudio3.ini.

Build the latest PgPool-II on Debian Etch

2010-12-14T00:00:00+01:00

After having build PgPool-II on Red Hat Enterprise Linux 5.5 without any problem, I tried to build it on a fresh Debian Etch. The catch is that I did not want to install PostgreSQL 9.0, but just extract it from the binary packages provided by Entreprisedb (with option --extract-only 1).

Whatever options I passed to ./configure, it resulted in the same error:

checking for PQexecPrepared in -lpq... no
configure: error: libpq is not installed or libpq is old

Here is the answer: the binary package contains the libpq with the name libcrypto.so.0.9.8 (the RHEL name) when pgpool is looking libcrypto.so.6 on Debian. The same applies to libssl. So a simple

ln -s libcrypto.so.0.9.8 libcrypto.so.0.9.8
ln -s libssl.so.0.9.8 libssl.so.6

before your ./configure will solve it!

Setup nginx for Mediawiki

2010-09-15T00:00:00+02:00

Two weeks ago, I migrated a server from Apache/mod_php to nginx/php-fpm. Only today did i succeed to remove all side effects. The latest one:

Static files must not go through php-fpm, but a simple test on extensions is ineffective, as url like http://server/File:name_of_the_file.png must be processed by PHP.

Here is my final setup, that corrects all the errors I encountered:

server {
    listen 80;
    server_name server_name;
    index index.php;
    root /path/to/www/;

    # Serve static files with a far future expiration 
    # date for browser caches
    location ^~ /images/ {
        expires 1y;
    }
    location ^~ /skins/ {
        expires 1y;
    }

    # Pass the request to php-cgi
    location / {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_param  SCRIPT_FILENAME $document_root/index.php;
        fastcgi_index  index.php;
        include fastcgi_params;
    }
}