gremu.net - Gregor Müllegger's Weblog

JPype - Bridging the worlds of Java and Python

Gregor Muellegger — Thu, 22 Jul 2010 19:54:07 +0100

JPype - Bridging the worlds of Java and Python

JPype is an effort to allow python programs full access to java class libraries. This is achieved not through re-implementing Python, as Jython/JPython has done, but rather through interfacing at the native level in both Virtual Machines.

Eventually, it should be possible to replace Java with python in many, though not all, situations. JSP, Servlets, RMI servers and IDE plugins are good candidates.

(via @ianbicking)

Using ipython's debugger as pdb replacement.

Gregor Muellegger — Thu, 22 Jul 2010 19:47:54 +0100

Do you know how to debug your python code? I usually do the following in my code to set something like a hardcoded breakpoint:

import pdb; pdb.set_trace();

This will bring up a pdb prompt when the code execution reaches this line. But pdb is a bit limited since it doesn't support code completion, syntax highlighting and so on.

So here is how you make your debug prompt more fancy by using ipython's debugger.

some ipython love for your debugging experience

Like always, make sure first that all dependecies are installed. In this case its ipython ¹. Install it with pip install ipython or easy_install ipython.

Now put the following file in ipdb.py somewhere in your PYTHONPATH (for me its ~/.local/lib/python2.6/dist-packages/ipdb.py):

import sys
from IPython.Debugger import Pdb
from IPython.Shell import IPShell
from IPython import ipapi

shell = IPShell(argv=[''])

def set_trace():
    ip = ipapi.get()
    def_colors = ip.options.colors
    Pdb(def_colors).set_trace(sys._getframe().f_back)

Know you can use in your code the following snippet:

import ipdb; ipdb.set_trace();

This will bring up ipython's debugger instead of the old pdb prompt. Test it out! It's so cool to have code completion while debugging.

¹ In case you don't know what ipython is, check their out homepage and try to run ipython in your console after installing it. It's definitly worth exploring. (For those who really don't know what it is: its a fancy, more feature rich python console).

Update

As Andrey mentioned in the comments: there is an easier way now to setup ipdb in your PYTHONPATH. Just install the ipdb package from the PyPI: pip install ipdb

Websocket support for django!

Gregor Muellegger — Tue, 20 Jul 2010 22:50:36 +0100

You know what WebSockets are? It's a new web protocol for bidirectional connections between the browser and your application server. This is a huge win over AJAX and plain HTTP since you don't need to establish a new connection for every tiny message.

Like TCP socket but for the web.

It's an exciting new technology - so I started experimenting with WebSockets and with my favourite webframework Django. Here is what I came up with, bundled into a reusable app called django-websocket.

What can I do with it?

You usually want to use WebSockets for single URLs - so the common design for making single views more powerful is by wrapping them with a decorator. Thats exactly how django-websocket will upgrade your views to accept websocket connections.

from django_websocket import require_websocket

# the decorator rejects all requests that aren't websockets
@require_websocket
def echo(request):
    while True:
        # wait() will return new messages as they arrive
        message = request.websocket.wait()
        # wait() returns None if the connection was closed by the client
        if message is None:
            return
        # simple method for sending messages
        request.websocket.send(message)

Above is a very basic example of an echo server that sends every message it receives back to the client.

You see its very easy to make websockets available. The decorator will attach a simple websocket-API-object to the request object. It has some primitive methods you can easily work with. Two of them are already used in the example above.

Another is for example request.websocket.read(). Unlike wait(), read() will not block your code if no messages are ready to be processed. It just returns None if new messages are received from the client.

Have a look into the documention to get an idea of the other methods that are available or to see more examples.

Explore

The project is still very experimental and has not been proved to be production ready. But its a lot of fun to work with. Dive into it and get creative!

The project and documentation on PyPI:
http://pypi.python.org/pypi/django-websocket
Code on github:
http://github.com/gregor-muellegger/django-websocket

I would really appreciate any feedback you can give. Flame, rant or make suggestions for improvements.

I'm especially interested in when you try to build something serious with django-websocket. Let me know about your awesome ideas and how it will make the django ecosystem even more colorful.

Enable setup.py test in your Django apps

Gregor Muellegger — Sat, 17 Jul 2010 14:07:50 +0100

Erich Holscher has posted a blog entry with the same title about a year ago. This helped me out quite a lot to get python setup.py test working with a reusable app.

Unfortunatelly the code in his blog post is not working anymore exactly like this. Django has changed its internal structure of its default test runner. So here is how you can get setup.py test working today.

Let's assume you have a repository layout for your app like this:

|-\ django_app/
| |- __init__.py
| |- models.py
| |- etc ...
|
|-\ django_app_tests/
| |-\ test_app
| | |- models.py
| | |- etc ...
| |- manage.py
| |- settings.py
| |- etc ...
|
|- setup.py

You would usually run your tests with:

cd django_app_tests
python manage.py test test_app

Right? Now lets create a new file django_app_tests/runtests.py:

# This file mainly exists to allow python setup.py test to work.
import os, sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_app_tests.settings'
test_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, test_dir)

from django.test.utils import get_runner
from django.conf import settings

def runtests():
    TestRunner = get_runner(settings)
    test_runner = TestRunner(verbosity=1, interactive=True)
    failures = test_runner.run_tests(['test_app'])
    sys.exit(bool(failures))

if __name__ == '__main__':
    runtests()

This will make running tests a bit easier:

cd django_app_tests
python runtests.py

Now lets tell setuptools what it should do when you call setup.py test. Go to your setup.py file and add test_suite='django_app_tests.runtests.runtests to the setup() function.

Thats it -- now you can run your app tests with python setup.py test.

Django admin with read-only permission

Gregor Muellegger — Thu, 01 Jul 2010 22:37:47 +0100

I work as a freelancer and do many django related projects. My clients care in most cases very much about django's admin site and want to have it customized in many different ways.

One client even has a django project that uses the admin site as only frontend. It's an intranet CRM system - but they need to assigned every employee change permissions because every user must be able to view every entry.

Since django doesn't support read permissions in the admin this was the only way to do it. Here is a quick-and-dirty fix without creating extra views for readonly pages, extra templates etc.

The basic idea is to reuse the change_view and changelist_view pages and just deny any POST requests for users without change permissions.

from django.core.exceptions import PermissionDenied from django.contrib import admin from crm.models import Customer class ReadPermissionModelAdmin(admin.ModelAdmin): def has_change_permission(self, request, obj=None): if getattr(request, 'readonly', False): return True return super(ReadPermissionModelAdmin, self).has_change_permission(request, obj) def changelist_view(self, request, extra_context=None): try: return super(ReadPermissionModelAdmin, self).changelist_view( request, extra_context=extra_context) except PermissionDenied: pass if request.method == 'POST': raise PermissionDenied request.readonly = True return super(ReadPermissionModelAdmin, self).changelist_view( request, extra_context=extra_context) def change_view(self, request, object_id, extra_context=None): try: return super(ReadPermissionModelAdmin, self).change_view( request, object_id, extra_context=extra_context) except PermissionDenied: pass if request.method == 'POST': raise PermissionDenied request.readonly = True return super(ReadPermissionModelAdmin, self).change_view( request, object_id, extra_context=extra_context) class CustomerAdmin(ReadPermissionModelAdmin): pass admin.site.register(Customer, CustomerAdmin)

The actual implementation does the following if a user tries to access the change or changelist pages:

Call the original change view from django - everything works fine if the user has change permissions.
If the permission is denied we check that no POST request is performed, otherwise we reject the request.
We set a readonly flag on the request to indicate that we have readonly access, this flag is used by has_change_permission to grant permission for this request. Note: this doesn't mean the user can change data now - we already have prevented this since we deny POST requests.

At the end some pro and contra:

Pro

Very easy implementation since nearly no custom code needs to be written.
Should work with most customizations you have already made to your ModelAdmin subclasses.

Contra

The links from the admin index page to the model's changelist doesn't show up. This was no problem in my case since the client does not use the default index page.
The user can still edit the fields in the change form since they are not marked as "readonly" or anything else. The page just shows a not very userfriendly Permission Denied after he tries to save.

Feel free to write any thoughts about readonly possibilities in django's admin page in the comments. Every critic about my (very simplistic) approach is welcome.

Howto reset your ASUS EEE 1101HA

Gregor Muellegger — Thu, 17 Jun 2010 18:18:14 +0100

I recently bought a cute little ASUS EEE 1101HA. The first thing I've done after receiving it was installing the latest Ubuntu version. So I hoped the best. But after trying some things and searching the Internet it seems that the Linux support for this netbook really sucks.

Display isn't working very well, multitouch touchpad doesn't work, microphone doesn't work properly and so on. I expected to have some small issues which I need to solve, but this was too much.

I decided to sell it again - but usually customers aren't that happy if you promise them a WinXP netbook but they receive one with a broken ubuntu ;-)

So I needed to do reinstall windows xp from the recovery DVD. This isn't as easy as it sounds, since the netbook has no optical drive.

I first expected to use my USB stick for the recovery as I used it to install Ubuntu. After following and testing some instructions from different EEE sites, I saw that this don't work for me.

For the rescue I found an old IDE DVD drive at home which I attached by an USB to IDE adapter to the netbook. This didn't worked either: the recovery DVD exited on boot with the message "NTLDR is corrupt. Cannot boot."! Isn't it nice of ASUS to provide such a good recovery experience (and no, ASUS don't provide instructions howto reinstall the netbook)?

After a lot of flaming, here is the way how it worked for me:

Insert the recovery DVD into another computer.
Copy the WINPE.ISO from the DVD to your harddisk.
Openup the WINPE.ISO file (I used MagicISO - a Windows program to modify ISO images) and change the AppPath value in I368\SYSTEM32\WINPESHL.INI to X:\GHOST32.exe. This is how the file looks for me now:
```
[LaunchApp]
AppPath = X:GHOST32.exe
[LaunchApps]
X:devcon.exe, disable usbstordisk
```
Burn the ISO.
Put the Recovery\EEEPC-GER.gho file from the recovery DVD on a USB memory stick that is large enough to handle it (the exact filename may vary, since it depends on your language).
Find a way to boot the burned ISO on your 1101HA (I used an old IDE CD drive with a USB to IDE adapter)
The Ghost tool pops up automatically. Select "Local > Disk > From image" in the menu.
Choose the *.gho file which you put on your USB stick earlier.
Reboot your EEE PC and see if everything is fine.

bye, bye little 1101HA. LLAP

Create patches for upstream with bzr

Gregor Muellegger — Wed, 21 Apr 2010 23:30:58 +0100

I searched a long time for a nice way to create clean diffs between two bzr branches. The command bzr diff remote-branch is not working ... Now I found a solution using a special revision identifier.

I found by accident in the bzr docs a page about some special revision identifiers. One of them is the branch: prefix which can be used to reference a remote branch.

With this knowledge its easy to create a diff between two bzr branches:

bzr diff -r branch:/path/to/remote-branch

You can also use instead of branch: the submit: identifier which only provides the diff back to the last common revision. This is exactly the behaviour you want to have for creating patches. You can also leave the argument for submit: blank and it will use the parent branch (the branch where you branched from) by default.

Here are now the steps I use when I want to provided a bug fix for an upstream project:

Branch the project's repository from launchpad (if the project is not available, I register it and import the upstream repository)
Then I branch the local copy of trunk to have a checkout dedicated for fixing the bug.
Hacking.
Run bzr diff -r submit: > ../bugfix.patch
Submit the patch to the developers.

Until today I was really disappointed by bzr for not having this feature. I was quite sure it MUST be there … but haven't found it. Ofcourse I know the bundle command etc. but this would create a merge directive which looks a bit like a patch but is actually only usefull if the upstream project also uses bzr.

Maybe this blog post helps someone else saving some time looking for this feature.

Compile csync from source on Ubuntu 9.10

Gregor Muellegger — Mon, 15 Mar 2010 13:49:53 +0100

Since csync is not available in the Ubuntu repositories and there are no .deb packages available on csync's website you have to compile it yourself from source. It's nearly straight forward but there were a few issues I ran into. Maybe this short walkthrough help you to make the installation of csync as easy as possible.

Note: I was a bit confused because there is a csync2 package for Ubuntu. But it seems that this is a totally different project and not related to csync.

resolving dependencies

csync has a few dependencies which are listed in the INSTALL file. They include:

CMake
check
log47
sqlite3
libiniparser
libsmbclient
libssh

So lets install these dependecies first:

sudo aptitude install cmake check liblog4c3 liblog4c-dev libsqlite3-0 libsqlite3-dev libsmbclient-dev libssh-dev

installing `libiniparser`

Only libiniparser is not available through aptitude so we must also download and compile it from source.

wget http://ndevilla.free.fr/iniparser/iniparser3.0b.tar.gz
tar xf iniparser3.0b.tar.gz
cd iniparser3.0b
make

These commands will download, extract and build the iniparser library. You will see that you now have the files libiniparser.a and libiniparser.so.0 in your current directory. Copy them to /usr/lib to make them available for other programs:

sudo cp libiniparser.* /usr/lib

download and configure `csync`

Now we can actually start to compile csync. Download your prefered release from the download page and extract it:

wget http://www.csync.org/files/csync-0.44.0.tar.gz
tar xf csync-0.44.0.tar.gz
cd csync-0.44.0

csync also wants us to build the project in a seperate directory. So lets create it:

mkdir build
cd build

Now we can run cmake to configure the environment. For me the first try raised an error which read like this:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files:
INIPARSER_INCLUDE_DIR

So we need to tell cmake which value the INIPARSER_INCLUDE_DIR option should have:

cmake -DINIPARSER_INCLUDE_DIR=/path/to/iniparser3.0b/src ..

Please note the .. at the end of the command. This tells cmake where to find the extracted source tarball.

build and install `csync`

Now we can try to build and install csync. On all Unix systems you usually do a make install which also should work with csync. But make install simply ignores Ubuntu's package system. We don't want that and use checkinstall instead to build and install a .deb package. So make sure checkinstall is available:

sudo aptitude install checkinstall

Before we compile the package we need to prevent an error which hit me on the first try. Copy the source files from the iniparser package to the build directory:

cp /path/to/iniparser3.0b/src/* .

Now you can run checkinstall from the build directory in which you configured csync:

sudo checkinstall

This will call a wizard which leads you through the process of creating the .deb package. Usually you can use the defaults to install the packages by hitting your return key several times. But since checkinstall uses the name of your current directory as package name and you don't want to have the package named build you need to change this default while going through checkinstall's wizard.

have fun!

Now everything should be installed and ready to use. I hope this helped you going through the few steps that are necessary to compile csync from source.

Please note that the urls I use with wget are just pointing to the current releases of csync and libiniparser. So maybe you want to check their websites before going through this tutorial if they have released newer versions.

Comment below if you have issues with this howto or suggestions how to make it better. Thanks!

A quote from Volker Dusch

Gregor Muellegger — Wed, 10 Mar 2010 11:29:44 +0100

Unittests are like a warm blanket you can wrap yourself into if you need to go out into the codeforest to refactor

by Volker Dusch

Test Driven Development - The Django Way

Gregor Muellegger — Wed, 10 Mar 2010 01:21:04 +0100

Test Driven Development - The Django Way

Nice little write up about test driven development with django. Nothing special or excitingly new but I wanted to post it because I followed nearly the same principles in my last big project.

It worked very nice and I will do it the same way with the next project.

gremu.net - Gregor Müllegger's Weblog

JPype - Bridging the worlds of Java and Python

Using ipython's debugger as pdb replacement.

some ipython love for your debugging experience

Update

Websocket support for django!

What can I do with it?

Explore

Enable setup.py test in your Django apps

Django admin with read-only permission

Howto reset your ASUS EEE 1101HA

Create patches for upstream with bzr

Compile csync from source on Ubuntu 9.10

resolving dependencies

installing libiniparser

download and configure csync

build and install csync

have fun!

A quote from Volker Dusch

Test Driven Development - The Django Way

installing `libiniparser`

download and configure `csync`

build and install `csync`