loutontheweb.co.cc: Latest entries

Python tip

Sun, 31 Jul 2011 02:58:37 -0500

Instead of:

if one == None or two == None or three == None:
    pass

use

if None in (one, two, three):
    pass

if any([x == None for x in one, two, three]):
    pass

Django nonrelational batch support

Tue, 05 Apr 2011 10:19:54 -0500

I have implemented custom backend for Django, that let you use batch operations with Django and Google App Engine. You should be aware that API is not stable yet and I did not get official approval from django-nonrel team yet (although there is some progress).

Get started

Get all needed libraries:

hg clone https://bitbucket.org/wkornewald/django-testapp batch-save-test
hg clone https://bitbucket.org/wkornewald/django-nonrel django-nonrel-bs
hg clone https://bitbucket.org/wkornewald/djangoappengine djangoappengine-bs
hg clone https://bitbucket.org/wkornewald/djangotoolbox djangotoolbox-bs
hg clone https://bitbucket.org/wkornewald/django-dbindexer django-dbindexer-bs

Apply patches:

hg -R django-nonrel-bs pull -u https://bitbucket.org/vladimir_webdev/django-nonrel
hg -R djangoappengine-bs pull -u https://bitbucket.org/vladimir_webdev/ djangoappengine
hg -R django-dbindexer-bs pull -u https://bitbucket.org/vladimir_webdev/django-dbindexer

Create symbolic links to fetched libraries:

cd batch-save-test
ln -s ../django-nonrel-bs/django
ln -s ../djangoappengine-bs djangoappengine
ln -s ../djangotoolbox-bs/djangotoolbox
ln -s ../django-dbindexer-bs/dbindexer

Try to run server:

./manage.py runserver

Now you should be able to use batch operations with Django.

Usage

The simplest example of using batch operations looks like this:

from __future__ import with_statement
from django.db.models import BatchOperation
with BatchOperation() as op:
    for i in range(100):
        op.save(Post(title='Title %d' % i, text='Text %d' % i))

That's it. Internally code above will create two pools: one for save operations and one for delete operations. When pool is filled entries will be flushed to backend and backend is responsible to batch save them. Available configuration options are:

pool_size (save_pool_size, delete_pool_size) - number of models instances stored in pool (500 by default). If you experience problems with memory you can try to lower this value.
batch_size (save_batch_size, delete_batch_size) - number of models instances that will be flushed in one batch operation (100 by default). If you experience problems with datastore timeout exceptions you can try to lower this value.

We need to make differences between pool_size and batch_size, because theoretically Django can be configured to use several databases. So pool can contain instances for different databases.

You can configure BatchOperation() like this:

config = dict(default=dict(pool_size=50,
                           save_batch_size=50,
                           delete_pool_size=50))
with BatchOperation(config) as op:
    for p in Post.objects.all()[:100]:
        op.delete(p)

Stats

I have tested batch saves with such views:

def plain_save(request):
    for i in range(100):
        Post.objects.create(title='Title %d' % i, text='Text %d' % i)
    return http.HttpResponse('Ok')

def batch_save(request):
    with BatchOperation() as op:
        for i in range(100):
            op.save(Post(title='Title %d' % i, text='Text %d' % i))
    return http.HttpResponse('Ok')

and got following results from appstats:

"GET /plain_save/" 200 real=2734ms cpu=1720ms api=6583ms overhead=21ms (100 RPCs)
"GET /batch_save/" 200 real=609ms cpu=193ms api=6516ms overhead=0ms (1 RPC)

Feedback

Feel free to add feedback/report bugs at django-nonrel user group.

Links for Jan. 9, 2011

Sun, 09 Jan 2011 11:25:11 -0600

Weekly links about django

DjangoWeekly is a free newsletter, keeping you you up-to-date with what’s happening in Django.
New App Engine Datastore API

Guido Van Rossum presents new asynchronous App Engine Datastore API, which is based on Python generators and "trampoline" framework.
Class Based Views and Dry Ravioli

Django's new class based views are very cool. I am starting to clean up an existing project using them, and lots of existing views are turning into declarative code. But it makes me worried about the ravioli effect.

Links for Nov. 11, 2010

Thu, 11 Nov 2010 07:50:56 -0600

Google Python Style Guide

This style guide contains many details that are initially hidden from view. They are marked by the triangle icon, which you see here on your left. Click it now. You should see "Hooray" appear below.
A Globals Class pattern for Python

A series of articles: A Globals Class pattern for Python, A Globals Module pattern, An Arguments Container pattern.
Bob: a Scheme interpreter, compiler, and VM in Python

A Scheme interpreter, compiler, and VM written in pure Python.
Working with Excel Files in Python

Python-excel.org seems to be down, but http://sites.google.com/site/pythonexcel/ is still alive. Hooray!

Links for Nov. 2, 2010

Tue, 02 Nov 2010 07:05:52 -0500

Python Module of the Week

PyMOTW is a series of blog posts written by Doug Hellmann. It was started as a way to build the habit of writing something on a regular basis. The focus of the series is building a set of example code for the modules in the Python standard library.
Python Exception Handling Techniques

Error reporting and processing through exceptions is one of Python’s key features. Care must be taken when handling exceptions to ensure proper application cleanup while maintaining useful error reporting.
Google App Engine: Best practices

Best practices for writing scalable applications for Google App Engine.
Configuring logging for Web applications

You can easily configure your web applications with Python logging, so that logged messages can go into web-application-specific log files (or other destinations) even for multiple web applications in the same Python process, and have those logs contain request-specific information such as remote user IP address, request method and others.
Python: Singleton pattern

The right way to implement Singleton pattern.

Links for Oct. 27, 2010

Wed, 27 Oct 2010 14:31:56 -0500

Django on Google App Engine via SQL (not nonrel)

There are good chances that Django 1.3 will be fully supported on Google App Engine.
Bleach, HTML sanitizer and auto-linker

Bleach is a whitelist-based HTML sanitizer and auto-linker in Python, built on html5lib, for AMO and SUMO and released under the BSD license.
Old and New Python Versions

Old and new versions of Python, forward- and back-ported from other releases of Ubuntu and Debian.
MySQL subqueries limitations

MySQL subqueries don't work as expected, use joins instead.
When App Engine went wrong

App Engine fail story.

Links for Oct. 21, 2010

Thu, 21 Oct 2010 16:21:31 -0500

Building Python extension with mingw and cygwin for Windows

Building Python extension with mingw and cygwin for Windows.
Monty Hall Problem

Monty Hall Problem coded in Python. So easy to be coded and so hard to be understood on intuitive level...
CSS: px vs em vs % vs pt

When using a value, you need to declare a unit of measure which itself has four options. Which is best?
Steps of creating a web app

Steps of creating a web app including modeling, sketching, HTML, Photoshop explorations and moving from static mockups to live running code.
Python, str, unicode, bytes and UnicodeDecodeError

Great article explaining differences between Python str and unicode types.

Split django settings

Thu, 23 Sep 2010 10:16:11 -0500

Some points

I don't see any sense "to move sensitive or machine/user specific settings like database passwords and such out of the main settings.py file". What's the point to share incomplete settings?
I want to be able to upload my application to production without modifying any lines of the code.
Host based solutions don't work with SSH / Command Prompt.

Let's start

settings.py - production version;
dev_settings.py - development version.

At the top of dev_settings.py:

from settings import *

# here we can redefine our settings for development machine

We also have to slightly modify manage.py file:

if os.environ.get('DEVELOPMENT', None):
    import dev_settings as settings
else:
    import settings

Finally we have to create environment variable DEVELOPMENT on development machine.

I think it is the easiest way to have different settings for development, staging, testing and production.