All Buttons Pressed

Goodbye

2011-11-24T12:24:00Z

Hi everyone,

unfortunately, Thomas and I have to say goodbye to our open-source community. We started contributing because we thought it was interesting and we had fun building something together. However, our projects have stopped being fun. We're not even using most of them, anymore (not even at our startup). This is why we've made a final decision: We've stopped maintaining our open-source projects. All of them.

This primarily affects djangoappengine users. The MongoDB backend has its own capable contributors, so django-mongodb-engine users should be unaffected by this. We're really sorry, especially to those who have a business that depends on djangoappengine. Thanks to everyone who has supported us.

If anyone is interested in continuing djangoappengine development, please send a mail to the Django-nonrel discussion group.

Best wishes,

Waldemar and Thomas

Update: Thanks for your warm response! We really didn't expect that. Wilfred Hughes offered to take over, so hopefully our projects won't die. By the way, I'll try to keep blogging about web development, NoSQL, and App Engine. There's a whole lot of stuff we've learned and keep learning at our startup and I'd like to share that with you from time to time.

Please also read the comments on Hacker News.

Leave a comment

Using dev_appserver with Python 2.7 on App Engine

2011-10-18T15:29:00Z

Starting with SDK 1.6.0 the App Engine fully supports Python 2.7. This article originally described a workaround for getting Python 2.7 running on SDK 1.5.5. Now it only describes how to port your app.yaml and request handlers to Python 2.7.

The following instructions work with any WSGI-compliant web framework. I'll explain how to use Python 2.7 with djangoappengine/Django-nonrel at the end of this post. Yes, Django-nonrel works with Python 2.7 on App Engine. This blog (All Buttons Pressed) is already running on Python 2.7 with threadsafe: yes.

Let's start with a simple app.yaml for Python 2.5:

application: yourappid
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: handler.py

The handler.py file looks like this:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

# This can be any WSGI handler. Here we just use webapp.
application = webapp.WSGIApplication(...)

def main():
    run_wsgi_app(application)

if __name__ == '__main__':
    main()

In order to use Python 2.7 you have to change your app.yaml like this:

application: yourappid
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /.*
  script: handler.application

Here we've changed the runtime to python27, added threadsafe: yes and modified the last line (script) to point to the WSGI handler directly instead of just the handler.py module.

Also, our handler.py can be simplified like this:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

application = webapp.WSGIApplication(...)

In other words, we've just removed main(). Note, webapp users would normally port their code to webapp2, but that's not really relevant for this post, so let's just ignore that detail.

That's all you need to use Python 2.7 on both the production server and dev_appserver.

Using djangoappengine with Python 2.7

If you're a Django-nonrel / djangoappengine user you can just change your app.yaml to look like this:

application: yourappid
version: 1
runtime: python27
api_version: 1
threadsafe: yes

builtins:
- remote_api: on

inbound_services:
- warmup

libraries:
- name: django
  version: latest

handlers:
- url: /_ah/queue/deferred
  script: djangoappengine.deferred.handler.application
  login: admin

- url: /_ah/stats/.*
  script: djangoappengine.appstats.application

- url: /.*
  script: djangoappengine.main.application

That's it. djangoappengine will take care of the rest. Note that the libraries section is currently necessary due to a bug in SDK 1.6.0. Since you're already deploying a custom django package it shouldn't be necessary to enable Django in the libraries section. The App Engine team will fix this bug in the next SDK release.

Happy porting!

Leave a comment

F() objects and QuerySet.update() support in djangoappengine

2011-07-07T14:55:00Z

As some of you already noticed, we added supported for QuerySet.update() and F() objects as part of some features we currently need for our startup pentotype. Finally, it's possible to use simple transactions on App Engine via Django-nonrel :) Let's see how it works.

Django's beauty

In order to show what we gain by using Django's QuerySet.update() method and F() objects let me illustrate the code differences between the old djangoappengine version and the new one using a simple transaction which increments the value of a counter by a specific amount. In both cases we have the following model definition:

# models.py
from django.db import models

class Accumulator(models.Model):
    counter = models.IntegerField()
    name = models.CharField(max_length=500)

The old version looks like this:

from google.appengine.ext import db

def increment_counter(pk, amount):
    obj = Accumulator.objects.get(pk=pk)
    obj.counter += amount
    obj.save()

db.run_in_transaction(increment_counter, 1, 5)

whereas the new one looks like this:

from django.db.models import F
Accumulator.objects.filter(pk=1).update(counter=F('counter') + 5)

Obviously, it's more elegant to update a counter using QuerySet.update() and F() objects because it's much clearer and we have to write less code, thus resulting in more productivity.

Additionally, it's possible to do more complicated transactions, for example, updating multiple entities using Django's elegant syntax:

Accumulator.objects.filter(name__startswith='simple').update(counter=F('counter') + 1)

Note, that this will update each entity in its own transaction. With the old version we would have to iterate over each entity explicitly and start the transaction:

counters =  Accumulator.objects.filter(name__startswith='simple')

for counter in counters:
   db.run_in_transaction(increment_counter, counter.pk, 5)

Again, we benefit from writing less code.

However, one of the main benefits of QuerySet.update() and F() objects is that the code becomes even more portable between backends than before. For example, the MongoDB backend also has support for F() objects and QuerySet.update(), so the same code can be used for transactional updates on App Engine and MongoDB. Moreover, the same code runs transactionally safe on SQL databases, too!

What's next

Support for QuerySet.update() and F() objects is the foundation of portable transactions over different databases. In combination with the django-dbindexer it now becomes possible to add support for automatically sharded counters as described in Sharding with Django on App Engine! In other words, we could add an index definition in order to tell django-dbindexer to automatically shard specific updates. Stay tuned!

Leave a comment

Minor updates and API changes

2011-05-05T10:18:00Z

Maybe you noticed that we've started to work on our projects again. ;) We unsurprisingly (:P) passed our final physics exams and are ready to work on NoSQL development again. Waldemar has already started to do some work in order to get our efforts into Django trunk. In the meantime I've started to work on some bug reports concerning django-dbindexer and nonrel-search. Both now have a dependency on django-autoload which ensures the loading of indexes or signal handlers before any request is processed. As a consequence, you have to adapt your code if you make use of django-dbindexer or nonrel-search. The documentation has been changed in order to reflect these changes. Maybe we'll take a few days off to go on vacation. Then, we'll come back and continue improving the way of NoSQL development.

Leave a comment

SimpleDB backend for Django-nonrel

2011-05-03T07:16:00Z

Dan Fairs has started developing a SimpleDB backend. That's the sixth backend for Django-nonrel! Check out django-simpledb from github. The code is experimental and far from finished, but in a recent tweet Dan mentioned that he thinks he's very close to having the Django admin running on SimpleDB. That's huge, Dan! :) We're very close to supporting every popular NoSQL database. Guys, if you want to use Dango-nonrel with SimpleDB please help Dan with the django-simpledb backend (or help Mirko with the Redis backend if you prefer that).

Leave a comment

Redis backend for Django-nonrel in development

2011-04-16T05:48:00Z

Mirko Rossini has finally made his Redis backend public. It's called django-redis-engine and according to him it's still in pre-alpha stage. From a quick glance at the code it looks like you'll be able to define database indexes in order to support queries that go beyond Redis' (meager ;) native query capabilities (much like with django-dbindexer). This is awesome because you won't have to maintain indexes by hand. If you want to see a stable and fully featured Redis backend please help Mirko. Just clone the repository from github and start playing with the code. Enjoy!

P.S.: Sorry for the long period of silence. We're still not finished with our final diploma exams, but we're getting close. Soon we can restart blogging more frequently and take the results of the previous survey into account.

Leave a comment

JOINs for NoSQL databases via django-dbindexer - First steps

2011-02-01T07:27:00Z

At the moment, we have a lot to do in terms of finishing our diploma thesis. However we're so excited about the early results of the currently refactored django-dbindexer that we couldn't hold back and keep django-dbindexer's simple JOIN support for non-relational databases a secret anymore. Yes, you didn't read wrongly, this post is about JOIN support for NoSQL databases! We'll show how to use in-memory JOINs and how to get JOINs working if the joined field's value doesn't change. So let's unpack our delayed Christmas present. ;)

Let's rock!

Let's take the example of a photo-user relationship, just like in our post JOINs via denormalization for NoSQL coders, Part 1:

# photo/models.py:

from django.contrib.auth.models import User
from django.db import models

class Photo(models.Model):
    owner = models.ForeignKey(User, null=True, blank=True)
    popularity = models.IntegerField(choices=[(0, 'low'), (1, 'medium'), (2, 'high')],
                                     default=0)
    published = models.DateField(auto_now_add=True)
    title = models.CharField(max_length=500)

Here, each photo entity is linked to a user via a many-to-one relationship and has a popularity tag assigned to it. Additionally we store the date when the photo has been published and give the photo a title. Now let's say you want to get all of a user's photos. In order to do so, NoSQL databases force us to use ugly workarounds. One possible way is to first get the user (or better the corresponding primary key in order to avoid fetching the whole user entity out of the database) and thereafter the user's photos:

def get_user_photos(first_name, last_name):
    user_pk = User.objects.values('pk').get(first_name=first_name, last_name=last_name)
    return Photo.objects.filter(owner__pk=user_pk)

However with our reborn baby, django-dbindexer, you can just use Django's spanning relationship syntax (double underscores) to which you are used to:

def get_user_photos(first_name, last_name):
    return Photo.objects.filter(owner__first_name=first_name, owner__last_name=last_name)

Feels a lot more comfortable, right? :) To get this working all you have to do is to install the django-dbindexer and add the following index definition:

# photo/dbindexes.py:

from models import Photo
from dbindexer.lookups import StandardLookup
from dbindexer.api import register_index

register_index(Photo, {'owner__first_name': StandardLookup(),
                       'owner__last_name': StandardLookup(),
})

and one of the new join resolvers (InMemoryJOINResolver or ConstantFieldJOINResolver) to DBINDEXER_BACKENDS in your settings:

# settings.py:

DBINDEXER_BACKENDS = (
    'dbindexer.backends.BaseResolver',
    'dbindexer.backends.FKNullFix',
    'dbindexer.backends.InMemoryJOINResolver',
)

That's it. django-dbindexer will handle anything else magically for you. :) Under the hook the refactored django-dbindexer now uses backends to resolve lookups. The BaseResolver is responsible for resolving lookups like __iexact or __regex for example and the FKNullFix backend will make __isnull queries work correctly on ForeignKey. In this example we choose the InMemoryJOINResolver which is used to resolve JOINs in-memory, just like we did manually in the example above. On the other hand, the ConstantFieldJOINResolver would denormalize the user's first_name and last_name into the Photo model and use these denormalized properties to execute the query. Anything described in JOINs via denormalization for NoSQL coders, Part 1 is then done automatically by the ConstantFieldJOINResolver for you. :)

JOINs aren't limited to simple examples like the one above. Of course you can combine a query with filters on the Photo model:

def get_popular_user_photos(first_name, last_name):
    return Photo.objects.filter(popularity=2, owner__first_name=first_name,
                                owner__last_name=last_name)

You can even span filters over multiple relationships if you have more complex models, just use multiple double underscores. If the photo model contains a ForeignKey to a group which itself contains a ForeignKey to a creator, a query getting all photos whose group creator is a specific user would look like this:

def get_creators_groups_photos(first_name, last_name):
    return Photo.objects.filter(group__creator__first_name=first_name,
                                group__creator__last_name=last_name)

Note that this query can return photos belonging to more than just one group because a user could have created multiple groups.

And as a nice side effect of the refactoring process, you can combine JOINs with other index definitions like an __iexact lookup on the user's first_name and last_name in combination with a __month lookup on published:

def get_user_photos_created_in_month(first_name, last_name, month):
    return Photo.objects.filter(popularity=2, published__month=month,
                                owner__first_name__iexact=first_name,
                                owner__last_name__iexact=last_name)

Just add the needed index definitions to your index module:

# photo/dbindexes.py:

from models import Photo
from dbindexer.lookups import StandardLookup
from dbindexer.api import register_index

register_index(Photo, {'owner__first_name': (StandardLookup(), 'iexact'),
                       'owner__last_name': (StandardLookup(), 'iexact'),
                       'published': 'month',
})

It couldn't be easier than that, right? :) First we hardly believed our own eyes when we tested JOINs on App Engine on production but it really works. :P Just give it a try and download the django-dbindexer-testapp. :) If you wanna see even more complex examples then take a look at the unit tests of django-dbindexer.

`InMemoryJOINResolver` or `ConstantFieldJOINResolver`?

At the moment you have to choose globally between either the InMemoryJOINResolver or the ConstantFieldJOINResolver in you settings. All queries involving JOINs will then use the join resolver chosen. This will be changed in a future release of django-dbindexer. For now, you may wonder which join resolver to use. In general, in-memory JOINs are useful for simple cases for which you know that you get only a small set of entities on the joined side i.e. the User side. This includes examples like the one above as well as one-to-one relationships. If you know that you have to deal with larger sets of entities on the joined side you have to use the ConstantFieldJOINResolver because in-memory JOINs don't scale in such cases.

Currently you can't combine in-memory JOINs with OR-queries and exclude-queries. Anything else should work. In-memory JOINs will try to query as efficiently as possible. On App Engine for example dbindexer will batch-get and use efficient values('pk') filters (which become a keys_only filter) where possible. Additionally, fields on the joined side won't be created for StandardLookup.

The ConstantFieldJOINResolver is more efficient than the InMemoryJOINResolver and doesn't have limitations neither for OR-queries nor for exclude-queries i.e. you can execute the following query for example:

return Photo.objects.exclude(Q(owner__first_name__iexact='Danzo') |
                             Q(owner__last_name__iexact='Shimura'))

Additionally it is useful for more than one-to-one-relations. However, at the moment it only works for constant fields i.e. if the user's first_name and the last_name doesn't change. In the future we will add support for non-constant values too.

For both resolvers, JOINs only work in the ForeignKey direction. No reverse lookups supported at the moment. :(

Expressive NoSQL - Beyond scalability

So the first steps towards JOIN support for NoSQL databases are done. We plan to extend the backends to handle JOINs for non-constant fields in a scaleable way, just like described in our JOINs via denormalization for NoSQL coders series. Additionally, django-dbindexer's API will be extended to allow to specify a join resolver on a per field/filter bases. With the refactored django-dbindexer and its backend system even support for aggregates becomes possible. This will allow you to write complex queries in a few minutes instead of hours (including unit tests and debugging ;). No more hand-written denormalization and map/reduce and aggregate code. Just tell the indexer what you want to do and it'll handle it for you in a way you specify (via a backend)! Developers still have to think about how to design their code. However, django-dbindexer allows them to use Django's expressive ORM and thus being much more productive. Another possible extension is a backend for nonrel-search which could be used in order to add full-text search functionality for the admin interface. :) It's hard to think of something that would not be possible! :P If you want to help with the next exciting phase of development you can drop us a mail: http://groups.google.com/group/django-non-relational

Leave a comment

Merry Christmas

2010-12-25T15:41:00Z

Merry Christmas and a happy 2011, everyone! Actually we wanted to finish a little present, but December was just too full with other stuff. Looks like Thomas will have to play Santa in January. ;)

This blog started more or less one year ago. Back then we still were on a blogspot subdomain. A few months later Django-nonrel was ready for hosting a simple site, so we moved everything on a custom domain (allbuttonspressed.com) hosted on App Engine with Django-nonrel. Now this blog has more than 700 subscribers and we really need to get some feedback from you, so we can improve our articles in 2011. Please help us by filling out this short survey.

Thanks a lot! Have a nice Christmas and a successful 2011.

Leave a comment

Porting from App Engine's webapp to django-nonrel

2010-11-30T17:19:00Z

Hey, you may have noticed it already, we've finally finished our top-secret article called Running Pure Django Projects on Google App Engine :). We thank Wesley Chun for his help, expertise and time so we could get the article published on App Engine's articles section.

Summarized the article goes through several steps explaining in depth how to port an App Engine webapp app over to django-nonrel with some additional notes on useful Django features. At the end of the article we summarize the advantages of using django-nonrel over other approaches and what awaits you in the future. But enough words, just read the article :)

Leave a comment

HTML5 offline manifests with django-mediagenerator

2010-11-23T15:17:00Z

This is actually part 3 of our django-mediagenerator Python canvas app series (see part 1 and part 2), but since it has nothing to do with client-side Python we name it differently. In this part you'll see how to make your web app load without an Internet connection. HTML5 supports offline web apps through manifest files.

Manifest files

First here's some background, so you know what a manifest file is. A manifest file is really simple. In its most basic form it lists the URLs of the files that should be cached. Here's an example.manifest:

CACHE MANIFEST
/media/main.css
/media/main.js

The first line is always CACHE MANIFEST. The next lines can list the files that should be cached. In this case we've added the main.css and main.js bundles. Additionally, the main HTML file which includes the manifest is cached, automatically. You can include the manifest in the <html> tag:

<html manifest="example.manifest">

When the browser sees this it loads the manifest and adds the current HTML and manifest file and all files listed in the manifest to the cache. The next time you visit the page the browser will try to load the manifest file from your server and compare it to the cached version. If the content of the manifest file hasn't changed the browser just loads all files from the cache. If the content of the manifest file has changed the browser refreshes its cache.

This is important, so I repeat it: The browser updates its cache only when the content of the manifest file is modified. Changes to your JavaScript, CSS, and image files will go unnoticed if the manifest file is not changed! That's exactly where things become annoying. Imagine you've changed the main.js file. Now you have to change your manifest file, too. One possibility is to add a comment to their manifest file which represents the current version number:

CACHE MANIFEST
# version 2
/media/main.css
/media/main.js

Whenever you change something in your JS or CSS or image files you have to increment the version number, manually. That's not really nice.

django-mediagenerator to the rescue

This is where the media generator comes in. It automatically modifies the manifest file whenever your media files are changed. Since media files are versioned automatically by django-mediagenerator the version hash in the file name serves as a natural and automatic solution to our problem. With the media generator a manifest file could look like this:

CACHE MANIFEST
/media/main-bf1e7dfbd511baf660e57a1f36048750f1ee660f.css
/media/main-fb16702a27fc6c8073aa4df0b0b5b3dd8057cc12.js

Whenever you change your media files the version hash of the affected files becomes different and thus the manifest file changes automatically, too.

Now how do we tell django-mediagenerator to create such a manifest file? Just add this to your settings.py:

OFFLINE_MANIFEST = 'webapp.manifest'

With this simple snippet the media generator will create a manifest file called webapp.manifest. However, the manifest file will contain all of the assets in your project. In other words, the whole _generated_media folder will be listed in the manifest file.

Often you only want specific files to be cached. You can do that by specifying a list of regular expressions matching path names (relative to your media directories, exactly like in MEDIA_BUNDLES):

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (
            r'main\.css',
            r'main\.js',
            r'webapp/img/.*',
        ),
        'exclude': (
            r'webapp/img/online-only/.*',
        )
    },
}

Here we've added the main.css and main.js bundles and all files under the webapp/img/ folder, except for files under webapp/img/online-only/. Also, you might have guessed it already: You can create multiple manifest files this way. Just add more entries to the OFFLINE_MANIFEST dict.

Finally, we also have to include the manifest file in our template:

{% load media %}
<html manifest="{% media_url 'webapp.manifest' %}">

Manifest files actually provide more features than this. For example, you can also specify FALLBACK handlers in case there is no Internet connection. In the following example the "/offline.html" page will be displayed for resources which can't be reached while offline:

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (...),
        'fallback': {
            '/': '/offline.html',
        },
    },
}

Here / is a pattern that matches all pages. You can also define NETWORK entries which specify allowed URLs that can be accessed even though they're not cached:

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (...),
        'network': (
            '*',
        ),
    },
}

Here * is a wildcard that allows to access any URL. If you just had an empty NETWORK section you wouldn't be able to load uncached files, even when you're online (however, not all browsers are so strict).

Serving manifest files

Manifest files should be served with the MIME type text/cache-manifest. Also it's critical that you disable HTTP caching for manifest files! Otherwise the browser will never load a new version of your app because it always loads the cached manifest! Make sure that you've configured your web server correctly.

As an example, on App Engine you'd configure your app.yaml like this:

handlers:
- url: /media/(.*\.manifest)
  static_files: _generated_media/\1
  mime_type: text/cache-manifest
  upload: _generated_media/(.*\.manifest)
  expiration: '0'

- url: /media
  static_dir: _generated_media/
  expiration: '365d'

Here we first catch all manifest files and serve them with an expiration of "0" and the correct MIME type. The normal /media handler must be installed after the manifest handler.

Like a native iPad/iPhone app

Offline-capable web apps have a nice extra advantage: We can put them on the iPad's/iPhone's home screen, so they appear exactly like native apps! All browser bars will disappear and your whole web app will be full-screen (except for the top-most status bar which shows the current time and battery and network status). Just add the following to your template:

<head>
<meta name="apple-mobile-web-app-capable" content="yes" />
...

Now when you're in the browser you can tap on the "+" icon in the middle of the bottom toolbar (update: I just updated to iOS 4.2.1 and the "+" icon got replaced with some other icon, but it's still in the middle of the bottom toolbar :) and select "Add to Home Screen":

Then you can enter the name of the home screen icon:

Tapping "Add" will add an icon for your web app to the home screen:

When you tap that icon the canvas demo app starts in full-screen:

We can also specify an icon for your web app. For example, if your icon is in img/app-icon.png you can add it like this:

{% load media %}
<head>
<link rel="apple-touch-icon" href="{% media_url 'img/app-icon.png' %}" />
...

The image should measure 57x57 pixels.

Finally, you can also add a startup image which is displayed while your app loads. The following snippet assumes that the startup image is in img/startup.png:

{% load media %}
<head>
<link rel="apple-touch-startup-image" href="{% media_url 'img/startup.png' %}" />
...

The image dimensions should be 320x460 pixels and it should be in portrait orientation.

Summary

The manifest file just lists the files that should be cached
Files are only reloaded if the manifest file's content has changed
The manifest file must not be cached (!) or the browser will never reload anything
django-mediagenerator automatically maintains the manifest file for you
Offline web apps can appear like native apps on the iPad and iPhone
Download the latest canvas drawing app source which is now offline-capable

As you've seen in this post, it's very easy to make your web app offline-capable with django-mediagenerator. This is also the foundation for making your app look like a native app on the iPhone and iPad. Offline web apps open up exciting possibilities and allow you to become independent of Apple's slow approval processes for the app store and the iOS platform in general because web apps can run on Android, webOS, and many other mobile platforms. It's also possible to write a little wrapper for the App Store which just opens Safari with your website. That way users can still find your app in the App Store (in addition to the web).

The next time you want to write a native app for the iOS platform, consider making a web app, instead (unless you're writing e.g. a real-time game, of course).

Leave a comment

All Buttons Pressed

Goodbye

Using dev_appserver with Python 2.7 on App Engine

Using djangoappengine with Python 2.7

F() objects and QuerySet.update() support in djangoappengine

Django's beauty

What's next

Minor updates and API changes

SimpleDB backend for Django-nonrel

Redis backend for Django-nonrel in development

JOINs for NoSQL databases via django-dbindexer - First steps

Let's rock!

InMemoryJOINResolver or ConstantFieldJOINResolver?

Expressive NoSQL - Beyond scalability

Merry Christmas

Porting from App Engine's webapp to django-nonrel

HTML5 offline manifests with django-mediagenerator

Manifest files

django-mediagenerator to the rescue

Serving manifest files

Like a native iPad/iPhone app

Summary

`InMemoryJOINResolver` or `ConstantFieldJOINResolver`?