The Daily Profeth

Apache CouchDB: External Process using Java and serving file

2009-12-26T03:13:00.002+01:00

Apache CouchDB supports invoking external processes - here is a very simple example in Java, which returns an image (or another arbitrary binary file):

import java.io.FilterOutputStream;

import java.io.IOException;

import sun.misc.BASE64Encoder;

public class ExternalProcessTest {

public static void main(String[] args) throws IOException {
System.out.print("{\"code\":200,\"base64\":\"");

new BASE64Encoder().encodeBuffer(
ExternalProcessTest.class.getResourceAsStream("logo.png"),

new FilterOutputStream(System.out) {

@Override

public void write(int b) throws IOException {

if (b != '\n' && b != '\r') {

super.write(b);

}

});

System.out.print("\",\"headers\":{\"Content-Type\":\"image/png\"}}");

System.out.println();
System.exit(0);

}

This can be bundled into a runnable JAR file.
Some hints:

You might want to make your output return another status code in case an error ocurred
You might want to replace the BASE64Encoder class which is part of sun.misc with your favorite implementation.
Most base64 implementations add CRLF every 76 characters, which need to be stripped, because CouchDB expects to get exactly one line returned.
The Content-Type header should match the MIME-type of the returned file. There are also automatic ways to recognize the MIME-type of a file.
Don't forget the ending newline after your output.

Within your local.ini of CouchDB (to be found within etc\couchdb), you need to register your external application:

[external]
test=java -jar C:\ExternalProcessTest.jar 2>1

[httpd_db_handlers]
_test = {couch_httpd_external, handle_external_req, <<"test">>}

Java needs to be within your PATH variable.
2>&1 suppresses error output to be logged to the CouchDB log.
http_db_handlers registers the handler on database level, not globally.

Additional hints:
CouchDB pipes some request data to the external process, you can receive this by reading from System.in:

String json = new BufferedReader(new InputStreamReader(System.in)).readLine();

In action:

Download the source code and the runnable JAR file.

Apple, please add adaptive navigation and context-awareness!

2009-12-13T02:15:00.000+01:00

My iPhone keeps on filling itself with apps (I wonder how it does that :-D) - some of them get removed directly after install if I don't like them, others are getting my favorites and again others I know I will use them in the future, probably when I don't have wifi or a good reception available, but I just don't need them now. You know about which type of apps I am talking about: white pages, hotel booker, dictionaries, traffic information, converter tools, etc.
Some of them are really useful, but I only need them at a certain time once, or over a certain period of time (mostly for some days or sometimes weeks) - but they still clutter my iPhone - and even more: new apps are added behind them, whether the newly installed app will become one of my favorites or not.

Now this is nothing new and exceptionally specific to the iPhone, we can also see this with webpages and/or information within webpages, also. I might check on eBay 10 times a day if I want to bid on something. But then again I might not use eBay for weeks if I don't want to bid on anything.

It's some sort of situational correlation.

Now with the iPhone,

the situation might be that I am currently travelling in a foreign country - let's take France for example, I would need my french dictionary often and definitely my other dictionaries are more or less useless while I am travelling - so are the apps I have for public transportation schedules in Germany.
the situation might be, that I have a certain task to do - search for a suitable name of an yet unborn child for example - I don't know if there is an app for that (there probably is!), but it definitely gets used less frequently once I found a name or the child was born.
the situation might be, that I am at home - in which case I probably will use my computer to check my newsreader or do some instant messaging instead of the according apps on my iPhone - but I might be still interested in my games or the birthdays of my contacts.
I am probably less interested in booking a hotel when I am in a 50 miles range to my home.
I probably won't need my compass when being at home.
etc.

For all of these cases, there are some keywords which might lead to an answer: context-aware computing, adaptive navigation and situational applications.

Now why do people use none of these keywords together in one sentence with the iPhone? It seems like a perfect candidate for such technologies:

My iPhone has the ability to know where I am.
My iPhone has the ability to make a good guess about which language I prefer.
My iPhone (probably) has the ability to know whats on my mind when I checked for a baby name in the [put a name here] app, the thousandth time this week.
My iPhone has the ability to know, that I didn't use this nameless application with the hideous icon within the last five weeks.

I want my iPhone to act accordingly.

Now in order to let my iPhone do this, there are some requirements.

Some sort of matching, using evidence sources to find out where exactly I am (what my current context is) - like MarcoPolo does it for example.
Some sort of awareness how often which application is used and in what context (see 1.).
Some sort of mechanism to adapt the navigation dynamically (e.g. re-sort and/or hide my apps) based on 1. and 2.

Those three requirements would allow for a full-fledged context-aware computing on the iPhone.

Example

By default my iPhone screen looks like this:

Lets make the assumption, that:

A is my text messaging app, which I use wherever I am.
I am currently at home and I definitely do not need apps I to L because they do something I can do better at my laptop and my laptop is currently also started.
I definitely don't need my car navigation, which is B.
There is a game R which I bought at the beginning of this week and have played this for some time when being at home.

Then the iPhone screen could look like this:

Lotus Expeditor: Using custom launch items

2009-11-28T17:26:00.001+01:00

When working with Lotus Expeditor and you want to add a custom launcher item to the "new" section in let's say Lotus Symphony for example, there is not much help available out there.

Sure, there is the description of different launchers in the Information center, but it only explains how to use urlLaunchItem, nativeProgramLaunchItem and perspectiveLaunchItem in detail.

So this article is for you out there searching Google for a how-to.

First define the Launcher in your plugin.xml:

(sorry Blogger didn't let me input XML here...)

Now this registers a new item (launchItem) and defines a launch item type (launchItemType) which gets invoked when clicking on it (see type attribute of launchItem and name atttribute of launchItemType).
The launchItemType invokes a class (see class attribute of launchItemType) which needs to be a subclass of com.ibm.rcp.ui.launcher.LauncherContributionItem (attention: ~~com.ibm.rcp.jface.launcher.LauncherContributionItem~~ is deprecated and should not be used).

The referenced class in launchItemType itself should look like this:

package com.xyz;

public class MyLaunchItemTypeClass extends com.ibm.rcp.ui.launcher.LauncherContributionItem {
    @Override
    public void launch(int arg0) {
        super.launch(arg0);
        System.out.println("LAUNCHED!");
    }
}

also don't forget to put com.ibm.rcp.swtex and com.ibm.rcp.ui on your required bundles list in MANIFEST.MF.

Google Reader integration for GMail

2009-11-24T13:53:00.000+01:00

Today I spent some minutes on writing a small Google Gadget which can be used to integrate the Google Reader into GMail.

You can add it to your GMail by enabling the "Add any gadget by URL" Labs feature within GMail and then add the gadget with this URL: http://gadgets.feth.com/reader/reader-gmail.xml.

It will display a small tab in your sidebar from which you can access Google Reader. The Reader will open in the area where you normally read your mails and will look like this:

Please feel free to write a comment and/or rate this gadget. If you encounter any problems, feel free to send me a note or use the comments section!

The Enterprisity

2009-11-18T22:18:00.001+01:00

I am so frustrated today...I definitely needed to translate an (relatively unknown) urban definition to English which exists in German since 2007 (Thanks to Michael Kukat):

Enterprisity - Measure for the adequacy of a method, a software or a person for his/her/its application in an enterprise, especially in major corporations. Important distinguishing features regarding enterprisity are the artificially boosted complexity, the resistance to counseling (primarily those of persons), the high demand for process orientation simultaneously paired with the preferred total circumnavigation of it, as well as the most miserable performance in comparison to competitive products (or persons) at the most terrible price.

Possible uses:

What the heck... I think the enterprisity got me - I didn't get any work done today!

God I love this company, there is such a low enterprisity, it almost feels like working on an Open source project.

Shoot! Why do I need to work with this rubbish - are they still trying to maximize enterprisity?

If you are a native speaker, please use the comments to help me remove any expression flaws from the definition. Thanks ;-)

Possible iPhone slide to unlock usability enhancement

2009-10-30T22:12:00.002+01:00

Update on 2 Dec 2010: Just found out that Sony Ericssons Xperia x10 actually has an ergonomic slide to unlock area. Yay!

Did you ever try to place a call with your iPhone while carrying your notebook?

Did you ever receive a call on your iPhone when carrying your meal?

It's not so easy to unlock the iPhone or accept that call while being only in possession of one hand, right?!

And also while it is possible to just do that little slide with only the thumb of the very hand also holding the iPhone, it is just not very convenient. Feels more like having played F1 Race on a Game Boy for the last 4 hours than using a phone designed by a company known for its excellence in creating products with an extraordinary usability.

When looking at a sequence of images about unlocking an iPhone with one hand, one can see, that the problem actually seems that the thumb itself needs to be bended too much to stay within the slider area:

The reason for this is that hitting the slider area implies moving the finger in a straight line, which is just not a natural movement for the thumb. A wipe movement in a curvy line is much easier to accomplish and also feels more natural:

So one possibility to improve unlocking with one hand would be realigning the slider area from a straight line (see "traditional" in image below) to a slightly bended line (see "thumb-friendly" in image below):

Realigning the slider area has one major drawback though: the background image gets overlayed by it. While I would still favor that approach and also would be willing to have my background image overlayed with a semi-transparent slider area, but I know that many of you will disagree with that, so the logical continuation of my approach to improve the slider usability is using a slider "knob" which can be dragged freely over they iPhone's display in any direction and with any bending:

As soon as the knob has been dragged far enough from its initial position this will unlock the iPhone (or accept a call certainly). This would maximize the ergonomics of the movement, as people with different hand sizes and/or different likes in thumb movement can choose the one which is the most natural to them.

To carry that thought a bit further, this could even allow for unlocking the iPhone not via entering a pass phrase or some numbers, but only by dragging the knob in a previously defined gesture:

Are these ideas feasible? Any additions? I'd love to hear what you think in the comments!

Free VoIP calls with sipgate on iPhone using XSPhone

2009-09-16T14:09:00.000+02:00

Today nothing developerish (sorry to all you frequent readers out there), but something not minor interesting. Yet, only for all of you out there being in possession of an iPhone or iPod touch (2nd generation).

On my search throughout the Apple App Store for neat applications I might miss, I came across a very simple (but nonetheless interesting) free application, which is called XSPhone.

XSPhone is a phone application officially designed for use with the dutch ISP XS4ALL, but implements the standard Session Initiation Protocol (better known as SIP) and is (according to XS4ALL) [...] suitable for use with XS4ALL’s Talk subscription, but also for other VoIP providers that use the SIP protocol [...].

I personally use sipgate, which is popular in Germany, but also is available in the U.S.. So I thought I should give XSPhone a try with sipgate. And, behold, it works! Even incoming calls are possible.
Due to the restrictions for calling apps on the iPhone, everything in this article only works over a WiFi connection, not with a mobile internet connection, though.

So let's get this up and working.

First, install the app (if you didn't already). You can either search for "XSPhone" or click the App Store link provided above.

Then, let's get your VoIP credentials.

For sipgate you can find them in your settings after logging into your sipgate account.

Sorry, screenshots are in german, too lazy to search if I can switch to english ;-)

Within your settings page, you should see a screen with two grey boxes, one of which reads "Account data":

and the other one "Server data":

Now go to your XSPhone settings (you can find them in the iPhone settings menu at the bottom, where the settings of any installed app can be configured) and enter the gathered credentials.

Number 1 is your user name (corresponds to your sipgate SIP-ID).
Number 2 is your password (corresponds to your SIP password).
Number 3 is the VoIP server to register to (this is sipgate.de in my case).
Leave the voicemail number empty (I couldn't get it working correctly with sipgate, it might work with other providers, though).
Check the experimental option "Allow incoming calls" - at least with sipgate it works. The app needs to be running to accept incoming calls, certainly (Doh! :-).

After you configured the app, you can fire it up. If everything was configured correctly, you should see a screen which looks like this:

In case of sipgate you can either try calling the test number (which is 10000), the echo service (which is 10005) or your mailbox (which is 50000). All of those numbers are free of charge when calling over sipgate.

Unfortunately the mailbox button does not work correctly, but you can add an entry for your sipgate voicemail to your address book and add it to the "Favorites" tab from XSPhone. Calling this favorite works fine (I wonder what difference using the mailbox button makes).

That's it, you are all set for VoIP calling (and being called) over sipgate.

There are some drawbacks, however, I need to tell you about this solution:

You can only be called if the app is open.
XSPhone only supports one single VoIP registrar at any time.
XSPhone does not look as nice as some iPhone applications do (there is some ugly background picture I did not dare to make a screenshot of, when you are calling someone) - although all the other VoIP apps currently in the App Store are also not exactly beautiful either, and at least, this one is free of charge!
No SIP-finetuning with proxy, STUN server, etc.

Apart from that: yep, that is a cool freebie!

I'd love to hear about your experience with XSPhone and your ISP (if you got it working, please include a short settings description for all the others out there) in the comments!

Update (2009-09-16): as of now there is an application from sipgate available within the App Store (search for "sipgate") - currently only working with sipgate team edition - in some weeks a version for the other sipgate plans should be available.

Circuit Breaker Pattern on Google App Engine

2009-04-19T02:38:00.004+02:00

This week I came across the Circuit Breaker Pattern, which seems to be a good approach for handling one or more dependencies to external services in cloud/mashup applications.
As applications on GAE tend to have such dependencies, I wrote a memcached and datastore-backed Circuit Breaker Pattern implementation for easy and straightforward usage on GAE Java (sorry, no Python this time - but should actually be easy to adapt and I am glad to post a link here to anyone who migrates the Java implementation).

So what exactly is that Circuit Breaker Pattern I am talking about?!

Think of your GAE application as a washing machine (pretty profane I know) - and think of your external service you like to use with your GAE application as the water you need for getting your clothes clean (thats what a GAE application should do, at last). So if you start your washing machine without the water, your clothes probably won't get much cleaner than they were before you put them into the machine - or worse: your washing machine may break down because its running hot and your clothes burst into flames.
To minimize casualties on that scenario, the Circuit Breaker Pattern comes into play: it basically tracks the health of your external services (the water, electricity and detergent) and trips if they are not healthy or completely unavailable (e.g. your washing machine will only start if all "external services" are available and healthy).

Technical stuff

From a technical perspective, it basically tracks how often a service request failed in the past and prevents you from making unnecessary calls to that service in the future (or act accordingly by informing the visitor to come back later, sending you an email, kicking the service in the butt or [you name it]).
To make this a little more dynamic, the Circuit Breaker Pattern allows you to add some sort of fuzziness to your logic, by not shutting down your whole app when an arbitrary external service fails only once (even though that could be done achieved the pattern) but only after a specific given number of times (the failure threshold).
Also, the Circuit Breaker Pattern defines a time period (timeout) which needs to elapse until the next call to that external service is allowed to get through and check if the service is back up again.

So how does this look in your GAE code?

Java pseudocode:

int failureThreshold = 3;
int defaultTimeout = 5; // seconds
ServiceCircuit ebayCircuit = CircuitBreaker
   .getOrCreateServiceCircuit("ebay", failureThreshold,
    defaultTimeout);

if (CircuitBreaker.isClosed(ebayCircuit)) {
    // circuit is closed,
    // we are allowed to request the ebay service
    
    // Here comes the call to your external service
    boolean success = ?;
    
    if(success) {
        // Service request returned successfully.
        
        // inform the circuit breaker,
        // the request was a success
        CircuitBreaker.success(ebayCircuit);
        
        //do something with your data
        System.out.println(yourData);
    } else {
        // Service failed
       
        // tell the circuit breaker, the request failed
        CircuitBreaker.fail(ebayCircuit);
        
        // inform your visitor
        System.out.println("ebay could not be reached!");
    }
} else {
    // circuit was not closed, so we are not allowed
    // to make a call to
    // our external service for now.
    // We need to wait for the timeout to
    // elapse
    
    // inform your visitor to come back later
    System.out.println("Outage: Please come back later!");
}

Thats more or less all the magic you need to add to your app to make use of the Circuit Breaker Pattern.
To use more than one service, you can either instantiate another circuit, or create a compound circuit (just change the ID from "ebay" to "ebay_and_anotherservice") to track the vitality of both services (depends on how they are needed: sequentially or parallel).

GWT sample application

Beside the code I also wrote a small GAE GWT sample application, which shows you how to use the Circuit Breaker Pattern and comes with some example code:

Code

Download the complete eclipse project. This includes the GWT sample and a whole lot of comments.

The Pattern implementation itself is in two files:

/CircuitBreakerExample/src/com/feth/appengine/circuitbreaker/CircuitBreaker.java
/CircuitBreakerExample/src/com/feth/appengine/circuitbreaker/ServiceCircuit.java

Please let me know what you think, if the Circuit Breaker is helpful or not, if you have additions or comments or encounter bugs...

Using reCAPTCHA with Google App Engine

2008-04-30T13:07:00.014+02:00

We have no image creation capabilities in Google App Engine (yet). So if you want to display a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) within a GAE application, you need to do this by using an external service provider.

One of these providers is reCAPTCHA - and the nice thing about reCAPTCHA is, that your visitors are helping to digitize books by solving the CAPTCHA.

So how do we start?
At first, you should get a service key on recaptcha.net.
After you registered, you can create an unlimited number of service keys for your domain.
Note: If you want to use reCAPTCHA with your dev environment, you need to get keys for localhost.
After generating one pair of keys, the screen should look like this:

Write down your public and private key - we will need them later. If you forget them, you can return to recaptcha.net to find them out. Don't tell anyone your private key.

After having registered for the service, you should get a simple service class I adapted from python.org to GAE. You can download the adapted class here. Rename it to captcha.py and copy it somewhere (I prefer putting it in a subdirectory named recaptcha/client/ - remember to add an empty __init__.py in every directory if you do this) within your App Engine project.

For demonstration, I will be extending the helloworld example from GAE.

Go to your helloworld.py and import the captcha and environ modules:

from os import environ
from recaptcha.client import captcha

Then go to your MainPage controller and add the following lines:

chtml = captcha.displayhtml(
  public_key = "YOUR-PUBLIC-KEY",
  use_ssl = False,
  error = None)

template_values = {
   ...
   'captchahtml': chtml
}

Exchange YOUR-PUBLIC-KEY with your public key - if you don't, you'll get a message:
"Invalid public key. Make sure you copy and pasted it correctly."

Now switch to your HTML template and add the captchahtml output within your form tags:

<form>
  ...
    {{ captchahtml }}
</form>

When you open the site in your browser, you'll see the recaptcha iframe:

Now when the form gets submitted, we need to check if the captcha input was correct. Therefore, we need to change into the post method of our Guestbook object and add some code:

def post(self):
  challenge = self.request.get('recaptcha_challenge_field')
  response  = self.request.get('recaptcha_response_field')
  remoteip  = environ['REMOTE_ADDR']

  cResponse = captcha.submit(
                 challenge,
                 response,
                 "YOUR-PRIVATE-KEY",
                 remoteip)

    if cResponse.is_valid:
        # response was valid
        # other stuff goes here
    else:
      error = cResponse.error_code
    ...

Exchange YOUR-PRIVATE-KEY with your previously generated private key.

The user inputs to the reCAPTCHA iframe will be validated together with the remote IP (your visitor's IP) and the challenge. We get a response from the reCAPTCHA API server and a RecaptchaResponse object will hold the answer.

The RecaptchaResponse object has two properties:

is_valid is set to True if the test was successful (otherwise it'l bee False)
error_code will hold an API error code if there was a problem.

Note: In the Getting Started Guide for GAE, the form gets submitted to the Guestbook controller. Normally you would submit to the MainPage controller and pass the error code from the RecaptchaResponse object (if there is one) to the displayhtml method of the captcha class:

chtml = captcha.displayhtml(
  public_key = "YOUR-PUBLIC-KEY",
  use_ssl = False,
  error = cResponse.error_code)

That will display a human readable error message to the user:

and lets your visitor redoing the CAPTCHA without losing his/her previously entered values in the other form fields.

Keep in mind: For every submitted CAPTCHA, a request to the reCAPTCHA server is made. This request is synchronous, so the response to your visitor will get delayed by the time it takes to fetch the response. If the reCAPTCHA server can not be reached, the error code recaptcha-not-reachable will be returned.

Using custom django template helpers with Google App Engine

2008-04-21T21:13:00.009+02:00

This is dedicated to all you django lovers out there. If you wondered how to use custom filter functions with your Google App Engine applications, here you go:

Create a new file (I created mine in common/templatefilters.py) to hold your template helper functions.

In this file, we need to register our custom filters, so:

# import the webapp module
from google.appengine.ext import webapp
# get registry, we need it to register our filter later.
register = webapp.template.create_template_register()

After we got the registry, we now can define our custom function:

def truncate(value,maxsize,stopper = '...'):
    """ truncates a string to a given maximum
        size and appends the stopper if needed """
    stoplen = len(stopper)
    if len(value) > maxsize and maxsize > stoplen:
       return value[:(maxsize-stoplen)] + stopper
    else:
       return value[:maxsize]

Then, we need to register our filter as following:

register.filter(truncate)

Now go to your bootstrap file (the one with your main function - in my case base.py - see image above) and add the library (if you used a different module than common.templatefilters, you need to specifiy it here):

webapp.template.register_template_library(
 'common.templatefilters')

After adding this line of code, your template functions will get added to django and are available to use within your html templates:

This will truncate the variable somevar to 20 characters and add "..." to the end, if needed.

ER-Modeling with Google App Engine (updated)

2008-04-16T16:08:00.003+02:00

3rd updated version with a lot of input from the GAE group. Thanks to everyone!

ER-Modeling with Google App Engine is somewhat different to "normal" modeling for a relational database.

Here is a small tutorial on how to create well-known relationship models One-to-One (1:1), One-to-Many (1:n), Many-to-Many (m:n) and a special one (Cascading relations) with GAE:

We need the following Model classes for our example. Also if you want to run it, don't forget to import the sys module - we need it for printing to stdout:

import sys #import sys module for printing to stdout

class Car(db.Model):
   brand  =  db.StringProperty(required=True)
   wheels =  db.ListProperty(db.Key)

class Human(db.Model):
    name    = db.StringProperty(required=True)
    drives  = db.ReferenceProperty(reference_class=Car)
    spouse  = db.SelfReferenceProperty()
    owns    = db.ListProperty(db.Key)

class Wheel(db.Model):
    isBroken =  db.BooleanProperty(default=False)
    position =  db.StringProperty(choices=set(["left_front",
                                              "left_back",
                                              "right_front",
                                              "right_back"]))

One-to-One (1:1)

A simple relationship between two entities.

Let's say a human called Jack drives one car.
We can model this relationship as following:

# one-to-one
jack        = Human(name="Jack")
mercedes    = Car(brand="Mercedes")
jack.drives = mercedes.put()
jack.put()
print >> sys.stdout, "Jack drives a "+jack.drives.brand

# Jack drives a Mercedes

As you can see, we create a human called "Jack" and a car "Mercedes". After that we assign the car to the drives-property of Jack and save Jack.

Hint (Thanks to Miguel for pointing this out): be careful with 1:1 relationships done with ReferenceProperty-properties - you could easily write something like this:

jack        = Human(name="Jack")
mike        = Human(name="Mike")
mercedes    = Car(brand="Mercedes")
mercedesid  = mercedes.put()

jack.drives = mercedesid
jack.put()

mike.drives = mercedesid
mike.put()

which won't be a 1:1 relation any more. So if you really need to make sure that an entity is only referenced once, you need to do this by your code design (by searching within your existing model kinds and throwing exceptions).
Even using a cascading-relationship with a parent entity (see later in this article) does not make sure, that there is only one car per human and therefore includes the same difficulty just the other way round.

Special: self references

Now a special sort of references are self-references. That means a Model references an entity of the same Model class (e.g. a Human references a Human):

# one-to-one self
bob     = Human(name="Bob")
jane    = Human(name="Jane")

bob.spouse = jane.put()
bob.put()
b_spouse = Human.get(bob.spouse.key())
print >> sys.stdout, "Bob's spouse is "+b_spouse.name

# Bob's spouse is Jane

We created two humans (Bob and Jane) and set Bob's spouse-property to reference Jane.
Now we can easily find out who Bob's spouse is.

Special: mutual references

But if Bob is married to Jane, Jane is also married to Bob, isn't she?
So let's do this semantically correct:

# one-to-one self mutual
bob     = Human(name="Bob")
jane    = Human(name="Jane")

bob.spouse  = jane.put()
jane.spouse = bob.put()
jane.put()

j_spouse = Human.get(jane.spouse.key())
print >> sys.stdout, "Jane's spouse is "+j_spouse.name
b_spouse = Human.get(bob.spouse.key())
print >> sys.stdout, "Bob's spouse is "+b_spouse.name

# Jane's spouse is Bob
# Bob's spouse is Jane

Be careful to check if reflexive references are semantically allowed or not – in our case it wouldn't be valid if bob's spouse is himself.

Also in a monogamous society, it wouldn't be valid if Bob is the spouse of more than one other Human entity (!)
If you don't want this behavior, you need to prevent it by throwing exceptions. You could use the validator-parameter of every Property class.

One-to-Many (1:n)

Sometimes we need to reference more than one entity from another.
When modeling 1:n relationships, a special of the Model class comes in handy: parent models (or ancestors).
Let's think of a car having four wheels. Those wheels belong exactly to one car, so we define the car as parent for every wheel we create:

# one-to-many using parent
bmw = Car(brand="BMW")
bmw.put()

lf = Wheel(parent=bmw,position="left_front")
lf.put()

lb = Wheel(parent=bmw,position="left_back")
lb.put()

rf = Wheel(parent=bmw,position="right_front")
rf.put()

# uh, snap, the 4th wheel is broken!
rb = Wheel(parent=bmw,position="right_back",isBroken=True)
rb.put()

# from car to wheels
bmwWheels = Wheel.all().ancestor(bmw)
print >> sys.stdout, "The BMW has the wheels: "
for wheel in bmwWheels:
    print >> sys.stdout, "- "+wheel.position

# The BMW has the wheels: 
# - left_front
# - right_back
# - right_front
# - left_back

# from wheel to car
brokenWheels = Wheel.gql("WHERE isBroken = :broken",
broken=True)
print >> sys.stdout, "The following cars are broken: "
for wheel in brokenWheels:
    print >> sys.stdout, "- "+wheel.parent().brand

# The following cars are broken:
# - BMW

We can see: it is easy to get the wheels for a car and it is also possible to get the car a certain wheel belongs to. Also an entity can only have one parent at a time - so we made sure, that a wheel is not used by more than one car at the same time. You could even add some spare wheels to a car (there is no way to define a maximum number an entity can be used as parent entity).

The other way round

It is also possible to do this the other way round. Lets say we have an additional Model like this:

class OwnedCar(db.Model):
   brand  =  db.StringProperty(required=True)
   owner  =  db.ReferenceProperty(Human, required=True)

then we could add cars to a person as following:

paul = Human(name="Paul")
paul.put()

pauls_bmw = OwnedCar(brand="BMW", owner=paul)
pauls_bmw.put()

pauls_mercedes = OwnedCar(brand="Mercedes", owner=paul)
pauls_mercedes.put()

pauls_cars = paul.ownedcar_set
print >> sys.stdout, "Paul's cars: "
for car in pauls_cars:
    print >> sys.stdout, "- "+car.brand

# Paul's cars:
# - BMW
# - Mercedes

This makes for example sure, that one car is only owned by one human at a time.

Also notice the part on how to get Paul's cars. You don't even need to create a GQL query - the property modelname_set holds the references. You can find more about this in the docs.

Thanks to Aprigio for his input!

Special: One-to-Many using a list

You can also create 1:n relationships using a list:

# one-to-many using list
dodge = Car(brand="Dodge")
w1 = Wheel(position="left_front")
w2 = Wheel(position="left_back")
w3 = Wheel(position="right_front")
w4 = Wheel(position="right_back")
dodge.wheels = [w1.put(),w2.put(),w3.put(),w4.put()]
dodge.put()

dodgeWheels = Wheel.get(dodge.wheels)
print >> sys.stdout, "The Dodge has the wheels: "
for wheel in dodgeWheels:
    print >> sys.stdout, "-"+wheel.position

# The Dodge has the wheels: 
# -left_front
# -left_back
# -right_front
# -right_back

but: be careful, this model does not make sure, a wheel does belong to exactly one car! You could reference a wheel from more than one car, which would be semantically wrong in our example. So rather use this model for n:m relationships!

Many-to-Many (m:n)

For creating m:n relationships, use the following model:

# many-to-many using list+db.Key
jack    = Human(name="Jack")
bob     = Human(name="Bob")

vw      = Car(brand="VW")
chevy   = Car(brand="Chevy")

carpool = [vw.put(),chevy.put()]

jack.owns = carpool
jack.put()

bob.owns = carpool
bob.put()

chrysler = Car(brand="Chrysler")
jack.owns.append(chrysler.put())
jack.put()

jackOwns = Car.get(jack.owns)
print >> sys.stdout, "Jack owns: "
for car in jackOwns:
    print >> sys.stdout, "- "+car.brand

# Jack owns: 
# - VW
# - Chevy
# - Chrysler

whoOwnsTheChevy = Human.gql("WHERE owns = :car",car=chevy)
print >> sys.stdout, "These humans own the Chevy: "
for who in whoOwnsTheChevy:
    print >> sys.stdout, "- "+who.name

# These humans own the Chevy: 
# - Jack
# - Bob

First we create our two guys Jack and Bob. Then they decide on sharing cars. Their carpool consists of a VW and a Chevy. After that, Jack decides on buying an additional car for himself (the Chrysler).

When adding to a list like in this example, there is no check if the value already is present within the list - so it would be possible to reference the same entity multiple times. If you want to prevent this, you need to run a function on the list, which makes sure there are only unique entities:

def unique(lst):
    d = {}
    for item in lst:
        d[item] = 1
    return d.keys()

jeep = Car(brand="Jeep")
jack.owns = unique(jack.owns + [jeep.put()])
jack.put()

You could also check to content of the list before adding an entity, if it is already referenced.

Most flexible: using mapping entities

It might be a better idea to implement a different entity to represent the relationship
between a human and the cars they own.
The reason for this is that you could add more fields to this which may be beneficial later in a query. Lets say we add a bought field that contains the date the car was bough. Then one could get cars owned by Jack that he bought after a certain date.

class CarOwner(db.Model):
 car    = db.Reference(Car, required=True)
 owner  = db.Reference(Human, required=True)
 bought = db.DateProperty(auto_now_add=True)

  @staticmethod
 def get_owner_cars(human, bought):
        """Returns the cars that the given human
           bought before a specified date"""
     if not human: return []
     query = db.Query(CarOwner)
     query.filter('owner =',human)
     query.filter('bought >= ',bought)                            
     return [entry.car for entry in query]

also we'd like to add a function to the Car-Model, so that it looks like this:

class Car(db.Model):
   brand  = db.StringProperty(required=True)
   wheels = db.ListProperty(db.Key)

   def human_owns(self, human):
       """Returns true if the given human owns this car."""
       if not human: return False
       query = db.Query(CarOwner)
       query.filter('car =', self)
       query.filter('owner =', human)
       return query.get()

When we have the Models defined like that, we can go on using them:

max = Human(name="Max")
max.put()
saab = Car(brand="Saab")
saab.put()

ownership = CarOwner(car=saab,owner=max)
ownership.put()

import datetime
delta = datetime.timedelta(days=-1)
yesterday = (datetime.datetime.utcnow() + delta)

maxs_cars = CarOwner.get_owner_cars(max,yesterday)
print >> sys.stdout, "Max's cars since yesterday: "
for car in maxs_cars:
   print >> sys.stdout, "- "+car.brand

# Max's cars since yesterday:
# - Saab

max_owns_saab = bool(saab.human_owns(max))
print >> sys.stdout, "Max owns the Saab: "+str(max_owns_saab)

# Max owns the Saab: True

As you can see, the additional property on the mapping entity comes in handy when selecting the cars Max bought since yesterday.
And also the method on the car-Model is pretty neat to bundle functionality related to a model within it.

Note: you can use this sort of referencing also use for 1:1 and 1:n relationships. You just need to make sure, that either one referenced entity (1:n) is unique, or both (the combination of those - for 1:1)

Thanks to Brian, who brought this up!

Cascading relations

With the parent-property, cascading relationships are possible. This is very useful, if you have a folder-like structure (e.g. categories and products of an online shop). Let's have a look on how to do this:

import sys

class Category(db.Model):
    name     = db.StringProperty(required=True)

class Product(db.Model):
    name       = db.StringProperty(required=True)
    price      = db.FloatProperty()
   categories = db.ListProperty(db.Key)

root    = Category(name="Products")
root.put()
tech    = Category(parent=root,name="Tech stuff").put()
books   = Category(parent=root,name="Books")
books.put()
fantasy = Category(parent=books,name="Fantasy")
fantasy.put()
scifi   = Category(parent=books,name="Science Fiction")
scifi.put()
scomics = Category(parent=scifi,name="SciFi comics")
scomics.put()

somebook = Product(name="Some book")
somebook.categories.append(books.key())
somebook.price = 9.99
somebook.put()

lotr = Product(name="Lord Of The Rings")
lotr.categories.append(fantasy.key())
lotr.price = 29.99
lotr.put()

allFantasyBooks = Product.gql("WHERE categories = :cat",
                  cat=fantasy)
print >> sys.stdout, "All fantasy books: "
for book in allFantasyBooks:
    print >> sys.stdout, "- "+book.name



path = []
p = scomics
while p.parent():
    path.append(p.name)
    p = p.parent()

path.reverse()
print >> sys.stdout, " > ".join(path)

# Books > Science Fiction > SciFi comics

Now getting all books in a category is pretty easy, huh? And also generating a breadcrumb-like "where am I" is simple by iterating through all parent entities for a given entity (here it is the science fiction comics Category).

Attention: while the use of parent relationships if fine for the use case here, it would not be very efficient in the case where there are a lot of children associated with a parent.
This is because this results in a very large entity group.
The more entity groups your application has - that is, the more root entities there are - the more efficiently the datastore can distribute the entity groups across datastore nodes.
Thus for efficiency you should avoid the case where there are a lot of children.
According to the docs:

"A good rule of thumb for entity groups is that they should be about the size of a
single user's worth of data or smaller."

Thanks to Ben, Michael and Brian for pointing this out.

Google App Engine & eclipse (PyDev)

2008-04-10T10:25:00.021+02:00

updated (Link to youtube video) thanks to mano marks.
updated (Python debugging) thanks to mkielar.
updated (CC-by license).

Update: If you like following a video more than reading some text, Mano Marks from Google created a screencast about how to get started with App Engine.

Since Google published their App Engine, I am highly interested in it. Too bad I didn't get an App Engine account, but at least I am able to test it locally...
As a fan of eclipse and being new to Python development, I searched for an eclipse Python extension and finally found PyDev sourceforge project.
I know this is basic stuff, but it took me some minutes to find out how to get code completion working with Google App Engine.
So here is a small HowTo for getting App Engine with eclipse and PyDev up and running:

Get Python and install it.
Get eclipse and unzip it.
Get the Google App Engine SDK and install it.
Open eclipse and go to Help -> Software Updates -> Find and Install.

Choose "Search for new features to install" and click on "Next"
Click on "New remote site"

choose a name
enter http://pydev.sourceforge.net/updates/ in the URL field
click "OK"

Select the PyDev update site you just added
Click on "Finish"
After the update site has been searched, Choose to install the PyDev plugin (see image below)
Click on "Next"
accept the agreement
Click on "Finish"
Click on "Install All"
Choose "Yes" to restart eclipse

After eclipse has been started

choose Windows -> Preferences to bring up the preferences dialog
Change to the "PyDev" -> "Interpreter - Python" section to configure Python
Click on "New" right to the "Python interpreters" field to add an interpreter
Search for the python.exe executable (normally in C:\Python25)

After you selected the executable, PyDev will search for libraries, and the following screen comes up:

Normally, the preselection done by PyDev is fine, so click "OK" to accept the system pythonpath entries.
In the Preferences Window also click "OK" to confirm your changes.

Now we are ready to create our first python project.

Do a right-click in the Package Explorer and choose New -> Other.
In the upcoming dialog select "Pydev project" from the Pydev folder.
Create a new helloworld project with (!important) Python 2.5 (see image below)
After clicking "Finish", eclipse should switch to the Pydev view.
Do a right click on your new project and choose "Properties"
In the properties dialog choose the "PyDev - PYTHONPATH" section to add the App Engine libraries (we need this for proper code assist)
Click on "Add source folder" to add the following folders (see image) from your google appengine folder:
Click "OK" to confirm your changes.

For a quick start, you can work on the Getting Started tutorial avalibale.
Use the "src" Folder as base folder for your project (see image to the right)
The last thing we need to do now is to add a user-defined starter for our Google App Engine development server.

Got to "Run" -> "Open Run Dialog"
Choose "Python Run" and add a new launch configuration (document icon with the plus sign in the top left of the dialog)
Name your run configuration
As project choose your python GAE project.
As main module, enter the location of the "dev_appserver.py" script.
Change to the "Arguments" tab and enter "${project_loc}/src" as first argument.

After this argument, you may add all available additional arguments listed on the Dev Webserver documentation page. (here for example we changed the port where GAE is listening to 9999)

Click on "Apply" to save your changes
Click on "Run" to run your project

From now on you can run your project by selecting your confuguration from the "Run" dropdown and access your app on http://localhost:9999/.
If an error occurs, you can see it in the console view of eclipse and click on it to jump to the error location within your scripts.

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

web 2.0 SVG badge

2008-04-03T22:24:00.007+02:00

Auf der Suche nach einem web 2.0 badge kam mir folgendes Photoshop Tutorium unter die Finger.
Da ich kein aktuelles Photoshop mein Eigen nenne, und eine Suche nach einem web 2.0 SVG badge bei Google nicht wirklich brauchbares Material zu Tage förderte, nahm ich mir geschwind inkscape und baute das badge aus dem Tutorium nach Augenmaß nach.
Das Ergebnis kann man links begutachten und hier herunterladen.

This work is licensed under a Creative Commons Attribution-Share Alike 2.0 Germany License.

EPS zu SVG konvertieren (EPS2SVG)

2008-03-18T18:25:00.000+01:00

Hin und wieder kommt man in die Verlegenheit ein Vektorformat in ein anderes überführen zu müssen. Beispielsweise EPS in SVG. Da inkscape (noch) kein EPS versteht bleibt nur der Umweg über Drittprogramme.

An dieser Stelle ein kleines Howto.

Benötigt wird:

pstoedit (z.B. Version 3.45)
Ghostscript (z.B. GPL Ghostscript 8.61)
GNU plotutils für Win32 (wählen Sie in diesem Fall den Download Binaries) oder andere (z.B. Version 2.4.1)

Laden Sie die Programme herunter und installieren bzw. entpacken (im Falle der plotutils) Sie sie.
Damit pstoedit auf Ghostscript und die plotutils zugreifen kann, müssen unter Windows noch die Pfade der /bin-Verzeichnisse von Ghostscript und den plotutils in die %PATH%-Umgebungsvariable aufgenommen werden. Exemplarisch an dieser Stelle der Vorgang für Windows - Nutzer anderer Systeme müssen diesen Schritt auf ihr System angepasst ausführen.
- Linux:
  Stichwort: PATH="$PATH:/path/to/bin"
- Windows Vista:
  Systemsteuerung -> System und Wartung -> System.
  Dort auf Einstellungen ändern (siehe rechts).
  In den Systemeinstellungen auf den Reiter Erweitert und dort auf den Button Umgebungsvariablen rechts unten klicken.
- Windows XP:
  1. Rechtsklick auf den Arbeitsplatz.
  2. Aus dem Kontextmenü Eigenschaften auswählen.
  3. Auf der Eigenschaften-Seite den Tab Erweitert wählen.
  4. Auf den Knopf Umgebungsvariablen klicken.
Im neuen Fenster bei den Benutzervariablen für XYZ (oben) die Variable PATH wählen und auf Bearbeiten klicken (Alternative: Doppelklick auf PATH).
Dort dann die Pfade zu Ghostscript und den plotutils an die bestehenden Einträge anhängen (mit Semikolon [bei Linux mit Doppelpunkt] von den vorhandenen trennen).

Beispiel:
Eine;Menge;alte;Einträge;C:\Program Files\gs\gs8.61\bin;C:\plotutils-2.4.1-4-bin\bin

Hinweis: Die Pfadangaben mögen sich bei Ihnen - je nachdem welche Pfade Sie beim entpacken/installieren gewählt haben - unterscheiden. Wichtig ist, dass Sie jeweils die Pfade zu den bin-Verzeichnissen eintragen und nicht nur den Stammpfad zum Programm.
Danach öffnen Sie eine Eingabeaufforderung und wechseln in das Installationsverzeichnis von pstoedit.
Hinweis: befinden Sie sich unter Windows Vista ist es sinnvoll die Eingabeaufforderung als Administrator zu öffnen (Rechtsklick -> Als Administrator ausführen), da es sonst sein kann, dass das in der Kommandozeile ausgeführte Programm die Ausgabedatei nicht schreiben kann. Hierbei gibt es keine Warnung seitens pstoedit, sondern die Ausgabedatei wird lediglich nicht angelegt.
Um eine eps-Datei mittels Ghostscript und den plotutils in SVG zu konvertieren geben Sie auf der Kommandozeile folgendes ein:

pstoedit -f plot-svg Pfad\zur\Eingabedatei.eps Pfad\zur\Ausgabedatei.svg
1. der Schalter -f legt das Format zur Konvertierung fest - in unserem Fall (plot-svg) sollen die plotutils zur Erzeugung von SVG verwendet werden. pstoedit versteht noch einige andere Formate, eine Übersicht finden Sie auf der Seite des Projekts.
2. Der zweite Parameter ist der Pfad zur Eingabedatei (in unserem Fall eine EPS-Datei).
3. Der dritte Parameter ist der Pfad zur Ausgabedatei (in unserem Fall eine SVG-Datei).
Beispiel:

pstoedit -f plot-svg x.eps x.svg

Dieser Befehl würde eine EPS-Datei namens x.eps welche im Verzeichnis von pstoedit liegt mittels den plotutils in eine SVG-Datei namens x.svg welche im Verzeichnis von pstoedit angelegt wird, konvertieren.
Denken Sie daran, dass Sie unter Windows Vista bei diesem Beispiel die Kommandozeile als Administrator geöffnet haben müssen, da die Ausgabedatei ansonsten nicht geschrieben werden kann.