Learning Technical Stuff

Installing Pylint for Python 2.5 on Mac

2011-06-23T05:37:00.000-07:00

In a previous post I discussed how to set up Google App Engine (Python) development environment on Mac OS X 10.6. I'm liberally mining some of my own notes on setting up Google App Engine on Ubuntu 10.10.

In my previous Ubuntu environment I had a neat Ant script for building, testing and uploading my Google App Engine applications such as My Web Brain. One of tests the script performed was Python code style checks using Pylint, which I have also mentioned before in the context of Ubuntu. I've set this up again on my Mac, so I thought I thought I would post my notes here.

In this post I talk about:

Installing Setup Tools for Python 2.5,
Installing PyLint using SetupTools, and
Configuring Aptana/Pydev to use PyLint

Lets go!

A couple of pre-requisites first: You need to have enabled the root user to be able to Sudo and I'm assuming you have already installed Python 2.5 (see my previous post for more information on that)

Installing SetupTools

I still prefer to install Pylint via SetupTools. SetupTools makes the easy_install command available from the shell to automatically install python eggs and sort out their dependencies. You might already have SetupTools on your system, but do you have the Python 2.5 version? I assume it is important (it does after all need to install packages into the correct site-packages folder). To install SetupTools, first download the egg file:

http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg

Open a terminal session and cd into the folder you saved it into. Type the following into the commandline:

$ sudo sh setuptools-0.6c11-py2.5.egg

This will install setup tools. When you run easy_install, make sure to use the command easy_install-2.5 to make sure you are addressing the correct Python installation on your system.

Installing PyLint

Type the following into your terminal shell:

$ sudo easy_install-2.5 pylint

This will install PyLint and all of its dependencies. You can now run PyLint from the command line (or from an Ant Task).

Using PyLint in Aptana

Aptana (now incorporating PyDev - again see my previous post) can integrate with PyLint to give you feedback on code style as you work and make changes to your source files. To configure Aptana to do this, open Preferences -> Pydev -> PyLint, check the the Use PyLint option and locate your Pylint executable. Despite the hint to look for a file named lint.py, I've never found such a file.

Instead, find the pylint binary. If you installed PyLint as per the above the following path should be where it is at:

/Library/Frameworks/Python.framework/Versions/2.5/bin/pylint

Click OK, open a python file and make some changes. Warnings and errors from PyLint should start appearing in the Problems view (Window -> Show View -> Problems if you can't see it). You can also specify a preferences file in the preferences page for PyLint to pass in any global settings (to ignore certain style rules, for example).

There you go. PyLint is installed and ready to use from the command line, an Ant Task or in Aptana. Let me know in the comments if you can suggest a better approach, have a question or any sort of general comment.

Getting Started with Google App Engine, Python 2.5 and PyDev on OS X 10.6

2011-06-19T05:54:00.000-07:00

Last week my shiny new Apple iMac 27 inch arrived and I thought I would share my steps in getting my Google App Engine (Python) environment set up. I have previously written about setting up Ubuntu 10.10 for the same purpose and that post remains one of the most popular on this blog.

To set up our Google App Engine environment we will need the correct version of Python, the Google App Engine SDK and a Python/GAE friendly IDE.

Installing Python 2.5

Firstly comes Python itself. Despite rumours of eventually upgrading the Python version supported on Google App Engine to 2.7, Google App Engine for Python still only supports Python 2.5. Mac OS X 10.6 ships with a distribution of Python 2.6. For hassle-free Google App Engine development it is always best to stick with the supported version.

Finding a packaged distribution of Python 2.5 for Mac is more difficult than it was the last time I looked, but you can still find a package here at pythonmac.org. Download the package and install it by running the installer. If you open a command prompt and type python --version you should see Python 2.5 is now the default version.

What happened to the existing bundled Python distribution? (or any others, if installed?) They are still present - the Python executable found by the shell is the one under /Library/Frameworks/Python.framework/Versions/Current, where Current is a symbolic link to a specific version's folder. Remember this if you ever need the shell to find a different version.

Getting the Google App Engine SDK

The Google App Engine SDK is easy to find, but the Google App Engine SDK download page is a touch misleading. In the download the links for the SDK, the link offered for Mac OS X is for the Google App Engine launcher. Contrary to my original assessment this download does in fact contain the SDK. When you finish downloading the package, open it and drag the launcher file to your applications folder. When you first launch it, the program will offer to create the necessary symbolic links in your /usr/local/bin folder.

If you would like more control over how and where the SDK is installed, you can grab the Linux/Other Platforms archive instead.

I never use the graphical launcher itself; I tend to start and upload my local App Engine programs via my IDE or an Ant build script (which is probably worth a post).

Setting up an IDE

I've previously posted about PyDev for Eclipse being my Python IDE of choice for Google App Engine and this has not changed. Fortunately getting it set up is now easier - PyDev is now a bundled component of Aptana Studio 3. Aptana Studio is built on the open source Eclipse IDE, and can either be installed as a standalone application, with all the extensibility of Eclipse, or added as a plugin to an existing Eclipse installation. Download the standalone version or find the details of required update site for a plugin installation here. The nice thing about Aptana for Google App Engine development is the inclusion of support for general web technologies and specific support for Django Templates.

You can also optionally install the Google Plugin for Eclipse. The vast bulk of this plugin is dedicated to assisting Google Web Toolkit development with Java, however, and out of the box Aptana seem to have included some or all of the limited Python/Google App Engine features in their studio, such as a way to launch the development server and a project wizard for setting up a Google App Engine project. ( To install the plugin, visit the installation page. Note that if you installed Aptana Studio as a standalone install you might need to satisfy some other Eclipse dependencies not included in the Aptana Eclipse distribution. I'm skipping the installation of the plugin at the moment.)

Within Aptana (or Eclipse), open the Preferences and add the Python 2.5 interpreter (Aptana -> Preferences -> PyDev -> Interpreter - Python). Click New and enter a name (eg. 'Python 2.5') and the path to Python 2.5. This will be the specific 2.5 path under /Library/Frameworks/Python.framework/versions/2.5/bin. Aptana/Eclipse will successfully find a range of folders to add to the PythonPath for that interpreter.

Starting a New Google App Engine Project

As mentioned, Aptana Studio now ships with a Google App Engine project wizard. This is helpful but not necessary - the most critical work to be done in making a Python project a Google App Engine project is to correctly add the SDK libraries to the Python Path recognised by the IDE. Doing this manually is described here. To use the project wizard instead, select File -> New -> PyDev -> Google App Engine project to create a new project. Make sure to change the Grammar and the Python Interpreter both to Python version 2.5.

You will be then be asked to locate the Google App Engine SDK files. If you installed the launcher application, the SDK files are actually within the installed .app file (folder?), which might make finding these files in Finder difficult. These easiest way to track down this path is through the terminal. Find where the dev_appserver.py symbolic link is (via which dev_appserver.py) and then inspect where it is pointing (ls -l usr/local/bin/dev_appserver.py). Grab this path from the terminal and paste it into the Aptana/Eclipse prompt.

You will then be prompted to confirm the folders to add to the Python path, with your only decision about which version of Django templates you will be using.

The new project wizard will close, and your new Google App Engine project will be ready.

A Better Prisoner's Dilemma Simulation?

2011-06-06T07:01:00.000-07:00

In my previous post I had started work on a python program named generations to try to replicate the simulations of Robert Axelrod in comparing differing Prisoner's Dilemma strategies. At the time I was simulating single organisms ('Critters') with different strategies accumulating food over a number of iterations with the critter with highest food at the end of the iterations judged the winner.

The result of this simple simulation was that two strategies - Grudger and Tit-for-tat - were continually vying for the top spot, with Grudger taking the mantle more often than not. In Axelrod's tournament Tit-for-Tat was the unambiguous winner, so I knew my generations script needed some work.

(Read my previous post for an introduction to Prisoner's Dilemma, why it interests me and what the various strategies were.)

I've been developing the simulation since. Technically it is a bit of a mess, but when I next refactor I will be focusing on new parts of the model aside from PD and so I wanted to deliver this conclusion to Prisoner's Dilemma now.

In this post I want to talk about how I changed the simulation and what effect these changes had. For the impatient - yes I have managed to tweak the simulation such that Tit-for-tat is the unambigously best strategy. Many critters had to die to make it so...

Go forth and multiply

In beginning my simulation only ran the same 5 organisms against each other, with each accumulating food indefinitely but at different rates as their strategies proved more or less successful. I specifically wanted to uncover how differing frequencies of strategies influenced the success of themselves or others, so I had to break free of this fixed model.

To this end I introduced reproduction into the simulation, and turned the simulation into one focused on measuring accumulated food to measuring reproductive success. My process for controlling reproduction was simple: When a critter accumulated enough food to pay an arbitrary cost of reproduction, another critter of the same strategy was created, and the cost deducted from the parent.

This change meant that any critter would eventually reproduce - and reproduce multiple times - if it was even marginally successful. More successful strategies would accumulate food faster and thus reproduce faster. Since the offspring could eventually reproduce, this change let to an exploding growth in the populations being tracked.

At the end of the simulation I could now see the success of the strategies: A count of the number of individuals with each strategy and their combined amount of food (which would become an increasingly irrelevant metric).

Adding reproduction and the resulting exploding population immediately pointed out the computational unfeasibility of running the simulation over a large number of iterations in its current form. Each additional critter had to interact with every other which form an exponentially growing computational task. Iteration after iteration, the simulation slowed down.

The results at this stage were unsatisfactory. Over the iteration counts I was prepared to run the simulation for, huge variations were seen from execution to execution. In some executions the Grudgers became the most prolific. In others it was Tit-for-tat. The random, cheater and sucker strategies were competitive on some runs, but not in others.

The source of this variance in outcome was the fortunes of the random strategy and those it interacted with. At this stage the random strategy produced the only non-deterministic part of the simulation. While growth was unrestricted the number of random interactions (interactions involving the Random strategy) was steadily increasing, even as the proportion of those interactions decreased as a part of the total number of interactions.

Everyone Eats... Everybody Dies

To put the breaks on the explosive population growth I next added the concept of death to the simulation. A Critter would die if it ever had an amount of food lower than 0. In theory a Critter could be 'suckered' through a string of interactions into this position, but with the success of the more altruistic strategies, this wasn't very likely. To make success in interactions essential, I added an attritional element to the simulation - That Critters, regardless of their success in interactions - expend a fixed amount of food each iteration.

These additions did not drastically change the outcomes of the simulation. Population growth was slowed, but it was not halted. Within a thousand iterations there were a thousand critters and every iteration more were added than were added the iteration prior. Reaching 1500 iterations took several minutes under these computation conditions.

The success of the random and cheater strategies declined noticeably, however. In the case of the random strategy, statistically assured runs of bad luck thinned their numbers. The cheater strategy could no longer afford to wait between successful cheats, and against the resurgent grudger and tit-for-tat strategies, success for a cheater most often came when encountering the offspring of these two strategies that had never met this individual cheater before.

A new form of Random

With the decline of the Random strategy in the population I wondered if a tweak to its strategy that made cooperation more likely than defection would prosper, or at least be reliably more successful than my initial equal probability implementation. In a simulated world dominated by altruists that held a grudge, defecting once ever two interactions was not performing well.

I changed the random strategy to be more initially flexible and allow the relative weighting of likelihood for cooperation, whilst breeding this weighting true in successive generations. I then added an additional random strategy to the simulation's initial state that was 3 times more likely to cooperate than defect, This strategy was known as Random 3.

To my surprise, the new cooperative random strategy faired no better in the simulation than the completely random strategy. I am still assessing why this might be the case. Grudgers would never forgive a defecting random, but a random strategy less likely to defect would be less likely to continually arouse the punitive defections from the tit-for-tat strategy, which should have been a noticeable advantage.

Limited Interactions

Until this point every iteration provided an opportunity for every critter to interact with every other critter. Universal interaction presented some problems. Apart from contributing to the exponential complexity of the computation, the sheer predictability of each iteration's interactions- varying only from iteration to iteration based on the addition of offspring- meant that altruistic strategies, once entrenched, gained an inevitable advantage that shielded every altruistic critter from risk. For example, if altruistic critters (ie critters with Tit-for-tat or Grudger strategies) made up 75% of the overall population, 75% of all an individual altruistic critter's interactions were certain to end in cooperation, making the consequences of the remaining interactions immaterial to the critter's eventual reproductive success.

Furthermore, as a simulation of non-zero sum social interactions, this universal interaction on every iteration was not realistic. No analagous organism has the capacity to continuously expand its capacity for interaction to all members of a population as the same population undergoes explosive growth.

Because of these issues I implemented a change to the available interactions for each critter each iteration. 5 iterations are chosen at random from amongst the possible combinations. Over multiple iterations and across the entire population, a given strategy will interact with a representative amount of each strategy in the population, but on any given iteration an altruistic critter might interact with entirely non-altruistic strategies. The effect of this change was not noticeable apart from an improvement in speed of the simulation.

The Constrained Environment

At this stage the population the Critter population within my simulation was unendingly growing, albeit at a slower rate, as countless Critters interacted and generated food out of... well nothing. In any real world ecosystem, however, the environment places a constraint on the population size that lives off of it. Apart from the need to achieve a minimum of success in the interaction stakes, my critters had an easy life of interacting and reproducing, and their success, especially for altruistic critters in an altruistically dominated world, did not come at the expense of any other critter.

I wanted to place a population cap on my Critters. This population cap would ideally not sacrifice the successful Critters from current and future generations in order to be maintained. I wanted the population cap to exert a selection pressure on the population that was analagous to the selection pressures in the real world.

To this end I decided not to directly enforce a population cap, but rather, enforce a hard limit to the amount of food available each iteration. With this limit in place, once interactions within an iteration had exhausted the environmental supply of food, no further interactions (and no additions or subtractions of food) were possible. I had no basis to prefer any particular strategy to always appear at the head of the line however, so I was careful to again randomise the interactions which were being considered.

Implemented this constrained environment had a dramatic effect. Firstly, the endless growth was over. For a particular environmental food limit there would be a set population that never varied more than 10 or 20 Critters either side. Suddenly running simulations across much larger numbers of iterations was possible. The effect on the outcome was startling as well: In simulations of 10,000 iterations, cheater and random strategies died out completely early on. The sucker strategy performed well for a time before dwindling and dying out. As before, both guardedly altruistic strategies grudger and tit-for-tat performed well out of the gate, but over the time period the grudger strategy fell by the wayside and leaving tit-for-tat to take the field - just like in the 1980's Axelrod Tournament I was trying to replicate.

Enabling Comebacks

That might have been the end of the story, but I wanted to ensure strategies which were extinct early in the simulation didn't fair better in the late game. For example, would a cheater strategy have more success against a field dominated only by Tit-for-tat critters as opposed to the tit-for-tat/grudger strategy duo seen in the early game? Would a Cooperatively slanted Random strategy do better without the Cheater strategy? Could any strategy upset an entrench tit-for-tat population?

Apart from answering these questions, the question of realism also begged the question. The history of evolution shows that once a trick is learnt, many organisms can retreat to old strategies and ways of life across evolutionary timespans when the environment changes.

I could have simply monitored the environment for strategies that had completely died off and reinjected candidate critters from those strategies at some future time. This seemed overly artificial to me however, so I instead engineered a constant (and yet small - 1%) chance of mutation in any Critter's offspring. When this mutation happened, a Critter with any strategy could emerge. This ensured that no matter which strategy or strategies were in pre-eminence, a 'background noise' of alternative strategies were always waiting in the background to take advantage of a beneficial change in the environment. (An environment is beneficial or not is solely dependent - in this simulation at least - on which other strategies are present in the environment).

There was no amazing change in outcome with this facility for mutation in place. The 'loser strategies' (the randoms, cheaters and later, suckers and grudgers) would invariable go extinct, reappear in small numbers, perhaps become extinct again before re-appearing.

Try it yourself

If I have been vague on providing specific numbers in this description (such as the amount of food consumed per iteration, the exact payoffs for cheating, defection and cooperation, etc) it has been because I have been acutely aware of how arbitrary some of my costs and allowances have been. Gross changes these arbitrary numbers will change the economics of strategy success.

The way these economic constants affect the simulations is interesting and worthy of experimentation.

Question: If the payoff for cheating is doubled, how what effect does this have on the population?

Answer: Cheaters lead early before the inevitable population crash. Grudgers rise highest out of the critter carnage.

Question: Does an environment with a higher food-cap, supporting a higher population, have more room for alternative strategies on the fringes?

Answer: Only slightly

If you want to find out for yourself, download my generations python script (use tag v1.5-PD for this version - as I mentioned in my previous blog generations will not track solely the Prisoner's Dilemma scenario).

The script should be easy to use - run the script without arguments to use default values

The -i or --iterations option lets you specify the number of iterations run
The -t or --track option lets you specify an arbitrary critter to track (the initial critter names are c1, s1, r1, r2, t1 and g1 after the strategies. Their offspring are c1_1, c1_2 and c1_1_1, for example, named after their lineage and their birth order from their parent).
The -r or --reporting options lets you specify the reporting interval
The -f or --filename option lets you specify a filepath to save a tab-separated datafile to for analysis in spreadsheet and charting programs.
Use -h or --help to get usage information.

For example:



Mimas:src bendavies$ ./generations.py -i 10000 -f cheatpayoff.tsv -r 50

Runs the simulation for 10000 iterations, reports every 50 iterations and outputs the population data to cheatpayoff.tsv

I wouldn't pay too much attention to the technical implementation, since the code is somewhat between plugin, event and direct manipulation architectures at the moment.

What is next?

This exercise was interesting to me since it allowed me to observe emergent population trends based on individual strategies. More-over, I am keen to leave the confines of the prisoner's dilemma simulation and try simulation of other behavioural strategies under evolutionary processes. Some answers I wouldn't mind pursuing would be:

Can I evolve cooperative strategies from scratch against a selfish population?
What factors influence parental investment?
How might embryological complexity weigh in reproduction strategies?
Can I define a critter genome that is flexible enough to 'mutate slightly', light enough to be computationally feasible and yet expressive enough to allow interesting emergent behaviours and qualities to emerge?
How would critters move, if given the choice, in an environment whose food resources did not reset every iteration, but rather depleted. How would fragmentation of population affect interaction strategies?
Would it be possible to evolve critters from a common base to two or more very different lifestyles?

All of this will take time, and I need to take a break to do some work on My Web Brain and he3-appengine-lib. Still, I'm tempted to simply dive right in....

If you have any comments on this above exercise please let me know. This has been an absurdly long blog post, but I wanted to share my experience and take the opportunity to examine how this experiment played out.

My Generations Project and Prisoner's Dilemma

2011-05-23T02:46:00.000-07:00

Apologies for the lack of recent posts. I have been 'learning technical stuff' mostly at work and mostly not in a form that can be easily shared. I have recently found the time to start a new project I call Generations - a python program I am developing to simulate some aspects of natural economy and natural selection.

Recently I re-read Richard Dawkins' The Selfish Gene and Robert Wright's The Moral Animal and was fascinated by the idea that computer simulations could tell us useful information about effective behavioral strategies, how those strategies prosper and in turn what effect these strategies may have on their own effectiveness and that of other strategies.

Currently this initial version of my program only simulates Prisoner's Dilemma. Wikipedia describes the Prisoner's Dilemma scenario, a classic game from the field of Game Theory, like this:

Two suspects are arrested by the police. The police have insufficient evidence for a conviction, and, having separated the prisoners, visit each of them to offer the same deal. If one testifies for the prosecution against the other (defects) and the other remains silent (cooperates), the defector goes free and the silent accomplice receives the full one-year sentence. If both remain silent, both prisoners are sentenced to only one month in jail for a minor charge. If each betrays the other, each receives a three-month sentence. Each prisoner must choose to betray the other or to remain silent. Each one is assured that the other would not know about the betrayal before the end of the investigation. How should the prisoners act?

Prisoner's Dilemma is an example of a non-zero-sum game, meaning that players are not attempting to beat other players, just achieve the best outcome for themselves. That is to say: winning doesn't automatically imply the other player or players have lost. When played repeatedly and scored (ie. 'Iterative Prisoner's Dilemma') the interesting question becomes which strategy, if any, is the most successful against the widest range of counter-strategies. Always cooperate? Always defect? Some mixture of the two?

The relevance of this game to the field of Darwinism (which both Dawkins and Wright were asserting) is that this situation of deciding to trust (or abuse trust) is a good abstraction of the issues underlying reciprocal altruism, one of the motivators of altruistic (or 'nice') behaviour in the neo-Darwinist worldview. Both Dawkins and Wright discussed a competition held in the 1970s and again in the early 1980's where Robert Alexrod challenged interested parties to design strategies for the game and test them out against the strategies of other players.

I am running my own tournament of sorts within Generations. Instead of the prisoner analogy however, my program simulates 'critters' adopting different strategies and receiving or losing 'food' as a result of the success or failure at the prisoner's dilemma game. I have currently implemented the following strategies:

Sucker, who always cooperates blindly
Cheater, who always defects
Random, who randomly cooperates or defects
Grudger, who cooperates with anyone who has never defected against them, and
Tit-For-Tat, who cooperates initially but punishes defections in the previous round by another player with a defection in the current round.

My ratio of scoring for defection and cooperation is arbitrary and by default the simulation runs over 1500 iterations. The most successful strategies so far are Grudger and Tit-For-Tat, With Grudger always just beating out Tit-for-Tat.

In Alexrod's tournament, Tit-for-Tat was judged the winner. One reason why my simulation might be playing out differently is that I have precisely one critter for each strategy, and whereas once bitten Grudger will never trust random again, Tit-for-Tat will 'trust' Random again after a random cooperation.

The specific economic rewards I have assigned might also be making a difference.

In the real-world, less successful 'critters' would reproduce in proportion with their success, and their hereditary strategies would be passed on their offspring. Critters with bad strategies would shrink as a proportion of the population, altering the dynamics of which strategy is more effective. Simulating this, as well as some other strategies I found on the Wikipedia page, is what I will do next.

In the meantime, you can get my Generations program on Github. Use the v*-PD tags to download the Prisoner's Dilemma only versions - I will be adding decidedly un-Prisoner's Dilemma elements (ie. other ways to simulate critters) in the near future. The current PD tag is v1.0-PD.

Let me know if you have questions, or comments.

PagedQuery not working on SQLite Google App Engine SDK..?

2011-02-18T23:18:00.000-08:00

A recent interchange with a helpful user gave me a heads up that PagedQuery - my simple query wrapper for providing paging in Google App Engine - might not work when testing locally using the SDK with the optional SQLite backend (there are number of open issues about SQLite on the SDK project page).

I haven't used SQLite backend personally, so for the moment I can't confirm this. From my discussion with the user it seems the SQLite backend does not implement cursors properly. I'm slowly reviving my he3-appengine-lib development environment so hopefully I will be in a position to confirm this soon.

(Thanks Milo for bringing this to my attention!)

Naked Domains on Blogger

2011-02-07T04:20:00.000-08:00

Google sometimes isn't great at updating its documentation. Or perhaps they need to improve their search. If you search for information about how to use a naked domain on Blogger, you will get the impression from results that this isn't possible.

The highest search result I found from Blogger seems to say the same thing:

To simplify custom domain configuration on your blog, Blogger only accepts non-naked domains in our sign-up process. If you try to enter a naked domain in the advanced settings of the Settings | Publishing tab, you will run into an error stating that Blogs may not be hosted at naked domains.

To fix this, simply enter the non-naked version of your domain and save your settings.

But you can use a naked domain! (sort of). No hosting, redirect pages or .htaccess files required. You can achieve the same result by configuring some A records on your naked domain to redirect to Google's server.

I knew I saw this somewhere while I was setting up Ferny Blog but it has taken me far too long to find the information in Google's documentation.

But found it I did - Look at the second half of this page

Eclipse-Based Wireframe and Mockup Designer: WireframeSketcher

2011-01-17T23:43:00.000-08:00

If you are in need of a capable wireframe and mockup designer for Eclipse, this post will interest you. I was contacted by Peter Severin from WireframeSketcher last month. Peter asked if I was interested in trying out his Eclipse-based wireframe and mockup designer. He also provided a free license key. Thanks Peter!

In truth, I had been looking around for an easier solution to wireframes and mockups. I had been trying to use Adobe Fireworks CS5 in combination with a course from Lynda.com for this purpose but I found the solution and workflow overkill for a web application designer working on his own projects. I need something simpler.

And then there is the issue of platform support. As I have mentioned in previous posts, I develop on Ubuntu and Mac OSX. My Windows license for CS5 would not help me on those platforms, and running additional virtual machines for the purpose simply complicated my workflow and source control.

Balsamiq looks like a great tool, and I have heard many good things about it, but it suffers from the same lack of platform support on Ubuntu and is otherwise just another tool to manage and integrate into workflow and source control.

WireframeSketcher, on the other hand, is built on Eclipse. It is available in a standalone application, but really comes into its own as a plugin into your existing Eclipse environment. WireframeSketcher, like Eclipse, is cross-platform, and works just as well on Ubuntu or Mac OSX as it does on Windows.

Working with wireframes and mockups in your IDE saves time and simplifies workflow. Without the need to switch applications I found I was using wireframes much more effectively. WireframeSketcher saves designs and mockups as individual .screen files, making them easy to manage within the same project as your source code and using the same source control.

WireframeSketcher is like other full-featured plugins for Eclipse. It provides an editor, several views and a perspective for easy switching to wireframe or mockup design.

Within those Eclipse conventions, WireframeSketcher is like many other wireframing solutions: you drag widgets from a palette onto your canvas. A properties panel allows fine-grained control and configuration. Clickable links can be defined between screens, and boilerplate or common layout can be defined in master screens that is reused across other mockups. These features make WireframeSketcher both easy to use and the equal of other wireframe tools in all the ways that matter to me.

Apart from the benefits from these features and that the integration with Eclipse provides, I will mention three features that surprised and impressed me: Firstly, not only can wireframes be exported to PDF, PNG and to the clipboard, this export can be done as a batch operation. Secondly, Wireframe styles and controls text and data tables using a powerful text markup language that provides a lot of flexibility. Thirdly, the plugin has been remarkably solid and well behaved, unlike many other Eclipses plugins I have tried. It feels polished and refined.

Which is not to say that WireframeSketcher could not be improved. The widget palette could be categorized and easier to navigate. Using the tool is quick, but I think the time spent trying to find a particular widget in the palette could be reduced. WireframeSketcher does not have as many out-of-the-box widgets in the palette as some other tools, although this has not caused me an issue yet. Lastly, the assets functionality, which complements the standard palette with images and master screens of your own, could be improved. Assets require a specifically named and located folder in your project structure. Personally I would prefer a customizable location for these assets to suit the variety of ways Eclipse may be used. Reuse of assets between projects is not addressed.

I should be clear, however, that these criticisms are relatively minor and have not prevented me from using WireframeSketcher effectively.

WireframeSketcher is an excellent mockup and wireframe tool, and I definitely recommend giving it a try, especially if you live inside the Eclipse IDE. For those of us that need to both design and develop applications the integrated nature of the plugin both enhances your productivity and simplifies workflow. In case I have not been clear, I really like WireframeSketcher!

I would like to thank Peter again for introducing me to WireframeSketcher and providing a free license key, and I hope this blog post plays a part in providing more exposure to his excellent tool.

Static Resources and Cache-Busting on Google App Engine Python

2011-01-03T20:11:00.000-08:00

Hello and happy new year!

In a couple of previous posts I discussed using Google Closure JavaScript Compiler in your build process to minimise the size of your JavaScript resources. Arguably a bigger win for the end user experience is to ensure the caching of all of your website static resources (JavaScript, stylesheets and images) is properly optimised. In this post I want to share how I think I have achieved this Google App Engine (Python).

Ideally, we would like the end user's browser to use its cache for every static resource, on every request, until that resource changes because we have deployed a new application version. Adding the (relatively new) default_expiration and expiration directives to your app.yaml file will achieve the first part of this requirement:

- url: /images
  static_dir: images
  expiration: "99d"

Behind the scenes, these directives add additional headers to the HTTP response, letting the browser know that they can safely serve these files from cache on subsequent requests.

But how do we tell the end user browser when our static resources have changed?

The solution is called cache-busting, and there are a number of similiar approaches. These involve changing the request URI for the static resources. This could mean appending a version identifier to the individual JS, CSS and image filenames. When the browser sees the request for a nominally different file it has no reason to pull the cached copy, so it requests the file again, with the specified caching headers in place again to prevent it being requested again:



First Request

URI: yourapp.blogspot.com\images\banner_v1.jpg (requested)



Second Request

URI: yourapp.appspot.com\images\banner_v1.jpg (not requested - cached)



Third Request

URI: yourapp.appspot.com\images\banner_v2.jpg (updated file - re-requested)



Fourth Request

URI: yourapp.appspot.com\images\banner_v2.jpg (not requested - cached)

This technique works, but how you consistently achieve it without introducing errors, confusing source control with trivial changes or simply absorbing too much development time is worth a discussion. Making manual changes to resource file names, the app.yaml static resource directives and the HTML for every application deployment is not very practical.

My approach to removing the development pain associated with cache busting works as follows: Instead of mangling the actual filename of the resource, I change the URI path. This means filenames themselves do not need to change. I also setup static file directives in the app.yaml such that the intermediate path variance (due to changing version) is ignored:

- url: /images/.*/(.*)
  static_files: static/img/\1
  upload: static/img/(.*)

Using a directives such as this, the position or naming of the actual static resource in the source code never changes. For example, the following URI's all serve the same static file:

yourapp.appspot.com/images/v1/banner.jpg » /static/img/banner.jpg
yourapp.appspot.com/images/v2/banner.jpg » /static/img/banner.jpg

Within the HTML (or any other resource specifying a URI) the source code contains a known token:

img src="/images/xx-replace-with-version-id-zz/banner.jpg"

My build script, written in Apache Ant, replaces the known token in the prepared build files with the version identifier. The original source files (whose known token works fine in development) are not updated:

<copy overwrite="true" todir="${path.build.src}">
    <fileset dir="${path.assets.src}">
      <include name="**/*.html" />
      <include name="**/*.css" />
    </fileset>
 
    <filterset begintoken="xx-" endtoken="-zz">
      <filter token="replace-with-version-id" value="${generated.id.timestamp}" />
    </filterset>
</copy>

The build script goes on to deploy the application either to its online test or production versions and then uses the same version identifier to tag newly deployed source code in my source control.

I know some developers prefer using a dynamically generated version identifier (using os.environ['CURRENT_VERSION_ID']) within their HTML or other URI carrying source. Since I already use a build script for JS compilation and Python linting, my approach of replacing a known token makes more sense to me - It removes complexity from my code and allows a full correspondence with the version identifier between the deployed version and the source control records.

I'd be interested in hearing if someone has an idea for improving this process or thinks another approach has distinct advantages I have not considered. If you have questions, leave them here as well.

Automating Closure Compiler with Ant

2010-12-12T15:45:00.000-08:00

In a previous post I talked about how to use Google's Closure Compiler to optimize and minify JavaScript files. Now I have shifted to a dual Ubuntu and Mac OS X development environment, I am determined to invest in a strong automatic build process. This is just as well, since compiled JavaScript is impossible to maintain or develop. Compiling JavaScript needs to be performed automatically and transparently.

I am building out my build scripts using Apache Ant. Ant is cross-platform compatible and is well supported in the Eclipse-based Aptana and Pydev development environments. Automating the JavaScript compilation using Ant proved a bit troublesome.

In Ant you use the Apply Task to run executable programs. Running Closure Compiler using the Apply task for each JavaScript file was the first hurdle, but Apply has obvious support for passing a FileSet to determine what those files would be. The bigger challenge was working out how to specify the input and output files using redirection.

My task was made a lot easier when I found a useful example of using Ant with JSMin as the last example on the documentation page, which is invoked in similar fashion to Closure Compiler. Modifying the example I came up with this Ant Target:


  <target name="-compilejs" depends="-copybuild"
   description="(runs the closure js compiler against all javascript files to replace the files in the build folder">
 
   <!-- modeled off example found at http://ant.apache.org/manual/Tasks/apply.html -->
   <apply executable="java" parallel="false" failonerror="true" addsourcefile="false">
     <arg value="-jar" />
     <arg value="${path.assets.bin.closurecompiler}" />
        <arg line="--warning_level QUIET" /> 
    <arg line="--compilation_level SIMPLE_OPTIMIZATIONS" />
     <fileset dir="${path.assets.src}" includes="**/*.js" />
     <redirector>
        <inputmapper type="glob" from="*.js" to="${path.assets.src}/*.js" />
        <outputmapper type="glob" from="*.js" to="${path.build.src}/*.js" />
     </redirector>
    </apply> 
  </target>

This isn't perfect. This approach always compiles all of the JavaScript files, even when they have not been altered. and JavaScript compilation with Closure Compiler is currently the longest running activity of my build. I will be trying to tweak Apply invocation to get this fixed.

The other problem is the suppression of warnings. Warnings appear inline in the output JavaScript, which breaks the JavaScript file but does not fail the build, which is what I would prefer. I am wondering if I redirect the error output and check its contents and manually fail the build whether that will catch the problem, at which point I can remove the (dangerous) suppression of warnings.

I will let you know when I surmount these problems, but if you have any thoughts let me know. I will also go into more detail about the rest of my build process - specifically as it applies to Google App Engine, which is probably the more unique aspect, and therefore, most useful.

Using Closure JavaScript Optimization and Minification

2010-12-07T04:02:00.000-08:00

Recently I attended cf.Objective(ANZ), a Melbourne based conference for Enterprise ColdFusion developers. A session which stuck with me was titled 'Speed Matters' and it was presented by Mark Stanton. Mark presented a compelling case for optimizing a web site's HTTP usage for a faster and more efficient experience for the user.

Now that I have rebooted my development environment in Ubuntu and Mac OS X I am looking forward to applying some HTTP optimization to my own site, My Web Brain. There are a lot of different areas I can (and will) target, but first off is JavaScript minification and optimization.

To get some experience with a new tool which did more than simply strip out comments and whitespace, I decided to try Google's Closure Compiler. Closure JavaScript Compiler is distributed as a simple jar file and works with standard input and output streams:

$java -jar compiler.jar < in.js > out.js

The compiler can also accept multiple JavaScript filenames from command line switches and output to a single combined file:

$java -jar compiler.jar --js in1.js --js in2.js > out.js

Closure Compiler has three level of optimization:

Remove WHITESPACE_ONLY - basically as per the venerable JSMin.
SIMPLE_OPTIMIZATIONS - removes whitespace, shortens private variables and other provably safe changes to the code.
ADVANCED_OPTIMIZATIONS - as per Simple Optimization but much more aggressive. This level relies on JSDoc comments to help the compiler understand which optimizations are safe for the code.

Passing the desired level optimization to the compiler is another switch:

$java -jar --compilation_level WHITESPACE_ONLY

You can read more about optimization levels and the differences here. Closure also helpfully provides warnings about poor or dangerous coding patterns. The compiler can also be accessed as a remote RESTful API, but I have not played with that yet.

Read the documentation for more information. For myself using closure compiler will likely be a gateway to learning about jsdoc, JavaScript code linting and better overall JavaScript coding. Awesome.

But wait up. How do you work with JavaScript code once you have thoroughly minified and optimized it? The short answer is that you don't. In an upcoming blog entry I will explain how I brought Closure Compiler into my build process.

Link: Importing Python Modules

2010-12-06T04:53:00.000-08:00

I ran into this blog post on importing Python Modules a couple of weeks ago and it help solidify my understanding of how import and from..import (and __import__()) operate in Python and what they actually do.

The PEP 0008 Python Style Guide does not state a preference for import style but this blog post is fairly blunt in recommending import (and its lower amount of name pollution) over other approaches when possible.

A good, clear read. Thanks Fredrik!

Python Code Style

2010-12-03T19:36:00.000-08:00

Code conventions are important in software development. In Python, where whitespace (specifically indentation) can have a functional impact on your software, coding conventions are critical, especially when code needs to be shared and reused.

Python does have a widely accepted (and widely tweaked) Style Guide called PEP 8. I saw it a while ago but didn't have the time to digest and then laboriously apply it to my active codebases.

When I recently rebooted my development environment on Ubuntu and Mac OSX I noticed the PyDev IDE (now shipping with Aptana Studio) had support for PyLint. PyLint - like other 'linting' programs - can examine your code and find non-compliances to a specific standard of code formatting. PyLint, when used as commandline tool, can provide a detailed report on your code style.

The great thing about the integration with PyDev, however, is that PyLint shows non-compliances in your currently open files and flags them as warnings and problems in your problems view. This makes deciding to implement and meet a code standard a lot less cognitively burdensome.

PyLint also highlights bad code smells - components and practices which don't smell right that might need refactoring. PyLint can be given global instructions to ignore certain issues, or specific source files can include comments to change how PyLint analyses the file. This gives some ability to customise the behaviour and ignore PyLint where pragmatic.

To enable PyLint in PyDev, first make sure PyLint is installed. Installing PyLint is as easy as installing any python program:

$sudo easy_install pylint

If you have different python versions in your environment, use the correct invocation of SetupTools for that version. I need to target Python2.5 for Google App Engine, so I use:

$sudo easy_install-2.5 pylint

Once pyLint is installed, within Eclipse (or Aptana etcetera) visit the Preferences page and specifically the PyDev > PyLint section in the hierarchy. Locate the actual PyLint script (lint.py) and enable PyLint.

PyLint is really helping me clean up my code (I admit it - I am a recovering tab-fiend).

Git Mac OSX Binaries from Google Code

2010-11-22T21:02:00.000-08:00

I was recently in Melbourne attending the cf.Objective() ANZ conference and was hopeful of getting some further setup done on my MacBook Air to make it ready for prime-time development. This will be my third development environment in use, after Ubuntu and Windows 7.

I wanted to jump to distributed version control using Git (a DVCS being essential for offline development) but found that both compiling from source and using MacPorts require a version of the C compiler not installed on my MacBook Air. Worse, the way to get the C compiler was to install Xcode. Xcode, if you didn't know (I didn't), is a hefty 2.9 GB download from Apple containing their development platforms.

Luckily I came across maintained binaries for Git hosted on Google Code, so the XCode install on my limited-space Air is not required. The binaries are working well. Thanks to the maintainers for keeping this out there.

Changing the Hostname on Mac OSX

2010-11-09T17:55:00.000-08:00

My new MacBook Air is my first Mac OS device, so I have some learning to do to come up to speed with the new environment. I've noticed that the OS setup routine at the start has set a daft hostname (containing my full name) so as a matter of priority I wanted to change it.

To change the hostname on a Mac OS X machine, use the following:

sudo scutil --set HostName mimas.local

where 'mimas' is the required hostname

I found this tidbit in a blog post (thanks Larry!)

You might see a few of these OS X related posts, sorry, as I come to grips with it all. I promise to graduate from the elementary stuff as soon as possible :)

Quick Note: Eclipse and Subclipse on Ubuntu (for Appengine)

2010-11-08T20:49:00.000-08:00

I have previously written about configuring PyDev on top of an Eclipse/Aptana installation for Google App Engine development. Now on the Ubuntu platform instead of the windows, the process is easier these days since Pydev ships with the new Aptana Studio 3 Beta.

I thought I would write a quick note to future Ben (i.e. me) about configuring Eclipse and Subclipse on Ubuntu. Subclipse is a Subversion plugin for Eclipse (the alternative is Subversive, which I haven't tried).

Briefly my Eclipse install (for reference):

sudo apt-get install eclipse (if using eclipse base install)
Install Aptana Studio 3 beta, either standalone or as Eclipse plugin
Install Google plugin for Eclipse from within Eclipse (page with download sites)
Configure Python 2.5 interpreter within Eclipse (assuming you have already installed Python 2.5)
Create new Google App Engine project (under PyDev grouping)
Configure Python Library Paths for Eclipse

To add Subclipse:

sudo apt-get install subversion
sudo apt-get install libsvn-java
install subclipse within Eclipse (download sites)

Since I eventually managed to install Python 2.5 I've had a great experience installing software on Ubuntu. I've experienced some detours but none of them have been too long. Let me know if I have missed anything, or could have done something better.

Using the provided OpenID functionality on Google App Engine

2010-11-07T23:42:00.000-08:00

I've just released a small, simple application to he3-appengine-lib demonstrate using the provided OpenID support Google added to App Engine in version X. I wrote the application from scratch to learn how a real application would behave, as I wasn't quite following the documentation.

When you set the authentication option for your application to OpenID to behaves very similiar to using the standard Google authentication. When you use Google authentication you redirect your users to the URL created by create_login_url() to login and create_logout_url() to log out. Before and after you can use the rest of the google.appengine.api.users API to find out the current logged in user, their nickname and email address.

Using OpenID works the same way with a only a few caveats:

When calling create_login_url(), you need to pass the required OpenID service provider or actual OpenID URL. This is because the the service provider provides the login screen (if required - the user might already be signed in).
Some service providers might not supply an email address, which may be problematic depending on your requirements. Even if you don't need to email the user, the email address is a dependable identifier, whereas nickname is not. For example, MyOpenID.com does not supply an email address by default, although this can configured at the time of the login. In my demonstration application, I detect when an email address has not be provided,. and reject the login.
If you use directives in the app.yaml file to mandate that login or admins are required for particular pages, you must implement a handler for /_ah/login and /_ah/login_required. Standard approach would direct the user to your dedicated login page (but I took the easy way out in my application and redirected to the home page)

Overall the transition from Google authentication to OpenID is not challenging from a technical standpoint. As Google themselves discuss, some of the challenges arrive from simple usability choices in how you communicate to your users what their options are.

You can find my demonstration application on he3-appengine-lib, and you can find a working copy at he3-openidtest.appspot.com, although I can't guarantee I will keep it around forever.

I am yet to use the OpenID support in a production application, and I am sure I will learn more when I do. I'll record anything else of use that I find out here.

Mapreduce to be built in to Google App Engine Environment

2010-11-04T06:00:00.000-07:00

If you read the release notes for the latest Google App Engine python SDK (1.3.8) you might have noticed the available declarations in the app.yaml application configuration has been increased. There are some interesting additions.

The include directive allows the app.yaml file to be broken apart for better reuse. Within the one project this might not make sense, but you could imagine entire utility modules suddenly being a lot more transportable between projects and easily added to an existing application.

The builtin directive, built on top the include functionality, provides easy inclusion of built-in handlers, including appstats, the administration console, the remote API and a 'datastore_admin'.

The 'datastore_admin' builtin seems to be built on the appengine-mapreduce project. As I have previously posted, MapReduce is a methodology of dividing a large task into smaller pieces of work that can be performed in parallel, before being reduced to a single a outcome.

I am excited about the new builtin - since I discovered appengine-mapreduce I have considered it essential for maintaining a large set of data in the datastore.

In case you were excited to try out the new Mapreduce functionality, you need to wait a little longer - until the next release of the SDK at least. Oddly enough the documentation lists the 'datastore_admin' as a valid builtin option, and the libraries are now included in the SDK, but some import errors prevent it from being used (namely simplejson and graphy.backends).

As this discussion thread on the Google Group shows, it isn't quite ready for primetime yet. Hopefully the next release will tidy things up.

Installing Python Google Appengine SDK on Ubuntu 10.10

2010-10-31T18:59:00.000-07:00

I am now back from my long hiatus from this blog (my excuse was a busy work schedule, wedding preparations and then the honeymoon) and I decided to have a look at Ubuntu 10.10. Ubuntu is a popular desktop Linux distribution which offers greater distribution-wide integration, consistency and polish than many other distributions.

I have a long but not terribly recent history of trying Linux but never fully adopting it. As an exercise I wondered how feasible it would be to port my development environment for Python Google App Engine (GAE) to my new Ubuntu virtual machine. Most things on a Linux platform are possible as long as you know what you are doing, but I don't, so getting GAE running required some investigation.

Ubuntu - at least as installed by the my VMware Easy Install process - presented some challenges. Ubuntu no longer installs the correct version of Python, for example, nor does it install source headers for libraries some Python extensions required by the SDK need to compile. After discussing my problem on the Google App Engine forums I found two very helpful blog posts that present the two different approaches to setting up Google App Engine Python SDK under Ubuntu 10.01.

Both approaches achieve the same thing. The harder approach (well, longer) is to download the Python 2.5 and related sources and install them manually. The easier (and shorter) approach is to download the maintained Python 2.5 packages from a third party and use setup tools to do to the rest.

The following is my recipe which is mostly a tweaked version of the easier approach.

Create directory structure for Google App Engine SDK and Google App Engine projects. In the terminal:

cd ~ mkdir Projects mkdir Projects/gae

(I am using exactly the same directory structure here as Boris does in his shorter approach. As mundane as it is I appreciate his detail here; As a Linux newbie I often wonder where to put things or even if it really matters)
Download and extract the latest Google App Engine SDK for Python (currently at 1.3.8) to ~/Projects/gae/google_appengine
Install Python 2.5. In this step I use the DeadSnakes repository for a maintained package of Python 2.5. I could also download the Python 2.5 source and compile it, but as wdenouden's Longer Approach shows you need to take care to already have SSL and Sqlite installed.

sudo add-apt-repository ppa:fkrull/deadsnakes sudo apt-get update sudo apt-get install python2.5
Install Python Setuptools (ie easy-install) for Python 2.5

cd /tmp wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg sudo sh setuptools-0.6c11-py2.5.egg
(optional) if you need the GAE imaging API, you will need to install PIL. PIL needs to be compiled for the local system, so it needs both the Python 2.5 headers and the headers for the JPEG library installed on ubuntu:

sudo apt-get install python2.5-dev sudo apt-get install libjpeg62-dev sudo easy_install-2.5 pil
Create a simple shell script to run the local development server and mark it executable by all. We'll put it firstly into the guestbook demo folder, test it and then copy it to the new_project_template.

cd ~/Projects/gae/google_appengine/demos/guestbook echo "#!/bin/bash python2.5 ~/Projects/gae/google_appengine/dev_appserver.py ./" > run chmod a+x run
Test the new file and our Google App Engine SDK install

./run
Assuming all works well, copy the shell script to the new project template:

cp ./run /home/ben/Projects/gae/google_appengine/new_project_template/

That is my pass at a recipe for getting the Google App Engine Python SDK ready to go on Ubuntu 10.10. If you see something I have missed or a better way of achieving the same thing please let me know in the comments.

Adding Library Paths to Google App Engine Python Projects

2010-08-24T23:07:00.000-07:00

Apologies - It has been a while since my last post in this blog. Wedding preparations and a focus on design work for a new version of My Web Brain (along with my day job as a SQL developer on a SAP rollout) has kept me busy.

Did you know you could alter the system path dynamically on Google App Engine Python? This tip is something that I am sure old hands at Python consider basic, but I hadn't seen the technique done before.

As part of some research of the next version of My Web Brain, I evaluated some Google App Engine frameworks, one of which was Tipfy. Tipfy uses this technique front and centre in its main.py file to alter the system path:

import os

import sys

if 'lib' not in sys.path:

    # Add /lib as primary libraries directory, with fallback to /distlib

    # and optionally to distlib loaded using zipimport.

    sys.path[0:0] = ['lib', 'distlib', 'distlib.zip']

Taking control of the system path frees your package structure from the normal constraints of the project folder structure. And as the Tipfy code demonstrates, it also provides a good way of separating framework, vendor and customised code bases with the desired fallback order.

Update 24 Nov 2010: Just a note on what is obvious in hindsight. You need to update sys.path prior to importing from modules on the affected path(s). The same is true for modules with dependencies on the affected path.

Timezones using Gae-Pytz

2010-07-03T01:09:00.000-07:00

This is a short post about my good experiences moving from standard pytz to the gae-pytz library for Google App Engine. Pytz is the standard python library for defining timezone information. Gae-pytz is a version of pytz optimised for Google App Engine.

If you have read one of my previous posts about implementing timezone support in Google App Engine, you'll know I have used the full pytz library in the past. In addition to improved performance, gae-pytz offers a couple of key advantages in comparison to vanilla pytz:

All timezone objects are packaged and deployed in a zip file. There are hundreds of timezone files in the normal pytz distribution, which makes the library particularly burdensome in the 1000-file maximum limit imposed by Google App Engine.
Gae-pytz has built-in memcache caching of individual timezone objects.

My experience with implementing gae-pytz in my application was great. For the most part it is a drop in replacement with no changes to code required. I could actually remove all of the caching code I had put in place since the provided caching already provided the functionality. If you have previously built an application with pytz for app engine, check gae-pytz out.

Maintaining the App Engine Datastore with MapReduce

2010-06-28T06:30:00.000-07:00

One of the items on the todo list for my application My Web Brain is to add features which will require changes to the data model used by the application. Making changes to an application's data model in the App Engine datastore is sometimes more painful than it should be, since the developer remains responsible for ensuring that existing entities are made consistent with the new model.

Having recently read about appengine-mapreduce I decided I would write a simple Mapper that looks for and fixes missing property values and invalid ReferenceProperty keys. You can find the ModelHygienceMapper (as I call it) at he3-appengine-lib in the new mappers.py file (some simple documentation coming soon).

The code for the ModelHygieneMapper that does the work very simple, and mostly worth listing in the event that someone can suggest an improved approach.

@staticmethod

  def process(entity):

  '''Checks and repairs model integrity of the passed entity 

  1. Removes dangling references

  2. Sets undefined datastore values to the default

'''

  props = [x for x in entity.__class__.__dict__.values()\

           if isinstance(x, db.Property)]

  changed = False

  for prop in props:

    if not prop.get_value_for_datastore(entity):

      prop.__set__(entity, prop.default_value())

      changed = True          

    elif isinstance(prop, db.ReferenceProperty):

      if not db.get(prop.get_value_for_datastore(entity)):

        prop.__set__(entity, None)

        changed = True

  if changed: yield op.db.Put(entity)  

The concept of Map Reduce and the App engine project is more interesting than my simple mapper. The Map Reduce is a framework for breaking up large problems into small parts that can be run in parallel (as I read on Wikipedia). Appengine-mapreduce is a project that has begun the task of implementing this concept on Google App Engine for Python and Java.

The work of using Map Reduce is in defining the mappers and reducers. Mappers perform work on the problem data in parallel, while Reducers (from what I understand), recombinate the outputs based on information in the original dataset. If you can structure a solution to your problem using this paradigm it becomes easier to distribute the work between computers and datacenters. For App Engine developers, Map Reduce is a way to traverse the datastore or other collection of data (for example, lines in a file) effectively without re-engineering the plumbing that allows you to do this.

Using appengine-mapreduce in your application, like I did with ModelHygieneMapper, is easy. Include the source code for the library in your application and add a URL mapping to your app.yaml file. Create a Mapper.yaml file which contains the configuration for the mapper or mappers you wish to use. If you haven't already, write or include the mapper. The getting started guide for python explains this pretty well.

I will be keeping an eye on appengine-mapreduce in the future for other tasks it will be appropriate for. In the meantime, I am happy that I had a way to implement ModelHygienceMapper without worrying about the plumbing. If you have any feedback about my mapper, please let me know.

PrefetchingQuery: Prefetch Reference Properties and Parents in App Engine

2010-06-16T04:34:00.000-07:00

Back in January Nick Johnson from Google wrote about the benefits of prefetching reference properties for a collection of App Engine datastore entities. Prefetching can dramatically reduce the amount of RPC calls* to the datastore by ensuring each entity referenced by a property is only retreived once.

It has taken me a while but I finally got a chance to go back to Nick's blog entry and build some infrastructure around his code to make prefetching reference properties easier. I call the effort PrefetchingQuery and you can find it over at he3-appengine-lib in performing.py

Like PagedQuery, PrefetchingQuery is a facade for a normal db.Query or db.GqlQuery object. You can configure the PrefetchingQuery with the properties to prefetch in its constructor, with a class attribute on your model entity or let it default to prefetching all reference properties. For example



#create a prefetching query

myPrefetchingQuery = PrefetchingQuery( MyEntity.all(),

(MyEntity.myReferenceProp1, ))

configures the PrefetchingQuery to prefetch the myReferenceProp1 property. Actual prefetching occurs when you call fetch() on the query.

Still with me? Here are some additional details.

After you have instantiated your shiny new PrefetchingQuery, you can use it like the underlying db.Query or db.GqlQuery object. This means you call filter(), order() or ancestor() for db.Query or count() or cursor() for either db.Query or db.GqlQuery.

You can use PrefetchingQuery with PagedQuery if you like, but you need to wrap your original query first with PrefetchingQuery and then with PagedQuery, like this:

#create a paged, prefetching query
myPagedPrefetchingQuery = PagedQuery(PrefetchingQuery( MyEntity.all(), (MyEntity.myReferenceProp1, )), 10)

PrefetchingQuery can also prefetch the Parent entity of your models. You request this by adding 'parent' (ie. the string) into your list of reference properties to prefetch. But be warned: Setting the parent or an element manually uses a private/undocumented attribute _parent and may break in future App Engine platform or SDK releases. See this discussion for background information.

Thanks and credit for the important code behind PrefetchingQuery goes to Nick Johnson, whose code I only had to modify slightly for my packaging. Ubaldo Huerta and his discussion with Nick about parent properties also helped out.

Further documentation about Prefetching will appear in course on he3-appengine-lib, but for now, you should peruse the comments at the top of performing.py, if you want to find out more.

I welcome any bug reports or feature suggestions. Give it a spin.

* the amount of performance benefit will depend on your situation and programming goals. Past performance is no guarantee of future performance benefits.

GRC Password Google Gadget

2010-06-08T06:02:00.000-07:00

Recently I mentioned I was doing some light experimentation with Google Gadgets. I decided to create a gadget to retrieve a password from GRC's Perfect Password Page. Previously I had done the same thing while exploring Google App Script, and it seemed a good way to get my feed wet. So I am happy to announce a gadget that will not change the world but did teach me something about Google Gadgets: the GRC Passwords Gadget.

This gadget does a very simple job. It connects to the Perfect Passwords page, retrieves the contents and parses out the requested password type (hex, printable and alphanumeric) and length. Nothing too tricky.

What is more interesting is the usefulness of Google Gadgets in general. Gadgets do not seem to get a great deal of attention from Google these days but many of their products support them. You can write one gadget which can be added to iGoogle, Project hosting on Google Code, Google Sites, Gmail, Google Calendar, Google Maps (ie. 'Maplets') and any other container which supports Gadgets (such as Apache Shindig). My GRC password gadget may never change the world but quick access to it within Gmail makes it very handy.

For companies trying to reach customers deployed on Google Apps through the Google Apps Marketplace, gadgets are an excellent opportunity to escape the silo of your application and integration your service more fully with the rest of the the Google Apps offerings. Of particular interest is the only real development in Google Gadgets that I have noticed in recent history: Gmail Contextual Gadgets. These gadgets can appear inline in email messages based on the content of the email itself.

For the moment I am hanging up my Gadget developer coat, but I have a feeling I might be back for when I have an application which needs good integration with the Google world - or when I get a decent idea to implement :)

PageLinks - a simple helper class for PagedQuery

2010-06-06T02:52:00.000-07:00

On May 26th I added PageLinks to the already published PagedQuery class available on he3-appengine-lib. PagedQuery is a python class to add a paging abstraction to the vanilla db.Query and db.GqlQuery used on Google App Engine (announced here). PageLinks is a simple helper class that generates a sequence of page number / URL pairs for use in page navigation, including previous and next links.

Usage of PageLinks is straight forward. Create and initialise an instance of the class and then call the get_links() method:



pageLinks = he3.db.tower.paging.PageLinks( page=2,

   page_count=3,  url_root='http://myresources/', page_field='page')

Sequence_of_links = pageLinks.get_links()

Sequence_of_links now equals:



[('prev', 'http://myresources/?page=1'),

('1', 'http://myresources/?page=1'),

('2', 'http://myresources/?page=1'),

('3', 'http://myresources/?page=2'),

('next',  'http://myresources/?page=3')]

This is simple helper class. I think I am most proud of the fact I finally made good use of a Python List Comprehension.

Google App Script: Now with Time-based Triggers.

2010-05-30T02:21:00.000-07:00

Google App Script is continuing to develop. You can interact with not only the spreadsheet that hosts the script but also email, Google Sites, Calendars, Forms, Documents and the DocList, Web pages and services, XML, Maps and Contacts. Check out the documentation. What I was not aware of is that you can now incorporate time-based triggers into your scripts, which makes them a lot more useful. I have resolved today to publish a simple site about my upcoming wedding to receive RSVPs and publish wedding information - no doubt using Sites and Google App Script.