Read the full story (in Dutch)

Interview John Nauta (YouTellMe) from mfredactie on Vimeo.

Thanks to all the FrankWatching readers for the positive feedback on the previous article.

charts

I was just browsing the code a bit. There is something peculiar there:

args, varargs, varkw, defaults = inspect.getargspec(func)

I don’t get the point of using the inspect functionality. Anyone care to explain?

This Thursday my fellow YTM entrepreneurs and me will be visiting the Open Coffee Amsterdam meetup. For me its a first timer on the long  standing Open Coffee tradition. Hope it will be fun :)

Choosing Django:

Django has a few killer features which make it a better choice for many projects.

High Level Fields

As a starter there’s the usage of high level fields when describing your data model. This is best clarified by an example of a model definition in django:

class Author(models.Model):
   ip = models.IPAddressField()
   email = models.EmailField()
   company = models.ForeignKey(Company)
   picture = models.ImageField(upload_to='images/profile_pics', blank=True)
   homepage = models.URLField(verify_exists=True, blank=True)

Fields are specified by their purpose, such as Email, Url and Image. From this definition all subsequent logic such as form validation and file uploads are handled. The homepage field’s validation will even ping the url to see if it exists.

Read more: Creating Models (django project)

Form Handling

Starting with the knowledge that you have an email field you will often want a nice text input in your form with a regex to check if the email is valid. Django has all these standard use cases worked out for you. The following example clarifies this by using Django’s ModelForm. A model form is basically a normal Form class, with the fields pre-populated as one would expect it based on the given model.

#Form specification
class AuthorForm(ModelForm):
   class Meta:
      model = models.User
      fields = ('ip','email','picture','homepage', 'company')

#Using the form in the view (controller in Symfony terminology)
form = AuthorForm(request.POST, instance=author_instance)
if request.method == 'POST':
  if form.is_valid():
     form.save()
     return HttpResponseRedirect(request.path) 

#In the template (view in Symfony terminology)

	{{ uform.first_name.label_tag }}
: {{ uform.first_name }} {{ uform.first_name.errors }}

Django will display a file field for the image field, a text field for the url (with validation), a text input for the email field and a select box for the foreign key relation. Saving the result of the form to the database is as simple as calling save on the instance of the form. Writing custom widgets and field types is straightforward. Currently many localized fields such as a Dutch postal code Field and widgets are available. Symfony has been trying to emulate the Django newforms library. Unfortunately the syntax doesn’t seem very friendly. (Pity php doesn’t have metaclasses)

Read more: Forms (Django book)

Superb ORM

Probably the largest difference is caused by the ORM. In PHP both Propel and Doctrine are nice projects, but simply quite inadequate. The Django ORM is syntax heaven if you are coming from php. A small example:

#Find the first 5 authors which have a relation to a company
#with the name YouTellMe (exact match) and site url that contains youtellme.nl
Author.objects.filter(company__name = 'YouTellMe', company__site__icontains = 'youtellme.nl')[:5]

The possibilities of the standard Django ORM system are quite good. Your queries will be optimized into joins if you call select_related, many to many relations are supported and polymorphic keys are as well. The only part it fails at is query optimization in terms of column based lazy loading and support for complex relations. Fortunately you can fall back to using SQL alchemy, which is Python’s most prestigious ORM layer. SQL alchemy allows you exact query control for performance tuning and many more options you did not know you needed.
However it isn’t (yet, i hope) fully integrated into Django. It would really be great to see a tighter integration with SQL alchemy, but even the Django ORM strongly outperforms Doctrine and Propel.

Read more: DB api.

Python

You could see this both as an advantage or disadvantage. Discussing the differences between Python and PHP is probably best left for a later post. Suffice to say that I found my programming productivity to be substantially higher with Python compared to PHP. The disadvantage is a smaller number of available scripts, developers and hosts. If you are in a position in which you can choose, you really should give python a go. There must be a reason why YouTube and Google do ;)

Read more: Python in 10 Minutes

Speed

When arguing with my colleague about the choice of framework we conducted a speed test between Django and Symfony using Apache bench. At the time I was arguing in favor of Symfony. We compared a lightweight PHP framework, with Symfony and Django. Symfony was stripped down for performance and was only about 30% heavier compared to the lightweight framework. When comparing it to Django however, the results showed that Symfony was only able to handle half the load Django could. Using Python and Django seems to have a substantial effect on your server hardware requirements. (Note that these tests were only intended as an indication for internal usage. We didn’t test enough scenarios to be certain how the outcome would hold up on a live site. )

Symfony Still Rocks

When working with PHP Symfony is still an awesome framework. In many aspects it is even superior to Django. There are quite a few things Django could learn from Symfony:

Generic validation Classes and support for automatic js form validation

When using Symfony your javascript form validation is automatically generated. This is possible because of the usage of generic validation classes. In Django this would be hard to achieve since the validation system is not based on reusable classes.

A debug toolbar

One Symfony feature which I really miss in Django is the debug toolbar. Having an overview of your DB Queries, config settings, logging messages and caching information is very convenient. Especially for debugging a site with caching the caching indicators in Symfony are awesome. These caching indicators simply show which part of the page are taken from cache and which are freshly generated.

DRY templates

Symfony has a simple but very pleasant template tag called a component. A component tag calls a specified view and renders the corresponding template. The typical use case for this tag is sidebar with news items. You will want to show this sidebar on many pages, but you wouldn’t want to have to call that code inside each and every view which needs the sidebar. Doing so would clutter your view and hinder template caching. A nicer approach is to use a component template tag which calls the view responsible for retrieving the news from the database and rendering the sidebar template. This way of allowing the template to invoke code allows for DRY views and templates.

Clear Javascript and CSS management

In Django including a javascript or css file comes down to writing the respective tags in the template. In Symfony these things aren’t left to the template, but are set by the code. This allows for a few neat features. For instance the usage of an ajax utilizing template tag (helper in Symfony) will automatically ensure that prototype.js is loaded on that page. If you would set a textarea to rich, Symfony will automatically figure out you need TinyMCE to achieve the desired effect. Furthermore it allows for a general config file where you specify which assets should be loaded for a certain combination of application and view (in Django terminology). The main benefit of such an approach comes when you combine your css and javascript files and want to optimize the groupings. Here an example of the Symfony config:

// In the view.yml - comparable to settings.py
indexSuccess:
  stylesheets: [mystyle1, mystyle2]
  javascripts: [myscript]

[php]
// In the Action - Controller in Django terminology
$this->getResponse()->addStylesheet('mystyle1');
$this->getResponse()->addStylesheet('mystyle2');
$this->getResponse()->addJavascript('myscript');

// In the Template - The view in Django

Both are great, but Django more so

Having used both Django and Symfony I believe the two frameworks can learn a great deal from each other. Fortunately many people seem to experiment with a wide variety of frameworks (Including at least one delicious developer, version 3 of delicious maybe? ;)). Django in general has some excellent features, which make it a better choice for web development. If you are somehow bound to PHP, Symfony is still a good choice. The Django community is buzzing and active like no other and I look forward to posting on the various features.

If you didn’t try it yet:
django-project.com
djangobook.com

Note: Looking to hire Python and Javascript Coders

YouTellMe.nl is currently looking to hire Python and Javascript programmers in The Netherlands. Drop me an email at thierry [at] youtellme.com if you would like to know more or want to suggest someone for the job openings.

Besides our constantly growing group of beta testers these are the first screenies to go out in the open. The feedback of our testers has been very enthousiastic and I look forward to a full launch!

Step 0: Analyzing performance

Prior to attempting any tweaks, be sure to get firebug and use their console.profile() and console.profileEnd(). Test results will vary substantially during subsequent tests, but they serve their purpose for finding the bottlenecks.

Step 1: Remove Double Dollar $$ and event binding

There are many small performance differences, but several things are likely to really kill performance. One of the most nasty ones is using prototype’s double dollar $$ function (or the similar Element.select). You can often avoid using the double dollar function. For example the use case of attaching events to all ‘report this’ buttons on your site. The simple (and often good enough) approach would be to use the following code:

$$('.report_this').each(function(report_button) {
   var id = report_button.id.split('_')[1];
   report_button.observe('click', this.respondToReportButton.bind(this, id);
});

Four things are slowing this code down: 1. the usage of the $$ function, 2. the usage of each instead of a native looping construct, 3. the retrieving of the id from the id string, 4. the repeated binding of functions.

There are several possible remedies against the above code:

1. Give all report_this buttons a unique id (say for instance that you have 15 or less in a list)
2. Pre generate a list of ids using your server side language of choice and pass it to javascript
3. Manually traverse the DOM; $(’container’).childNodes can do wonders
4. Bind once to a common parent element
5. Find items by name instead of class
6. Forget about all the initializing and fall back to old school onclick=”classinstance.respondToReportButton()”

This last option sort of goes against many webdevelopment principles, but is often a very pragmatic choice. Joseph Smarr from Plaxo holds a similar opinion on the topic.

A better implementation using technique 1 would be:

this.respondToReportButtonBound = this.respondToReportButton.bind(this);
for(x=1;x<16;x++) {
   button = $('report_button'+x);
   if(!button) break;
   button.observe('click', this.respondToReportButtonBound);
}

In practice I’ve experienced speed improvements of over 40 times using this technique. Firstly the repondTo function is bound only once. Secondly the $$ function is no longer needed. Thirdly the each implementation has been replaced by a blazingly fast for loop. Fourthly the id is no longer extracted. A better practice is to extract it when respondToReportButton is actually executed. This last point actually brings us to the next main item.

Step 2: Be Lazy!

The trick here is to actually put in a bit of effort to make your code lazy. Don’t do anything until it is needed. (With the one, but large exception if doing so would hurt the user experience.) If some items are currently not visible to the user simply don’t bind events to them. If you need to extract id’s don’t do so until someone actually clicks on the item in question. Furthermore also make it lazy in the regular sense of the word. If your code only need to change one item, figure out which one it is and don’t loop about changing all just in case. This point is different for every application, but it can achieve great speed gains for a creative programmer!

Step 3: Stop using prototype functions if you don’t need them

Often you do not really need (as in that it barely saves you development time) some of the functionality of prototype. When comparing the speed of element.innerHTML = ‘hello world’ versus element.update(’hello world’) the differences are substantial (60 times with large chunks of html). Also the each iterator is often not needed and can be replace by a simple for loop with checks on nodeType and tagName. The same goes for the templating system. These tools barely save you time, but really hurt performance. I am a big fan of the prototype library and for most use cases these nice helpers are great. When you really need speed, be sure to refrain from using them though.

Step 4: Lower level optimizations

When you are done implementing the really important optimizations there are quite some lower level optimizations which will speed up your code.

• Write to innerHTML instead of using document.createElement
• Use for loops instead of for in loops
• Use Array.join instead of += on a string, see SitePen
• Cache variables and functions
• Limit the usage of Eval
• Limit the usage of Try Catch statements
• When manipulating the DOM copy the element out of DOM change it and stick it back in

Two generally great resources on the topic are:

1. Joseph Smarr (Plaxo)
2. Opera Dev

The missing performance data

One item affecting the performance of javascript interface elements is the speed by which the browsers redraws the page. Unfortunately I have barely found any documentation on how browsers determine what to redraw and what affects redraw speed. One certainty is the negative influence of transparent images on redraw speed, but other than that little seems to be known. Furthermore most browsers won’t redraw the page until your javascript function returns. This can make your interface elements feel very unresponsive. A solution to this problem is to use the setTimeout functionality.

Message for Dutch Programmers

YouTellMe is an innovative startup located in the center of Rotterdam. Talented programmers are always welcome. Have a look at the job page to get an impression of the job openings (We develop using Python/Django).

Have a look:

var cookieJar = Class.create();
cookieJar.prototype = {
initialize: function() {},
write: function(name,value,days) {
if (days) {
var date = new Date();
date.setTime(date.getTime()+(days*24*60*60*1000));
var expires = "; expires="+date.toGMTString();
}
else var expires = '';
document.cookie = name + '=' + value + expires + '; path=/';
},
writeJSON: function(name,value,days) {
var JSONvalue = $H(value).toJSON();
this.write(name,JSONvalue,days);
},
read: function(name) {
var nameEQ = name + '=';
var ca = document.cookie.split(';');
for(var i=0;i < ca.length;i++) {
var c = ca[i];
while (c.charAt(0)==' ') c = c.substring(1,c.length);
if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length);
}
return false;
},
readJSON: function(name) {
var value = this.read(name) || '{}';
value = value.evalJSON();
return value;
},
erase: function (name) {
this.create(name,"",-1);
}
};
cJar = new cookieJar();

After one of my posts got featured on Ajaxian many interesting offers hit my mailbox. One of them was actually from a startup right here in Rotterdam called ZeroBubble. I was happily surprised to find an IT startup in Rotterdam. Especially since after talking to them it became clear that they operate at the highest level of technical possibilities and have an absolutely amazing team. Two months ago I happily joined their ranks.

The project we are working on is called YouTellMe. I don’t want to share too much about it right now, but surely I will have plenty of exciting blog posts coming up in the next months.
Currently we are working with some of the nicest tech on the net. To give some examples: our admin interface is written entirely in ExtJs, the site’s search is powered by Lucene, we use prototype 1.6 for great object extending, for ajax functionality we use yahoo history manager to keep it bookmarkable, the entire site has been optimized according to the Yslow principles and we are doing some interesting things with openSocial. Given all these you can’t help but wonder why we aren’t using Symfony.

Why no Symfony?

Personally I am a huge fan of the Symfony framework. The team at Sensio has done an absolutely amazing job. My opinion on the framework is best described by these blog posts Part1, Part2. However the current project we are working on has some special requirements. First of all the application’s calculations are very harsh on the servers. Combine this with a large amount of AJAX and you have some serious performance issues. The calculation speed has been pretty optimized by a colleague of mine, who wrote a python daemon for the task. Still it is essential to keep the PHP framework’s overhead to a minimum. Prior to my employment at this company it was decided that Symfony would be too slow to handle the task. This is a topic which often nags Symfony.

I am curious how fast Symfony can be. For the YouTellMe site I need a bootstrapped and blazingly fast framework. In the coming weeks I’ll be setting up some tests too see how Symfony compares to our home build framework. As a starters I’ll definitely relieve Symfony from the ORM and fancy routing. From there on I will need to test to see which components are slow and can be removed. The clean and flexible programming in Symfony should make this easy to do.

Our current framework is very lightweight. It even does not do auto loading. I for one have no clue what the speed gain is from not using auto loading and it would also be interesting to test it. The MVC structure is entirely home build, but the rest of the features use Zend.

If there are readers of this blog, which have gone through the process of stripping Symfony, be sure to leave some tips in the comments!

Jobs at ZeroBubble

ZeroBubble is currently recruiting talented PHP programmers in the Rotterdam area. We are located in the Beurs World Trade center. If you are into the latest technology and like to work with great people, software and hardware be sure to email me at thierry [at] zerobubble [dot] nl or my boss at joost [at] zerobubble [dot] nl. As mentioned we work with fun software such as Ext Js, Lucene, Zend, object oriented js with Prototype 1.6, yahoo history, Yslow principles and openSocial.
We are looking for both full and partime PHP programmers. Python, ExtJs, prototype, server admin, subversion and memcached knowledge are all nice extras. As a main quality though, you have to be excited about building a unique and amazing web application.

The php:
var_dump(array('mynumber'=>42.2));
var_dump(json_encode(array('mynumber'=>42.2)));

The result:
array(1) {
["mynumber"]=>
float(42,2)
}
string(17) "{"mynumber":42,2}"

Notice the 42,2.
‘42,2′ or “42.2″ or 42.2 would have all been fine, but 42,2 obviously invalidates your json object.

The problem seems to arise from my ‘nl’ internationalization setting in php. I’m guessing the same problem will occur with any language which uses the comma for decimal separations.

Currently I am solving this problem by typecasting my number to a string, but there has to be a better approach.

Here 3 Pake tasks to automate your mysql dumping:

1. mysql-dump-data
2. mysql-load-data
3. propel-build-all-save-mysql

Download all three pake tasks. To install them simply copy the file to myproject/data/tasks/

You will have to edit the file to configure your database settings.

Note that this is only a temporary solution. I personally would be very happy to see Propel making database structure changes.

I didn’t see any manual on creating Pake tasks. Improvements and suggestions are more than welcome.