• pubsub, i.e. pub/sub, i.e. publish/subscribe, which is a common pattern in, well, the world, where multiple parties subscribe to one or more publishers, to receive notifications on updates.
• hub, is a center of communication, often used for efficiency purposes, like an ethernet hub.
• bub, you tell me. I haven’t a clue.

So what is it, really? It’s a protocol to enable near real-time update notifications among pubsubhubbub-supporting parties. Whereas RSS and Atom feeds required polling to receive updates, pubsubhubbub pushes information to parties. Currently various Google properties are have pubsubhubbub-support, such as Google Reader and Google Buzz. So, when you install a pubsubhubbub plug-in for a wordpress blog, like I have and I push the publish button on a post, it will almost instantaneously appear in all of your Google Readers and your Google Buzz, whereas before, it may have taken a few minutes or hours to appear.

Want to learn more about the details? Check out these slides, or the website:

I could have never have guessed that HTTP, HTML, CSS and Javascript would once not become the main way to access information, but also replace a lot of desktop applications. The abilities of the new HTML5 and other web technologies like SVG never cease to amaze me.

Here are 10 things I had not expected these open web technologies would be able to, but can in 2010:

1. Interactively render physics of a cloth
2. Live motion tracking
3. Play YouTube videos without Flash
4. Collaboratively edit source code in a browser IDE
5. Do weird interactive stuff like this
6. Animate simple 3D landscapes
7. Read books in a mobile browser, while disconnected from the internet
8. Play Wolfenstein 3D
9. Play MarioKart
10. Render flash files using Javascript/SVG

function Person(name, age) {
  this.name = name;
  this.age = age;
}

Person.prototype.getNameAndAge = function() {
  return "Name: " + this.name + " and age: " + this.age;
};

var p = new Person("Zef Hemel", 26);
console.log(p.getNameAndAge());

What we do here is define a constructor function Person, which starts with a capital P, to indicate that it should be used in conjunction with the new operator, as demonstrated in the second last line. The second statement adds a method to the Person prototype, which in practice makes that method available to all Person objects (including ones already created). The last two statements instantiate a Person object and calls the getNameAndAge method.

Using this mechanism you can also implement inheritance. Let’s introduce a LivingBeing “super class”, which has an age property:

function LivingBeing(age) {
  this.age = age;
}

LivingBeing.prototype.getAge = function() {
  return this.age;
};

Alright, now we’ll define the Person contructor again, but assign a new instance of LivingBeing to its prototype, which will add all fields and methods available in LivingBeing to all Person objects (again, including existing ones). Then, we add an additional method to Person objects: getNameAndAge:

function Person(name, age) {
  this.name = name;
  this.age = age;
}

Person.prototype = new LivingBeing();

Person.prototype.getNameAndAge = function() {
  return "Name: " + this.name + " and age: "
         + this.getAge();
};

var p = new Person("Zef Hemel", 26);
console.log(p.getNameAndAge());

For more information on how inheritance and prototypes work, read this excellent page in the Mozilla documentation.

Now, Crockford suggests that there is a nicer, cleaner and more natural way to do object-oriented programming and inheritance in Javascript, which he calls functional inheritance. The functional-inheritance style version of the above program looks as follows:

function livingBeing(age) {
  return {
    getAge: function() {
      return age;
    }
  };
}

function person(name, age) {
  var that = livingBeing(age);
  that.getNameAndAge = function() {
    return "Name: " + name + " and age: " + that.getAge();
  };
  return that;
}

var p = person("Zef Hemel", 26);
console.log(p.getNameAndAge());

Note that the invocation style of the constructors changed here too, no new keyword should be used. I agree that this is a nice style, although it makes extending existing objects with additional methods/fields hard, but one could argue this is a bad idea anyway. Although clean, it also seems more expensive to execute, because you’re basically composing an object from scratch every time — you start out with an empty object and that push in all its methods and fields — while with the pseudo-classical approach you create one prototype object with all the methods, and then simply point to that object. Theoretically calling a method would then be more expensive, because at invocation-time the prototype hierarchy has to be traversed. But maybe Javascript engines have a clever solution to all of this and in practice it doesn’t matter. I decided to investigate.

I benchmarked two things:

1. The performance of object creation, by creating 10x 1,000,000 objects
2. The performance of method calling on a single object, by invoking a method 10x 1,000,000 times

I did each 10 times, so that I can take an average time on each. I executed this benchmark on four browsers running on my Mac (Macbook Pro):

• Firefox 3.5
• Chrome 5 (dev)
• Safari 4
• Opera 10.50b

Disclaimer 1: The reason I executed these on these four browsers is not to compare their performance, this is not a good way to compare browser performance, but mainly to check that the results in different browsers do not diverge too much.

Disclaimer 2: What I’m testing is not representative for real programs. If one approach is going to be twice as fast as the other, this does not imply that your programs are going to be twice as fast, it means that object creation is twice as fast, or method invocation is twice as fast. Whether that matters to you depends on the amount of objects you create or methods you call.

Benchmark 1: Object creation

This benchmark creates 1,000,000 objects. First using the pseudo-classical style and then through the functional style. The code to this benchmark can be found here. The times reported are in milliseconds for 1,000,000 objects being created. As mentioned, every test is performed 10 times of which I took the average:

Update: The previous version of the benchmark script contained a major flaw that increased the execution time of the functional style considerably. This has now been adjusted in the graph. Thanks Adrian for noticing this.

This chart clearly shows that creating objects using the pseudo-classical style is cheaper in all browsers. This varies from about 35% cheaper (Chrome) up around three times as cheap (Firefox). If you create huge amounts of objects in your code, you may want to take this into consideration.

Benchmark 2: Method invocation

This benchmark creates one object and then invokes the same method on it 1,000,000 times. First using the pseudo-classical style and then through the functional style. The code to this benchmark can be found here. The results are as follows:

I would have expected that method invocation would have been cheaper in the functional style of object creation, but it turns out it’s not. The differences here are almost negligible, so I’d say that in practice it doesn’t really matter what style you choose if your application is heavy on method calls.

Still, I felt that the functional style must be cheaper especially if you use inheritance. So I adapted my benchmark script to introduce a level of inheritance (like the LivingBeing “super class”). However, method calls using the pseudo-classical style are still cheaper, although less so. Perhaps, if you use 4+ levels in your object hierarchy, the functional style method calls may start to become cheaper (although I expect creating those objects will be much more expensive).

It seems that method invocation on objects created using the functional style are roughly as expensive as using the pseudo-classical style.

Conclusion

There are two properties to take into account when deciding what object-creation style you’re going to use in your code.

1. Which style produces clearer code? This depends on the taste of both on the developers of the code, but if you’re developing a library for others, possibly also on the expectations of your audience. Scanning through some Javascript libraries it seems the pseudo-classical approach is much more popular. Consequently, your audience is likely going to expect and be more comfortable with this approach. That’s something to take into account.
2. Is the code CPU/memory intensive, does it create lots of objects? If so, the pseudo-classical approach is superior.

• “Markdown for WordPress and bbPress”, this plug-in (just search for it through the wordpress plugin manager) contains the PHP Markdown parser, which supports the whole of Markdown, plus a few extras such as footnotes¹. It parses every post as Markdown, including the old ones, which wasn’t a problem for me because the HTML of previous posts is also Markdown and seems to look OK.
• “WP MarkItUp”, this plug-in contains a reasonable editor for editing Markdown, it doesn’t do much, but has some buttons for common Markdown markup (hahaha), such as headings, bold, italic and so on. The plugin is based on MarkItUp.
• “highlight.js”, is a pure-javascript syntax highlighter that automatically detects the language of code blocks, which is exactly what you need in Markdown, because there’s no way to mark a code block with a language name. You cannot install this plugin through the plugin manager, as far as I’m aware, so you have to download it and install it yourself (see the README of the download). Example of syntax highlighting (of Javascript):

for(var i = 0; i < ar.length; i++) {
  console.log(i);
}

Now let’s see how this works in practice.

On twitter we long had this problem with the same links and retweets showing up in our streams every so often. It would have been nice if those had automatically been grouped together like in the Buzz timeline, but they weren’t. Twitter fixed this problem by implementing native retweet functionality, which sort of solves the issue. So, the question is: why did Buzz group these particular updates together? To Let’s step into the algorithmic brain of the Google and try to figure this out. Let’s ungroup the updates:

Those with eye to detail will notice the number of updates here. How many were there when they were grouped? 1, 2, 3, 4, 5. Right? And when expanded? 1, 2, 3, 4… huh?

Anyway.

Let’s see if we can discover the reason behind the grouping. Is it content-based aggregation? Probably not, unless Google aggregates programming related content here, which I doubt. Did it summarize the four updates into one? Not at all, the javascript framework update has absolutely nothing to do with the other updates. So what is it then, that made Google feel the other updates were somehow less important than the Javascript one? And why would this be useful to me? The only answer I can come up with is that they were all posted by me around the same time. Except they’re really not posted all at 10:47am, they were posted in the course of about an hour, but Google only polls twitter every few hours or so (it seems). Is time-based aggregation really a useful thing, though?

Google, I think, or at least hope, you can do better than this.

Quite out of the blue, Joel Spolsky (of Joel on Software and StackOverflow fame) has published a Mercurial tutorial online. And even if you don’t give a rat’s ass about Mercurial, I suggest you do read at least the first "chapter": Subversion re-education, which points out the differences between the subversion and mercurial/git mindset:

Want to know something funny? Almost every Subversion team I’ve spoken to has told me some variation on the very same story. This story is so common I should just name it “Subversion Story #1.” The story is this: at some point, they tried to branch their code, usually so that the shipping version which they gave their customers can be branched off separately from the version that the developers are playing with. And every team has told me that when they tried this, it worked fine, until they had to merge, and then it was a nightmare. What should have been a five minute process ended up with six programmers around a single computer working for two weeks trying to manually reapply every single bug fix from the stable build back into the development build.

And almost every Subversion team told me that they vowed “never again,” and they swore off branches. And now what they do is this: each new feature is in a big #ifdef block. So they can work in one single trunk, while customers never see the new code until it’s debugged, and frankly, that’s ridiculous.

Keeping stable and dev code separate is precisely what source code control is supposed to let you do.

Recognize that? Yes indeed, it’s the complete list of standard Javascript objects of Javascript 1.5. 16 objects, of which 7 are error objects. Of course, this is just the Javascript language by itself. In practice Javascript executes inside a browser, which gives it access to additional objects like the DOM, and all kinds of fancy HTML5 and non-standard browser-specific features.

When Javascript was just a little language that people would use to do simple mouse-over events, this was not a problem. However, now, more and more applications are written in Javascript in the browser and Javascript is used in more and more places other than the browser. Better yet I dare to bet you that in a few years time Javascript is going to be one of the most used programming languages in a growing number of domains. The growing body of Javascript code also means Javascript needs more than just those 16 objects. It needs a larger standard API and it needs a standardized way to modularize code.

At the very least it needs a de-facto standard way of doing object-oriented programming, having the choice between classical, prototypal, or, sure, why not lazy inheritance is not doing it for me. Different libraries use different styles. Worse yet, different libraries develop their own utility objects to produce classes, and if they’d at least agree on a common way of doing that, but no, there are a few dozen different way of implementing exactly the kind of inheritance you like. I’m not the biggest fan of Java, but at least they made a decision on this at the language level.

But I’m getting side tracked. The main two things Javascript needs to be a proper language that we can apply "grown-up" software engineering principles to:

1. A standard library
2. A module system

First, the standard library. Those 16 objects are no longer sufficient. So, what happens is that programmers step in to build a larger, more extensive library of objects. It would be great if they all teamed up to come with the one framework that everybody uses, but of course they didn’t, so we now have Prototype, MooTools, jQuery, MochiKit, ExtJS, YUI, Google Closure, and I’m probably forgetting about a dozen. This divides the Javascript world into different camps. Either you’re a jQuery guy or a Prototype guy, a ExtJS guy or a MochKit guy. There’s a YUI-y way of doing things and a Prototype way of doing things. There’s a jQuery calendar, an ExtJS calendar, and a Prototype calendar. Of course, once you picked a framework you stick to it, because if I want to use a prototype calendar from my jQuery application, I have to pull in x kilobytes of additional code that essentially duplicates jQuery functionality. A lot of effort is wasted because of this.

What Javascript’s standard library should contain is pretty tough thing to determine. First thing that comes to mind is a set of standard data structures. Of course Javascript has arrays and objects. Objects are typically also used as maps (although the keys can only be strings, and not even all strings) and sometimes even as set (where the property names represent values).

A better library of data structures is required, including a proper map and set implementation. Additionally, APIs for other common tasks such as:

• JSON parsing and serialization
• Testing
• Cryptography
• Date handling
• DOM querying (I think most libraries agree that CSS selectors are a good way, right?)

Some stuff like DOM traversals, querying and widgets will be hard to agree on probably, but would be nice to have.

Because Javascript is increasingly being used as a non-browser language, for instance on servers, it will also need non-browser stuff like IO, inter-process communication, sockets etc. The CommonJS initiative is working on these. CommonJS is an initiative, mostly among Javascript server vendors, to agree upon certain standard interfaces to, e.g. IO, threads, sockets etc. Its main contribution to date, however, is its module system.

Javascript does not have namespaces, but you can use objects for this purpose. That’s fine. In the past, some frameworks have built their own module system around this. Dojo and Google Closure offer a remarkably similar API to export and load modules:

dojo.require('dijit.widget.Editor');
dojo.require('myproj.Something');
dojo.provide('myproj.MyObj');

myproj.MyObj = function() { };
myproj.MyObj.prototype.initialize = function() { ... };

Replace ‘dojo’ by ‘goog’ and you basically have the Google Closure version. Quite nice and reasonably clean. However, there’s also JSAN’s module system. CommonJS’s module system is really nice, however according to some reports cannot be implemented in the browser properly. There’s an asynchronous version of the CommonJS module system called RequireJS, but, well, it’s not the CommonJS standard. There’s a proposal to standardize the RequireJS system as part of CommonJS, but it’s not entirely clear what the status is of that proposal.

I feel these issues have to be resolved and the good thing is that it does not require any changes in the Javascript language itself, nor its browser support. If the different framework vendors would just agree on a single base library that they all use, because, let’s face it, everybody needs a function to trim strings and a proper set implementation, a clean way of doing inheritance, plus a module system to go with that, be it dojo/Google’s system or a CommonJS variant, I don’t care.

It would be oh-so-nice to have a de-facto standard library for this stuff.

var elements = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
for(var i = 0; i < elements.length; i++) {
  var el = elements[i];
  console.log(el);
}

What does this code fragment print? Of course, it prints each of the elements in the array, so first 0, 1, 2, 3 etc. Easy enough. Now let’s adapt that a little bit:

var elements = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
var fns = [];

for(var i = 0; i < elements.length; i++) {
  var el = elements[i];
  fns.push(function() {
      console.log(el);
    });
}
fns[2]();

This code fragment uses a powerful feature of Javascript: first order functions. Functions are values that can be put in a variable, including arrays as this code demonstrates. In this fragment, the for-loop builds up an array called fns with functions that print an element of the list. Later on we can then pick one of the functions from that array and invoke it. The last line demonstrates this, it executes the 3rd function in the array. So what will that print?

If you’re a C/Java/C# programmer you’ll probably be guessing that it would print 2. In every iteration a variable el is declared that contains the current element from the array. The function that is generated on the fly contains a reference to el and will therefore be part of its closure. Logically, this means that in the first function captures the value 0 for el, the second function value 1 etc. Sadly, this is not the case. Javascript only has function scopes, not block scopes (for constructs like the for-loop). In effect, what you’re actually executing here is the following:

var elements = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
var fns = [];
var el;
for(var i = 0; i < elements.length; i++) {
  el = elements[i];
  fns.push(function() {
      console.log(el);
    });
}
fns[2]();

Alright, fine. Why is that a problem? You have to realize that the closure of the generated functions contain pointers to variables, not a snapshot of the values at the point of definition, meaning that if variables change, these changes are visible from within the function. This is a very powerful feature, but also a potentially very confusing one. Since there is only one variable el, to which a new value is assigned every iteration, and each function points to this same variable, every function in the array will produce the same value: 9 (the last value assigned to el in the for-loop).

If you have problems getting your head around that, don’t feel bad, it took me a while too.

So, how can we fix this? Well, we can artificially introduce scopes by creating new functions and immediately invoking them: 

var elements = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
var fns = [];

for(var i = 0; i < elements.length; i++) {
  (function() {
    var el = elements[i];
    fns.push(function() {
        console.log(el);
      });
  }());
}

fns[2]();

Incidentally, if you’re not already following me on Buzz, do so now!

What Buzz is definitely not good for right now is following popular people; people that get loads of comments on every buzz, meaning these buzzes jump to the top all the time. But I’m sure that will be fixed.

Buzz is likely to go a bit more mainstream than Twitter, which will mean that my wife, mom and dad are likely to start using at some point. Should they really be following me though. I’m likely to push out stuff that they really do not care about (like work related things). Currently this is why I use both facebook and twitter. Facebook is for personal things. Twitter for work/hobby. What should I use Buzz for, if anything? Time will tell.

Anyway, my point is: follow me on buzz.

Person p = em.find(Person.class, "somePersonId");
println(p.getName());
println(p.getFather().getName());

Easy enough. But hang on, what did that third line do? I turns out that it had to execute a SQL query to fetch p’s fatherly Person object. Not a big deal, you’d say. But what if you have code like this in a for loop iterating over 200 person objects? At the face of it, this should be cheap, you’re just doing simple property access, right? However, behind the scenes it executes another query for every iteration, making this simple loop quite expensive at 201 required queries. Sure, you can tell Hibernate to prefetch the father property, and you should, but if you forget you may not notice what’s hogging the database until you start inspecting query logs.

In Java, it is difficult to predict which statements are going to be expensive to execute by inspecting the code.

This is true for practically every programming language, but it’s also a shame. It may be useful to have some kind of syntactical overhead for performing expensive operations. If expensive operations are inconvenient to write, will that not encourage programmers to write more efficient code?

It turns out Javascript has a mechanism for this. Javascript environments are typically single threaded. If you only have a single thread available in an interactive environment like the browser you want to block it as little as possible. Therefore, in Javascript, expensive operations are performed asynchronously. I complained about asynchronous programming before, but it has the nice property of making explicit what operations are going to be expensive.

Let’s see how this affected persistence.js, my asynchronous Javascript ORM: 

persistence.transaction(function(tx) {
  Person.load(tx, "somePersonId", function(p) {
    println(p.name);
    p.fetch(tx, 'father', function(father) {
      println(father.name);
    });
  });
});

That’s one piece of annoying code to write, right? Indeed it is, because it’s quite expensive to execute too. As you can see there are three callback functions, which is also the number of database operations that are required to execute this code. One to start a transaction, one to load the person object and one to load its father object.

So how does this work in loops? We can use the each method on query collections:

persistence.transaction(function(tx) {
  Person.all().each(tx, function(p) {
    println(p.name);
    println(p.father.name);
  });
});

Again two defined callback functions, which gives you a feeling of how expensive this code is. One callback for starting the transaction and another one that iterates over the query result. However, if we execute this code we will get exceptions:

Property ‘father’ with id: 0BEEC2CB6AF64A72A7647DF09BCD62C3 not fetched, either prefetch it or fetch it manually.

Rather than lazy loading the father property in each iteration, persistence.js simply throws an exception, it does not support lazy loading without syntactical overhead. However, as the error explains, we can fix the problem by prefetching the father property, so we will:

Person.all().prefetch("father").each(tx, function(p) {
  println(p.name);
  println(p.father.name);
});

Which results in an efficient predictable performance in terms of number of queries executed (1 query for this piece of code).

Kind of nice, right?

There is one other asynchronous Javascript ORM framework that I know of (Impel), but it has a dependency on MooTools, which would be yet another framework you need to pull into your web application. persistence.js has no dependency on any Javascript framework (other than a tiny Google Gears initialization script in case you want to use Gears).

persistence.js is an asynchronous framework, which means that operations that can take some time, are provided with a callback function that will be invoked when the operation has completed. This asynchronous nature makes persistence.js harder to use than synchronous frameworks like e.g. ActiveRecord.js, but the fact is that if you want to take advantage of the new HTML5 database capabilities, you have no other choice than to do it asynchronously. persistence.js tries to make it as easy for you as possible.

The code is fairly well documented (for my doing anyway), and the README explains how to use it and gives some code examples as well. To give you flavor, the following code opens a database connection, defines two entities (Task and Category), and synchronizes the definition with the database (creates the tables etc.):

persistence.connect('testdbnm', 'My test db', 5 * 1024 * 1024);

var Task = persistence.define('Task', {
    name: "TEXT",
    description: "TEXT",
    done: "BOOL"
});

var Category = persistence.define('Category', {
    name: "TEXT"
});

Category.hasMany('tasks', Task, 'category');

persistence.schemaSync(null, function(tx) {
  alert('Successfully synchronized the schema!');
});

Instances of the defined entities can then be created in a natural way, and subsequently marked to be persisted:

var task = new Task();
task.name = "My new task";
var category =
    new Category({name: "My category"});
persistence.add(task);
persistence.add(category);

One-to-many and many-to-many relationships can be used as collections:

category.tasks.add(task);

When modifications to objects have been made, these have to be flushed to the database:

persistence.flush(null, function() {
  alert('All objects flushed!');
});

A nice feature of persistence.js is QueryCollections, which are virtual collections that can prefetch relations, can easily be filtered and sorted (and in the future paginated): 

Task.all().filter("done", '=', true)
          .prefetch("category")
          .order("name", false)
          .list(null, function (results) {
    results.forEach(function (t) {
        console.log('[' + t.category.name + '] '
                    + t.name);
    });
});

In the future I intend to add synchronization support to it so that the local database can transparently be synchronized with a (view on) a remote database, which is a typical use case of applications like these.

However, when you’re more in a browsing mode "let’s see what e-mail we got today", you flip the device around and get a view with a message list on the left and e-mail at the right:

If find this a rather fascinating user interface idea and wonder if we can not apply it more iPhone applications as well. There’s one iPhone application that I know of that really uses this, and that’s the Calculator application, which in portrait mode looks like this:

And when you flip it:

In the framework for the development of mobile web applications that I’m developing, I have support for orientation events. Here’s a simple todo application I’m working on (looks best on an iPhone or desktop webkit browser) that takes advantage of this feature. I’m not entirely sure this is the best way to use the feature, but it’s cool nonetheless. In portrait mode the application looks as follows:

You can swipe any of the items to show a delete button:

However, if you’re in a destructive mode you can also flip the device around and switch to edit mode, in which all items become immediately deletable:

(You can emulate this behavior in a desktop browser by resizing the window, making it wider than it is long or vice versa).

A user interface design concept to think about.

With that come the usual restrictions: only install software through the App Store and no multi-tasking. Although I truly believe the no multi-tasking support limitation is going to be resolved in June, when Apple usually releases new iPhones and iPod touches (and an operating system to go with that), there is a way around both these problems:

The iPad, like the iPhone and iPod touch comes with an excellent, fast browser.

If you build iPad web applications, you can build whatever you like, it doesn’t have to be approved by Apple and you can roll out updates instantly. In addition, web applications, like on the iPhone/iPod touch are the only type of application that "keep running" (in the sense of not being killed) when switching applications. You can have several web apps open in safari, and they’re still there, in the same state, after you exit Mobile Safari and return to it. Although the applications do not run simultaneously, you can easily switch between them, in a kind of task switcher inside Safari, which is its ability to switch between "tabs":

And yes, indeed. This exactly the direction in which I’m heading with my DSL for mobile applications (which may actually include the iPad as a target).

By the way, I’m trying to come up with a new name for the DSL, now that MobiDSL is apparently taken. Name suggestions? It does not have to include the word DSL at all, preferably not. I’m thinking about something with "Touch" in it, or possibly something completely different. Any ideas?

Asynchronous operations are typically used to perform tasks that might take a long time to complete, such as opening large files, connecting to remote computers, or querying a database. An asynchronous operation executes in a thread separate from the main application thread. When an application calls methods to perform an operation asynchronously, the application can continue executing while the asynchronous method performs its task.

Asynchronous programming clearly has performance benefits, as mentioned in the explanation I just quoted. What I have complained about before is the programming model that follows from it. In Javascript this becomes painfully clear. In Javascript you do not have threading (although it’s coming), therefore, anything that is not going to be instantaneous, needs to be executed asynchronously or it will freeze the browser. The typical example of this is Ajax (Asynchronous Javascript and XML, a term that is way too general, when you think of it), but you get the same things when you start interacting with local databases. In a previous post I showed the following code:

What I do in this code is attempting to implement simple sequential execution of

1. opening a database connection
2. starting a transaction
3. creating a table
4. inserting a task
5. inserting another task
6. selecting all tasks in the table
7. render list items for them

In a synchronous programming model this would roughly be:

var db = db.openDatabase(...);
db.startTransaction();
db.executeSql("CREATE ...");
db.executeSql("INSERT INTO ...");
db.executeSql("INSERT INTO ...");
var results = db.executeSql("SELECT * FROM ...");
for(var i = 0; i < results.length; i++) {
  ...
}

I think you will agree that this is much more legible than the asynchronous counterpart, except that you should never run it like this in Javascript, because it will completely block the UI thread for the duration of the code (which could be several seconds).

I looked at the Google Web Toolkit because I hoped they had solved this problem, but it turns out they even made it worse, because now you have to deal with the Java overhead of implementing interfaces:

(Nothing bad about GWT by the way, using it makes a lot of sense in many larger projects.)

Getting back to my point: would it be possible to automatically rewrite synchronous code to asynchronous code? I’m starting to feel that’s very possible indeed. Let’s take a simple example. Let’s say we managed to implement an adder function that is extremely slow (maybe by executing it on a server far, far away):

console.log(adder(3, 4));

This code would block for several seconds, possibly, while the adder function is adding the numbers 3 and 4. In an asynchronous manner this would be written as:

adder(3, 4, function(result) {
    console.log(result);
});

Asynchronous functions always provide a callback function, that is passed the result of the computation once it completes. The result of the computation is what we call a return value in synchronous programming. So, what if we sequence a number of synchronous statements:

var a = 0;
a++;
var n = adder(3, a);
var m = adder(n, 4);
console.log(m);

What would that look like in asynchronous code?

var a = 0;
a++;
adder(3, a, function(n) {
  adder(n, 4, function(m) {
    console.log(m);
  });
});

As you can see, this transformation is not complicated. It could easily be automated, so that in the DSL that we’re designing you could write synchronous code, and it would be translated to asynchronous code automatically.

However, there is one thing that worries me about this approach. The advantage here is that you can write simpler code, which does indeed run asynchronously. However, we miss out on the opportunity to parallelize things. For instance, the two INSERT queries in my first example can easily be executed in parallel. There is no reason why I should execute them sequentially (when we look at just the two INSERTs). This may be a lot faster even. In this particular case I did this, because I wanted to make sure that both INSERTs had taken place before I did the SELECT. If I would have parellized the two INSERTs, I would to somehow track when both INSERTs had happened so I could do the SELECT. That’s kind of cumbersome. In the general case, however, there may in fact be many things that may be easily parallelized, and although I want to provide the user with a synchronous model because it’s so much more convenient to write, I also don’t want to lose the parallelizability of asynchronous code.

Maybe it would be possible to analyze the code for any dependencies. In the last example, for instance, the second adder call cannot be executed in parallel to the first adder call, because it depends on the result of the first, but in other cases there may be no such dependency. But I’m afraid to open up the world of hurt that is called concurrent programming here, so maybe I should stay out of that kind of stuff.

Any ideas or experiences with this?

• Javascript: The Good Parts, excellent book about Javascript, helping you avoid all that bad parts
• jQuery in Action, good book about jQuery
• Building iPhone Apps with HTML, CSS and Javascript, free online book from O’Reilly

In addition there are a number of great blog posts and libraries helping to develop native-looking web applications for the iPhone:

• Make the most of the iPhone SDK
• Universal iPhone UI library, many nice CSS tricks
• QTouch, jQuery extensions for iPhone web apps
• iPhone Native looking skin

And a number of performance related posts from Google (from the Gmail team that developed the excellent Gmail mobile application):

• HTML5 and Webkit pave the way for mobile web applications
• Suggestions for better performance
• Using AppCache to launch offline – Part 1
• Using AppCache to launch offline – Part 2
• Using AppCache to launch offline – Part 3
• Cache patterns for offline HTML5 applications
• Auto-growing textareas
• A Common API for web storage

Because eventually I want to add automatic data synchronization support to our DSL, I’m also looking into synchronization strategies. I found a nice post about that. Of course, Apple’s own iPhone web app documentation is also very useful.

You may say: "but Zef, that is absurd. The mobile domain is huge. There are so many phone platforms out there!" But I say, no. I recently investigated the importance of the iPhone and Android platforms compared to the rest and found that people care mostly about iPhone and Android and not so much about anything else. That’s right. Anything. The following graph proves this:

So we shall not worry about this issue.

Now that we decided on a domain (mobile applications) and target platforms (iPhone and Android phones), the next essential step for success is choosing a good name. But, do not worry, I already decided what it is going to be. The name of our language shall be: MobiDSL, which is pronounced Mobi Diesel. It’s fuel for your phone, which is just what we need with the sucky battery life that the phones that matter have these days. So MobiDSL is the name, at least until I can come up with something better. And do not worry, I registered MobiDSL.org already, which redirects to this very post. Is it coincidence? I will leave that for you to figure out.

But, you will ask, if we are going to target both iPhone and Android, what kind of code are we going to generate? If we generated Objective-C code, it will only run on iPhone, and Java will only run on Android. Are we going to generate both? My answer to this question isseth "no".

My original thought was to use PIL, the universal platform-independent super language, which I would let generate the Objective-C and Java code for me. But then I came up with something much more radical, something much more forward thinking, my friends, a solution that may just hand me some other mobile platforms for free (not that they matter).

We are going to build applications the way Steve intended them to be built, back in 2007 at the launch of the iPhone. I was entusiastic about this idea from the very beginning:

Thanks Steve! "Yeah, we have this new amazing modern way of developing applications for your phone, it’s called a website!" Awesome. "It integrates great with all the iPhone features. For example you can create a "call:0123223222" link to call somebody!" Great. Except you always need an internet connection to use it, it’s slow, you can’t create an icon for it in the menu. And… oh yeah… it’s a frickin’ website! Apparently this was done for "security reasons". What about Java, Steve? Every frickin’ phone supports Java. It all runs in a sandbox, it’s all secure. It even runs without an internet connection and it’s responsive (yes, that has become a feature in the web 2.0 day and age). Why no Java on the iPhone, explain it to me.

Yes indeed. We are going to mobile web applications. But not to worry, web applications on mobile phones have advanced far beyond the killer "click to call" feature mentioned. Heck, the Google Nexus One has the same clock speed as my wife’s iBook. HTML 5 gives us access to this power by offering:

• Local databases (SQLite, in the browser)
• Location information (GPS coordinates)
• Threading (WebWorkers)
• Gestures (swipes, mostly)
• Canvas, for some pretty drawing
• Offline support, the entire application is cached on the phone and is fully functional even without internet connection
• The application can have an icon in the phone’s menu
• Full-screen mode, applications can run without the location and browser navigation bars (on iPhone at least)
• One platform to rule them all: simply generate HTML, CSS and Javascript and run on both platforms

In addition we work around the App Store which can be seen as both an advantage and disadvantage. Advantage is that we can build whatever application we like and Apple does not need to like or approve it. Rolling out updates is also much faster. The disadvantage is that web applications don’t appear on the App Store, which is a great promotion tool, but then, tools like PhoneGap can generate wrappers around web apps that can be submitted to the Android marketplace and iTunes App Store.

As I said, we meet in exciting times. I spent most of last week playing around with Javascript and jQuery and applying them to mobile applications. I managed to build a functional todo application that stores all of its data locally on the phone and looks very much native to the iPhone (some CSS trickery can probably make it Android-like too). The application can be used without an internet connection in full-screen and has a icon on the iPhone menu. I currently work on a few jQuery extensions to make writing such applications simpler. Then, when that’s done I shall proceed to build a nice, easy to use and analyze DSL on top of it. Isn’t HTML, CSS and Javascript easy to use, you ask? Well, I don’t know about you, but Javascript’s asynchronous coding is not always ideal in my opinion:

Trust me. It will be great.

Therefore, I shall declare 2010 as the year of the mobile web application.

I shall keep you informed, my valued surfer of the Internets.

I don’t mean to be mean, because I really like your blog and I read it all the time… but could you just explain to me the point of WebDSL? I honestly don’t mean it in any kind of negative way, I’m just wondering why you’re dedicating your valuable time to building something like that when RoR, php, et al. already exist.

Just from a cursory look, it seems like the syntax is halfway between Visual Basic (yikes!) and C. Why do you prefer that type of syntax to the Lisp-style syntax of Clojure/Compojure?

Good questions!

WebDSL was started about 3 years ago by Eelco Visser as an exercise the the design and implementation of domain-specific languages. His focus up to then had been on parsing and meta-programming, but it was time to focus on a new domain: the web. When he started Eelco had never built a web application. He investigated a number of Java frameworks as a basis and eventually decided to use JBoss Seam as a target.

The goal of WebDSL is to get rid of the boilerplate code you would have to write when building a Java application and raise the level abstraction. The vision was to have simple, domain-specific sub-languages that allow a programmer to specify a certain aspect of the application and the WebDSL compiler would generate all the implementation code for that aspect. Initially there were three sub-languages: a data modeling language, a user interface language and a simple action language to specify logic. As others joined the project (including myself), we added more sub-languages and more features: access control, workflow, data validation, ajax support and more recently search. Work is also done in the area of data evolution (i.e. migrating databases as you change your data model).

Although WebDSL is mainly a research project, we are increasingly working to make it useable by anybody with some programming experience. We currently have a few websites in production built using WebDSL (researchr, tweetview, webdsl.org and pil-lang.org) and the manual is growing.

The idea of building abstractions for the web itself is hardly novel. As Jay mentions, there are many web frameworks that already do this: Rails, Django and so on. There are a few things that we do differently in WebDSL, compared to existing frameworks:

• We create our own custom syntax. Whereas Rails and Django are struggling to express everything using Ruby and Python, respectively, we designed our own clean syntax. Whether you like this syntax is a matter of taste. Personally I like it, although, indeed, it inconsistent here and there.
• WebDSL is a statically typed and checked language. I wrote a number of posts about this issue and its advantages.
• WebDSL compiles to low-level Java code, which has good performance characteristics. The code we generate does not rely on run-time meta-programming and reflection features of the language which are typically rather slow.
• WebDSL is platform independent. We generate Java code now, but it can be ported relatively easily to .NET, Python or PHP. We have prototypes of this utilizing the PIL language that I developed.
• Within the next few months WebDSL will have excellent IDE support for Eclipse, built using Spoofax/IMP. My colleagues are working on this. It will feature syntax highlighting, as-you-type error reporting, code completion and eventually refactoring support.

A drawback that WebDSL has today is that it’s not trivial to install, but with the IDE plug-in and Java-version that should become a lot easier soon.

So, why am I putting so much effort into this? As you may be aware I’m doing a Ph.D. in the area of domain-specific languages, so we investigate how to best build them. WebDSL is a case study for us. Soon I intend to work on another DSL, in the domain of mobile applications (yes, a DSL to build iPhone and Android applications, people!). It’s interesting from a research perspective to see how to best do this.

In addition I regularly experiment with alternative ways of creating DSLs, like in Clojure and Scala. I’d like to see how far you can push these languages to build the DSLs you like. Clojure allows you to define your own custom syntax, in some sense, as long as you adhere to the rule of the parenthesis. Static error checking is much more problematic. Clojure is also rather tied to one platform, sure, there’s also ClojureCLR, but to write programs that work on both CLR and JVM is, well, challenging. IDE support for a Clojure DSL is also non-trivial.

On the other hand, the flexibility of a DSL like WebDSL also has its downsides. Basically you can design the language any way you like, both its syntax and semantics, you don’t get much for free. Whereas an internal DSL built on Clojure or Scala gets a lot for free: some error reporting, support for namespaces (something we still don’t have in WebDSL), a type system (in Scala’s case), an escape to a powerful language (Clojure or Scala) and a rich set of libraries you can use. In WebDSL we have to design all of this from scratch.

So in the end both approaches have their advantages and disadvantages. I intend to continue to explore them both.

open(FILE, "bla.txt");
while(<FILE>) { print; }

In case you don’t speak Perl, this is equivalent to:

open(FILE, "bla.txt");
while(<FILE>) { print $_; }

Still unclear? Alright, once more:

open(FILE, "bla.txt");
while($line = <FILE>) { print $line; }

Perl is developed by linguist Larry Wall, who likes to put all kinds of natural language things into Perl. $_ refers to the subject of the sentence, it’s "it" as it were. print $_, or simply print means "print it". How do we know this variable exists? We don’t. Unless we read the manual. Although they might be useful, magic variables like these are generally bad practice, it’s one of many reasons that Perl code is often called write only code.

PHP also has some magic variables whose behavior do not adhere to any usual rules, they are called super globals. In particular there are $_POST and $_GET. They do not adhere to the scoping rules defined in PHP.

I don’t know about you, but I like to see my variables declared. I like to see where they come from, are they local variables, parameters, or imported from a module that I can lookup? Magic variables are confusing, because they have different behavior than other variables in a language, they do not adhere to the usual rules.

As part of Adia, I played around with Compojure, a Clojure web framework. Generally, Compojure is a well-designed framework, but I came across one instance where it uses magic variables: in the defroutes macro. An example:

(defroutes webservice
  (GET "/"
       (str "Hello, "
         (:name (:query-params request)))))

This defines a mapping from the "/" URI, to code to be executed when that URI is requested. As you can see, I use the variable request in this example. Upon inspection, it not clear where it comes from. At first, you may assume it’s a global that is dynamically bound (Clojure has dynamically scoped variables). But some refactoring of the code shows that this is not the case:

(defn say-hello []
  (str "Hello, " (:name (:query-params request))))

(defroutes webservice
  (GET "/"
       (say-hello)))

This results in a "symbol cannot be resolved" error for the request variable. So clearly, request is not a dynamically bound variable either. So what is it, and where does it come from? I had to dig into the Compojure source code to find it, it is defined in the with-request-bindings macro, which defines some magic symbols: params, cookies, session and flash that are only accessible from inside the macro’s body:

(defmacro with-request-bindings
  "Add shortcut bindings for the keys in a request map."
  [request & body]
  `(let [~'request ~request
         ~'params  (:params  ~'request)
         ~'cookies (:cookies ~'request)
         ~'session (:session ~'request)
         ~'flash   (:flash   ~'request)]
     ~@body))

I find this confusing and therefore bad practice. So, what is the alternative? It turns out there’s no perfect solution here, it’s all about trade-offs.

The problem magic variables try to solve is enabling quick access to data that is otherwise not accessible in a concise manner. In this case it could have been solved by declaring the request and other variables somewhere, for instance as parameters to defroutes:

(defroutes webservice [request params cookies session flash]
  ...)

A reason not to go with this solution is likely to be its verbosity, nobody likes to write functions with a large number of arguments. A second alternative is only passing the request parameter and letting the user pull out the other from that map every time they need it. A reason not to go with that solution is user inconvenience, users likely want quick access to all five of these values and don’t want to look them up in the request map every time. A third alternative is using dynamically bound global variables. In Clojure these can be bound to a new value only for a certain thread within a certain code execution path:

(def request nil) ; root binding is nil

(defn print-request []
  (println request))

...
(binding [request (build-request ...)] ; rebind
  (print-request) ; prints request
  ...))

I use them in Adia for access to the request, post and get parameters. For this I declare 3 variables in the adia.web module:

(def *request* nil)
(def *form* nil)
(def *query* nil)

Values have been dynamically bound to their respective values by the time they are used within a webfn:

(defwebfn say-hello []
  (str "Hello, " (:name *query*)))

My thinking is that by using the *name* convention, it is at least clear these are not normal locally defined variables, they are different. Secondly, they are only available if you import the adia.web module and are part of that module’s interface, i.e. you can look up their declaration. What is still not clear is where their value comes from, but I suppose the user will have to abstract from that. The alternative to this approach would be to pass these variables as parameters to every webfn, but I decided against this because it would make writing webfn defintions too verbose. It was also not very clear what the syntax should have been: (defwebfn fn-name request form query [arg1 type arg2 type] ...)? (defwebfn fn-name [request form query] [arg1 type arg2] ...)? Neither of these options seemed very elegant to me, which is why I decided to be pragmatic and go with dynamically scoped globals.

I think this is an acceptable compromise. While the five globals are essentially still input parameters, they do not have to be passed around all the time. However, if you invoke a webfn outside a path where values are bound to the *request*, *form* and *query* variables, you are likely to receive NullPointerExceptions.

As a general rule: to keep programs readable and comprehensible, it should be easy to see where your variables and symbols come from. Are they local variables, parameters, globals? Magic variables should be avoided if possible.

Objective-C is C with some Smalltalk bolted on to it, which gives it a strange feel:

MyObject* o = [[MyObject alloc] initWithNum: 20 andString: @"Hello world!"];

In C, a function call has the syntax function_name(arg1, arg2). However, when moving into Objective-C object land, a method call looks like: [object aMethodCall: arg1 andArg: arg2]. Alien, if you ask me.

In Lisp land, an example of this is the common-lisp loop macro:

(loop for x in '(a b c d e)
      for y from 1
      if (> y 1) do
        (format t ", ~A" x)
      else do
        (format t "~A" x))

If you’ve ever written any Lisp code, you’ll see that although this is very readable and concise, like Objective-C, by the way, it feels completely weird in a Lisp-style language.

I have a similar problem with ClojureQL, a query language for Clojure. Queries expressed in ClojureQL change the meaning of Clojure in a way that I feel is bad language design because it it breaks assumptions that hold true for the rest of Clojure.

Consider the following snippet of code:

(let [first-name "zef"]
   ...
   (= first-name "zef") ...)

This piece of code binds the value "zef" to the symbol first-name. The programmer’s expectation is that when the first-name symbol is used anywhere within the let, its value will be "zef", unless it is rebound to something else with another let. However, this assumption breaks when using ClojureQL:

(let [first-name "zef"]
  (query users * (= fname first-name)))

This is legal in ClojureQL, although it is a bit unclear where fname would come from, it comes from the * there, we can make this more explicit:

(let [first-name "zef"]
  (query users [fname lname] (= fname first-name)))

This is perfectly valid ClojureQL code, except it doesn’t do what you would expect it to do. It does not find all users with first name "zef", no, it will throw an SQL exception saying that the table users does not have a field "first-name". Huh?

It turns that when we use the query macro, we step into a different world, a world where we have to let our previous assumptions go. When first-name is used, it no longer refers to the value bound to it before, instead it’s simply a name referring to a column in a table. It is still possible to escape to "normal" Clojure semantics by escaping back into the Clojure world with a ~ prefix:

(let [first-name "zef"]
  (query users [fname lname] (= fname ~first-name)))

I’m not very fond of this type of language design. It would probably be better if a ~ would not be necessary, in that case you could read the query as a kind for loop where each result row is destructured and bound to [fname lname], which are then used in the body expression. However, still, intuitively in this interpretation the names of fname and lname should not refer to column names in the users table, but instead are only to be used for binding in the code, referring to the first and second column in the result set. Still confusing.

A syntax that is more Clojuresque, if you will, albeit more verbose would be:

(let [first-name "zef"]
  (query [u users] (= (:fname u) first-name)))

Intuitively, the query iterates over all users binding each user to u and filtering on the value of the :fname key of each user entry. I’m still not confortable with the use of users there, which seems some type of magic symbol, but I suppose that could be fixed too. Maybe of having a (deftable users) statement somewhere else in the code, or replacing with with (table :users), which, again, would make it slightly more verbose:

(let [first-name "zef"]
  (query [u (table :users)] (= (:fname u) first-name)))

The point is that with great power comes great responsibility. The macro facilities of Lisps give you enormous power to create your own language extensions, which is great. It makes experimenting with languages very easy. However, it turns out that language design is very difficult. The language syntax is the user interface of your language. Whereas typical languages like Java and C# evolve very slowly and are designed by experienced language designers, in a Lisp anybody can do it, which can result in very confusing abstractions.

Abstractions like these have to be designed very, very carefully.