Getting a C++ game engine running in JavaScript, using WebGL for rendering, is a massive feat and is largely due to the toolchain that Mozilla has developed to make it all possible.

Since the release of the Unreal Engine 3 port to Asm.js I’ve been watching the response on Twitter, blogs, and elsewhere and while some developers are grasping the interesting confluence of open technologies that’ve made this advancement happen I’ve also seen a lot of confusion: Is Asm.js a plugin? Does Asm.js make my regular JavaScript fast? Does this work in all browsers? I feel that Asm.js, and related technologies, are incredibly important and I want to try and explain the technology so that developers know what’s happened and how they will benefit. In addition to my brief exploration into this subject I’ve also asked David Herman (Senior Researcher at Mozilla Research) a number of questions regarding Asm.js and how all the pieces fit together.

What is Asm.js?

In order to understand Asm.js and where it fits into the browser you need to know where it came from and why it exists.

Asm.js comes from a new category of JavaScript application: C/C++ applications that’ve been compiled into JavaScript. It’s a whole new genre of JavaScript application that’s been spawned by Mozilla’s Emscripten project.

Emscripten takes in C/C++ code, passes it through LLVM, and converts the LLVM-generated bytecode into JavaScript (specifically, Asm.js, a subset of JavaScript).

If the compiled Asm.js code is doing some rendering then it is most likely being handled by WebGL (and rendered using OpenGL). In this way the entire pipeline is technically making use of JavaScript and the browser but is almost entirely skirting the actual, normal, code execution and rendering path that JavaScript-in-a-webpage takes.

Asm.js is a subset of JavaScript that is heavily restricted in what it can do and how it can operate. This is done so that the compiled Asm.js code can run as fast as possible making as few assumptions as it can, converting the Asm.js code directly into assembly. It’s important to note that Asm.js is just JavaScript – there is no special browser plugin or feature needed in order to make it work (although a browser that is able to detect and optimize Asm.js code will certainly run faster). It’s a specialized subset of JavaScript that’s optimized for performance, especially for this use case of applications compiled to JavaScript.

The best way to understand how Asm.js works, and its limitations, is to look at some Asm.js-compiled code. Let’s look at a function extracted from a real-world Asm.js-compiled module (from the BananaBread demo). I formatted this code so that it’d be a little bit saner to digest – it’s normally just a giant blob of heavily-minimized JavaScript:

Technically this is JavaScript code but we can already see that this looks nothing like most DOM-using JavaScript that we normally see. A few things we can notice just by looking at the code:

• This particular code only deals with numbers. In fact this is the case of all Asm.js code. Asm.js is only capable of handling a selection of different number types and no other data structure (this includes strings, booleans, or objects).
• All external data is stored and referenced from a single object, called the heap. Essentially this heap is a massive array (intended to be a typed array, which is highly optimized for performance). All data is stored within this array – effectively replacing global variables, data structures, closures, and any other forms of data storage.
• When accessing and setting variables the results are consistently coerced into a specific type. For example f = e | 0; sets the variable f to equal the value of e but it also ensures that the result will be an integer (| 0 does this, converting an value into an integer). We also see this happening with floats – note the use of 0.0 and g[...] = +(...);.
• Looking at the values coming in and out of the data structures it appears as if the data structured represented by the variable c is an Int32Array (storing 32-bit integers, the values are always converted from or to an integer using | 0) and g is a Float32Array (storing 32-bit floats, the values always converted to a float by wrapping the value with +(...)).

By doing this the result is highly optimized and can be converted directly from this Asm.js syntax directly into assembly without having to interpret it, as one would normally have to do with JavaScript. It effectively shaves off a whole bunch of things that can make a dynamic language, like JavaScript, slow: Like the need for garbage collection and dynamic types.

As an example of some more-explanatory Asm.js code let’s take a look at an example from the Asm.js specification:

function DiagModule(stdlib, foreign, heap) {
    "use asm";

    // Variable Declarations
    var sqrt = stdlib.Math.sqrt;

    // Function Declarations
    function square(x) {
        x = +x;
        return +(x*x);
    }

    function diag(x, y) {
        x = +x;
        y = +y;
        return +sqrt(square(x) + square(y));
    }

    return { diag: diag };
}

Looking at this module it seems downright understandable! Looking at this code we can better understand the structure of an Asm.js module. A module is contained within a function and starts with the "use asm"; directive at the top. This gives the interpreter the hint that everything inside the function should be handled as Asm.js and be compiled to assembly directly.

Note, at the top of the function, the three arguments: stdlib, foreign, and heap. The stdlib object contains references to a number of built-in math functions. foreign provides access to custom user-defined functionality (such as drawing a shape in WebGL). And finally heap gives you an ArrayBuffer which can be viewed through a number of different lenses, such as Int32Array and Float32Array.

The rest of the module is broken up into three parts: variable declarations, function declarations, and finally an object exporting the functions to expose to the user.

The export is an especially important point to understand as it allows all of the code within the module to be handled as Asm.js but still be made usable to other, normal, JavaScript code. Thus you could, theoretically, have some code that looks like the following, using the above DiagModule code:

document.body.onclick = function() {
    function DiagModule(stdlib){"use asm"; ... return { ... };}

    var diag = DiagModule({ Math: Math }).diag;
    alert(diag(10, 100));
};

This would result in an Asm.js DiagModule that’s handled special by the JavaScript interpreter but still made available to other JavaScript code (thus we could still access it and use it within a click handler, for example).

What is the performance like?

Right now the only implementation that exists is in nightly versions of Firefox (and even then, for only a couple platforms). That being said early numbers show the performance being really, really good. For complex applications (such as the above games) performance is only around 2x slower than normally-compiled C++ (which is comparable to other languages like Java or C#). This is substantially faster than current browser runtimes, yielding performance that’s about 4-10x faster than the latest Firefox and Chrome builds.

This is a substantial improvement over the current best case. Considering how early on in the development of Asm.js is it’s very likely that there could be even greater performance improvements coming.

It is interesting to see such a large performance chasm appearing between Asm.js and the current engines in Firefox and Chrome. A 4-10x performance difference is substantial (this is in the realm of comparing these browsers to the performance of IE 6). Interestingly even with this performance difference many of these Asm.js demos are still usable on Chrome and Firefox, which is a good indicator for the current state of JavaScript engines. That being said their performance is simply not as good as the performance offered by a browser that is capable of optimizing Asm.js code.

Use Cases

It should be noted that almost all of the applications that are targeting Asm.js right now are C/C++ applications compiled to Asm.js using Emscripten. With that in mind the kind of applications that are going to target Asm.js, in the near future, are those that will benefit from the portability of running in a browser but which have a level of complexity in which a direct port to JavaScript would be infeasible.

So far most of the use cases have centered around code bases where performance is of the utmost importance: Such as in running games, graphics, programming language interpreters, and libraries. A quick look through the Emscripten project list shows many projects which will be of instant use to many developers.

• A number of game engines have already been ported. A good demo of what is possible is the BananaBread FPS Game (Source Code) which is playable directly in the browser and features multiplayer and bots.
• A port of LaTeX to JavaScript, called texlive.js, using Emscripten, allowing you to compile PDFs completely within your browser.
• A port of SQLite to JavaScript capable of running in Node.js.
• NaCL: A Networking and Cryptography Library

Asm.js Support

As mentioned before the nightly version of Firefox is currently the only browser that supports optimizing Asm.js code.

However it’s important to emphasize that Asm.js-formatted JavaScript code is still just JavaScript code, albeit with an important set of restrictions. For this reason Asm.js-compiled code can still run in other browsers as normal JavaScript code, even if that browser doesn’t support it.

The critical puzzle piece is the performance of that code: If a browser doesn’t support typed arrays or doesn’t specially-compile the Asm.js code then the performance is going to be much worse off. Of course this isn’t special to Asm.js, likely any browser that doesn’t have those features is also suffering in other ways.

Asm.js and Web Development

As you can probably see from the code above Asm.js isn’t designed to be written by hand. It’s going to require some sort of tooling to write and it’s going to require some rather drastic changes from how one would normally write JavaScript, in order to use. The most common use case for Asm.js right now is in applications complied from C/C++ to JavaScript. Almost none of these applications interact with the DOM in a meaningful way, beyond using WebGL and the like.

In order for it to be usable by regular developers there are going to have to be some intermediary languages that are more user-accessible that can compile to Asm.js. The best candidate, at the moment, is LLJS in which work is starting to get it compiling to Asm.js. It should be noted that a language like LLJS is still going to be quite different from regular JavaScript and will likely confuse many JavaScript users. Even with a nice more-user-accessible language like LLJS it’s likely that it’ll still only be used by hardcore developers who want to optimize extremely complex pieces of code.

Even with LLJS, or some other language, that could allow for more hand-written Asm.js code we still wouldn’t have an equally-optimized DOM to work with. The ideal environment would be one where we could compile LLJS code and the DOM together to create a single Asm.js blob which could be executed simultaneously. It’s not clear to me what the performance of that would look like but I would love to find out!

Q&A with David Herman

I sent some questions to David Herman (Senior Researcher at Mozilla Research) to try and get some clarification on how all the pieces of Asm.js fit together and how they expect users to benefit from it. He graciously took the time to answer the questions in-depth and provided some excellent responses. I hope you find them to be as illuminating as I did.

What is the goal of Asm.js? Who do you see as the target audience for the project?

Our goal is to make the open web a compelling virtual machine, a target for compiling other languages and platforms. In this first release, we’re focused on compiling low-level code like C and C++. In the longer run we hope to add support for higher-level constructs like structured objects and garbage collection. So eventually we’d like to support applications from platforms like the JVM and .NET.

Since asm.js is really about expanding the foundations of the web, there’s a wide range of potential audiences. One of the audiences we feel we can reach now is game programmers who want access to as much raw computational power as they can. But web developers are inventive and they always find ways to use all the tools at their disposal in ways no one predicts, so I have high hopes that asm.js will become an enabling technology for all sorts of innovative applications I can’t even imagine.

Does it make sense to create a more user-accessible version of Asm.js, like an updated version of LLJS? What about expanding the scope of the project beyond just a compiler target?

Absolutely. In fact, my colleague James Long recently announced that he’s done an initial fork of LLJS that compiles to asm.js. My team at Mozilla Research intends to incorporate James’s work and officially evolve LLJS to support asm.js.

In my opinion, you generally only want to write asm.js by hand in a very narrow set of instances, like any assembly language. More often, you want to use more expressive languages that compile efficiently to it. Of course, when languages get extremely expressive like JavaScript, you lose predictability of performance. (My friend Slava Egorov wrote a nice post describing the challenges of writing high-performance code in high-level languages.) LLJS aims for a middle ground — like a C to asm.js’s assembly — that’s easier to write than raw asm.js but has more predictable performance than regular JS. But unlike C, it still has smooth interoperability with regular JS. That way you can write most of your app in dynamic, flexible JS, and focus on only writing the hottest parts of your code in LLJS.

There is talk of a renewed performance divide between browsers that support Asm.js and browsers that don’t, similar to what happened during the last JavaScript performance race in 2008/2009. Even though technically Asm.js code can run everywhere in reality the performance difference will simply be too crippling for many cases. Given this divide, and the highly restricted subset of JavaScript, why did you choose JavaScript as a compilation target? Why JavaScript instead of a custom language or plugin?

First of all, I don’t think the divide is as stark as you’re characterizing it: we’ve built impressive demos that work well in existing browsers but will benefit from killer performance with asm.js.

It’s certainly true that you can create applications that will depend on the increased performance of asm.js to be usable. At the same time, just like any new web platform capability, applications can decide whether to degrade gracefully with some less compute-intensive fallback behavior. There’s a difference in kind between an application that works with degraded performance and an application that doesn’t work at all.

More broadly, keep in mind the browser performance race that started in the late 00′s was great for the web, and applications have evolved along with the browsers. I believe the same thing can and will happen with asm.js.

How would you compare Asm.js with Google’s Native Client? They appear to have similar goals while Asm.js has the advantage of “just working” everywhere that has JavaScript. Have there been any performance comparisons?

Well, Native Client is a bit different, since it involves shipping platform-specific assembly code; I don’t believe Google has advocated for that as a web content technology (as opposed to making it available to Chrome Web Store content or Chrome extensions), or at least not recently.

Portable Native Client (PNaCl) has a closer goal, using platform-independent LLVM bitcode instead of raw assembly. As you say, the first advantage of asm.js is compatibility with existing browsers. We also avoid having to create a system interface and repeat the full surface area of the web API’s as the Pepper API does, since asm.js gets access to the existing API’s by calling directly into JavaScript. Finally, there’s the benefit of ease of implementability: Luke Wagner got our first implementation of OdinMonkey implemented and landed in Firefox in just a few months, working primarily by himself. Because asm.js doesn’t have a big set of syscalls and API’s, and because it’s built off of the JavaScript syntax, you can reuse a whole bunch of the machinery of an existing JavaScript engine and web runtime.

We could do performance comparisons to PNaCl but it would take some work, and we’re more focused on closing the gap to raw native performance. We plan to set up some automated benchmarks so we can chart our progress compared with native C/C++ compilers.

Emscripten, another Mozilla project, appears to be the primary producer of Asm.js-compatible code. How much of Asm.js is being dictated by the needs of the Emscripten project? What benefits has Emscripten received now that improvements are being made at the engine level?

We used Emscripten as our first test case for asm.js as a way to ensure that it’s got the right facilities to accommodate the needs of real native applications. And of course benefiting Emscripten benefits everyone who has native applications they want to port — such as Epic Games, who we teamed up with to port the Unreal Engine 3 to the web in just a few days using Emscripten and asm.js.

But asm.js can benefit anyone who wants to target a low-level subset of JavaScript. For example, we’ve spoken with the folks who build the Mandreel compiler, which works similarly to Emscripten. We believe they could benefit from targeting asm.js just as Emscripten has started doing.

Alon Zakai has been compiling benchmarks that generally run around 2x slower than native, where we were previously seeing results anywhere from 5x to 10x or 20x of native. This is just in our initial release of OdinMonkey, the asm.js backend for Mozilla’s SpiderMonkey JavaScript engine. I expect to see more improvements in coming months.

How fluid is the Asm.js specification? Are you open to adding in additional features (such as more-advanced data structures) as more compiler authors being to target it?

You bet. Luke Wagner has written up an asm.js and OdinMonkey roadmap on the Mozilla wiki, which discusses some of our future plans — I should note that none of these are set in stone but they give you a sense of what we’re working on. I’m really excited about adding support for ES6 structured objects. This would provide garbage-collected but well-typed data structures, which would help compilers like JSIL that compile managed languages like C# and Java to JavaScript. We’re also hoping to use something like the proposed ES7 value types to provide support for 32-bit floats, 64-bit integers, and hopefully even fixed-length vectors for SIMD support.

Is it possible, or even practical, to have a JavaScript-to-Asm.js transpiler?

Possible, yes, but practical? Unclear. Remember in Inception how every time you nested another dream-within-a-dream, time would slow down? The same will almost certainly happen every time you try to run a JS engine within itself. As a back-of-the-envelope calculation, if asm.js runs native code at half native speed, then running a JS engine in asm.js will execute JS code at half that engine’s normal speed.

Of course, you could always try running one JS engine in a different engine, and who knows? Performance in reality is never as clear-cut as it is in theory. I welcome some enterprising hacker to try it! In fact, Stanford student Alex Tatiyants has already compiled Mozilla’s SpiderMonkey engine to JS via Emscripten — all you’d have to do is use Emscripten’s compiler flags to generate asm.js. Someone with more time on their hands than me should give it a try…

At the moment all DOM/browser-specific code is handled outside of Asm.js-land. What about creating an Emscripten-to-Asm.js-compiled version of the DOM (akin to DOM.js)?

This is a neat idea. It may be a little tricky with the preliminary version of asm.js, which doesn’t have any support for objects at all. As we grow asm.js to include support for ES6 typed objects, something like this could become feasible and quite efficient!

A cool application of this would be to see how much of the web platform could be self-hosted with good performance. One of the motivations behind DOM.js was to see if a pure JS implementation of the DOM could beat the traditional, expensive marshaling/unmarshaling and cross-heap memory management between the JS heap and the reference-counted C++ DOM objects. With asm.js support, DOM.js might get those performance wins plus the benefits of highly optimized data structures. It’s worth investigating.

Given that it’s fairly difficult to write Asm.js, compared with normal JavaScript, what sorts of tools would you like to have to help both developers and compiler authors?

First and foremost we’ll need languages like LLJS, as you mentioned, to compile to asm.js. And we’ll have some of the usual challenges of compiling to the web, such as mapping generated code back to the original source in the browser developer tools, using technologies like source maps. I’d love to see source maps developed further to be able to incorporate richer debugging information, although there’s probably a cost/benefit balance to be struck between the pretty minimal source location information of source maps and super-complex debugging metadata formats like DWARF.

For asm.js, I think we’ll focus on LLJS in the near term, but I always welcome ideas from developers about how we can improve their experience.

I assume that you are open to working with other browser vendors, what has collaboration or discussion been like thus far?

Definitely. We’ve had a few informal discussions and they’ve been encouraging so far, and I’m sure we’ll have more. I’m optimistic that we can work with multiple vendors to get asm.js somewhere that we all feel we can realistically implement without too much effort or architectural changes. As I say, the fact that Luke was able to implement OdinMonkey in a matter of just a few months is very encouraging. And I’m happy to see a bug on file for asm.js support in V8.

More importantly, I hope that developers will check out asm.js and see what they think, and provide their feedback both to us and other browser vendors.

These are a few arguments against the switch that I’ve seen so far:

A browser switching to WebKit will result in stagnation

That is demonstrably not true. KDE created KHTML, Apple created WebKit based upon that, Google created WebKit/Chromium based upon that. I don’t think anyone can successfully argue that Chome/Chromium isn’t a better browser than Safari which isn’t a better browser than Konqueror. The Chrome team has proved that stagnation when using WebKit is merely a choice, as a contributor to WebKit you have the complete ability to drive it in a direction you wish (often for the better). I see no reason why the highly-skilled development team at Opera won’t be able to do the same. They can implement a number of their Opera-specific features into WebKit and it’s likely that those features will start to trickle back into other WebKit-using browsers as well.

This is helping to making WebKit a de facto standard, bugs-and-all

I don’t see this argument as being relevant any more, WebKit is already a de facto standard. I mean, everyone remembers when the browsers decided to implement -webkit vendor prefixes? It’s obvious that the “WebKit is an de facto standard” horse has already left the gate. As to the bugs: WebKit is a common code base that a number of browsers contribute to, however every browser vendor has the ability to make changes to their own fork of the code base. I see no reason why these “now-standard” WebKit bugs wouldn’t be fixed by any single vendor. Having a bug in the engine does not mean that a single browser vendor is incapable of fixing it — they would just be willfully not fixing it (as the browser vendors currently willfully clone -webkit prefixes).

In the case of JavaScript libraries virtually everyone has standardized upon jQuery at this point. This didn’t result in stagnation, which was a major concern, instead it’s resulted in a number of interesting and hyper-popular second-tier frameworks which build upon jQuery, such as: Twitter Bootstrap, HTML5 Boilerplate, and Backbone.js.

This will affect Opera’s ability to influence standards

I don’t see the switch to WebKit causing this. I do see Anne van Kesteren‘s move to Mozilla as being a massive blow to Opera’s ability to push standards though. I don’t know anything about the situation but if his moving was caused by the switch to WebKit then yes, Opera’s move to WebKit has affected their ability to influence standards.

Opera switching to WebKit is a slippery slope and/or Opera is a small player, Firefox or IE switching to WebKit would be a bigger problem

I think one this is clear already: WebKit has completely and unequivocally won mobile at this point. They are nearly the only rendering engine used on the vast majority of mobile browsers, including the soon-to-switch Opera Mini/Mobile browsers too. There is no reason to worry about a slippery slope, the slope has already been slid down. In order for any other browser to remain relevant in the world of mobile (which, you must admit, is quickly becoming the only world we live in) they must keep feature parity with WebKit. Let’s follow this to its logical conclusion: In a world in which WebKit is now virtually the only mobile browser vendor Mozilla and Microsoft will feel increased pressure to switch their browser engines over to WebKit in order to keep pace. Google has proved that with Chrome that WebKit stagnation is simply a choice so there’s no reason why these other companies shouldn’t be able to build off of WebKit (and possibly create WebKit hybrids, such as WebKit + IonMonkey).

The big question becomes: Should they switch?

At this point it’s honestly a business/engineering decision for Mozilla and Microsoft (as it always has been). If some percentage of your developer force is spending all of their time implementing the same standards that everyone else is implementing then switching to a common code base will give you the ability to free up some of your developers to work on something else. You saw what happened in the case of Chrome: They used their extra developer time to completely crush the competition on performance. This resulted in the everyone-wins race to become the fastest browser.

Ultimately it’s important to remember that WebKit is not a monolithic entity. It’s a shared codebase that a number of corporations contribute to. (In this respect it’s different from jQuery: Almost all contributions to jQuery comes back into the main codebase, whereas with WebKit some come back to the main codebase and some stay in a fork.) There is a lot of code sharing going on but it isn’t the be-all and end-all of browser development. Innovation can clearly still occur when working on a shared codebase and performance will almost certainly continue to improve.

I’ve found that this occurs when people are doing inline event tracking and are assuming that the Google Analytics functionality will always exist. Ghostery-or-not this is generally not a good practice to have: Assuming that some external, 3rd-party, script will always correctly load will result in many a broken web page.

You see cases where people work around this issue, for example in the HTML5 Boilerplate project:

This attempts to load the jQuery library directly from Google’s CDN and if it fails (for whatever reason — Google’s down, bad Internet connection, or some other weird reason) then it falls back to attempting to load a local version of jQuery. One question that comes of this is: Why not just always load jQuery from your local source rather than rely upon a third party? In this case it makes a lot of sense to pull jQuery from a CDN as you’ll benefit from the performance and caching benefits that a CDN provides, all while reducing the total bandwidth that you’ll have to expend on your end.

In the case of Google Analytics you typically don’t have that luxury: Google recommends that you load the ga.js library directly from their site (which can make sense as then they can push out seamless API changes and bug fixes). However this still causes issues for when Google Analytics simply doesn’t load, wether due to network conditions or due to Ghostery or some other browser extension.

For this reason, in a site I recently built, I added in the following code as a light wrapper around Google’s event tracking API:

and then when I wanted to track an actual event I would do something like this:

If Google Analytics has failed to load for some reason then the track() call just silently does nothing – and for users who have Google Analytics installed it tracks the event. I should note that this same technique applies for other event tracking APIs as well, such as Mixpanel or KISSmetrics (they’re blocked the same as Google Analytics).

Even libraries like Segment.io, which act as a unified wrapper around tracking, are prone to the same issue. Even though they’ve gone through all the hard work of creating a unified API for event tracking they are still blocked by Ghostery, which will likely still result in errors in your application. In that case you would just write a similar track(...) wrapper around their analytics.track('Name', properties) call.

Update: The current version of the Google Analytics script always exposes the _gaq object, even if the code isn’t loaded, so this may be a non-issue for Google Analytics (although the principles here hold true for any third-party-included libraries). I’ve gotten word from the Ghostery team that Ghostery now handles this case, automatically exposing missing global objects for cases like this (see below in the comments). I’ve also gotten word from the Segment.io team that, as of now, Ghostery does not block their analytics tracking so you may not need a wrapper there. Additionally Segment.io does check to make sure that the potentially-blocked global objects are actually there before attempting to use them. In general though, and this goes for all third-party script, you’ll probably want to have a wrapper (or an alternative script) if you’re loading code from a third-party domain name.

Happy news! My book, Secrets of the JavaScript Ninja is finally in stock on Amazon! It’s been available on Manning.com for over a month now but I think Amazon has been struggling to keep the books consistently in stock.

You can get an ePub or Kindle version of the book at Manning.com.

I’ve written about the writing process for this book previously (and why it’s taken so long to get out). I especially want to thank my co-author Bear Bibeault who worked hard to get this book out the door.

One question that I’ve been asked recently: “How relevant is this book to writing JavaScript in 2012?” I think it’s still exceedingly relevant. This book has two major focuses:

1. Teaching aspects of the JavaScript language that are under-served or poorly explained. (For example I’ve been getting a number of messages from people saying how much they appreciated the explanation of closures in the book and that it finally helped them to understand that feature of the language.)
2. Helping people understand the internal workings of cross-browser DOM-centric JavaScript libraries (such as jQuery). Considering how popular jQuery is (64% of the top 10,000 web sites use jQuery) and, relatively, how few people truly understand what the library does or is trying to do – there is still a huge market for people to learn what exactly jQuery is and how it works.

I’m really happy that this book is finally out and to celebrate I’ve been trying to blog more (as you may have noticed). Are there any particular topics that you’d like me to write about? Let me know!

It has to deal with the eternal question: How do you store sensitive configuration options (such as usernames, passwords, etc.) in source control? Typically what I’ve done is to just punt on the problem entirely. I create a dummy configuration file, such as conf/sample-settings.json which has the basic structure but none of the details filled out. For example:

conf/sample-settings.json

If someone else needed the details I would just email it to them, or some such (not ideal). Especially when it came time to add additional information to the file or make other changes.

The technique I picked up from Craig was to, instead, keep an encrypted version of the configuration file in source control and then provide a means through which the user can encrypt and decrypt that data.

In this case you can still have the a dummy config file, if you wish.

To start you’ll want to make sure you have your source control ignore the configuration file — just to make super-sure that no one ever accidentally commits it. In Git you’d add a line like this to your .gitignore file:

.gitignore

Next you’ll want to create your actual config file and populate it with the real values.

conf/settings.json (* Do not check this in to source control!!)

Finally you’ll want to create a script (I’m using a Makefile) that the user can run to encrypt and decrypt the file. This script uses OpenSSL, and specifically CAST5, to encrypt/decrypt the file. OpenSSL was chosen in particular as it worked out-of-the-box on both Linux and Mac machines.

OpenSSL reads in the appropriate files (depending upon if you’re encrypting or decrypting) then will prompt you for a password to encrypt/decrypt the file. (You’re free to use any encryption scheme that OpenSSL supports, of course.)

Makefile

With all this in place the next step is simple, you’ll run:

and you’ll enter in a password with which to encrypt the config file:

Make sure you write this down and don’t forget it — it’ll be very hard (if not impossible) to get your config file back if you forget the password.

At this point you’ll have a conf/settings.json.cast5 file and you can commit all the changes, using something like:

Now whenever someone downloads the code from source control they’ll need to either fill in their own values into the config file or they’ll need to get the password from you (the one you entered when you ran make encrypt_conf — or even better, use a shared password safe to manage this). Once they have the password they just run the following and enter it:

If you ever need to update the values in the config file, it’s really straight-forward. Just update the config file, run make encrypt_conf again, and commit the new conf/settings.json.cast5 file.

One extra bit that you can add to your application, to make this process more intuitive, is a check for a missing config file and output with instructions for using the Makefile.

For example if you were using Node.js you could do:

Also, you may want to consider having a check to see if the decrypted file is out of date (which can happen if some changes were made in the source control, then were checked out, but you didn’t also run make decrypt_conf). Perhaps something like the following:

And that’s it! Simpler than passing around config files manually and you still get all the benefit of using revision control to manage the file and changes.

The module is now available on NPM and can be installed by running:

npm install i18n-2

The code and documentation for the module is available on Github here:

https://github.com/jeresig/i18n-node-2

I’ve designed the API to work as simply as possible with Express.js – while still supporting a simple Node.js usage or usage with another web framework.

The module includes a number of features built-in:

• Automatic integration with Express as middleware and integration with templating solutions.
• Support for handling the setting of locale based upon query strings and/or subdomain.
• Dynamic reading and updating of translation files during development mode.
• Automatic caching of translation files in production mode.

Usage is simple, in your Express app.js:

Inside your Express view:

Inside your templates (this example uses the Swig templating system).

{% extends "page.swig" %}
{% block content %}
<h1>{{ __("Welcome to: %s", title) }}</h1>
<p>{{ desc }}</p>
{% endblock %}

My module is based off of the i18n-node module by Marcus Spiegel.

I used the original i18n-node module as a basis for further work as it provided the closest solution to full Express.js integration that I needed. After some initial usage though I saw that it was very much lacking. As I described in my previous post there is potential for severe problems to occur in an asynchronous environment like Node.js.

The problem that I was having was rather severe and, unfortunately, required a complete rewrite of the module. The original i18n-node module stored the locale information in a closure. Calling setLocale changed the locale but did so for every instance that was currently active, resulting in templates being rendered in the wrong language.

Here is the issue that is especially surprising (but not so much once you know how modules work in Node.js): Since the state was stored in a closure repeated calls to require('i18n') resulted in the current locale state being held the entire time, not reset. This meant that when a new request came in, the locale was set, changing the locale for all other active requests.

The only way to fix this particular problem is to simply encapsulate all state into a single object — I did this by making an object that you instantiate: new I18n(options).

I looked at other JavaScript translation solutions, including the excellent Jed and messageformat and while their translation capabilities are top-notch what they provided was overkill for my particular application and did not provide the Express.js integration that I needed.

I hope others will find this module to be useful, especially when used in conjunction with my strategy for i18n and Node.js.

I’ve used enough internationalized web sites, and travelled to enough foreign countries and attempted to use English language sites back in the US, that I knew what kind of features I wanted:

• Full parity between languages. Wherever possible the same content should be available to everyone.
• Use sub-domains to contain different language versions. It’s overkill/expensive to use different TLDs and it’s annoying to have to twiddle query strings or paths to match a language.
• No automatic translations of content to the user’s native language. There is nothing worse than arriving at a site and being forced into a version that you can’t read either because it’s poorly translated, or you’re being GeoIP detected, or you’re on a computer whose language settings don’t match your own. If you visit a URL it should always be in the same language
• No automatic redirects to a site with the user’s native language. Same as the last one. If I visit “foo.com” I should not be automatically routed to “es.foo.com” because it thinks that I speak Spanish. Instead, give the user a notification in their native language and allow them to visit the page themselves.

The end result would be a simple URL system that works like so:

• domain.com – Main site (English)
• ja.domain.com – Japanese Site
• XX.domain.com – Other languages

and due to the full translation parity and the lack of URL modification it meant that for every page that you visited you could visit the same page in another language just by changing the sub-domain.

For example:

   domain.com/search?q=mountain
ja.domain.com/search?q=mountain

Both work identically, just the second one is presented in Japanese.

This has the benefit that in the header of the page I can link the user to the same exact page but in their native language.

Additionally I can use the rel=”alternate” hreflang=”x” technique to help Google understand the structure of my site better. I can put this in the header of my page and Google will show the language-preferred version of the site in the Google results.

<link rel="alternate" hreflang="ja"
    href="http://ja.domain.com/" />

Server

Encouraging users to find the correct content is a key implementation detail, considering that the content is not translated automatically nor is the user redirected to content in their native language. While the links in the header are a good start I also wanted to show a message at the top of the page encouraging the user to view the content (with the message being written in their native language).

As it turns out this can be particularly tricky to implement. The easiest way to do it is to simply check the user’s request headers and look at what’s listed in their “Accept Language” and then display the message based upon that. This really only works if your content is always dynamic and is never being cached.

If that’s not the case for your application, how and where you do your caching matters.

In my particular application I’m using nginx in front of a proxied collection of Node/Express servers. This means that everything coming from the Node server is cached (including any messages to the user telling them to visit another page).

As a result, in order to display this message to the user we’re going to need to manage the logic for this on the client. Unfortunately this is where we hit another stumbling block: It’s not possible to reliably determine the desired language of the user using just JavaScript/the DOM.

Thus we’ll need to get the server to pass us some extra information on what the user’s desired language is.

To do this I used the nginx AcceptLanguage Module and then set a cookie with the desired language and passed it to the client. This is the relevant nginx configuration to make that happen.

set_from_accept_language $lang en ja;
add_header Set-Cookie lang=$lang;

And now on the client-side all the needs to happen is reading the cookie for the desired language and displaying the redirect message if the current language and the desired language don’t match.

This gives the best of all worlds: nginx continues to aggressively cache the results from my Node servers and the client displays a message in the user’s native language encouraging them to visit the appropriate sub-domain.

i18n Logic

I’ve written a new Node i18n module which follows the makes the following strategy possible.

None of the i18n logic is particularly out of the ordinary but there are a few strategies I took that helped to simplify things.

• All translations are stored in named JSON files.
• Those files are loaded and used for in-place translation in the application.
• Translations are done using the typical __("Some string.") technique (wherein “Some string.” is replaced with the translated string, if it exists, otherwise “Some string.” is returned instead).

Since all requests to the server are handled by a single set of servers this means that translation logic cannot be shared – it must be initialized and used on a request-by-request basis. I’ve seen other i18n solutions, like i18n-node that assume that the server will only ever be serving up pages in a single language, and this tends to fail in practice – especially in the shared-state, asynchronous, realm of Node. For example:

Since whenever a request comes in and set the current language, it sets the current language for the shared i18n object – and given the asynchronous nature of Node it’s possible that other requests may be happening at the same time, and thus changing the displayed language of another request as a result.

You’ll want to make sure that, at minimum, your current language state is stored relative to the current request to avoid this problem. (My new i18n node module fixes this, for example.)

In practice it means that you’ll be adding a i18n property to the request object, likely as a piece of Express middleware, like so:

app.use(function(req, res, next) {
	req.i18n = new i18n(/* options... */);
	next();
});

Workflow

I have it so that the i18n logic behaves differently depending upon if the server is in development mode or in production mode.

When in development mode:

• Translation JSON files are read on every request.
• Translation files are updated automatically any time a new string is detected.
• Warnings and debug messages are shown.

In production mode:

• All JSON translation files are cached the first time they’re read.
• Translation files are never updated dynamically.
• No warnings or debug messages are shown.

The two major differences here are the caching and the auto-updating of the JSON files. When in development it’s quite useful to have the translation files reload on every request, in case a change has been made to their contents. Whereas in production they really should be considered static files.

Additionally, the workflow of having the translation files update every time a new string is found is actually quite useful: It helps you to catch strings that you may have forgotten to translate. Naturally doing this in production (frequently hitting the disk) is not a good idea.

Marking Up Strings

Strings that need to be translated can be found in a number of locations: Inside your application source, inside templates, inside JavaScript files, and even (god forbid) inside CSS files.

In the case of my application I had no strings inside my JavaScript or CSS files. This worked out nicely because I had already written my application in such a way that content is never being dynamically constructed from strings inside my client-side JavaScript. If I were to do that I would use a template and put the template directly into the HTML of my page, using something like my JavaScript Micro-Templating solution.

I consider it to be especially important that you try to avoid having any translatable strings in your JavaScript or CSS files as those files you’ll want to heavily cache and likely put onto a CDN. Naturally, you could dynamically replace those strings as part of your build process and just generate a number of script/css files, one for each language you support, but that is likely up to you and how much extra work you want to introduce into your build.

Inside my application I made sure that the only time I ever attempted to translate a string it was inside of a Express view handler (meaning that I had access to the request object, which is where I bound my i18n object).

An example of using the i18n object inside a view:

For my templates I use the confusingly-named swig, but the technique for actually using the i18n methods will be roughly the same for most templating systems:

A string is wrapped with a __(...) call and the string is replace with the translated string, as it is called.

Translation

So far the actual translation process has been relatively simple. It’s just me doing the translation and I’m not out-sourcing anything (at least not yet). Additionally my site is relatively simple, only a couple dozen strings at the moment.

I’ve been able to temporarily “cheat” in a few ways (at least until I hire a real translator):

• Use Open Source Translations. There are already massive Open Source projects out there that have done a lot of hard work in translating their UIs. For example Drupal has all of their translations online in easy-to-download formats. I was able to find a number of strings that I needed by going through their files.
• Look at already-localized sites. This is another cheat but look for other sites that have some of the same features of your site and have already gone through the hard work of localizing into multiple languages, like Google. (In my case I was working on a search engine so a number of Google’s strings directly matched strings on my site.)
• Google Translate. I know, I know – but I was really surprised at how much better Google Translate has gotten as of late, especially for translating single words or concepts. It’s able to tell you the exact meanings for different possible translations, which is really impressive.

Conclusion

I’m only a couple weeks in to having implemented this process on my site, so I’m sure some things are likely to change as I start to scale up more. I’ve already substantially increased traffic to my site, especially as Google has started to index the newly-translated site. As I mentioned before I have a new Node i18n module that complements the above process and hopefully make it easier for others to follow, as well.

I recorded the talk on my laptop (so please excuse the echo as it wasn’t properly mic’ed up) and you can watch it here:

A slightly older (stats are slightly out of date) version of the slides can be found here:

We’re releasing a completely new platform that targets people with no programming knowledge and gives them an engaging and fun environment to learn in.

Over everything else we wanted to emphasize creativity and exploration and make it approachable for people of all ages, including young kids.

Sal Khan and I recorded a video giving a good introduction to what we’re releasing:

To get started feel free to visit the Khan Academy Computer Science site:

http://www.khanacademy.org/cs

Explore, play around, and build something!

Jamie Wong, who worked on the CS platform this summer, has also written up a blog post on his experience developing the platform.

Methodology

Earlier this year I was working on an iteration of the CS platform which was heavily focused on structured curriculum leading towards exploratory projects.

Until I saw Bret Victor‘s mind-blowing talk on using a responsive programming environment.

After watching that video I couldn’t shake the feeling that what he presented was a fundamentally different way of approaching how to interact with, and learn how to, code.

In an environment that is truly responsive you can completely change the model of how a student learns: rather than following the typical write -> compile -> guess it works -> run tests to see if it worked model you can now immediately see the result and intuit how underlying systems inherently work without ever following an explicit explanation.

When code is so interactive, and the actual process of interacting with code is in-and-of-itself a learning process, it becomes very important to put code front-and-center to the learning experience.

Rather than starting Computer Science education off by explicitly teaching how a computer works or fundamental programming concepts (like variables, logic, control structures, etc.) you put the student into code of graduated complexity and encourage them to manipulate, explore, and write their own programs.

Once they start to explore and figure out things for themselves then they can begin to dig into all the explanatory tutorials and documentation that are provided to clarify how things work.

The desire to learn and understand can be a powerful accelerant for students and it’s something that is completely missing from almost all Computer Science education.

When I look back at how I became interested in programming, or ask other programmers about how they started to program, the answer is very frequently: “I was given a [Basic/QBasic/Python] compiler and a bunch of programs and I wanted to understand how it all worked so I could write my own!” It was only after digging in to the code that the student wanted to understand how to tackle certain programmatic challenges – but they then did so with great gusto!

The Khan Academy Computer Science program is a distillation of this process into it’s purest form: Directly encouraging experimentation, exploration, and creativity as a direct gateway to deeper learning.

How do we teach?

On the site we provide dozens of programs, at a variety of skill levels, across a number of disciplines (including art, mathematics, and science). We wanted students to find the niche programs that could capture their excitement while also being able to provide a direct line from normally theoretical concepts (such as algebra, geometry, or physics) towards direct evidence of their utility in a tangible medium that the student can control.

We provide a number of tutorials that cover concepts that come up frequently in the programs that we’ve developed (from programming concepts like variables and conditionals to other concepts inherent to the medium such as drawing and animation). These tutorials should cover most of the questions that students have when they’re just beginning to program and can be accessed at a student’s leisure.

I’ve talked before about how I think that JavaScript, and specifically learning how to code in-browser, is inherently a better way of learning. Reducing the complexity of getting started down to zero will result in more people learning. Additionally the ubiquity of JavaScript only serves to educate people in a language that will be generally useful.

I also feel that learning how to code in an environment where the output is visual and highly enticing can better excite the student towards wanting to produce, and thus learn, more. I feel that purely textual coding environments (or coding environments that favor textual output over visual output) serve as a highly uninteresting introduction to programming.

At a fundamental level we want the students using the platform to be self-directed learners, choosing to drive the direction of their education in a way that best suits their interests.

We’ve taken some steps to provides students with some level of possible direction which they could follow, if interested. For example when viewing an introductory program that only does basic drawing we encourage the student to explore variables or animation — ways of bringing their program “to life” which simultaneously getting them excited about attempting to learn those, comparatively, complex subjects.

We already have a number of programs for students to explore, and we will be adding many more (highlighting some of the best from the community as well). With that being said we don’t want to be the sole arbiter of programs that a student might find to be interesting. For this reason we’re working to put community, sharing, and collaboration as pillars of the platform.

We want students to work up to the point at which they feel comfortable writing their own projects. We’ve already seen some pretty phenomenal output from both people here at Khan Academy and from middle-school-aged students who’ve never programmed before.

Over this past summer we held a number of “Discovery Lab” sessions at Khan Academy in which kids of roughly middle school age did a number of learning activities. We had the opportunity to watch them interact with the Computer Science content and were blown away with what they created. In the first hour they were designing and drawing their own characters, in the second they had added in variables and were starting animations, by the end of the third hour they had functional draw loops and were adding in logic to control the animations.

We’ve never seen students of that age progress so quickly through such challenging concepts. It was noted that they were moving faster than most college age students taking a normal intro Computer Science course.

One project that I particularly enjoyed was made by a student who really loved Minecraft. He designed some characters from the game (drawing them using a number of points and rectangles) and then added in some user interaction. He made it so that you could control the the position of a creeper by moving your mouse, as it came closer to “Steve” he would start to hiss and blink, upon hitting Steve he would explode into a firey cloud. This was done over the course of maybe four hours and two days.

To say that I’m excited to see how people are going to use this platform is an understatement.

Coding Environment

The environment is built around a two-panel view: The left being an Ace editor and the right being a canvas.

All of the user’s code is written using JavaScript and uses Processing.js to generate all the graphics.

All code is executed in real-time. Not only is the code re-run (which is relatively easy to do, just re-evaling the code) but the code that you change is live dynamically injected into the runtime — this gives you the ability to change variables, colors, or even functions on the fly without the program ever restarting.

We did a lot to try and take advantage of the real-time nature of the environment, giving the user all sorts of tools to adjust the code, and thus the state of the program as well.

We have a number picker:

We have a color picker:

and we even have an image picker (using images from the always-excellent Lostgarden.com):

We’ve also put a lot of work into the error correction. We build off of the existing linting that we do to try and provide extra levels of hinting. We do spelling correction, provide argument suggestions, and try to make suggestions for common beginner mistakes.

In our experience students who’ve never programmed before can be left unattended using this environment and be able to resolve almost all of their issues via the error hints or the tutorials.

As it turns out it’s really, really, hard to build the real-time execution environment that Bret Victor proposed, in practice — even harder in a browser. I plan on writing many posts about the nuts-and-bolts of the implementation when we release the code-and-editor as Open Source (which should be soon).

To make the environment we have to use a lot of features that only the latest browsers support. For this reason we are only actively supporting Chrome, Firefox, Safari, and IE 9+. Chrome, Firefox, and Safari will have the best experience as they also have Web Worker support (which we make extensive use of).

Thankfully we’re able to make this hard call as less than 6% of our user base fails to match the above criteria — and that number is quickly shrinking.

Side note: One area of the implementation that would benefit from an Open Source library is some form of in-browser, pure-JavaScript, ECMAScript 5 linter. We use JSHint right now and we’d really appreciate being able to enforce true strict mode (JSHint’s strict mode support is rather limited in nature).

Collaboration and Community

With the Khan CS platform we wanted to provide an environment that effectively mimicked the best aspects of great programming communities like Github.

Students can always see the code associated with a project and it’s always displayed next to the output. Students can change any code they see and and save it to their own personal account (aka “forking” in the Github parlance).

(Note the authorship details and the “Based on” link with screenshot that shows up below a project that has been “forked” from another project.)

Community feedback and discussion is also large part of the benefit that we hope students will be able to derive from the platform. Students will be able to solicit feedback from others and have discussions within the context of their code. We’re not to the point of having line-level comments yet but we hope the initial discussion functionality will be very useful to everyone using the site.

To a large degree the Khan Computer Science platform is influenced by the Open Source communities that I’ve participated in: All the code is open, experimentation is encouraged, and feedback is easily provided. I feel that this particular environment is highly conducive to learning as you can easily see many ways of tackling a problem. You can also get feedback on your personal attempts, creating a tight feedback loop.

(I should note that we do have versioning baked in to the system as well but we haven’t exposed it yet, though probably will at some point.)

We also really wanted students to feel excited about sharing their work with others. We provide the ability to share it through traditional social media (Facebook, Twitter, and email) but also on the site and through their profile.

On a student’s profile they have the ability to choose some explorations that they wish to explicitly share with others. We want students to take pride in their best work and feel open to sharing it with the world.

Tutorials and Next Steps

While students exploring the content of the platform will certainly learn some things simply by exploring and manipulating code, we wanted to go a step further and provide tutorials and a clear narrative to help students better understand some of the complex subject matter that they’re seeing.

The primary vehicle for learning is through the tutorials on the site. These are unique in that they both behave like a normal code editor but also as a “video” player. All of the tutorials include audio walkthroughs (playing live from Soundcloud using their API), combined with live playback from the presenter, right in the code editor itself.

We wanted to have the playback take place directly in the editor to try and provide an experience for the student that was more immediate and tangible. Producing hard-coded video explanations or forcing the student to walk through step-by-step written instructions is both tedious and lacks interactivity. With this playback mechanism the student will be able to see the results live, exactly as they would always run in the editor, while also being able to pause the playback and manipulate, or copy, the code that they see. This ability to dig into the meat of a tutorial should serve to deepen, and reinforce, a student’s understanding of the subject matter.

In addition to the tutorials we have a large amount of documentation on Processing.js and JavaScript, all of which is written up and available in a code editor (again, for further exploration). All documentation is available on a dedicated page and beneath every code editor, making it an easy reference for students.

Next to many of the projects that are on the site we also provide some suggested “Next Steps” that the students can follow for some deeper engagement and learning.

We don’t explicitly evaluate the student’s work here and instead we encourage them to share their work with others, providing an external motivator to try and write the best possible code they can. With the active and excited Khan Academy community that already exists students will be able to receive constructive feedback on their work and make a number of improvements thereon.

One question is obviously: How much “Computer Science” do we actually teach? Obviously, right now we’re not teaching very much in the way of traditional Computer Science content, we’ve placed far more focus onto the platform and feel that a lot can be gained from just what we’ve already built. We’re still very early on in the production of content and will be producing much more, of increasing complexity, over the upcoming weeks and months.

We decided that it was much more fruitful for us to tackle the most challenging problem that exists with Computer Science education first: Getting people excited about programming. If we can get people excited about programming, and build (or point them to) the resources they need to learn more then we will have been successful.

Eventually I think we’ll work to effectively replicate the materials one typically finds in a “Computer Science 101″ but in a way that is far more engaging and self-paced than what you would find at a traditional university.

So what’s next? We want to work on providing even more direction and tooling around students writing larger projects. We want to provide easy APIs around storage, network connectivity, and dependencies that will allow for some fascinating projects. Additionally we want to improve peer-to-peer and mentor-student collaboration, finding ways of pairing struggling students with skilled developers. Finally we want to produce a lot more content of a wide range of difficulties and topics.

We’re only just getting started and how the community responds to this release will largely help to drive the direction of the platform as a whole. Feedback is most welcome!

Thank Yous

This is the culmination of a bunch of work by a number of people and I want to take this opportunity to thank them — without them this project would not have happened and would not have come out as well as it did.

Most importantly I want to thank Jamie Wong and Jessica Liu — both of whom worked this summer on the Computer Science platform and have both made tremendous contributions in ways that I couldn’t have even hoped for.

Jamie primarily worked on the Khan CS platform, providing a superb level of quality and scalability that we’ll be enjoying for a long time. He provided contributions above-and-beyond what was needed and even helped to improve some of our development processes at Khan Academy. His expertise has been a massive boon to the project, quickly escalating it to encompass functionality that we didn’t even hope we’d be able to provide. In the short time he’s been with the company he’s already made a tremendous impact upon the organization.

Jessica worked on creating Computer Science tutorials, meticulously researching methodologies and iterating after gathering user feedback. The structure and content of the Computer Science curriculum is largely of her design (if you view some of the CS tutorials you’re sure to enjoy much of her expertise and wit). She also worked on improving the usability of the editor and the curriculum, making it far easier for newbies to get up to speed. Her innate understanding of a new programmer’s mind, combined with her ability to execute on a solution that will aide them has made the platform an incredibly effective tool for self-paced learning.

I strongly hope to have the opportunity to work with both of them in the future.

I also want to thank everyone at Khan Academy for their input into the design process and their help in creating the awesome projects you see on the site. I especially want to thank Sal Khan, Shantanu Sinha, Ben Kamens, Jason Rosoff, Ben Komalo, Ben Alpert, Tom Yedwab, and Mathias Hiron for their help and advice in making the platform what it is today.

I also want to thank the teachers and students at the Khan Academy Discovery Lab for being such willing, and excited, participants in experimenting with the Computer Science platform. Your feedback was instrumental in identifying many issues that we were able to make much better as a result.

And so, on that note, I hope you will all enjoy the new Khan Academy Computer Science platform!

I wanted to give a brief status update on the book and how it’s going. I started the book in early 2008 and was actually quite productive, finishing nearly the entire book that year (with some missing gaps that I fixed up in 2009). There was some work left to do to make it a better book but, honestly, I got caught up in coding and stopped focusing on writing. I had to prioritize my time and I chose to prioritize doing more development and focusing on my personal life. Some time last year Manning brought on a co-author, Bear Bibeault, to finish the book and get it out the door. He’s done a bunch of revision bringing it up-to-date and just the other week I finally wrote the preface. It’s in final revisions now and should be out-the-door very soon. This is a huge relief for me and it’s great that there’s one less thing to worry about.

To the point of Jeff’s post, I would absolutely not write a technical book again. It’s a tedious process and unless you LOVE writing and are really good at it (like Nicholas Zakas or Dave Flanagan) then I suggest that you stick with the medium that is truly successful: Writing long-form articles/blog posts and possibly spinning them off into purchasable ebooks. (As an example, I’d point to Juriy Zaytsev and Peter-Paul Koch both of whom could get any JavaScript position in the world purely based upon the quality of their articles and sites, without ever having written a book.)

I realized at some point in late 2008 that that’s really what I should’ve done with Secrets of the JavaScript Ninja but I was already neck-deep in the book with most of it complete. Ironically working on the book (or not working on it, however you look at it) actually compelled me to NOT blog more as every time I wanted to write a technical blog post I was forcing myself to make the decision “I’m writing about 1000 words on a technical matter, shouldn’t this just be going towards my uncompleted book?” and would just end up writing nothing as a result.

I do feel bad for, and apologize to, the people that pre-ordered hard copy version of my book over the years only to have to wait on a time scale comparable to George R. R. Martin’s writing schedule. Amusingly the book has been extremely successful as a pre-order e-book at Manning.com – it’s the best-selling MEAP book of all time. I’ve gotten numerous emails from readers who’ve gotten enormous benefit from the book, even in it’s rough form, and this has pleased me greatly.

I’m excited to finally have this book be out in the “wild”, as it were. It’s no longer (as Jeff put it) “permanently unfinished”.

I’ve included the preface from my book below, to give some more detail:

When I started writing Secrets of the JavaScript Ninja four years ago, in early 2008, I saw a real need: There were no books providing in-depth coverage of the most important parts of the JavaScript language (functions, closures, and prototypes) nor were there any books that even attempted to cover writing truly cross-browser code. Unfortunately it appears as if the situation has not improved very well, which is truly surprising.

More and more development energy is being put into new technologies (such as the ones coming out of HTML 5 or the new versions of ECMAScript). There isn’t a point to diving into new technologies, or using the hottest libraries, if you don’t have a proper understanding of the fundamental characteristics of the JavaScript language. While the future for browser development is very bright the reality is that most development needs to make sure that code continues to work in the majority of browsers and for the majority of potential users.

Even though this book has been under development for a long time, thankfully it is not out of date. The book has been given a solid set of revisions by my co-author Bear Bibeault. He’s gone through and made sure that the material will continue to be relevant for a long time to come.

A major reason why this book has taken so long is largely due to the experience that I was drawing upon to write the material for the later chapters on cross-browser code. Much of my understanding of how cross-browser development happens in-the-wild has come from my work on the jQuery JavaScript library. As I was writing the later chapters on cross-browser development I realized that so much of jQuery’s core could be written differently, optimized, and made capable of handling a wider range of browsers.

Perhaps the largest change that came to jQuery, as a result of writing this book, was a complete overhaul from using browser-specific sniffing to using feature detection at the core of the library. This change made it so that jQuery could be used almost indefinitely, without assuming that browsers would always have specific bugs or be missing specific features.

As a result of these changes jQuery directly anticipated many of the improvements to browsers that have come during the past couple years: Google released the Chrome browser, the number of useragents have exploded as mobile computing has increased in popularity, Mozilla, Google, and Apple have gotten into a browser performance war, and Microsoft has finally started making substantial improvements to Internet Explorer. It can no longer be assumed that a single rendering engine (such as WebKit or Trident, in Internet Explorer) will always behave the same way. Substantial changes are occurring rapidly and are spread out to an ever-increasing number of users.

Using the techniques outlined in this book, jQuery’s cross-browser capabilities provide a fairly solid guarantee that code that you write with jQuery will work in a maximal number of browser environments. This guarantee has led to explosive growth in jQuery over the past four years, with it now being used in over 57% of the top 10,000 websites on the Internet, according to Builtwith.com.

The portions of this book that cover features that are relatively un-changing, such as code evaluation, with statements, and timers are continually being used in interesting ways. There are now a number of active programming languages that are built on top of, or compiled to, JavaScript: Such as CoffeeScript or Processing.js. These languages require complex language parsing, code evaluation, and scope manipulation in order to work effectively. While dynamic code evaluation has been maligned due to its complexity and potential for security issues, without it we would not have had the CoffeeScript programming language – which has gone on to influence the upcoming ECMAScript specification itself.

I’m personally making use of all of these features, even today, in my work at Khan Academy. Dynamic code evaluation in the browser is such a powerful feature: You can build in-browser programming environments and do crazy things like inject code into a live runtime. This can result in an extremely compelling way to learn computer programming and provide all sorts of capabilities that wouldn’t be possible in a traditional learning environment.

The future for browser development continues to be very strong and it’s largely due to the features encapsulated in JavaScript and in the browser APIs. Having a solid grasp of the most crucial parts of the JavaScript language, combined with a desire for writing code that’ll work in many browsers, will result in your code being elegant, fast, and ubiquitous.

I’ve been trying a number of the Open (and non-Open) Source tools (a great list of which is on Wikipedia here). Thus far none of the tools that I’ve found are of high-enough quality to warrant further pursuit. They either do simple color comparison, basic wavelet/outline comparison, or some form of hashing – none of which appears to work very well beyond basic images. Some of the best algorithms are either caught up in University research programs (generally unreleased) or are available as corporate search engines.

In an ideal world I’d like something with the quality of TinEye (I’d even be open to using TinEye’s commercial services but they haven’t gotten back to me as of yet – I suspect that they’re mostly interested in dealing with large corporate clients).

In short: Does anyone have a lead on a high quality image similarity search tool (using Content-Based Image Retrieval)? I’m open to Open Source, closed source, or even paid API service – as long as it works well.

Note: I’m looking to use this on a private collection of images, so a service like Google Image Search (or TinEye’s normal commercial API) are not suitable alternatives – they both search images on the open web.

We’re in a very unique position as we’re primarily aiming to teach students who’ve been through our previous math and science-centric curriculum. Because of this we can create some rather compelling exercises and projects that never would’ve been feasible otherwise.

The prospect of teaching the JavaScript language as a first language is actually really exciting. Teaching prototypal inheritance to experienced classical-inheritance-using developers is normally rather frustrating (and results in many libraries springing up attempting to replicate the classical style of inheritance in JavaScript, which is a whole realm of weirdness in-and-of itself). Teaching prototypal inheritance to someone who has never seen any form of inheritance before will decidedly be an easier task. The same goes for learning functional programming. JavaScript is a great language for experiencing functional programming and can be a major focus of our curriculum as a result.

As we’ve begun to look at the prospect of JavaScript-as-a-first-language a number of obvious warts stick out (as is obvious to anyone who has worked with JavaScript for any duration). To make sure that general warts don’t crop up we will be using some form of linting (either JSLint or JSHint or similar) in the code editor to give the users contextual information on what’s happening with their code and why they should be writing their code in a certain way.

We want to go beyond basic syntax tweaks though and find ways of using the language that’ll result in an easier learning experience. In particular there are two changes which will likely result in a much simpler on-ramp to learning.

Note: These particular recommendations really only make sense if you’re teaching JavaScript to someone who has never seen the language before and is really only programming with a set of specific, well-coded, libraries. Obviously more will have to be taught in order bring the students up to the level of “see any random piece of cross-browser JavaScript code and understand what it does.”

Type Coercion

Type coercion is just a complete mess, as many many others have pointed out and as what Douglas Crockford typically teaches, as in JavaScript: The Good Parts.

It might make sense to discuss it far later in the education cycle… like after learning about prototypes, functional programming, and closures. Basically after everything that’s actually important.

The first change that I’m recommending is that the students will only ever see, and use, === (and !==). While using ‘==‘ does have the advantage of being syntactically shorter there is so much type coercion baggage attached to it as to make it an exercise in futility to try and teach early on in the learning of programming.

The one exception that might be worthwhile to teach later on is the case in which you wish to see if a variable contains a null or undefined value. This can be done easily with a simple someVar == null check and is likely the one useful case of ==. (Another noted exception is the browser bug in IE where === checks against Window objects will always return false, but it’s unlikely that we’ll cover such specific browser issues in our curriculum.)

Falsy Values

For the same reasons that == can be messy, so can falsy values. Enforcing strict boolean checks would result in less edge cases but would certainly result in longer code. Perhaps education of falsy values can be limited to booleans, null, and undefined with number and string falsy values left for a later exercise.

Function Declarations

Perhaps the most interesting change that we can make is a rather subtle one, but it’s eschewing normal function declarations for creating anonymous functions and assigning them to a variable.

There are a number of good habits that are instilled when you use this particular technique.

• Makes it easier to understand “functions as an object”. I’ve found that when you show new developers a function being assigned to a variable it suddenly becomes much more obvious that a function is actually an object and can be manipulated as such (and that a function can be passed as an argument to another function). Thus students are advanced along the path towards a better understanding of functional programming.
• It enforces good semicolon habits. Traditional function declaration is the only situation in which semicolons aren’t needed (save for conditional statements and loops, naturally) and it makes it much more obvious when they’re required all the time.
• Doesn’t have much of the baggage traditionally associated with functions and scope.

Block Scope

This is the remaining area that’ll certainly be a challenge for any introductory student to understand and yet I don’t see a particularly good solution here. The issue of variables declared within for loops hoisting up is more than enough to make most developers heads spin. I’ll have to see if we can’t come up with some intuitive ways of explaining how variable declaration works (and combining it with vigilant lint enforcement) rather than having a purely technical solution.

(While (function(){ ... })(); is a solution I’m skeptical that we’ll be able to teach that early-enough on as to make it worthwhile.)

JavaScript as a First Language

It should be noted that while we’re starting with JavaScript as a first language – largely due to its ubiquity, desirability in the larger workforce, lack of prior installation requirements, and ability to create something that’s easy to share with friends – we’re not going to be myopic and only focus on JavaScript. There are so many things that can be learned from other languages, not to mention entire skillsets that aren’t terribly relevant to in-browser JavaScript, that it behooves us to try and bring as much of them into our curriculum as possible.

I talk a little bit more about our choice to use JavaScript and some of the browsers that we’ll want to support in our development in the following video:

By all accounts I want to try and avoid any features that would cause cross-browser weirdness to spring up. As a result we’ll be making extensive use of libraries (for drawing to a canvas or manipulating the DOM) and using only JavaScript language features that work consistently in the browsers that we end up supporting.

Today we’re pushing live a complete rewrite of the Khan Academy Exercise framework (live demo).

A big push at Khan Academy has been to write more-and-more exercises for students to practice with. Naturally, to increase the number of exercises that we have, we needed to make it easier for team members, and casual committers, to write them.

When I was first looking in to Khan Academy I did a bunch of digging through their code base (which is Open Source) and noticed that a lot of effort was going in to writing exercises. Interestingly all the exercises were being written in pure JavaScript and were dependent upon the server component in order to run properly. To me this seemed like a massive barrier to entry.

Brand New Framework

When I started just over two months ago I immediately went about rewriting the framework to use semantic HTML and not rely upon any server-side components whatsoever. To provide an example of the change that was made here is one particular exercise, Significant Figures 1, written for the old framework and written for the new framework. You can view the live version of the exercise, as well.

Significant Figures 1 went from 191 lines of code down to 58.

Note the dramatic lack of “code” – most logic is encapsulated and reduced to a series of variable declarations (some with minimal logic). Additionally utility functions were made significantly more useful, and generic, shared across exercises (some of this was done in the old framework but a complete rewrite allowed us to find areas of obvious overlap).

The result of the new framework is an interesting mix of semantic markup and logic – wrapped in a form of highly specialized templating. Tons of details regarding the specifics of the framework can be found in the Writing Exercises tutorial.

In short, some of the cool new features of this framework include:

• A full module system for dynamically loading specific utilities (such as graphing).
• A brand new templating framework, including templating inheritance, that makes it easy to reduce repeated code in exercises (some examples).
• A brand new graphing framework (written by Ben Alpert, named ‘Graphie’) that’s built on top of Raphael. This framework provides a simple API for creating all sorts of graphs and charts, it’s used extensively throughout the exercises.
• An extensible plugin system for creating new answer types (to expand beyond basic multiple choice or fill in the blank).

Of course, that’s saying nothing of the actual user interaction as well. Most of the new framework’s logic is contained on the client-side and is generated directly off of Khan Academy’s REST API.

By moving logic to the client-side we’re able to have a much more interactive experience (doing all problem generation on the client-side, doing Ajax-y background submissions, things of that sort).

Testing

A big concern for us, with this rewrite, was to ensure that the quality and integrity of exercises was maintained well into the future. We wanted to make sure that the content of the exercises were correct and that regressions do not occur over time.

In fact we have three levels of testing:

1. We have unit tests for our modules and utility functions (ensures that stuff like the templating will continue to work correctly).
2. We have an interactive interface for generating unit tests for any given exercise (details below).
3. We have a “Report a Problem” interface for sending post-deployment issues directly to our bug tracker.

The second stage is particularly interesting. On a fundamental level it’s hard to unit test something as complex as an exercise. While we can ensure that they still work at a functional level we can’t verify if the problems make sense, or if graphs look correct, or if it’s even possible to pick the right answer.

For these reasons we’ve introduced a new testing interface in which we use human developers as unit testers. When in test mode an additional panel is displayed asking the user to provide additional details regarding wether the problem is working correctly, or not. We ask them to really dig into the exercises and make sure that they’re working correctly, hints are being displayed properly, and other things of that nature.

All of this information is collected and distilled into a piece of JavaScript code that is dropped into our test suite, all pushed into QUnit. At the moment we have around 2414 unit test collections with over 10,000 assertions.

At the moment we’re working on ways of having better reporting and integration with our development process, for all those unit tests.

Development Process

I’m rather pleased with how the overall development process worked out. We actively kept both the code open (it’s up on Github and it’s MIT licensed) and the process open as well. We’ve had contributions from over twice as many outside contributors than members of the Khan Academy team (although the vast majority of the development was done by members of the team).

We have a detailed Getting Involved guide that explains how exactly anyone can participate in the development of the framework (or in the authoring of new exercises).

Predominantly we’ve been coordinating through a public HipChat room, using the Github Issue Tracker, and managing all the exercise work through a series of Google Spreadsheets.

We’re going to be transitioning now into creating a ton more exercises and we’d love to have even more help from the community. There is a ton of information on how to get involved in the wiki.

Next Steps and Thanks

There are a few things that we’re looking to do in the near future: Bring offline support to the exercises (for those with spotty internet connections or those using the upcoming iPad/Android application), client-side translation of content into other languages, and exercise answer checking using Node.js and JSDOM.

I also want to take this opportunity to thank everyone at Khan Academy, and all the community contributors, for helping to make this project happen. I especially want to thank Marcia Lee, Ben Alpert, Jeff Ruberg, Igor Terzic, Omar Rizwan, and Ben Kamens for their help in getting this out the door.

I should also mention that Khan Academy is hiring devs. We’re looking for both server/client devs (Python/JavaScript) and also mobile devs (iOS and Android).

This ends up being a double-edged sword. While it’s great to provide a ton of problem variety to the students it’s fundamentally tricky as we want to have a manageable sample size of problems so that we can analyze student behavior and responses. For this reason we want to reduce the pool of possible problems to a more-manageable size (like about 200 or so – we’ll experiment with exact figures as time goes by). Most students will only do about 10-20 problems before moving on (since we define a ‘streak’ as having completed 10 problems in a row correctly) – although we want to provide enough of a pool to allow adventurous users to explore more.

Having a smaller pool size creates another issue though: Potential for student overlap. If there’s one thing that we’ve learned so far it’s that students are quite resourceful and identify patterns very quickly. They will realize if they start on the same problem together and if the problems go in the same order.

On top of this we need to make it so that every time the student hits the exercise they are presented with the same exact series of exercises. The first problem will always be the same – and the 50th problem will always be the same for that user. (This will allow us to reproduce the problems at a later date, showing them their problems/answers or analyzing the results.)

Thus we need two pieces of information to determine which problem should be presented to a user: The user’s ID and the problem # that they’re currently tackling.

We start by placing the user into a simple “bin”: We take the CRC32 of the student’s ID and mod 200 to start the user at one of the 200 possible exercises. (We picked CRC32 as it’s a simple function and we only need a basic level of granularity in the results.) Next we use the CRC32 to figure out how the user should jump through the exercises.

We can’t jump through the exercises 1 by 1, since the students will instantly recognize that pattern, so we have to take a random approach – one developed by Khan Academy intern Ben Alpert.

We start with a pool of prime numbers (the first 23 primes, to be exact) and use the user’s CRC32 to determine which of those primes will become the “jump distance” for that user.

For example:

Thus when I visit a particular problem I will start at problem # 158, then go to problem 29, then problem 100, then problem 171, etc. On the other hand, if user “jeresig2″ visits the same problem they start at problem 171 then visit problem 182, 193, 4, etc.

(Note that we leave out 2 and 5 to avoid creating a circular loop of exercises – 2 will result in 100 repeating exercises, 5 will result in 40 repeating exercises. Of course we could just make the number of exercises prime to achieve a similar result – we’ll see!)

Even though we’re not using any built-in random number generator in this process, the distribution of chosen problems appears to be similar to a random distribution (since we’ve effectively implemented one ourselves):

To re-emphasize: We’re not trying to stop problem overlap – in fact we want overlap to occur, to give us data to study – but a lack of repeatable patterns has definitely been arrived at here.

We chose a collection of 23 primes because it is sufficiently large – but it also doesn’t divide into 200 cleanly (giving users that might have an identical starting position a different jump amount).

Also note that in the final solution we’re going to be combining the user’s ID with the name of the current problem – so that even if, somehow, two students end up with identical start positions and jump rates on one exercise they will have a different start position and jump rate on other exercises.

If all goes well we’ll be opening up the new exercise rewrite for beta testing within the next week or two.

Note: I’ve been posting interesting dev updates over on Google Plus. Feel free to follow me there if you wish to find out more about what I’m working on with Khan Academy or jQuery.

A relatively controversial change in 1.6 was regarding how attributes and DOM object properties were handled. In 1.6 we wanted to take the major step of completely separating the two, allowing us to create an .attr() method that wasn’t quite so mealy-mouthed with regards to how attributes were handled. We did this in 1.6 because we felt that it was a substantial change (we only do major changes in the 1.x major releases of jQuery) and had the possibility of affecting people.

We did a considerable amount of testing on the code and we were quite confident that the amount of problems that people would encounter, while upgrading, would be quite minimal. The biggest pain points, we surmised, would be regarding how boolean attributes were handled (attributes like “disabled” or “selected”). However most of this would be mitigated and would likely have worked fine for users that consistently used .attr() to access and update their attributes.

After making the changes, and publishing 1.6, there were enough complaints that we had changed the API to cause us to reconsider and return .attr() to its sometimes-attribute, sometimes-property, state.

jQuery is in an incredibly tricky position now (and has been for some time). We very rarely add features to the library, for fear of bloat and added API maintenance overhead, and are rarely able to make any sort of API change, for fear of preventing people from upgrading.

Thankfully even though we’ve reverted some of the changes in 1.6, we’ve done it in a way that still maintains the performance gains that we achieved with the 1.6 release.

I wanted to try and summarize the .attr() method, to explain how it currently works in 1.6.1, but ended up writing up a sample function instead (note that this is a bit of an over-simplification, please read the code for more details):

Ironically this isn’t much shorter than the actual .attr() implementation, I recommend that you check it out.

There’s a very good chance that your code, written targeting 1.5.2, will continue to work just fine in 1.6.1 using this particular technique.

However this point now begs the question: Why does .prop() exist?

In short, for two reasons:

1. There are legitimate use cases for interacting with some DOM properties (such as nodeName, selectedIndex, or defaultValue) and we want to provide a simple solution for accessing, and mutating, them.
2. Accessing properties through the .attr() method will be slightly slower than accessing them directly through .prop() (as .attr() calls .prop() internally in order to handle all property-related mutation).

In jQuery 1.5.2, and older, in order to access a DOM property you would have to do something like this:

In 1.6+ you can just do:

Summary: There’s a good chance that your code won’t be affected at all with the changes that’ve happened, especially so with the changes in 1.6.1, and jQuery now has a convenience method for handling DOM properties in a simpler, and slightly faster, manner.

Tangentially this reminds me of a common question that I hear: What will be in jQuery 2.0? I have no idea what will be in that release, should it ever arrive, but I do know what won’t be in it: A massive API change of any sort. Even when we make, relatively minor, API changes like in the 1.6 release the amount of negative feedback that we get is monumental. If we’ve learned anything after doing 31 releases of jQuery it’s that people like having stability in their API and will cherish that over everything else.

I do want to thank the community though for being so vocal and working to communicate with the team so actively. Without the community’s communication and support it’s doubtful that the team would be able to continue operating.

I would like to take this opportunity to encourage everyone to get involved with the development of jQuery. We hold active discussions every day in IRC and hold public meetings once a week. We also post weekly status updates if you wish to follow along. Right now we’re working on the 1.7 release of the library and are actively encouraging contributions and feedback. If you want to help ensure the quality and stability of the next release of jQuery, the best way to do so is to get involved. Hope to see you around the bug tracker.