pmuellr

AMD - the good parts

2012-01-19T13:58:00.000-05:00

Based my blog post yesterday "More on why AMD is Not the Answer", you might guess I'm an AMD-hater. I strive to not be negative, but sometimes I fail. Mea culpa.

I do have a number of issues with AMD. But I think it's also fair to point out "the good parts".

my background

I've only really played with a few AMD libraries/runtimes. I've been more fiddling around with ways of taking node/CommonJS-style modules, and making them consumable by AMD runtimes. The AMD runtimes I've played with the most are almond and (my own) modjewel. My focus has been on just the raw require() and define() functions.

But I follow a lot of the AMD chatter around the web, and have done at least reading and perusing through other AMD implementations. To find implementations of AMD, look at this list and search for "async". The ones I hear the most about are require.js, Dojo, PINF, and curl.

And now, "the good parts".

the code

Browsing the source repositories and web sites, you'll note that folks have spent a lot of time on the libraries - the code itself, documentation, test suites, support, etc. Great jobs, all. I know a lot of these libraries have been battle-tested by folks in positions to do pretty good battle-testing with them.

the evangelism

Special mention here to James Burke (not this one), who has done a spectacular job leading and evangelizing the AMD cause, both within the "JavaScript Module" community, and outside. In particular, he's been trying to get AMD support into popular libraries like jQuery, backbone, underscore and node.js. Sometimes with success, sometimes not. James has been cool and collected when dealing with jerks like me, pestering him with complaints, questions, requests and bug reports. Some open source communities can be cold and bitter places, but the AMD community is not one of those, and I'm sure that's in part due to James.

Thanks for all your hard work James!

the async support

AMD is an acronym for Asynchronous Module Definition. That first word, Asynchronous, is key. For the most part, if you're using JavaScript, you're living in a world of single-threaded processing, where long-running functions are implemented as asynchronous calls that take callback functions as arguments. It makes sense, at least in some cases, to extend this to the function of loading JavaScript code.

If you need to load JavaScript in an async fashion, AMD is designed for you, and uses existing async function patterns that JavaScript programmers should already be familiar with.

support for CommonJS

This is a touchy subject, because when you ask 10 people what "CommonJS" is, you'll get 15 different answers. For the purposes here, and really whenever I use the phrase "CommonJS", I'm technically referring to one of the Modules/1.x specifications, and more specifically, the general definitions of the require() function and the exports and module variables used in the body of a module. These general definitions are also applicable, with some differences, in node.js.

When I use the phrase "CommonJS", I am NEVER referring to the other specifications in the CommonJS specs stable (linked to above).

AMD supports this general definition of "CommonJS", in my book. Super double plus one good!

Aside: I'm looking for another 'name' to call my simplistic notion of CommonJS. So far, things like simple require()y support, seem to capture my intent. But you know how bad I am with naming.

the design for quick edit-debug cycles

One of the goals of AMD is to facilitate a quick edit-debug cycle while you are working on your JavaScript. To do this, AMD specifies that you wrap the body of your module in a specific type of function wrapper when you author it. Because of the way it's designed, these JavaScript files can be loaded directly with a <script src=> element in your HTML file, or via <script src=> elements injected into your live web page dynamically, by a library. This means that as soon as you save the JavaScript you're editing in your text editor/IDE, you can refresh the page in your browser without an intervening "build step" or special server processing - the same files you are editing are loaded directly in the browser.

the resources

Beyond just loading JavaScript code, AMD supports the notion of loading other things. The things you can load are determined by what Loader Plugins you have available. Examples include:

Coming from the Java world, this is familiar territory; see: Class.getResource*() methods. For the most part, I consider this to be amongst the core bits of functionality that a programming library should provide. It's a common pattern to ship data with your code, and you need some way to access it. And, of course, this resource loading is available in an async manner with AMD.

Aside: I'm not sure what the analog of this is in the node.js world. You can certainly do it by hand, making use of the __dirname global, and then performing i/o yourself. I'm just not sure if someone has wrapped this up nicely in a library yet.

the build

At the end of the day, hopefully, your web application is going to be available to some audience, who will be using it for realz. It's unlikely that you will want to continue to load your JavaScript code (and resources) as individual pieces as you do in development; instead, you'll want to concatenate and minimize your wads of goo to keep load time to a minimum for your customers.

And it seems like most AMD implementations provide a mechanism to do this.

More on why AMD is Not the Answer

2012-01-18T10:38:00.000-05:00

Follow-on to Tom Dale's blog post "AMD is Not the Answer" and James Burke's response "Reply to Tom on AMD". I pretty much concur with everything Tom said. Here are some points brought up by James:

need an alternative

James says:

what is missing is a solid alternative to AMD. In particular, "use build tools to wrap up CommonJS modules" is not a generic solution. It does not allow for generic dynamic loading of resources, particularly from CDNs. Dynamic loading is needed for big sites that still do builds but want to stage loading of the site as the user uses it.

I don't think using build tools precludes you from building wads of modules that can be loaded dynamically. Generic solutions bother me. See: J2EE.

ceremony

James says:

On his concern about too much ceremony -- this is the minimum amount of ceremony:

    define(function(require) {
        //Module code in here
    });

On the other hand, here's some code pulled from dijit.Calendar.js:

    define([
        "dojo/_base/array", // array.map
        "dojo/date",
        "dojo/date/locale",
        "dojo/_base/declare", // declare
        "dojo/dom-attr", // domAttr.get
        "dojo/dom-class", // domClass.add domClass.contains domClass.remove domClass.toggle
        "dojo/_base/event", // event.stop
        "dojo/_base/kernel", // kernel.deprecated
        "dojo/keys", // keys
        "dojo/_base/lang", // lang.hitch
        "dojo/sniff", // has("ie")
        "./CalendarLite",
        "./_Widget",
        "./_CssStateMixin",
        "./_TemplatedMixin",
        "./form/DropDownButton"
    ], function(array, date, local, declare, domAttr, domClass, event, kernel, keys, lang, has,
                CalendarLite, _Widget, _CssStateMixin, _TemplatedMixin, DropDownButton){
        //Module code in here
    });

This isn't just ceremony. It's ugly ceremony of the sort that's difficult to debug when you screw up.

module consumption

James says:

the alternative he mentions does not handle the breadth of JS module consumption

The simple require()y story of npm modules seems to handle the JS module consumption pretty well, for node.js. For the browser, you can use npm modules with Browserify or other "build tools". Where is the equivalent of npm for AMD? Or maybe npm already contains AMD modules? Or let's say I want to just use a single module out of Dojo in my app. How do I do that?

making Web SQL easier to use

2012-01-12T20:12:00.000-05:00

One HTML5-ish area I've not yet played much with, but am starting to now, is Web SQL. Web SQL is the W3 standard for "SQL in the browser", which has been orphaned by the W3C. Here are some warnings from the Web SQL Database spec at w3.org:

"Beware. This specification is no longer in active maintenance and the Web Applications Working Group does not intend to maintain it further."
"This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path."

Despite the fact that the W3C no longer endorses Web SQL and will never 'standardize' it, implementations of Web SQL are shipping in browsers, including some mobile platforms. It's not dead yet.

what's the api like?

Here's an example right out of the spec, to count the number of rows in a table:

A function showDocCount()that calls a function db.readTransaction() that takes two anonymous functions as callbacks. One of those callbacks calls t.executeSql() passing it yet another callback, which finally, mercifully, gets the row count.

It's executing a single SQL statement.

But wait, it gets better.

If you need to run multiple SQL statements within a single transaction, you'll have to chain those executeSql() invocations within increasingly nested callbacks.

It's basically a horror show. At least as far as I'm concerned. Nested callbacks are things I hate to have to write, and things I really hate seeing other people have to write. The result is almost always unintelligible.

can we make this easier to understand?

For weinre, I implemented a stepper module used by the inspector's database interface. Not very clear what's going on there though, sorry.

The basic idea is that you provide an array of functions - the steps - that the stepper will arrange to run in order, by managing the callbacks itself. Double bonus good - you get to linearize your database transaction steps and you don't have to deal with callbacks!

I found another instance of this kind of processing in Caolan McMahon's async library - specifically the waterfall() function. Not really appropriate for Web SQL, but at least it seems to validate the approach.

I decided to implement a clean implementation of the stepper, trying to make it easy to use and understand. The result is available as WebSQLStepper at GitHub. As a use case, I took the Todos sample from Backbone, and modified it to use this library to persist the data to a Web SQL database. The complete sample is included in the GitHub repo, and here are the steps used to read all the rows of a database:

references

WebSQLStepper at GitHub
Web SQL Database spec at W3
SQLite documentation - SQLlite is used in the Web SQL implementation for WebKit-based browsers

wr - an auto-build command for the command-line

2012-01-12T13:20:00.000-05:00

For much of my programming career, I've been a user of "IDE"s, including BASIC, Turbo Pascal, various Smalltalks, IBM's VisualAge products and Eclipse.

Lately, I've been spending most of my time working without any IDEs. My tools are text editors, terminal windows, a boat load of unix commands built-in to my OS, some additional command-line tools from various open source locales, and web browsers. I think of this loosely organized/integrated set of tools as my IDE, they're just not packaged up in a nice single app.

But there's one thing I dearly love and really depend on in most IDE's that's missing from this command-line-based environment: an "auto-build" capability. Specifically, when I save a file I'm editing, I want a "build" to happen automagically. Without an "auto-build" capability, I'm left in a world where I save a change in my text editor, jump over to the terminal, run make, then jump to my browser to test the build. The jump to the terminal is not desired. Using some existing editor's capability of running shell commands doesn't work either, because I use a wide variety of tools to "edit" things - all the tools would need this capability.

(Quick aside about "builds". A lot of people see "build" and think "30 minutes of waiting for the compiles to be done". Yes, those builds are long and painful. I'm looking at you WebKit. But in all the IDEs I used to use, builds were insanely fast. Sub-second fast. If you're living the "big C++ project" lifestyle, I feel your pain, and "auto-build" is probably not applicable for you.

So, I'm talking about quick builds here. As an example, once you've done at least one build of weinre, subsequent builds take on the order of 6 seconds or so - I consider that slow, but it's doing a lot of work, and I could probably cut the time in half if I needed to - I just haven't felt the need to.)

a command to run something when a file changes

A few years ago, I wasn't aware of a tool that could watch for arbitrary file changes and run a command for me, so I wrote one - run-when-changed. This script served me well for a long time, but there were always little thingees I wanted to add to it. And since run-when-changed polled the file system to see when files changed, I was always interested in finding a better story than polling and the inevitable waiting for the polling cycle to hit - even waiting 3 seconds for run-when-changed to realize a file changed was bothersome.

A port to node.js seemed like it might be in order, especially since it has a non-polling file watch function fs.watch().

Earlier this week, I spent a few hours and now I have a replacement for run-when-changed, called wr, available at GitHub, written using node.js. The name wr comes from "watch and run".

Some features:

installable via npm
colorized stdout and stderr
reversed video status messages
when run with no arguments, will look for a .wr file in the current directory to use as arguments

Here's what it looks like, using it with one of my current projects:

Click the image for a slightly larger version.

Legend:

the reverse video blue lines are status messages
the reverse video green lines are command success messages
the reverse video red lines are command failure messages
the blue text is stdout
the red text is stderr

As another example, I'm generating this blog post by running the following command:

    wr --exec "markdown wr.md > wr.html" wr.md

Breaking this command down, we have:

wr - the command
--exec - a command-line option (see below in gotchas)
"markdown wr.md > wr.html" - the command to run
wr.md - the file to watch for changes.

The arguments to wr are options, the command to run, and the names of files or directories to watch. When the wr command is run, it waits for one of the specified files to change, runs the specified command, reports the results of running the command, and then starts waiting for a file to change again. Exit the program by pressing ^C.

The ability to store a .wr file in a project directory makes life even easier. I used to create a target in my make files to invoke run-which-changed ; now I just put the wr invocation for a project in the .wr file, and run wr with no arguments.

My work-flow for using wr goes something like this:

open a terminal window
cd into my project's home directory
launch an editor, returning control to the command-line
run wr
move the terminal window where I will always be able to see the last few lines of the window, to see the live status
switch between text editor and browser, for my edit-save-reload-test cycle, all day long, keeping an eye out for red stuff in the wr output
when the day is done, ^C out of wr, and then commit work to my SCM

This can all be yours, with a simple command-line incantation:

    sudo npm -g install wr

current gotchas

wr has two ways of invoking the command to run, determined by the command-line option --exec; see the README, in the ENVIRONMENT section, for more information on the differences.

The big gotcha though is the non-polling file watching. This is an awesome feature, as it means as soon as you save a file, wr will wake up and run a command. The problem, on the Mac anyway, is that there is a relatively small limit on the number of files that can be watched this way. Like, 200 or so. If you go over the limit, you'll see some error messages from wr, and you'll have to resort to using the --poll option, which uses a polling file watcher.

accessible backed up git repos for transient work-related projects

2011-12-16T12:00:00.003-05:00

preface: this is basically a "note to self", but thought others might find it useful

I tend to create a lot of "projects" at work. A "project" is a directory in my "~/Projects" directory. Who knows what might be in it; probably code, or maybe just documentation. Many times, it's a "project" that I think may be useful to other people, and I may eventually "publish" to an either a public SCM repo farm like GitHub or an internal IBM SCM repo farm. I currently have 317 projects in my "~/Projects" directory.

Before "publishing" it though, I'll usually want to crank on it for a bit, see if I can get the crap to work, and figure out if it's actually useful. So there's a window where my project isn't being "backed up" or available to anyone else, because it's sitting on my laptop's spinning disk of rust. That's not great.

In the past, I've used Subversion and/or Mercurial to help manage this "back up" process for personal, non-work-related projects. What I'd do is set up a repo on my shared server, and then just use it like any other remote SCM.

Problems:

What about work-related stuff? I don't want to publish something that's work-related on a non-IBM site.
I made it sound like "set up a repo on my shared server" was easy. It isn't. Depending on the capabilities of your server, it might be impossible.
Yeah, we do have various internal-only SCM repo farms in IBM, but those still require some "setup" and "maintenance", just a different flavor than shared hosting, and even that can be too much sometimes. Especially for a project which may only be useful for a couple of days.

Bummer.

a new hope

What I'm doing now is using Git, and storing my Git repos on IBM's internal, backed up, rsync-/smb-/http-accessible cloud file store. Even if you don't have rsync or smb access, there are likely ways you can do something similar, with different tools, on your own server.

Here's how it works:

mount the file store on my local machine, create a directory for the new repo in my "git-repos" directory, which is in a part of my cloud file store that is publically accessible via http, unmount the file store.
initialize the project on my local machine with git (eg, "git init")

write a "backup" script, which looks like this:

#!/bin/sh

#--------------------------
# make the git repo accessible via:
#   git clone http://{host-name}/~pmuellr/git-repos/{project-name}/.git
#--------------------------

cd `dirname $0`

git update-server-info
rsync -av . {host-name}:{path-to}/git-repos/{project-name}

Instead of using a "git add/commit/push" workflow, I use a "git add/commit; ./backup" workflow.
As the comment in the script notes, the git repo is accessible via http, to anyone who has access to that URL. For the general case for me, that's anyone in IBM.
The "backup" script, as written, also backs up my working directory. Which is a win as far as I'm concerned. Now I can also point folks to web-accessible/-renderable resources in my working directory.

I've found this to be a very useful, very lightweight process to keep my immature projects backed up. I don't doubt there are ways to do similar things with SVN or Mercurial or whatever as well, I just never tried.

I can, but haven't yet, set up a similar story for personal projects, using my shared server, but that should be pretty easy; it supports rsync, but not smb. Instead of "mount and create project directory", I can "ssh and create project directory". Even that's probably not needed; rsync can likely create that project directory for me, but I'm an rsync n00b and I kind of like having to make that step explicit.

debugging concatenated JavaScript files

2011-11-16T10:31:00.000-05:00

In James Burke's blog post "Why not AMD?", the subject of debugging briefly came up. In response to the claim that AMD is "Inefficient by default", James responds: (bold emphasis applied by me)

I assume this means that it allows separate modules to be loaded async. FWIW, the current state of ES harmony modules will allow separate file loading for each module. This works better for debugging.
For AMD, you can get the "one script file at the bottom of the page" loading with the 750 byte almond AMD shim and runtime http loading.
I cannot see a better solution to this issue. If you have ideas I would like to hear them.

I'd like to point out some futures/options for making concatenated scripts easier to debug. None of solutions deal with mangled names, they're just ways of letting the debugger show a concatenated script as the set of individual scripts that make up the concatenated script. Baby steps!

Source Map support in the web debuggers

Werner Schuster's blog post "Mozilla, WebKit To Support Debugging Minified JS As Well As CoffeeScript and other JS Languages" covers most of the ground for the Source Map work that's ongoing. Chris Heilmann's recent presentation from Velocity "Firefox Performance" mentions some work going on at Mozilla.

Of course, we need both ends working here; minizer's are going to have to be able to spit out Source Maps, as well as language translators like CoffeeScript. The CoffeeScript bug for Source Map work is here (I think).

cheating with eval() and //@ sourceurl

Back in the day, a feature in Firebug that I copied into Web Inspector was "sourceURL" support, described in "Give your eval a name with //@ sourceURL". I recently made use of this in my "unminified" version of the injected target weinre script. weinre ships with two versions of the code that gets injected into the target: target-script.js and target-script-min.js. The "min" version is what you should be using for typical weinre debugging, the non-"min" version is what I use when debugging the debugger.

The file target-script-min.js contains gently minized JS files, just concatenated together. The target-script.js file is a bit different. For every JS file, it adds a "//@ sourceurl" comment to the bottom of the file, converts the script content to a string with JSON.stringify(), and then writes an eval([that string here]) to the concatenated file.

The difference between debugging these in a browser which supports sourceurl is night and day. Try it: after visiting these pages, open your web debugger and look at the scripts which are loaded. Note, this works on Google Chrome 16.0.912.36 beta on a Mac (and likely other versions).

what's new in weinre 1.6.0

2011-10-30T15:43:00.001-04:00

Notable changes in weinre 1.6, which was released last week. More weinre links here.

Element Highlighter Color Changes

The "element highlighter" is the coloring you get on your web page when you hover over DOM elements in weinre's Elements panel. Previously, the colors were very muddled.

Below are the new colors, with the orange-ish margin, turquoise-ish padding, and purple-ish body. Not shown is the border (this element has none), which is a bright yellow.

There are still some issues with this highlighter, which was about the 6th different one I had tried. As a result of all those attempts, the element highlighter is now fairly pluggable, and it should be straight-forward for you to create a better one. I dare ya.

JavaScript Callback Error Diagnostics

I often hear people talk about debugging their JavaScript with weinre, but weinre really just provides a REPL and console logging. New in 1.6 is an attempt to provide some additional JavaScript diagnostics.

For a set of built-in functions in JavaScript that have callbacks as parameters, weinre overrides those functions to run an instrumented version of the callback instead. The instrumented version just runs the specified callback within a try/catch statement and reports if an exception occurred. The functions that weinre overrides are:

window.setTimeout()
window.setInterval()
window.addEventListener()
Node.prototype.addEventListener()
XMLHttpRequest.prototype.addEventListener()

The Node.prototype.addEventListener() override should handle document and all DOM elements in your document. The XMLHttpRequest.prototype.addEventListener() will handle your XHR's. Unfortunately, I'm not quite sure how to override the on-family of event properties, because browser host objects suck.

As an example of what you'll see in the Console window when an exception occurs, here's some JavaScript code which will generate a runtime error when the specified button is clicked:

Below is what you see in the console when the button is clicked.

Description of the five error messages in the console:

This is the toString() of the exception that was caught when the callback was invoked.
The callsite is an informational description of the call which registered the callback. In this case, it tells us the callback was registed to an <input> element's click handler, and that the callback registered was a function named buttonClicked().
If the caught exception has a stack property, the stack will be dumped following this line. Otherwise, you won't see this line or the stack.
This is the stack at the time the runtime exception occurred, if available. Note that the last two lines (at <error: TypeError: ...) seem to be bugs of some kind in the browser (couldn't be in my code!).
This is actually a line from the new onerror handler that weinre registers, unrelated to the new callback exception logging described here.

Not all browser's JavaScript engines support the error.stack property (eg, WebKit's JavaScriptCore), so this is what you'll see if you're running on one of those lame engines:

Also note the third line is, again, from the new onerror handler, which seems to like to not contain useful information. Why not? See here, here, here, and here.

The New Resources Panel

The Resources panel is brand new for version 1.6. It provides diagnostics for all the XHRs that you code issues. Below is an example of a web page that has issued five XHR's.

Clicking on one of those rows will display additional information in the right hand side of the window, as shown below.

There are two tabs here, one to show "header" information, the other to show the "content". The content for the XHR is shown below.

To dismiss the detail view and get back to the table view, click the circled X button to the left of the Headers/Content tabs.

weinre 1.5.0 is now available

2011-07-22T11:46:00.001-04:00

weinre 1.5.0 is now available at https://github.com/phonegap/weinre/downloads.

ChangeLog: http://phonegap.github.com/weinre/ChangeLog.html

I also went through and neutered the bits at the old "pmuellr/weinre" site. Everything should now, somehow, point you to the "phonegap/weinre" site/repo. Let me know if you see dangling pointers. (note, I'm not going to change links in my old blog posts)

public weinre server hosted on the intertubes

2011-05-26T12:11:00.001-04:00

Although I knew it was coming, the twitter post from Andrew Lunny yesterday made me very happy:

hosted weinre for PhoneGap Build is now live at http://debug.phonegap.com

Awesome! The fine folks at Nitobi are running this server, not me. Thank you so much! I owe everyone involved a cool beverage of your choice, next time we meet (OSCON?).

Of course, if you have any questions about using weinre, including this new public debug server, don't hesitate to ask:

Google Groups: https://groups.google.com/group/weinre
IRC: #phonegap @ irc.freenode.net
GitHub issues - https://github.com/pmuellr/weinre/issues, for bug reports, feature requests, etc

some additional things to do for multi-user weinre servers

There's always more work to do, isn't there?

make the front page easier to use and understand, if required
optimize client and target resources - ok, this is kinda embarrassing, but the weinre server is not making use of some of the well-known best practices to optimize HTTP resource access
provide some hints / tips / best practices for folks wanting to run their own multi-user weinre server

In case you're wondering why someone would want to run another multi-user weinre server, now that we have one "in the cloud":

um, you're going to depend on the cloud?
there's not much security around the current weinre client/server interaction patterns, and there is at least the problem that the current debug packets are flowing across the internet in plain text. That includes, at a minimum, any serialized versions of DOM nodes that you are seeing in the DOM elements panel.

Given that, seems like it would make sense for organizations to run their own in-house weinre servers.

I'm planning on setting up a weinre server inside the IBM firewall for IBMers to use. I'll post news on that subject to the new weinre "Community" I set up on the IBM internal "Connections" page. To find that community, search for weinre, mobile, debug, etc.

callback debugging - better mobile error diagnostics

2011-05-20T07:23:00.000-04:00

One of the common issues that comes up regarding debugging mobile web apps, is that the error diagnostics for JavaScript are terrible. Sometimes non-existent. Current state of the art is to turn on the "Debug Console" switch in Mobile Safari, which is kinda horrifying, but does provide some information. Or use low-level debug tools like Android's logcat. These tools provide additional clues about JavaScript errors that have occurred.

But I've wanted to add something to weinre, and a while back opened issue 38 - "provide better error handling - by hook or by crook". My first attempt was to try overriding some addEventListener() functions, which would instrument the callback methods passed to them. The instrumentation was to wrap the invocation of the original callback in a try/catch, and if something was caught, report it. The reason this is a good avenue to go down is that after the top-level JavaScript code in your HTML and JavaScript files runs, everything else is run because of a callback somewhere. If you can control the callbacks, you control JavaScript.

Aside, there's a number of interesting links in that bug report:

v8's stack trace API, callable from JavaScript
what happens when you override Function.prototype.call() in JavaScriptCore (Holy Shit Really?!)

I decided to spend some time on the error diagnostics issue this week, but first did a cursory Google search, and found an article by Tom Robinson on the subject. Basically describes the approach I was heading down, so things still sounded promising.

Since I didn't see any kind of existing library to do this kinda stuff, I wrote one - CBWrapper. Putting it in a separate GitHub repository for now, because it's kind of a general purpose callback diagnostic tool. Could be useful in Node, perhaps. If it turns out to be useful and safe for weinre, I'll add it, but for now you can use it as a separate set of script files.

Here's what it can do. Keep in mind that the normal amount of JavaScript error handling you get in weinre is - nothing. Add the CBWrapper scripts as indicated on it's page, and you'll instead see the following output in the weinre console. Note that the information displayed is different depending on your web browser's JavaScript engine. I only tested under Mobile Safari and Android's Browser.

under JSC - all Apple devices and older Android devices

exception: TypeError: Result of expression 'xScript' [null] is not an object.
location:  https://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:8
callback:  window.setTimeout(aScript(), 2000)

under V8 - newer Android devices

exception: TypeError: Cannot call method 'doSomething' of null
callback:  window.setTimeout(aScript(), 2000)
           [http://[[url-elided]]/CBWrapper/tests/test-weinre-runtime-function-script.html:10]
stack:
TypeError: Cannot call method 'doSomething' of null
    at bScript (http://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:8:11)
    at aScript (http://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:3:5)
    at cbWrapper (http://[[url-elided]]/CBWrapper/CBWrapper.js:253:27)
    at http://[[url-elided]]/target/target-script-min.js#test:3674:14

Curiously, the exception messages are wildly different here. The line that threw the exception was:

    xScript.doSomething() // where xScript == null

Note that besides the other information provided, there's a line labelled "callback", which gives you some information about the function that registered the callback that just threw the error. In this case, it was a window.setTimeout() call, and the message provides some contextual information on the call itself. The registration function is no longer on the stack, but I keep a reference to it in case you want it. Like in this case.

CBWrapper can do a lot more than log errors. It provides before/after callbacks before every callback that it wraps, so you can do things like get the elapsed time of all the callbacks, log callback events on the Web Inspector timeline, etc. In theory.

Practice, however, is a different story. Playing with it a bit, under weinre, led me to some issues. See, weinre itself uses some of the same callbacks that CBWrapper wraps, so it's pretty easy to fall into a trap where you do something in a CBWrapper callback (write to the console), which causes weinre code to set a timer or do an XHR, and you've just created an endless chain of console logs / CBWrapper callbacks that will never end. The callback-based async version of infinite recursion.

Finally, Kent Beck posted the following message on Twitter today:

the problem with callbacks triggering callbacks triggering callbacks is that when something goes wrong, reading the code doesn't help much

Seems like maybe you could whip something up with CBWrapper that could show you callbacks registered from other callbacks, and then show a trail of those callbacks when an error is thrown. Would that help? Only one way to find out and I'll leave that as an exercise for the reader.

weinre 1.4.0 available

2011-05-09T13:20:00.000-04:00

weinre 1.4.0 is now available in the usual place: https://github.com/pmuellr/weinre/downloads. Changelog is available here: http://pmuellr.github.com/weinre/ChangeLog.html.

Major changes for this release:

Google Groups mailing list
extensions
multi-user support

As always, for bugs or feature requests, create an an issue. Or use the new Google Group (see below).

Google Groups mailing list

I've created a Google Group for weinre, here: http://groups.google.com/group/weinre. Usual stuff applies, the group is set to moderate messages for "new" members.

extensions

Like all great platforms, Web Inspector has an extension mechanism allowing users to write code to extend the capabilities of the tool. The extension support for real Web Inspector (used in Safari/Chrome/etc) is not currently enabled, but I went ahead and enabled it in weinre.

I've written two extensions so far.

https://github.com/pmuellr/weinre-ext-dom-monster - runs DOM Monster on your device, displaying the results in weinre
https://github.com/pmuellr/weinre-ext-sample - an example extension you can use to base a new extension on.

Here's a screen-shot of DOM Monster running in weinre - click the image to see a larger version. Isn't that panel icon so cute!

The extension support is best-effort at present. Definitely open an issue for bugs or feature requests. If it's API related, I may forward them to the WebKit bugzilla component for Web Inspector.

I presume in real Web Inspector, all the various .html files that are loaded will be sandboxed, like Chrome and Safari extensions, but for weinre they're running as iframes of the main inspector window. As such, they have full access to everything in Web Inspector. Powerful. Dangerous. Have fun.

I decided to use a deployment structure that is easily mapped to a VCS, like Git/GitHub. This makes it easy to install and update extensions. To install the DOM Monster extension, run the following shell commands:

mkdir ~/.weinre
mkdir ~/.weinre/extensions
cd ~/.weinre/extensions    
git clone git@github.com:pmuellr/weinre-ext-dom-monster.git

To update the extension, after I make and publish changes, just run:

cd ~/.weinre/extensions/weinre-ext-dom-monster
git pull

multi-user support

This support will only be useful to folks who want to host a weinre server for multiple people to use simultaneously. For more information, see: http://pmuellr.github.com/weinre/MultiUser.html.

Although it's noted in the doc, I will mention here also. There is no real security here. Your client and target need to know a "shared secret" to connect to each other. On the bright side, there's never any information persisted on the weinre server, everything is transient, so if you aren't connected to a weinre server, there's nothing for anyone to steal.

There's likely more work to be done here, but looking for feedback.

weinre 1.3.0 available

2011-04-05T09:39:00.001-04:00

weinre 1.3.0 is now available in the usual place: https://github.com/pmuellr/weinre/downloads. Changelog is available here: http://pmuellr.github.com/weinre/ChangeLog.html.

Major changes for this release:

CSS style editing now available. I'm surprised more people haven't complained about this - I think this is one of my favorite features of big Web Inspector. If you're not familiar with it, there's some documentation on Chrome's DevTools documentation site about style editing.
The Resources panel has been enabled, but currently only the localStorage and sessionStorage entries contain values. I couldn't resist adding localStorage support after seeing Steve Souder's blog post "Storager case study: Bing, Google".

A few notes on the style editing:

Those little yellow warning icons to the left of the property names are error indicators; either the property name or value is incorrect. Unfortunately, for weinre, we can only catch such errors for properties you edit. We can't catch errors that are present in your CSS that the browser originally sees. The CSSOM doesn't provide access to error-ish stuff.
Adding new style rules doesn't work yet. Vote on this issue if you really need it; pretty low priority for me.
If the Matched CSS Rules section is empty for most of the elements you click on, it may be because of same origin restrictions for accessing CSS rules via the CSSOM. In other words, weinre may not be able to see the CSS rules for CSS files that you import from a different origin than the original document. Still investigating.

As always, for bugs or feature requests, create an an issue.

WTF are these scoop files?

2011-03-16T20:03:00.001-04:00

At some point, someone's going to look at the weinre source code and ask, "WTF are all these .scoop files?"

They're almost JavaScript files, intended to preprocessed into legal JavaScript files. The lines of the file which are not indented are "directives" that indicate how the following indented bits are to processed. The indented lines are plain old JavaScript which isn't touched by the preprocessor except to deal with "super" calls.
They're my response to wanting to use classical OOP in JavaScript, without dealing with the crufty the way it's typically done in JavaScript. I like JavaScript, but I hate cruft.
Since I'm also playing with CommonJS, I ended up folding some goodies for that in too.

Assuming you know JavaScript, and have some familiarity with classical OOP, I suspect you will get the gist of what's going on if you look at one of the weinre .scoop files.

I knew I'd be writing a lot of JavaScript for weinre, so I wrote scoop first to make it a pleasant experience.

The source for scoop is at GitHub. The doc is spartan, but my existing user base hasn't complained too much about that (existing user base == me).

Here's what I learned in the process of using scoop to build weinre:

I like it. It's become second nature to author scoop files.
The preprocessor runs fast enough that it doesn't impact my build in any significant fashion. In practice, all software I create has some kind of automated build that runs when I touch a source file. Including web sites. Adding one second to that process, if that, doesn't impact me. (weinre builds take 5 seconds, which includes compiling Java source - I could optimize that build more, but haven't needed to)
There's no new language to learn. It's just JavaScript with some structuring. Just a few directives to learn, and rules to follow (indent your code). No offense to the altJS crowd - I love all that stuff, but really don't want to invest in learning another language, figure out how to debug it, etc.
scoop and the CommonJS module runtime I use (modjewel) both take turns processing the source, but don't change the line or column numbers of the original source. When the JavaScript runtime complains about a syntax or runtime error with a line number, no mental mapping is required to find the line in the scoop source. The meat of code I step through in the debugger, function bodies, is the same as the scoop code I write.
Exposing classes as modules seems to work fine, but requires the module system to support overriding the exports value that the module ends up populating. In practice, this works with node.js and modjewel, so, works for me. Specifically, the first class defined in a scoop file is assigned to module.exports, and thus any code which does a require() on that module will get the class as a result.
The big downside of overriding module.exports is that recursively require()'d modules don't work right. I added a requireClass directive which will identify this condition at runtime, and fails fast with a decent diagnostic. Fixing these situations typically isn't difficult.
Syntax-coloring text editors generally do the right thing for scoop files, once you lie to them and tell them that a scoop file is just a JavaScript file. The directives are usually colored oddly, but that's not a big problem. The actual JavaScript code is always colored correctly, for the text editors I use.

Part of my rationalization of doing this is based on Smalltalk's class structuring story. The Smalltalk language doesn't define any class structuring features like a "class statement" - all that is handled by the tooling and runtime. And it has a great classical OOP story. If Smalltalk can do it, then so can JavaScript.

I have a long laundry list of things I'd like to do with scoop, especially since I consider it just a stepping stone to having a nice, dynamic, live development environment for JavaScript, just like Smalltalk. The current incarnation is the ascii, text-file version of where I want to go. The thing I want to work on first is some read-only browsers for the module system and the class structure, that you can run right in your browser (or debugger). Next would come live editing, etc, etc. Probably would have worked on some of that stuff had my JSConf 2011 proposal for scoop been accepted. The pressure is off on that front though.

mobile debugging updates

2011-03-03T10:51:00.001-05:00

Yesterday, mobile web developers got some refreshed debug tools to play with. I released a new version of weinre, and Remy Sharp released some YouTube movies showing off his JS Console tool.

The big new feature for weinre is the Timeline. The Chrome Dev Tools version of Timeline is documented here, and weinre implements some of it. A little of it. Timers, intervals, XHRs, some global events. It handles onerror, but I haven't seen onerror available on any mobile WebKit browsers, since it's a relatively new addition to WebKit.

Aside, if you're not familiar with WebKit Web Inspector / Chrome Dev Tools (same thing, different names), the Google Chrome Developer Tools site seems to be the best source of information.

With both jsconsole and weinre, there are limits to what you can do. Specifically, since they are both written in user-land JavaScript, and there are no user-land JavaScript APIs that deal with breakpoints, there's no way to do breakpoints (well, ...). Even just getting error information / stack traces is difficult-to-impossible.

What we really need is the real Web Inspector to work on mobile platforms. There's some code in WebKit to handle the remote-y-ness, as shown by this movie. Not shipping yet, though, AFAIK. Maybe on iOS 4.3 or already in Honeycomb but hidden? Would be a nice surprise for sure, but would make weinre - my baby - obsolete. That's fine with me, I've been developing software long enough I know that sometimes your creations need to die. There's plenty of other stuff to do in this arena.

What's next for weinre? You tell me. Or post a message on Twitter mentioning weinre, and I'll probably see it.

Here's the new movie for weinre:

Mobile, Debugging, Challenges, Hacks

2011-02-03T19:30:00.001-05:00

Alex Russell has a great post today on software performance, titled Performance Innumeracy & False Positives. Having worked in the embedded/mobile space for the last decade (on and off), I certainly feel his pain. On the other hand, this is one of the reasons why I love the mobile space: it's challenging.

Performance is just one of the challenges that we face in the mobile space. Other issues are program and data size, RAM usage, and network speed. All constrained compared to desktop-class machines.

For some of these challenges, we need tools, because it's often not obvious where the problems really are. Sometimes the only tools you have is your eyes staring at the code and intuition. Performance, in particular, is an area that needs tools. If there's one thing I've learned about performance problems over the years, it's this: your intuition is almost always wrong.

Alex showed a stack trace from WebKit in his blog post, without any kind of time numbers in it - a pretty crude tool. But most people don't even have that luxury (instrumenting WebKit), for applications running in a browser. On some mobile platforms, recompiling parts of the OS isn't even a possibility for most developers.

In lieu of actual tools, I'm always up for crazy hacks. That's basically what weinre is - a crazy hack. Take the Web Inspector user interface, and plug in a fake runtime behind it running on the target browser - just enough to get the user interface to display something interesting. Until we have the real Web Inspector showing us the browser innards running on a mobile platform, maybe this cheat will be useful to folks.

On a slightly related note, an interesting bug got entered against Web Inspector the other day: Bug 53659 - Web Inspector: Better support for finding "leaked" DOM. The summary describes a debugging scenario for which some new tooling could help. Here's the sad part: if this were Smalltalk, there's a chance you could write a specialized tool to collect the desired information with user-land code. The ability to enumerate all the live objects in a running image was an especially helpful feature. How useful would it be, if you could do that in JavaScript? In a special debugging mode, of course - not exactly a function you want to expose to everyone. But it would be nice to not have to wait for the platform developers to write tools for us, if we could write our own.

Could you help diagnose the scenario described in Bug 53659 with a crazy hack? If not, what's the bare minimum additional runtime functionality you'd need in JavaScript to do it?

weinre at GitHub

2010-12-17T11:07:00.001-05:00

I finally pushed my weinre project up to GitHub last night:

weinre - web inspector remote - pronounced "winery"
http://pmuellr.github.com/weinre/

It's a debugger for mobile web apps. Uses the Web Inspector debug front-end from WebKit (implemented in HTML/CSS/JS), plugged up to some plain old JavaScript running in your web page. There's an HTTP server in there somewhere.

It's early days - it may not be the easiest thing to set up and run - and there's a lot of function not implemented (showing all the CSS style info, for instance). But it's something.

I intend to continue enhancing it over the next while. First order of business will be to get it back, function-wise, to where my earlier prototype was, as shown in this video. I've been spending more time on infrastructure stuff than adding debug function lately - it'll be fun to dive back into the innards of Web Inspector again.

My intention is for this work to eventually be folded into the PhoneGap project. PhoneGap (wisely IMO) requires a Contributor Agreement in place before accepting contributions, so I won't be able to accept any contributions into my repo unless you've signed up.

If you have questions, feature requests, etc, create an issue at GitHub. I also hang out on #phonegap on freenode during business hours (US east).

Tasty Innards Too!

Besides this project's subject matter being a fun thing to work on (building a working debugger), I've had a lot of fun building and using smaller thingees. If I can get my blogging going again, hopefully I'll write more about some of these goodies, soon:

All my JavaScript code is structured as CommonJS modules, using modjewel.
All my JavaScript is written in a pidgin dialect of JavaScript called scoop. Basically just a lexically simpler way to define classes, methods, etc in JavaScript (if you didn't know already, I'm an old Smalltalk dude). Here's an example - the implementation of the Console class used by the debug target.
The interface between the large grained components of weinre is specified in a subset-but-extended bastardization of WebIDL. The Web Inspector code already does this itself (because WebKit in general does this also), and it worked out well for me. I wrote a WebIDL-to-JSON compiler, and the runtime makes use of the JSON to create proxies, etc. Lots more interesting stuff can be done here, I'm sure.

tech books on the Kindle

2010-12-14T09:11:00.001-05:00

Three images from David Flanagan's JQuery Pocket Reference, as they look on my 3rd generation (latest) 6" (not the big one) Kindle.

The first image below is from the PDF version of the book.

The font size is a little too small for me, and I would assume, most people. It also wastes a lot of space on the left and right margins in general, on the indentation of the code sample, and the footer information.

Note that PDFs are not "resizable" in the same way a web page is; you can't make the font larger. What you see is what you get.

The next image below is from the Mobi version of the book. Mobi is a publishing format that has been used by various reader software over the years, and is the "native" format for the Kindle.

What I'm showing here is the text at the smallest font size supported. I typically read at one font size up from that. There are 8 font sizes available when you're reading Mobi files, along with being able to tweak the line spacing a bit, etc.

I'm showing the smallest font size because it shows one of the biggest problems with using Mobi for tech documents - the code sample is too big for this font size (the smallest!) and it wraps. It just looks terrible.

I have some experience with the Mobi format, converting HTML documents to Mobi using Calibre. It appears the Mobi format is pretty limited. I've tried all sorts of styling of <pre> elements, and the best you can do is force some vertical whitespace (not the default!) in front of your code samples. It's just awful.

What's a boy to do? Turns out O'Reilly also offers an ePub version of their eBooks. You can easily turn your ePub into a Mobi with Calibre, but you're not going to be able to do much better than the image above.

So I did something different:

unzip the ePub - ePub files are zip files with an .epub file extension
concatenate all the .html files in the epub together, into one file - ePub files are XHTML files for their main text content
```
cat ch*.html > combined.html
```
remove the <head> sections with a find/replace regex pattern, etc

add a simple <style> section:

    h1.title {
        page-break-before:  always;
    }

    body {
        font-size:          200%;
        font-family:        Georgia;
    }

    pre {
        font-size:          70%;
        background-color:   #EEE;
        padding:            0.5em;
        overflow:           hidden;
        border-radius:      0.5em;
        -moz-border-radius: 0.5em;
    }

    .sidebar {
        padding:            0.0em 0.5em;
        overflow:           hidden;
        border-radius:      0.5em;
        -moz-border-radius: 0.5em;
        background-color:   #EEE;
    }

    .sidebar p.title {
        font-weight:        bold;
    }

bring the resulting file up in FireFox
print, but save to PDF instead of actually print (I'm on a Mac)
copy the PDF to the Kindle

Here's the result. Not perfect, but much better than the other two files. The font is larger, and the code sample is readable and well separated from the other text.

Note: FireFox is not my day-to-day web browser, but is the only browser that produces usable PDF files for files like this. PDFs produced from both Safari and Chrome only last about 30 pages or so before the text disappears, when viewed on the Kindle. Who knows.

Update on 2010/12/14 at 1:20pm - the images aren't working from flickr, so put on dropbox

bind() considered harmful

2010-06-04T12:12:00.000-04:00

A recent passive-aggressive twitter message from me:

Oh yeah. Keep on bind()'ing folks, because bind() rocks. http://bit.ly/aPyfuV

In case you can't tell, the message is facetious. bind() does not actually rock.

what is bind()?

bind() is a JavaScript function implemented by many JavaScript libraries and included in ECMAScript 5. If you aren't already familiar with bind(), this post isn't going to make much sense, but if you're curious anyway, see Prototype's documentation for their bind().

Let's look at what bind() is actually doing:

associates an object to use as this (aka the receiver) when the function is invoked
associates additional parameters to be passed to the function when the function is invoked (aka currying).

bind() does this by returning a new function, which when invoked, arranges for the object you want as the receiver and the additional parameters to be passed to the original function. It hides these values in the newly created function, associated with the closure the function was returned from.

bind() is frequently used for callbacks and event handlers, which typically allow you to pass a function as the callback, but don't provide a way to specify the receiver or additional parameters for the actual call to the callback.

why does bind() not rock?

gets in the way of debugging
Take a look at the bug I linked to in the Twitter message above: Bug 40080: Web Inspector: better Function.prototype.bind for the internal code
The bug concerns providing a better debug story around functions that have been bind()'d. Because the current story isn't very pretty. When you stumble upon a bind()'d function in the debugger, you see the source for the bind() function itself, which isn't what you want to see. In addition the resulting bound function is usually anonymous, which means your stack traces, profile reports, etc, will be filled with (anonymous function) entries.
garbage
Calling bind() creates a new function object every time it's called. Not just a string, or empty object, or array. A function. Which I'm guessing is more expensive than simpler objects. [yes, I should do some measurements.] A closure is also created and associated with the function (which is where the receiver and curried arguments are stored). More garbage.
Now imagine that, for whatever reason, you need to add and then remove a callback frequently, over and over again, for some reason. Or set a timer over and over again. And you're using bind(). Think of the garbage you're creating.
Alex Russell posted some thoughts on bind() in a post to the es-discuss mailing list (item 3).
So why does this [using bind()] suck? Two reasons: it's long-ish to type, and it doesn't do what the dot operator does -- i.e., return the same function object every time.

It is longer, and therefore yuckier. It's also often not DRY (obj referenced twice):
```
    setTimeout( obj.method.bind(obj, "John"), 100 );
```
The second point is explained by Alex in his post:
Many functions, both in the DOM and in the library, accept functions as arguments. ES5 provides a bind() method that can ease the use of these functions when the passed method comes from an object. This is, however, inconvenient. E.g.:
node.addEventListener("click", obj.method.bind(obj, ...));
Also, insufficient. Event listeners can't be detached when using Function.prototype.bind:
// doesn't do what it looks like it should node.removeEventListener("click", obj.method.bind(obj, ...));
So the trick here, if you want to use removeEventListener(), is that you have to store the result of a single call to bind() and then use that value in subsequent addEventListener() / removeEventListener() paired calls. Alex's post suggests a new language feature to work around this (btw, I'm not in favor of his proposed language feature).

why are we in a bind with bind() today?

It's pretty obvious to see how we got to the point where you need to use bind() in your code today.

Historically, JavaScript was a glue language that let you do a light amount of programmatic processing against stuff in your page. When specifying a callback/listener, you didn't have to worry so much about the receiver of the function you passed in; you were probably using global variables instead of creating your own little objects.

And so the places that take callbacks, like setTimeout(), onload handlers, etc, didn't really have a need for you to specify the receiver of of the callback when it was invoked. The receiver was always ... well, whatever it was for your callback.

Fast-forward a decade, and now we have people building huge systems out of JavaScript, using some sort of "class" story, or living the hippy prototype lifestyle, or who knows what kids are doing today. In any case, there's often an "object" in the picture, and you'd often like to arrange to have that object be the receiver of the callback. Quite often, you'd like for the callback function to be a method of an object, and have the receiver of the callback be that object.

The problem is, there's no where to put the receiver; all the pre-existing callback patterns just allowed the use of a function parameter. The trick with bind() is that it attaches the receiver, and possibly curry'd arguments, to an invisible bag wrapped around a newly created function which is a delegated version of your callback function. Nature will find a way.

how can we fix the evils of bind()?

For me, the root of the problem is that we're passing the method receiver in a secondary channel, the bound function. So, stop doing that. Pass it explicitly.

Let's play with changing the addEventListener() function to accommodate a new receiver parameter. Here's the current function signature:

    target.addEventListener(type, listener, useCapture)

We can add the receiver parameter to the end of the function:

    target.addEventListener(type, listener, useCapture, receiver)

or we could allow listener and receiver to be combined together in an array and used where the existing listener value is today:

    target.addEventListener(type, [receiver, listener], useCapture)

This second flavor tastes better to me.

what does the fix smell like?

Let's compare the code. For the examples below, obj is the receiver of the callback, callback() is a method available on the obj object.

Using ECMAScript 5's bind() method:

    node.addEventListener("click", obj.callback.bind(obj))

Here's the four-arg addEventListener():

    node.addEventListener("click", obj.callback, false, obj)

Here's the two-element-array-listener addEventListener():

    node.addEventListener("click", [obj, obj.callback])

These examples could be made even DRYer, if instead of passing a function reference, you pass a string, which will be used as a property name to obtain the function from the receiver object:

    node.addEventListener("click", "callback", false, obj)
    node.addEventListener("click", [obj, "callback"])

In terms of being able to handle the removeEventListener() case as well, when using the ECMAScript 5 version of bind() you would have to arrange to store a copy of the bound function, so you can send the exact same function both addEventListener() and removeEventListener(). My proposed versions could do a compare against the parameters or array elements, allowing you to use the exact same parameters on addEventListener() and removeEventListener().

In other words, here's how you do it in ECMAScript 5:

    var boundFunction = obj.callback.bind(obj)
    
    node.addEventListener("click", boundFunction)
    ...
    node.removeEventListener("click", boundFunction)

And here's how you do it with my proposal:

    node.addEventListener("click", [obj, "callback"])
    ...
    node.removeEventListener("click", [obj, "callback"])

This invocation pattern works the same for the other form of call that I proposed.

Curried arguments can be handled the same way the receiver parameter is handled; passed as additional arguments to addEventListener() (not needed for removeEventListener()?), or an additional element in the array where the listener argument was previously used. One simplification would be to allow a single curried argument - other callback systems typically refer to this as userData or clientData - rather than deal with a variadic list. It's simple enough to combine multiple curried arguments into an object or array for use as a userData argument.

actually, bind() isn't always evil

Although I've spent this entire post complaining about bind(), I will acknowledge it's power and usefulness. Particularly in meta-programming and function programming.

My primary complaint is having to use bind() is something pedestrian as callbacks. That's too much.

the obligatory Smalltalk reference

My views on this subject are biased by my exposure to OTI/IBM Smalltalk. Read up on the Callbacks section on page 150 of "IBM Smalltalk: Programmer's Reference".

You'll note that my suggestion here is no different than the addCallback:receiver:selector:clientData: method described in that manual, just a shorter name.

coopitition

2010-06-02T11:05:00.001-04:00

A neat thing happened last month. I entered a feature request against the Web Inspector debugger shipped with WebKit, to get some new function added. Two weeks later, that function was implemented in the FireBug debugger for FireFox. Coincidence?

The feature request was this one: Bug 38664: Web Inspector: add a "table" method to console, to allow output of tabular data. I got the idea for this from looking at another Web Inspector bug: Bug 30897: Web Inspector: Support Debugging via HTTP Headers like FirePHP and FireLogger. Turns out that FirePHP (another interesting project) has had support for tabular console output for a while. I immediately felt left out.

So how did the FireBug folks beat the Web Inspector folks (me included) to the implementation? Turns out that IBM employs folks who work on both Web Inspector and FireBug. My primary contact for FireBug stuff in IBM is John J Barton, and I thought this new function would be interesting enough they'd want it too. I guess I was right. John and I trade notes on stuff like this from time to time.

Once Jan Odvarko (aka "Honza") posted his blog entry Tabular logs in Firebug, my interest in this bug grew, for some reason :-) So, had some conversation with Jan, on the bug, last Friday. It was great for Jan to provide additional feedback in WebKit's Bugzilla. And then I was totally surprised yesterday morning to find that a bunch of additional great ideas around the feature request got posted back to the bug over the weekend, from Jan and other Web Inspector developers.

I decided to try to summarize where we were, provide some examples, and ended up writing a little console.table() simulator to play with, all of which is currently available in this file: console.table() proposal. Please post comments on that file back to the bug, for now.

I love friendly competition!

the weinre experiment - WEb INspector REmote

2010-04-28T13:18:00.001-04:00

For the last few months, one of the things I've been working on is an experiment called weinre, which stands for WEb INspector REmote. The basic ideas are:

reuse the existing Web Inspector UI code, implemented in HTML, CSS, and JS
get that Web Inspector code to run as a plain old web page
have that Web Inspector code debug a web page running in another browser window, perhaps on another machine

You might ask - "Why?" The answer is, largely: mobile. How am I supposed to debug mobile web apps on an iPhone, Android device, or similar? Today the answer is: painfully. Tomorrow the answer should be: via a nice debug user interface like Web Inspector.

The source and an executable demo are available on WebKit's wiki, currently at the bottom of the Web Inspector page:

https://trac.webkit.org/wiki/WebInspector

I have an introductory write-up of the experiment at my web site, and it's also included with the source and executable demo:

http://muellerware.org/papers/weinre/manual.html

If you are truly lazy and don't want to bother with those, but can afford 3 minutes for a demo movie, you can watch it on YouTube. Best viewed in 720p (HD) in full screen mode, if your browser supports that.

http://www.youtube.com/watch?v=MxmFrFEGInU

Let me emphasize that this is an experiment. Although parts of it do work, as shown in the movie, it's a bit unwieldy and insecure to use as a production tool. I was at a good place to stop, and so thought I'd archive the wad up for other people to experiment with.

So, what's next?

Security, as noted in the write-up, is a big concern of mine. Must do something there.
Remove the requirement for the separate agent/server. Meaning that one of the debug client or the debug target should act as a server to it's peer. Or perhaps both debug client and debug target should be able to act as the server. Or use some other whacky IPC mechanism.
Formalize some structure around the messages sent between the target and client. Today the messages sent between the debug target and debug client are very specific to the Web Inspector implementation. It would be nice to have something less specific. One of them things called an A-P-I.
Not sure where my time is going to be spent over the next year, but it might not be here, so the answer to "What's next?" might be: "nothing, from me". But the tea leaves seem to be reading: "remote web debug".

prior art

2010-02-26T10:59:00.001-05:00

A few caveats before starting:

I am not a lawyer, just an old country programmer, currently working at IBM.
This blog post is my own and does not necessarily represent IBM’s positions, strategies or opinions.
The patent infringement suit I mention below does not involve IBM as a litigant. It does involve me as a fact witness.
No need to go into details on the patent infringement suit, as they are irrelevant to this blog post.

Tim Bray recently wrote a blog post, "Giving Up On Patents", providing his opinion on the state of patents and linking to a number of interesting patent-related texts on the web. My thoughts are currently roughly aligned with Tim's, so there's no need to restate them.

Since I was recently involved as a fact witness in a patent infringement suit, I thought I'd provide some tips if the following are true for you:

You believe that there are a lot of patents granted for which there already exists prior art, and so those patents really shouldn't be granted in the first place.
You make things.

The tips:

Document the crap out of the things that you make.
Put your documentation on the web, and make sure search engines can find it 20 years from now.

For the suit I was involved with, an attorney of one of the defendants did a Google search on a term relevant to the suit, and found a paper I had written 13 years ago, on my personal web site. The paper they found lead them to believe I had information relevant to the suit; they thought I had evidence of prior art. Eventually, a lot more documentation was excavated from various people and places.

I found it interesting that a couple of movies were added to the pile of evidence I was asked about during my deposition, all of which appeared to be digitized versions of tape. They were all also around 13 years old. I suspect video will become even more popular as evidence in the future, given the ease in creating and disseminating it, and the amount of information it provides. Lawyers love information. Think screencasts, or perhaps ASCIIcasts are more relevant for the thing you made.

In case you're morbidly curious, I don't believe the statements I provided during my deposition actually helped the defendant in any way. Based on the questions I was asked, I have my guesses as to whether I helped anyone, but I have no way of knowing. I didn't study the patent in question, so I don't know what claims the plaintiff is making, nor do I know how the defense is planning to defend against those claims. All that is irrelevant for a fact witness; I was there to objectively answer questions that were asked of me. Just the facts, ma'am.

A couple of slightly humorous notes:

During the deposition, the court's Windows machine could not be coaxed to play the videos (missing codecs). One of the younger attorneys saved the day with his MacBook.
Making some small talk (heh) with my counsel during a break, I mentioned that I might claim "software patents are against my religion" if I was ever asked to do something I really didn't want to do. He had a good laugh, and said he'd never heard that before. Note taken.
The paper that the attorney found on the web had a self-referential link in it. I always do that, never know what might happen to an HTML file. The link, however, was to a web site long since gone, which today renders as a parked 404 page somewhere. Of course, the paper had moved somewhere else, and of course, Google knew where it was. But it was still funny to see that old link.
Being a fact witness doesn't pay well. I received a $55 check for my time; the deposition was an all-day event.

modjewel 0.2.0 - a CommonJS require() function for the browser

2010-02-09T10:19:00.001-05:00

I decided to get off my butt and bring my "for the browser" CommonJS require() implementation at http://github.com/pmuellr/modjewel into the 20-teens.

some background

Just over a year ago the "ServerJS" group was formed based on the "What Server Side JavaScript needs" blog post from Kevin Dangoor. This effort was renamed "CommonJS" later in the year, and lives on the web here: http://commonjs.org/. Kris Kowal has a nice write up on CommonJS at Ars Technica in the article "CommonJS effort sets JavaScript on path for world domination", if you don't know anything about it. Note: I'm a sucker for software-based world domination.

One of the first items looked at by the CommonJS community was a "module" system, centered around a function named require(). More information on modules and require() is available here: http://wiki.commonjs.org/wiki/Modules/1.1. While most of the folks in the group were playing with server-side JavaScript, I was interested in the browser, so cooked up a crude version of require() in a github project just over a year ago; the crufty old hack is here.

I figured someone else would do a proper version, but haven't really come across anything besides David Flanagan's efforts, documented in his blog posts "CommonJS Modules implementation" and "A module loader with simple dependency management". I have to admit I don't watch the CommonJS stuff all that closely, I probably missed some other implementations.

David's code doesn't have any kind of a license associated with it, so I didn't really feel it was appropriate to use it for a real project. And there were some things I wanted to modify. So, at the very least David shamed me into sprucing up modjewel and I will admit to stealing bits of his design as well.

One thing I learned from David's blog post was that compliance tests exist. Loves having tests. They currently live here: http://github.com/kriskowal/commonjs. Note that David's blog post points to an older version of the tests.

Onto the goodies in the newly improved modjewel ...

nice ui for the compliance tests

As I mentioned above, there's a set of test cases for the require() function. Problem is, they're built for command-line invocation; remember, this CommonJS stuff is more server-side than client-side JavaScript. I was recently pawing through the source for the SunSpider JavaScript benchmarks, and realized I could use the same "run the tests serially in IFrames" technique here, so I lifted that design. I now have a single page that will load and run all the tests serially, and produce a green/red report card when done. It's not perfect, but it's very handy.

One weird thing to note was that I couldn't find any kind of license on the tests; I ended up importing them directly into my project, which is kind of silly. I should probably run them right out of the github repo.

support for module.setExports()

The existing facilities with modules make it easy to provide access to a number of objects within a particular module. However, for those times you only have a single object in a module, things get verbose. Here's an example: say you want to build and use a module called WebDav (so the file is named WebDav.js) which exports a single function, WebDav.

In the file WebDav.js, you would define the WebDav class as:

   exports.WebDav = function WebDav() {
       // the function is 'named' for debug purposes
   ...
   }

You can then use the WebDav class in another file as:

   var WebDav = require("WebDav").WebDav

All the RY of Java. The "WebDav" passed to require() specifies the module, and the next WebDav selects the function as a property of the exports object the module populated.

Alternatively, with the module.setExports() support you can define the class in WebDav.js as:

   module.setExports(function WebDav() {
   ...
   })

and then use it in another file as:

   var WebDav = require("WebDav")

Better! Very convenient if you have a 1-1 correspondence between classes and modules.

module.setExports() isn't official yet; see the wiki page for more information: http://wiki.commonjs.org/wiki/Modules/SetExports.

preloading modules via <script src="">

The way modjewel's version of require() works is like every other browser-based one: it use synchronous XHR. Which is ok for development-time, but completely unacceptable at deployment-time. So they say. Flanagan came up a preload story for his require2.js file, but the mechanism seemed a little crude.

Instead, I created a command-line utility which will convert a module into a file which can be embedded with <script src="">. So then the story would be that you run this utility as part of a build step, perhaps minize or whatever you kids do to your JavaScript afterwards. In the end, you have a "plain old" JavaScript file to "include". All the utility really does is wrap your code in a function. When this code is executed, it registers the module and the function, such that the first time require() is called to load the module, the function is executed and the module is loaded.

A nice aspect of this is that the order in which the <script src=""> are placed doesn't need to be in some kind of special pre-req order. Including the modules doesn't really load them, it just pre-loads them.

Anyway, it's one answer to the "you can't use synchronous XHR!" argument.

portability

I've run the tests successfully on Mac Chrome, Mac Safari, Mac Opera 10, the IPhone Simulator, Windows Chrome, and Windows FireFox 3.0. IE 7.0 was a no-go.

There is an issue with Chrome, where some tests fail when running against a file:// URL. See issue 1 for more info.

There are some jiggly issues with Windows - of course those being path separator issues (\ instead of /) and new-line issues (CR LF instead of LF). Should be easy to fix. Not sure what to do about IE 7, do I use FireBug Lite to debug that?

remote web inspector - baby steps

2010-02-01T15:31:00.001-05:00

Last week, I did a little work getting a remote debug story between an iPhone and WebKit's Web Inspector limping. Barely limping. More like crawling. Anyhoo, the write-up is here:

http://muellerware.org/papers/remote-web-inspector-baby-steps.html

who called that function?

2009-12-02T20:59:00.003-05:00

A question came up today in Web Inspector land - someone wanted to know why setTimeout() was being called so often in some web application. The suggestion was that a capability could be added to Web Inspector's timeline panel to show a function's invocations (setTimeout() in this case) as well as the stack traces for those calls, to figure out who was actually calling the function.

My immediate thought was - can't you do this with user-land code? If you can do it with plain old JavaScript, then I see less of a need to add the function to the debugger. Especially since:

The timeline already collects a LOT of information, keeping track of stack traces just adds more. The UI will get even busier, and Web Inspector will eat even more memory.
Some new gestures would be required to manage the list of functions that you wanted to see on the timeline, and whether you wanted the stack traces or not (if optional). Note that it's not possible to use the existing breakpoint gestures for this, as you can't set a breakpoint on built-in function (like setTimeout()).

I wasn't sure you could do this with user-land code, so I felt the need to figure out if you could or not. I got close, maybe close enough, and the result is up at GitHub as a gist with a wrapWithBackTrace() function. The idea is that you pass the function you want traced and back traces displayed for, into the wrapWithBackTrace() function, which returns a new function to use in place of the original. Invoking that function will dump information on the console, including a back trace of functions that called this function. I think there are enough comments in the code to figure out what's going on, but here are a few caveats:

The existing code makes use of Array's new methods indexOf(), map(), and forEach(). How nice to have these! If you don't have them, there's a link in the code to a site to get portable versions of them. Note that all the browsers I tested with (listed below) seemed to have them already. Wonderful!
The whole functionality revolves around the bizarre caller property of Function instances. Look it up in your favorite JavaScript reference. Supposedly this property will disappear in future implementations of JavaScript engines, and is already not available in some (IE, I think).

Some sample code that uses the wrapWithBackTrace() function is available in the same gist. Here's a bit of it:

function legend(message) {
    var dashes = new Array(60).join("-")
    console.log(dashes)
    console.log(message)
    console.log(dashes)
    console.log("")
}

// tests with user-land function

function factorial(n) {
    if (n <= 0) return 1
    return n * factorial(n-1)
}

function printFactorial(n) {
    console.log(n + "! == " + factorial(n))
    console.log("")
}

function a() { b() }
function b() { c() }
function c() { 
    printFactorial(0)
    printFactorial(5)
}

legend("calling factorial() before wrapping")
a()

// install the replacement function
factorial = wrapWithBackTrace(factorial)

legend("calling factorial() after wrapping")
a()

legend("calling factorial() at top level")
factorial(0)

This code exercises some of the edge cases for the function - recursive functions, and functions run at the top level of a script (no back trace available). The output of running the code above, in the console, is:

-----------------------------------------------------------
calling factorial() before wrapping
-----------------------------------------------------------

0! == 1

5! == 120

-----------------------------------------------------------
calling factorial() after wrapping
-----------------------------------------------------------

backtrace for factorial()
   - printFactorial()
   - c()
   - b()
   - a()

0! == 1

backtrace for factorial()
   - factorial()
   - wrappedWithBackTrace()
   - ... recursion on factorial()

(... repeated invocations elided ...)

backtrace for factorial()
   - factorial()
   - wrappedWithBackTrace()
   - ... recursion on factorial()

backtrace for factorial()
   - printFactorial()
   - c()
   - b()
   - a()

5! == 120

-----------------------------------------------------------
calling factorial() at top level
-----------------------------------------------------------

backtrace for factorial()
   - {top level call}

Sample code is also provided that wraps setTimeout(), which worked the same way. Just wanted to make sure it worked with built-in functions as well as user-land functions.

The code works in the latest WebKit nightly, Safari 4.0.4, Firefox 3.5.5, Chrome 4.0.249.12, and Opera 10.10, all on Mac OS X 10.6. Note that Opera does not have console.log(), so I gave it one:

console = {log: opera.postError}

The code also works node.js. The node system also doesn't have a console.log() function, so I gave it one:

console = {log: require('sys').puts}

Actually, there's one tiny little WIBNI with node that shows up in the output - can you spot it? (you'll need to install node and try it yourself)

In the end, I kinda doubt the usefulness of this - the problem being that so much of JavaScript these days is anonymous functions. Without being able to get the "file name" and line number a function is defined on, and without a name for the function itself, having your backtrace be a list of {anonymous} is less that helpful. Seems like it would be handy to have a getStackTrace() method on Arguments to give you this information. This still wouldn't help with eval()'d code though, as it doesn't even have a "file name" associated with it.

evil eval

2009-11-03T16:27:00.001-05:00

With all the nascent alt-JS things people are playing with, talked about on this InfoQ blog post, I got to thinking a while back how we could make debugging a little better for these folks in WebKit's Web Inspector.

With my familiarity of Java's innards, I happened to think about supporting a mode of using an alternative source file, and line number mappings between the original source and the generated source, to allow folks to have some level of source-level debugging in their original language. That would be for systems which actually generate JavaScript from the original language.

Java supports meta-data in it's .class files for this sort of information, and I've seen it used for things like debugging Groovy in Eclipse using the stock Java debugger. Not a perfect story, but useful.

So I opened up Bug 30933 - Web Inspector: support for debugging alt-JS files which generate JS. Feel free to drop ideas there.

Anyhoo ...

While chatting with someone who could make use of this, they noted that their biggest problem was that eval() doesn't do a very good job of providing information about syntax errors in the eval()'d code. Really? Time for some experiments.

First, why care about eval() in the first place? Since it's evil. If code is being generated from some alternate source, don't you end up with JavaScript code you can just use in the browser with a <script> element? Sure, you can do it that way. But why do an off-line compile if you could do the compile in the browser?

It's also the case that some 'traditional' JavaScript libraries like Dojo may end up (depending on your configuration), downloading your source via XHR and then running it via eval() instead of using <script> elements.

Getting good diagnostic information from eval() would be good for lots of people.

On to the experiment:

Here's the source of an HTML file to try out with different browsers. There's a link in the HTML which, when clicked, will run the clickMe() JavaScript function. That function builds a string which will be passed into eval(). I added 100 or so newline's in the source before a string that all the browsers would actually choke on (took me a minute; my first guesses at things that would cause syntax errors didn't). The 100 empty lines are to see what kind of line numbers JavaScript will tell us about, if at all; if line numbers with syntax errors are reported, we want to see a line number like 100, not 13 (the line in the source that invokes eval()).

The page is also available on my web site.

Below the source are the results you see in the browser after clicking; these are the properties of the exception object generated by eval(), one per line.

HTML source for test:

     1  <html>
     2  <head>
     3  </head>
     4  <body>
     5  <a href="javascript:clickMe();">click me!</a>
     6  <p>
     7  <div id="output" style="font-family: monospace"></div>
     8  <script>
     9  function clickMe() {
    10      var lines99 = new Array(100).join("\n");
    11      evalString = lines99 + "for ]](x does not compute)]]\n"; 
    12      try {
    13          eval(evalString); // this is line 13 of the source file
    14      }
    15      catch(e) {
    16          dumpException(e);
    17      }
    18  }
    19  
    20  function dumpException(obj) {
    21      var result = "Exception:<br>\n";
    22      for (var key in obj) {
    23          result += "   " + key + ": " + obj[key] + "<br>\n";
    24      }
    25      document.getElementById("output").innerHTML = result;
    26  }
    27  </script>
    28  </body>
    29  </html>

WebKit nightly r50423 on Mac (basically, Safari):

Exception:
   message: Parse error
   line: 100
   sourceId: 4837249224
   name: SyntaxError

Opera 10.00 Build 6652 on Mac:

Exception:
   message: Statement on line 6: Syntax error stacktrace: n/a; see opera:config#UserPrefs|Exceptions Have Stacktrace
   opera#sourceloc: 6
   stacktrace: false

Google Chrome 4.0.223.11 on Mac:

Exception:
   message: Unexpected token ]
   stack: SyntaxError: Unexpected token ]
    at clickMe (http://muellerware.org/exception-in-eval.html:12:8)
    at unknown source
   type: unexpected_token
   arguments: ]
   name: SyntaxError

Firefox 3.5.4 on Mac:

Exception:
   message: missing ( after for
   fileName: http://muellerware.org/exception-in-eval.html
   lineNumber: 112
   stack: eval("[100 \n's elided]for ]](x does not compute)]]\n")@:0 clickMe()@http://muellerware.org/exception-in-eval.html:13 @javascript:clickMe();:1
   name: SyntaxError

IE 8.0.7100.0 on Windows 7 RC:

Exception:
   name: SyntaxError
   message: Expected '('
   number: -2146827283
   description: Expected '('

Google Chrome 3.0.195.27 on Windows 7 RC:

Exception:
   message: Unexpected token ]
   stack: SyntaxError: Unexpected token ] at clickMe (http://muellerware.org/exception-in-eval.html:13:8) at unknown source
   type: unexpected_token
   arguments: ]
   name: SyntaxError

Yikes, not a pretty picture! WebKit was the only one that got the line number I wanted. But it gave a pretty crappy "message". Nice seeing the "stack" entries for some of the implementations.

To be honest, things like the "message" - the reason why you got the syntax error - I expect to be somewhat implementation dependent. Although I expect something (looking at you, JavaScriptCore!).

There's a question of semantics, of course. The exception generated from WebKit, with the line number of the eval()'d source, would be confusing had I not caught the exception (though perhaps I'd see something different in the debugger had I not caught the exception). In reality, what you may really want to see, structurally, is something like Java's nestable InvocationTargetException, for eval(), because there are multiple levels of things going on here; for the ultimate evil, eval() may end up doing some of it's own eval()'s.

In any case, it would be nice to see some of this stuff standardized. In ECMAScript 6, I guess.