ecmanaut

Plan B 4.0

2009-10-15T13:28:00.002+02:00

It's Blog Action Day, and 2009's topic is climate change. With the most enlightening read on my 2008 list being Earth Policy Institute's Plan B 3.0, I was really glad to see Plan B 4.0 released a while ago -- it is available in its entirety in both HTML and PDF formats online, which happily means we can augment the material using tools like Greasemonkey and MashLogic (I am one of the developers and maintainers of both extensions).

Both extensions, in different ways, allow us to easily improve our reading experience by running code that someone else already wrote once, for material that was published by yet another unrelated person or organization, without any sort of organized cooperation among either of us. I find that notion incredibly powerful, and an uplifting thought about where decentralized cooperation is taking us online.

When I read Plan B 3.0 (the paper copy I bought at the release party of the Swedish translation, as it were), I found myself wishing I could more effortlessly look up the many high quality source references that are strewn about the book for the immense amount of facts about recent macro events affecting our planet, on all sorts of levels, instead of flicking back and forth as soon as I got curious. Footnotes work really well in hypertext. The online versions actually don't yet do that for footnotes.

So I wrote a handy user script that not only adds bidirectional footnote hyperlinks, but also pulls them into the end of the paragraph that references them, the first time you click them. Try it (direct install link); it's a very comfortable reading style.

If you run MashLogic too, you get essentially the same feature for bringing in content from Wikipedia (and sometimes references to people from LinkedIn), without losing more focus on the material than you find you want to, getting up to date about relevant data about, say, the World Food Programme's operations, or where the Xinjiang Province is. I find it a rather rich and low-effort way of broadening your general knowledge about everything.

It would be delightful if Google Books would export an index of their entire repository of books, and provide a linking scheme so we could turn those book and page references in the footnotes into actual hyperlinks too, the way we currently do with Wikipedia article names. Take the first reference in the introductory chapter, for instance (full precision of the deep-linkage only available after you install the above mentioned Greasemonkey script :-) -- with a catalog of all book titles to book id:s available, we could turn that into a link like "[1] Sandra Postel, Pillar of Sand (New York: W. W. Norton & Company, 1999), pp. 13–21.]" -- just like that.

The web is a very exciting medium, only just starting to show its potential. And as for Plan B 4.0, I can warmly recommend sinking into it, letting yourself fascinate with richer understanding of where we're heading, and all the fascinatingly intricate ways our global economy and ecology connect, affect each other, consequences and solutions.

There's even a good number of tips for ways we can all join in making the world a better place on the Earth Policy Institute's site. Happy Blog Action Day, and do spread the word!

How to convert a FAT32 disk to NTFS

2009-09-05T00:52:00.004+02:00

I had a near-Windows experience today.

I have had an old 500GB USB disk sitting around, that I used for manual backups in pre-Time Machine times, back when I deemed FAT32 to be the comfy file system choice for hooking up a disk with any computer I might want to reach it from. Today, I think that sweet spot is NTFS.

Especially as Time Machine doesn't want anything to do with FAT32 devices as backup targets.

I figured the easiest way to convert the filesystem in place (as I didn't have any other disk around) was to boot up a Windows XP virtual machine in Parallels and let Windows do it.

That procedure was perilous. If you ever embark on doing it yourself, here is how you probably want to do it:


> net stop wuauserv
The Automatic Updates service is stopping....
The Automatic Updates service was stopped successfully.

> convert E: /fs:ntfs
The type of the file system is FAT32.
Enter current volume label for drive E: [...]

> net start wuauserv
The Automatic Updates service is starting.
The Automatic Updates service was started successfully.

If you skip that first step, you are wise to remember that a Windows machine needs baby sitting not to do bad things to itself. Like deciding it has found and applied security updates that it really wants to restart to complete in ten, nine, eight, ...all the while that long file system conversion pass is humming about in the background.

Fixing $(document).ready() to work for any document

2009-08-08T05:09:00.003+02:00

The first time I saw jQuery's $(document).ready(callback) idiom, I thought it was a great idea: not only is it readable as plain English, grep(1)able (it is available as $(callback) too, but I advise against using that in larger code bases, where you might need to dig up all references to it, at some time) and intuitive, it also lets you provide a different document object you are waiting for.

The first time I used it in the latter capacity I was a bit saddened to find that it was not quite that good yet; it always tests the same document that jQuery itself was loaded into. (At present time, meaning jQuery 1.3.2 and earlier.) The document object you pass isn't actually even fed to all the code that together implements the ready function.

As we needed it to do that at MashLogic though, my colleague Dave took to fixing it so it does indeed work that way, as per the jQuery ticket that has been sitting around for two years (from someone else that apparently wanted and expected the same thing), and I submitted a patch upstream. I'm hopeful it gets accepted for next jQuery release.

We even prepared a little test case that loads jQuery in an iframe and puts document ready callbacks on both itself and its parent window, which seems to work fine in all browsers and OSes we have tested it in so far.

I think the original ticket may have gotten closed over it often being a bit tricky to get a reference to the document object of an arbitrary frame or iframe for some frame you want to track -- the javascript security model forbidding DOM access to content in documents loaded from a different domain, for instance, and timing issues with grabbing a reference to the document object of a frame that is inlined in a document's source -- before we can get a handle to the document, it is a little difficult to have that parameter to pass to jQuery for listening in the first place, after all. But I hope such issues don't prevent the fix from landing, for the cases when we do have one.

Tracking multiple svn branches in git

2009-07-23T05:57:00.005+02:00

I've been using git-svn for a while, to track projects still trapped in the subversion age, with git. It's not too messy to set up, though in order to get nice names addresses you need to map all the subversion identities to their corresponding name and email address. I recommend taking the time to suck out the entire repository history -- it's a one-time operation, and you get free offline history browsing and inspection.

If you have the names and email addresses of all svn committers, list them in a newline-separated file, each line reading svn-id = Person's Name <email@address> -- I kind of started tracking a list of jQuery committers, but got bored and/or side-tracked before finding the last ones. Anyway, if you go through the trouble, pass along this file on the command line (git svn clone -A committers.txt ...) when you create your local copy.

The docs for git-svn are not the best, and didn't mention how to import news into your local git repository for more than the branch you have checked out at the moment. The trick, which might be obvious for some but was not for me (thanks, singpolyma!), is to run git svn fetch --all to update your local repository with data for all branches, instead of (well, or besides) git svn rebase that just updates your currently checked-out branch.

And Bob's your uncle.

Updating Firefox extensions from 3.0 to 3.5

2009-06-19T13:37:00.001+02:00

When I started writing this post, two weeks ago, 46% of the top 95% most used Firefox extensions on AMO (according to Firefox Add-on Compatibility Report) still did not manifest 3.5 support. The Mozilla team has been checking in with at least the topmost few of those projects (being one of the Greasemonkey maintainers, I got one of those pings, a few weeks ago), about whether we were having any issues upping the max-version of their extension to 3.5. We had not tried, at the time, but (happily) did not see any big issues, so we just let the release already out claim 3.5.* working.

For other extensions, like MashLogic which I hack on at work, that transition was not as trivial. Our integration surface towards Mozilla being larger than Greasemonkey's, more things changing underfoot get a shot at breaking things, and did.

The information about backwards incompatible changes available at MDC was and is still a bit sketchy, and did not cover issues we were hitting, at the time -- it seemed XUL documents were getting XPCNativeWrapper treatment in 3.5, that they did not have in 3.0, so we had to start adopting Greasemonkey-like roundabout practices to read and write properties of the XUL DOM. That's the kind of information that is good to be able to read about -- and there, specifically.

It's a wiki, though, so if you find something that isn't already there, while you are scratching your head about why things are not fine and dandy in 3.5 (due rather soon), please add your findings too. I did a brief writeup on my own findings.

For MashLogic, we actually ended up removing the xul page we had used for configuration to do the same thing in HTML instead; work we had already mostly done for our IE port, anyway. Other tidbits remain here and there, though, before we can claim having fully working functionality under 3.5.

On top of that, there is the not so enjoyable multi-week delay at AMO, getting our latest release through their review system. Our current release is thus so far only available if you look at their see all versions page, and grab the so-called "experimental" build on top. User reviews seem to help get the ball rolling a little, though, so if you try it out, we're really glad if you share your opinion with other users there by writing a review.

Whether you are a user or extension developer, sharing your experience like this really helps others. Especially if you do it where others go looking for it. MDC and AMO are rather good funnels for developer and user centric Firefox extension related things, and the more we use both, the better they get. Thanks for your help!

Greasemonkey and bottom-up user scripting

2009-06-07T03:42:00.004+02:00

I reread Paul Graham's essay Programming Bottom-Up yesternight, and mused for a while about a set of ideas Olivier Cornu launched on Greasemonkey-dev around Christmas last year, about offering user scripts some functionality to cooperate with each other. It is, I realize now, the same territory.

That initial message quickly grew to astronomous proportions as we developed the territory (him and I sharing similar experiences of wanting user scripts to be able to share functionality with each other), trying to come up with how, while also selling the idea to remaining Greasemonkey maintainers Aaron and Anthony, neither of which wrote huge site specific user script applications. I will not attempt to sum up much of the proposals or their implementation, but Olivier's fork of the project has the latter, in what state it reached at the time, plus a healthy bit of refactoring and polishing of corners here and there, in case someone is curious.

What I will do in this post, however, is to transplant Graham's post to the domain of web development in general, and user scripting in particular -- moving it into the language domain of Javascript and the implementation domain of web browsers in general, and, lastly, user script programming under Greasemonkey in particular. A few steps away from Lisp and web server applications, but where all the same principles still apply. (The remainder of this post will not make as much sense if you haven't read Graham's Programming Bottom-Up.)

In client side web development, we face diverse host environments where most specified APIs work (and break) subtly different, dependent on the user's browser. The largely adopted solution to that has been the breadth of ajax libraries such as jQuery, YUI, Dojo, Prototype, Base2 and others. To the client side web world these are the first step towards changing the language to suit the problem: bridging the (largely unwebby) DOM API into something more tractable.

Thus your javascript programs become about implementing sought functionality, more than about jumping the hoops of the browser environment.

User scripting, by comparison, is a much smaller domain in terms of body of programmers that do much of their work there, and in terms of ajax library coverage. Greasemonkey, probably still its biggest sub class, in addition to the hoops of the web, adds all the hoops of the Mozilla javascript sandbox, which is an even more hostile environment than the web: there are XPCNativeWrappers around your DOM nodes and lots of other gotchas to watch out for.

And due to that added baggage on top, you will likely find that most or all of the ajax libraries subtly break here and there, if invoked as you would any other code under Greasemonkey, and/or need slight changes here and there in order to work as intended. So writing user scripts for this environment, becomes even more in need of libraries if you aim to do anything beyond very tiny hacks, or you waste dev time jumping the hoops.

Greasemonkey got @require support last year, to give some syntactic sugar for inlining code from a remote URL at installation time, letting us at least hide away them from the main body of code, which was a slight blessing. But that only does install-time composition.

This is where Olivier's work comes in, letting scripts export API methods, usable from other user scripts -- effectively acting as a middleware between a web page or site and the other user script. This is yet another step towards changing the language to suit the problem, or more precisely still, changing the provisions of the host environment to suit the problem.

Because not only do Greasemonkey user scripts suffer the penalties of normal client side development and the traction penalties of the Mozilla javascript sandbox, but also the constraints of working with whole web sites (where data or functionality needed by a user script could be spread across different urls) through the eyes of one single web page at a time. When you develop a browser helper user script for GMail, a dating site, some web forum, game or similar, some of the functionality you want to add on one page might require keeping track of data only found at other pages.

That quickly becomes a great endeavour keeping track of in a user script, meaning that you will spend more time jumping through hoops and wasting more code in your script on building and maintaining site interfacing code than the function of the script itself. The host environment starts invading your core logic, again, leading you off what your script is really about.

Attempts at bridging that with @require will force you (read: your script's user base) to reinstall every script you use that library in every time something on the site has changed that requires a fix in the library. Slight improvement, but we could do better.

What Olivier and I wanted (and evolved through much discussion, together with all the other smart voices on the dev list) was a system that would let a user script abstract away that functionality for other scripts, maintaining rules for which pages to run on, keeping local storage for whatever state info it see fit and so on, exporting some abstract API to third party scripts that in turn want to focus solely on site related functionality, and not on site plumbing.

It proved staggeringly difficult to reach consensus about the soundness of these ideas on-list, though two or three of four or five had a very good grasp of what value they would bring to user script development. I think we may soon see a somewhat different take on how the Greasemonkey project evolves and is maintained. I hope we will keep about the same track record of stability and security in what we release to you at addons.mozilla.org.

I also hope we might make it easier to bring about new tooling like this in the future.

XPath bookmarks

2009-06-01T13:08:00.004+02:00

A few years ago, annoyed with web pages that don't offer in-page permalinks where they ought to, I wrote a tiny XPath bookmark user script to solve half of the problem -- scrolling to the location in the document to which a URL fragment formatted as #xpath:some-xpath-expression was pointing, when visiting any such URL.

I didn't end up using those URLs much in practice, largely because it was a bit of a hassle crafting those XPath expressions, and to some extent because they wouldn't do much for people that did not run that user script too.

At some later point in time, I seem to have realized that I didn't, and that my custom keyboard bindings script, which I mainly use to scroll search results and other (typically paginated) similarly item-oriented pages, one item at a time (via m/p -- don't ask :-), knowing about XPath expressions for the stuff it was navigating, could easily offer the feature of creating those XPath bookmarks for me. [1]

So I apparently implemented that feature, called it ? and promptly forgot all about it. Needless to say, I still didn't use them any more than I used to before.

Today, I realized that my (forever growing) fold comments script knows a thing or two about XPath expressions to stuff in a page (visitor comments, typically) and ought to expose that to this magic bookmark creation script. Especially since it is typically bookmarking some specific comment I wanted, anyway. It at the same time dawned on me that if I just let it share this XPath expression referencing comments (that it would fold) with the items-xpath meta tag of the pagination microformat that the above keyboard bindings script groks, and that script, in turn, always updated the URL when scrolling to a new place in the document, my primary use case would be covered for most of the sites I frequent a lot, at least.

I was quite bemused finding that I had implemented the ? thing, and in short work updated it to do that automatically. So, at long last, setting XPath bookmarks to in-page items of opaque, permalink deprived web pages, is a breeze. I wonder how much they will end up being used. As I never even blogged about the XPath bookmark resolver script in the first place, it hasn't picked up any mentionable user base (245 installs at present time of writing). I think it is a feature it would do the web a whole lot of good, if something like it gained traction and widespread adoption.

Not all XPath expressions make good bookmarks, of course, on web pages that change over time, but very many do, and for static content especially, it's pretty much a guarantee -- even in the face of stuff like CSS layout changes, just as permalinks do.

[1] Zipping through template-formatted items one item at a time with some keyboard command, so your eyes do not have to shift focal point, and can focus on just the changing content (minus static template administrativia, which suddenly cognitively completely disappears from view), is a positively huge UI improvement over either of the normal practices of reading a little, scrolling a little, and reading a little more, or skimming a page, hitting page down, reorienting yourself, and repeating the procedure.

It gets even worse with paginated web pages like search results, where, in addition to that, you have to click "Next" and wait for a moment somewhere in the work flow. Fixing this problem by pushing "endless pages" (that automatically load another pageful of content once you scroll to the end of the page) onto all your visitors, is the wrong way; many will hate you for it. As for my own browsing, I sooner or later tend to end up adding a pagination microformat producer script (if a site already has those meta tags present with correct XPath expressions in them, I of course would not need to do so myself) for my unpaginate pagination microformated web pages script.

Those scripts work well in unison, so I can key down an item at a time for pages upon pages, at worst having to wait for a moment here or there while next page is loading.

Resolving IDNA urls in browser javascript

2009-05-16T02:11:00.005+02:00

A handy feature I discovered a while ago: resolving (for want of a better word -- I can't make up my mind whether to call turning Unicode to ASCII encoding or decoding in this context :-) IDNA URLs (RFC 3490: International Domain Names in Applications) is a breeze:

function resolveIDNA(url) {
  var a = document.createElement("a");
  a.href = url;
  return a.href;
}

Test your browser, or try your own IDN domains!

It doesn't seem to work in all browsers yet, though; of those I've tested, Firefox 3 and Safari 3 both support it, whereas Opera 10.4.11 and Google Chrome 1.0.154.65 do not.

Update: Both as my google memory didn't find this when I sought it out (decoding a non-ASCII, non-IDNA-encoded hostname to its punycode/IDNA-encoded ASCII counterpart), and for having found a use for this in the wild, here is the altered function that just takes a unicode hostname and turns it into its punycode/IDNA form:

function IDNAtoASCII(hostname) {
  var a = document.createElement("a");
  a.href = "http://" + hostname + "/";
  return a.hostname;
}

Little projects I'm warming up to

2009-05-10T22:10:00.006+02:00

I've been cooped up a lot between real life, work and a low energy budget for a while now, but think I'm getting a hankering for some ideas I'd like to develop or see developed, again. Here's a little off-the-top-of-my-head list of a few of them, small and large:

Data SVGs: Craft a few examples of how to encode data in its original raw form in SVG graph(ic)s and have the SVG render them right via coordinate transforms. Forward goal: get Wikipedia to incorporate this principle in their guidelines for how to represent data in vector graphics. Erik Möller (Deputy Director at Wikimedia Foundation) and Brion Vibber (CTO) both found this a cool idea. I ought to at least blog it if I'm not going to be the main advocate rallying people around the idea.

Migrate Kronos Utils (user script here), which I'm not maintaining or developing as much as I used to, and other ikariam related hackery I've done at some time or others, to github, so others that want to hack at it have an easier time forking it. I apparently started doing that, at some time or others, in an ikariam repository, and then promptly forgot about it. *pushes recent changes*

So I guess this pretty much boils down to migrating my updates off of Google Code and onto githubg. And figuring out how to have my pushes and/or commits trigger a tweet to @ikariam_tools, which apparently people have already spotted and started following, despite its severe lack of content. :-) Does anyone remember how that is done? I seem to recall some setting somewhere for it at github, but wasn't successful finding it in all of a minute's scouting around.

Oh, and figuring out how to have vc-mode in Emacs do git things.

Run the jQuery test suite from a Greasemonkey script, to figure out how much breaks under the Firefox javascript sandbox. It'll be nice to have fine grained data on which parts that remain useable and which don't, so maybe someone would even care to fix the latter.

Revive the live Exhibit web scraper thing I crafted some year ago and never got to the point of happiness with where I ever blogged about it. With recent development in Ext JS, it seems it could get a nice UI indeed, this time in a pop-open window, so I don't have the CSS invasion problem, and don't invade the DOM we're trying to scrape.

Don't fold comments on pages we arrived at through a permalink with a fragment pointing to something in the foldable section (at that point, we know we foremost wanted to read that specific comment, and not the article, so don't tuck it away). Edit: Done!

Taking down that specification in text instead of code seems indicative I'm not really in a coding mood today and ought to go have lunch. See you around!

A Mozilla bug

2009-04-03T02:26:00.002+02:00

CSS rgba(...) colours are fun. Firefox 3 doesn't handle them quite perfectly when applied to the <html> tag, though. I haven't seen this sort of reflow bug (leaving shadows behind dragged objects) much since I, pretty much, ceased using Windows machines a few years ago.

Nostalgia to be had! Try this bookmarklet, say, on this page and play around with the mock window at your heart's content.

Wishlist items

2008-12-11T10:06:00.005+01:00

I attended Add-on-con's open house meetup at Mozilla headquarters here in Mountain View earlier tonight, airing a wish list item of mine in the trailing discussion about what we want from Firefox, after some talks about what's coming up next, and where Mozilla want to go at the moment. It was a good session meeting friends and new acquaintances, and I even recognized a few faces from Mashup Camp a few weeks ago. Being the season for wishlists, I thought I'd sum up some of mine, on a more interesting scope than those we might share with our friends and family about items, activities and whatnot. Anything goes, high and low -- I even mention the point I brought up earlier during the evening.

This is a rather scattered list of stuff I want, and would find most incredibly practical, useful, powerful extensions of will, mind and capability. Consider it food for thought, if you will, or inspiration to wild, liberating, wonderful hackery, or even for being my Santa this year. If neither, please post your own, to your own blog. I've tagged mine "tech_wishlist" on Del.icio.us, and encourage you to do the same (both tagging your own post with it, and any other person's post that contains items that you want, too). That's a powerful way of connecting brains, folks, sharing and taking inspiration and cooperation to pretty wild levels. We owe it to ourselves and humanity to make the most of the tools and ideas we generate and adopt, and one very important aspect of that is sharing them.

(This also includes music and other media, by the way, but those industries are too busy not understanding how to monetize it right, and doing all they can to tie us up in pre-wired ages where distribution and marketing cost money, to notice. But I digress.)

I want file system access to the web: File systems are often conceived of a way of organizing information on disk, when the important part of them really is a way of accessing information. You'd be surprised how powerful an augmentation tool file systems are, and how easy it is today to write one today, as compared to the old bad days, now that we have great things like FUSE or MacFUSE. File systems let us reach data and resources in a heavily standardized fashion using all the computing tools there is, whether made for the web, desktop, or any other conceivable environment. The entire web is a walled garden not available to all the tooling there is. We should do something about this.
I want XPath file system access to nodes in XML and HTML documents: File system paths don't have to have any posix semantics built into them by any necessity; they are identifiers that discern data storage location, or reference particular information, if you will. The file system is as little bound to following specific conventions about what a path means as a web server is forced to pick any particular semantics to how it interprets path and query segments of its URLs. In practice, this means that a file system path can very well contain an XPath expression digging out information of a web document, serving a smaller document than accessing the document itself would, after filtering out all the junk you were not interested in. Similarly, a file system path can contain SQL queries or poke search words into google. Doing these kinds of operations has historically locked you into particular tool chains, programming languages/environments and whatnot, for no good reason. Computers are here to liberate and augment our will, not force it into constraining walled gardens.
I want file system access to my running browser session: If you've become dependent on and enamoured with any kind of interactive command line / read-eval-print-loop environment like Firebug, favourite programming language and/or the unix shell, and do any kind of Firefox extension development, chances are you've encountered or would benefit from acquainting MozRepl, formerly part of the MozLab extension, which lets you telnet into your running Firefox session (I hear it works in most of the other stuff based the Mozilla toolkit too, but haven't tried myself). This lets you play around in chrome space as you would in the sandboxed content space with Firebug via a simple telnet session, or indeed right from inside of Emacs, if that's your fancy.

Well, I think browsers have been divorced from the unix and the rest of the tool chain for too long, and I want to operate on and work with the files I have in my brower session without any silly fuss about having me store dead documents on disk, picking a location for them, directing my tool of choice to it, and losing all the benefits of live computing and convenience of access that entails in one blow. The web is mine, and I want to interact with it in my fashion, whichever that might be at the time. I like some aspects of accessing it through the glossy chrome that is Firefox, or Google Chrome, or even the iPhone, but I don't want to be walled in there either, and getting file system access to my live session's tabs and documents means getting access to a specific view pointed at me with the cookies and other me-centric data that some random tool like wget wouldn't see without a lot of laboursome work.

And it would let me into the same walled garden that my current web browser session has already opened the doors to, when I first logged in to whichever site I am on, to do whatever I do there; socializing, shopping, blogging, banking, researching, mailing, playing, working, or whatnot. Cd:ing from window to window, tab to tab, and reaching the documents, images, audio, video and linked documents other referenced from there to do whatever my bidding. Scores of operations you wouldn't even come to think of as possible become downright simple, and useful, in an environment like this. Again, it breaks down silly walls that block our view and perception today.

I'll wager our kids and grandchildren will find the web and computing environments of the early century very backwards indeed, scoffing at the insane amounts of effort spent theming, skinning and reshaping all the walls we locked ourselves up behind, with stuff that did not evolve together as an ecosystem, but separated, and screened off from one another in different dimensions pasted together only very lightly via human-bound I/O, and the monitor/speaker-mouse-keyboard read-eval-print loop. Silly, really. But I guess we really like these walls. It's so easy thinking inside the boxes we grew up with.

Neither of these ideas are new, by the way; there have been wonderful pearls like Plan 9 in the past, but they were, you guessed it, walled away from the other stuff that was evolving at the time, and died out in niches few humans ever colonized and made their own. It's not enough making the best stuff humanity has ever seen, if humanity never sees it, for anyone but yourself, and your legacy and contribution back to the world is measured by how you both execute it, evangelize and show and tell it to others so that they, too, can benefit from it and keep evolving it, adapt and repurpose it, bring it across other walls you didn't care to cross, came up with the idea to, or even figured possible. Note also, how they tend to be orthogonal to one another, and how well they stack up when combined together again, building atop one another.

Combine with all the tooling you had which was previously prevented from being part of the ecosystem, and not even the sky is your limit. Which begs my next item:
I want file system accessible cloud storage: This already exists, but not for long enough to have been commoditized yet, the way web mail is; Amazon S3, for instance, can already be combined with S3FS (free, besides applicable S3 fees) and FUSE or MacFUSE, or SubCloud (expensive enterprise software).
I want browser standardized access to commodity cloud storage: In our walled-in browser reality of today, it would already be a big win having a user provided bucket in the sky to tie browser data to, rather than sticking the goo on local disk, and figuring out complex syncing schemes to connect one computer's local disk data with another one's. Just access the web with any device and have them all phone home to your bucket in the sky to pick up the you bound data.

Note that it should be your bucket, which you handle with ease and can transfer to some other bucket cupboard provider, should you for some reason want to; we want to avoid lock-in. Even though I kind of like the shape and sound you get from Amazon's buckets. Everybody won't. Heck, Google might give you a free bucket, some day, or at least charge you fewer bucks. Or maybe Mozilla would find some way of remaining a nonprofit by routing bucks their search engine integration, or a Mozilla held app store or market for extensions would give them, from your usage of said search engines, or a percentage of the profits of commercial apps sold through such an entity -- into the payment of your Mozilla sponsored bucket, for browsing with Firefox, when browsing the web through their particular chrome.

Which reminds me I promised to share my tech-wishlist item from earlier tonight:
I want Firefox to remember the URLs it installed extensions from: That way, us extension authors (I speak as one of the maintainers and developers of Greasemonkey and MashLogic alike) can pick up configuration data passed in, for instance, the query and/or fragment parts of that URL.

This is primarily useful for improving and streamlining a most seamless installation experience for users, so that -- in the case of Greasemonkey for instance -- a user script link, when clicked by a user that has Greasemonkey installed, installs the script, and when clicked by a user not having Greasemonkey installed, prompts with the Greasemonkey extension installation sequence, which installs Greasemonkey, which then proceeds to install the script.

Via the same link, which points to the Greasemonkey xpi parametrized with the URL for the user script to install. The same kind of streamlining and on-web-sharing of initial configuration data applies very much to other extensions too, perhaps particularly so commercial grade extensions where business models benefit from shaving down the install and setup threshold for users as much as conceivably possible, and what better way to do it than to allow the prospective user to do it on the familiar web rather than to mandate doing it within some walled-in confines of a XUL garden?

I know; I wouldn't want to have to do it on the web, but that's a familiar setting for web users, and it allows people to help one another on the web with doing actual configuration presets, rather than describing in words and pictures how to do manual labour in often contrived and messy user interfaces, which makes non-expert users feel it is way too difficult and well beyond their understanding. But clicking an install link after reading a TechCrunch or LifeHacker post about a specific use case and setup for it, shared in same post, is available to everybody. This is the kind of direction at least we at MashLogic want the web to take, but it's presently awful messy doing that kind of thing. (Not that we'll let that discourage us, but it's an unnecessary shortcoming of the platform forcing you to bring your own duct tape to do it.)

A few of these ideas probably constitute quantum leaps in computing, if or when pulled out of idea space and reshaped as implemented functionality, given more detailed shape and form, convention, and figuring out idealized notions of how to come across the many mismatched and yet unmet problems that will come along the way, as we evolve the tools of the past into the tools of the future (what we always do as wizards and programmers, somewhat romanticized, and sometimes with more or less merit than otherwise). Figuring out ideal solutions is not important upfront; getting started and making incremental improvement over the present is. Doing even a subset of even one or a few of these will enable us to do many things previously much more difficult or undoable, and you, and/or others can keep polishing it up later. That's some of what open source and publication does best.

No-interface interactive maps

2008-12-07T08:15:00.003+01:00

I've wanted to do this kind of hack for ages, since social software became commonplace on the web. Being both very bad at geography (even worse so, US geography, having moved here only just recently), and rather well versed with mucking about with other web developers' user interfaces, I finally took to it yesterday night (I've admittedly done a similar Greasemonkey hack for Facebook at some time, but it wasn't nearly as fun, and has most likely rotted since then, anyway).

The idea is basic: you tend to see location names (cities, addresses and the like) associated with users or events strewn out all over the place. Those don't tell me a whole lot unless I've been there, live near them, or similar. Maps do, though -- at least if I have some reference point I know to relate them to. For a site like OkCupid which is all about getting to know people (or yourself, but that's a different story), the natural point is of course home. And these days, Google Maps offers some really nifty geocoding and travel directions APIs. Add some Greasemonkey magic to the mix, and voilà -- a context sensitive map section which shows you where people are, in relation to yourself, as you hover a location name. It's quite snappy, and completely effortless from the user's point of view; no typing, clicking, dragging or otherwise involved:

This really is my favourite kind of interface: it doesn't have a learning curve, takes no clicking about, and has intuitive discovery properties; hover a place, and the same place shows up, name and all, in the map view.

It can of course use some additional polish like drag-a-corner-to-resize, a get-out-of-my-face icon to turn it off and the like, but suddenly it's a whole little software project rather than a one-nighter. That said, I think I like it well enough to want it across the entire web. Add those, a little toolbar to drag it around with and whip up geo and adr micro formats detection for it, and all of a sudden it's a neat all-round tool.

To finish off, closing the circle of the original OkCupid hack, replace that one with one that just sweeps OkCupids pages and converts its place name HTML to the adr micro format -- and the two tools snap together forming a stigmergy of cooperating Greasemonkey scripts. (It's a development pattern I really enjoy, for the unanticipated emergent bonus effects it tends to bring.)

Anyway, I thought I'd shine some light in the rest of the post on how I put it together, as it's painful doing things with Google Maps from Greasemonkey (can't inject the Maps API comfily, as its loader mucks about with document.write, which only works as intended for inline script tags, as opposed to dynamically injected code), and since my technique is rather easy to borrow.

The map area and geocoding is all done in a little iframe I load from here, starting it off with some URL encoded query parameters like at=Mountain View, California, United States (to center the initial map and put a little blob there), country=United States (which helps the geocoder, IIRC), nav=2 (if we want a medium-size zoom/pan control) and z=9 for a useful default zoom level. (The latter adapts to an appropriate scale when we ask for routes.) It will fill the whole page, or iframe, so we don't need to state any pixel dimensions.

So how do we tell it to look up places for us? This is where it's getting fun. Using window.postMessage on the iframe's window object, we send the geocoder its instructions, and using window.addEventListener("message", listener, false), we listen to whatever it figures out for us (Firefox 3+, and other modern web browsers).

For convenience, I went for a rather minimal messaging API; send it a newline-separated list of locations, optionally preceded by a "focus" (to move the view and blob there) or "route" line (to try drawing a route between the first and last address). All places in the list get geocoded, and the return value passed is a JSON structure {"coords":{"place name":[long, lat], "other place name":[long, lat], ...}, "took":t, "type":"geocode"} where t is the lookup time in milliseconds. Route requests are in addition populated with "distance" and "time" properties, either containing a "html" property (for a human readable form), and a "meters" or "seconds" property respectively.

And that's really all you need to use it.

Wanting it somewhat speedy (as the iframe only has page longevity, and thus won't learn and retain address to lat/long mappings over time), I opted to add a little local cache of places already looked up in the past with globalStorage (Firefox 2+, non-standard). Quick and dirty, but functional, it puts places in a little JSON tree indexed on country, region, city, and so on, however deeply the hierarchy goes. As OkCupid's place names are all on a tidy machine-readable comma-separated list format with consistent capitalization this is a rather small representation; place names in the wild might find it less useful, if still not worthless. As your web browser's IP might get blocked from using the Google Maps geocoder if you use it too recklessly (>15,000 requests per day -- but still), the persistence might be good to have there, just in case.

As usual, feel free to adopt, embrace, extend and hack about madly with it; code wants love too. The full source (and script, docs, et cetera) of the Greasemonkey script is on userscripts.org, and the Google Maps part is linked above. Happy hacking!

MashLogic

2008-10-11T19:38:00.006+02:00

I'm taking a few moments' refuge from the growing absence of furniture in my echoing soon-to-be former home in Sweden (as my household and me are finally moving to the California Bay Area now -- after about two years more or less stuck in that ghastly red tape/inertia land), to join the little crowd of my coworkers at MashLogic, that are just cheerfully lifting the veil off our Firefox/Flock extension of the same name: MashLogic sets sail!

Update: TechCrunch liked us too. Yay! :-)

A close sibling of Greasemonkey's, we try to shift how the web works, from the walled gardens we have today (where a feature you like would stay within the confines of a particular web site, or, at best, group of web sites) -- towards working more like how communicating vessels or a sea of information would: add a feature you like once, and have it propagate across the entire sea, wherever you may roam.

Let's say you love how some web site offers explanations of the difficult words it uses, or even links advanced concepts to their Wikipedia article counterparts. While that is a neat feature, the rest of the web remains stuck on its now annoying lower evolutionary notch, devoid of it.

We think such features should not be implemented in server side logic, once per site, to give you the benefit of it only on a small fraction of the web: they should be part of your browser's toolbox and normal web browsing experience, and subject to your own control and choices.

That is what we have browser extensions for in the free software world. The extension user experience in Firefox isn't without its flaws, though; easy toggling of functions you might want or not is not is always subject to whether the extension implements its own such toggle, and if they do, they all end up having Yet Another User Interface, yet another icon or menu entry somewhere, and so on. And while you can disable any extension, you have to restart the browser to be rid of it, and for reversing the operation, restart yet again. There is a very distinct lack of convenience there.

Greasemonkey improved that picture a lot -- Greasemonkey user scripts all show up the same way in the monkey menu, and you have both an effective off switch for them all in one go, and each individually by checking or unchecking them, at the less insane cost of just reloading the page. MashLogic lets you toggle them and have them apply or undo their effects in real time instead, thus letting you play even comfier still, without even reloading the page, and much less restarting the browser.

It is some rather boring menial labour doing much of the text crunching that goes into a user script or extension that scurries through document node trees too, if you were to make one doing things like the above -- or perhaps an automated currency converter that translates weird foreign price tags, or articles or blog posts naming amounts in foreign currencies, into your own currency of choice.

The lion's share of the work isn't in mashing up that currency conversion web API you bookmarked the other day with data for it, or even in the challenge of recognizing free form mentions of amount + currency, but in hacking up a little document crawler that finds the stuff and then changes it (without completely locking up the browser). And by the time you realize that, it's not very fun hackery any longer.

We sort of wanted to address that too, making it more like plugging in your regexp, or a little dictionary, in a polished crawler that already does that boring job for you, so you don't end up reinventing those icky wheels over and over again client side too.

The particular examples above are, unsurprisingly, stuff we already provide, among a handful of other features. There is plenty of room to grow in still, even in features available to Greasemonkey users already (for instance, we lack the provisions for installing new mashes, which is what we call the separate MashLogic features), but we'll mature over time, as any tech worth having does.

I'm eager to get back to the hackery again soon, but you would not believe how long this whole real-worldly stuff of moving from one country to another is taking me; it's almost as if this world of physical matter of ours is immune to my wizardry skills. It just does not compute. I think it's broken somewhere. Where do we file tickets on it?

Fun with Greasemonkey: @require

2008-07-24T19:44:00.005+02:00

After the release of Greasemonkey 0.8, the door is open for @require helper libraries, the user script take on Ruby's require, or the C family's #include pre-processor directive. For security reasons, the required library is downloaded a single time at script installation time, and from a functional perspective, is equivalent to pasting the script or scripts referenced at the top of your script, in the order of the @require lines listed. (==UserScript== headers in the referenced files are presently ignored, so a library may not yet use @require or @resource lines of its own.)

@require in practice makes user scripting scale down better, which is essentially to lower the threshold towards getting a job done, to a level where implementation cost to scratch an itch is so low that rather than feeling that mild annoyance with something twice, you address the issue once, and never see it happen again -- on that site, or the web at large. We are still far from there (in as much as "getting there" would ever happen -- that threshold can always be chipped off further, I assure you) -- but @require is healthy progress.

This is what the feature can look like in an example script of mine which adds image captions to online comics, by default Dinosaur Comics and xkcd (expect another upcoming post about $x and $X):

// ==UserScript==
// @name      Image title captions
// @namespace http://code.google.com/p/ecmanaut/
// @require   http://ecmanaut.googlecode.com/svn/trunk/lib/gm/$x$X.js
// ==/UserScript==

I'll be sharing some of my own helpers here, in a little series of "Fun with Greasemonkey" posts. I encourage you to write, and share, your own, on your blog, as well. User script authorship is a very social form of software, especially when you share your results on userscripts.org, and your best practices, for instance by way of blogging.

Javascript books

2008-06-05T12:45:00.003+02:00

I just received two javascript books I recently ordered from Amazon (the US branch, for once, since the dirt cheap dollar made shipping across the Atlantic notably favourable to shipping across the English Channel): John Resig's Pro javascript techniques and Douglas Crockford's Javascript: the good parts.

I had for the longest time (since the nineties) been of the opinion that the only javascript book worth getting was David Flanagan's Javascript: the definitive guide, which I at the time found excellent in it's third edition and still good (and much updated) in its fourth (by my cursory looks and used for reference, at least), but as two of my favourite authors on the subject of javascript (the last being Dean Edwards, whose primarily means of expression on the subject, so far, at least, is terse, solid code) have been published since then, it was time to challenge that assumption. And I'm quite glad to say that both were up to par.

Pro javascript techniques really shows Resig's experience in library writing, and shares lots of useful input and practices, and provides an enjoyable and rather useful walk-through of the popular javascript libraries of its day, which manages to still be fairly relevant, in spite of the evolution they have seen since its publication. Much the same applies to the bits and details it shares on browsers and the tool-chain, but that is to be expected. (I am really looking forward to its successor, Secrets of the javascript ninja, much of which can already be read online.) Particularly enjoyably and usefully, he shows off strengths of each library without much fuss.

As for Crockford's Javascript: the good parts, it follows his usual seasoned narrative which has earned him nicknames such as "the Yoda of javascript", and keeps a close focus on the more timeless parts of the core language itself, devoid of all the clutter and temporally challenged aspects of the document and browser object models. It follows a clean-slate approach to describing the beautiful parts of the language most worth learning, while warning about the pitfalls of some of its worst warts, and is just as enlightening as the rest of Doug's work. It's something of a feel-good book for javascript buffs wanting to meditate over their use of the language -- or at least it is for myself.

Great companions for a season where the weather merits some good outdoors rather than indoors geekery. ;-)

Expensive, yet responsive, live interaction

2008-04-10T09:40:00.001+02:00

Let's say you have a slider widget that users can drag to filter a largeish table so it only shows rows where some column is greater than the a threshold, defined by the slider. That's an application with rather strict real-time constraints, but filtering the table is expensive; a naïve solution that refilters the table with every drag event you get, or even just every event that changes your threshold value, will still be too slow for the interface to feel responsive.

I just came up with a rather elegant solution for that problem, which makes sure that your expensive code won't make the interface feel sluggish, by postponing calls to it that come in a more rapid succession than the time it would have taken for it to run. Or, to be precise, the time it took last time it was called. It's a simple wrapper function you can reuse anywhere:

// returns a function that only runs expensive function
// fn after no call to it has been made for n ms, or
// 100, if not given, or the time fn took last time, if
// it has been run at least once and no n was given.
function expensive(fn, n) {
  function run() {
    fn.timeout = null;
    var timer = n || new Date;
    fn();
    if (!n) duration = (new Date) - timer;
  }
  var duration;
  return function postpone() {
    if (fn.timeout)
      clearTimeout(fn.timeout);
    fn.timeout = setTimeout(run, n || duration || 100);
  };
}

You might of course want to add some argument and this re-binding to it, for cases where the callback isn't self-contained, but then it gets a bit more cluttered, and I really just wanted to show the principle here. Typical usage is replacing your node.addEventListener("mousemove", callback, false) with the wrapped node.addEventListener("mousemove", expensive(callback), false) version, and suddenly, the UI is not only bearable, but probably even rather pleasant to use.

Google site index bookmarklet

2008-03-10T12:05:00.002+01:00

The bookmarklet first, for anyone who wants to opt out of the verbiage: I call mine @site, which fits snugly in my bookmarks toolbar. It searches Google for any page Google has indexed on the site you are presently visiting, and presents them in Google's usual interface which you learn how to navigate once, and can then use for the rest of your life, pretty much without ever thinking about it.

I am well aware that one of the things that made the web so hugely popular with content producers, is its very anarchic non-standardization of the user facing interface. Yes, your graphics and interaction designers are at full liberty to devise pretty much any user interface and interaction scheme they damn well want to, and they happily do so, too. The same applies to DVD menus too, though, thankfully, the dvd consortium at least recognized that their medium was one of entertainment primarily (or, through dumb luck) and somewhat constrained the technical possibilities a little.

Web sites are fairly unconstrained about their navigation (im)possibilities, though, and as every web site ever made is a country of its own with conventions of their own (most -- thankfully -- even shared with or at least inspired by others, modeled from similar components, and so on), so lots of features you might want the medium to provide, like "show me the most popular pages of this site" have to be found, when they exist, identified as such, and learned, before you can use them, once per site, or even once per content producer and site, when shared between several.

That most of the web imitates most of the rest doesn't change the fact that visitors still have a level of interface burden that is not associated with media like books, where every language has a common convention for which order you flip the pages, where to start reading, and so on, and, even across languages, most agree on left to right, top to bottom. The web could have conventions like these, but has not evolved to that stage yet (and might never do). There is some consolation in that browsers have a set of basic functionality for finding text within a page, scrolling it and the like, which is rarely intruded upon by web pages (but little prevents them from doing so).

While I don't know the particular sort criteria Google uses for open searches like the one this bookmarklet does, it often does seem to list the more popular pages first, at least on web sites like Alertbox, some of the first hits are the basic sections of the site. Reading Jacob Nielsen's top-10 application-design mistakes, how many perceive the web as one application? I do. I think most naïve, non-technical users who could not care less about the detail inner life workings of the web, do too.

And I think you should, too, despite knowing a lot more about it than that. Your web sites and design thoughts will surely benefit from it. I am rarely happier than when taming the web to behave more like one single, internally consistent, application, that responds to my actions in the same way, regardless of whichever of its zillion dialogs I happen to interact with. Try not to waste their brain power on working your part of that application, if you can. Yes, it is more fun for your own ego to have them adapt to your whims than you to their expectations and there are counter-examples to every rule, but those account for miniature fractions of a percentage, and it is pretty safe to assume that your application is not one of them.

That said, most of my web sites and hacks venture far into the territory of doing things backward, one way or another. I can assure you that the visitor appeal of said hacks, measured in raw numbers, are inversely proportional to their obscurity. The same is true of my obscure writing.

JSEnv live XPath visualizer

2008-03-08T08:20:00.007+01:00

I've had some productive hack weekends recently -- last weekend, among other things, I took a plunge into the Firebug 1.1 beta, both to re-apply a rather useful feature patch I submitted upstreams a year ago (which was lost, and found, and lost and found again -- might make 1.2) and to add some handy keyboard shortcuts to the HTML view (deleting nodes with delete and backspace, and both folding/unfolding and stepping up/down the DOM tree with arrow left and right).

Anyway, I ended up excavating Firebug's highlight-a-node code to turn it into a tiny lib and then use that lib for beefing up Jesse Ruderman's jsenv bookmarklet with a live edit XPath field highlighting all matches in the page as you type:

It tells you what the expression matches in the title bar of the jsenv window, and signals some additional colour coded info via the XPath input field's border:

red for illegal expressions,
green for matching nodes,
gray for 0 matched nodes,
yellow for a string,
blue for a number and
magenta for a boolean value.

Ah, one more feature -- you move the focus node you see with arrow up and down in the input field, or scroll wheel up/down when hovering it.

The ideal spot for this feature would probably be as an extra feature to toggle on and off in Firebug's HTML view -- to filter the view by, for instance. The more I think of it, the more useful this feature would be in that shape. I hope the Firebug Working Group is as open to additional community developers as it seems to aim for.

Anyway, here is the XPath jsenv; enjoy!

Google safing passwords

2008-03-01T19:38:00.005+01:00

Apparently, lots of places on the net, prominently including all WordPress blogs, use unsalted hashes for password verification, i e you get to log in if hash(given password) == hash stored in the password database; commonly using cryptographic hash functions like MD5 or SHA1. If that password database leaks, people can fairly effortlessly use Google as an O(1) lookup function from hash to plaintext, for most weak passwords.

The problem is easily avoided by using salted passwords -- i e hashing the password concatenated with a random component, say 32 bits long, and storing this salt next to the hash, in plaintext. Net effect: your password database, if compromised, will not be susceptible to the same attack, as the search space is 4,294,967,296 times as large, making it less likely that Google knows how to reverse map the hash for you.

I whipped up a quick tool to see whether a password is susceptible to the Google lookup or not (for MD5 and SHA1, and optionally, whether the plaintext password itself is found on the web too, though I discourage using it for reasons of potential snooping of your traffic). The hack uses some neat client side tools to query Google's Searchmash, using Yahoo! Pipes to decorate its JSON result with the JSONP callback they apparently lack (Google unfortunately still rarely gets that right), and some micro-libraries I've started accumulating recently:

Sample Google for a password dictionary attack on:

Check the checkbox to query google for the plaintext version of your prospective password, and not just cryptographic hashes of it. (Anyone snooping your, Google's or Yahoo's network traffic, will see it.)

User scripting new GMail messages

2008-03-01T15:27:00.002+01:00

With the recent change in how GMail works, lots of prior user scripts improving the service, like a small hack that removes <font size="+1"> tags from mails broke, in the wake of the DHTML improvements (which make the site more difficult to augment via user scripting).

The GMail developers, generously, were not very far behind to release a smallish Greasemonkey API, somewhat alleviating the problem -- but it still is less than trivial to target mail bodies with your user scripts.

I took a stab at it the other night, to figure out what would go into it, and managed to come up with a new font manhandler (it targets a rather local nuisance on the Greasemonkey mailing lists). As I have been trying out MailPlane for a while for reading my mail I decided to see if I could make the script run both in Firefox on Greasemonkey and Safari/WebKit/MailPlane on the GreaseKit user script manager. That turned out quite doable. The script, in its entirety:

// ==UserScript==
// @name           Gmail - deBill:ifier
// @namespace      http://code.google.com/p/ecmanaut/
// @description    Manhandles all font size tags to stop all the yellin'
// @include        https://mail.google.com/*
// @include        http://mail.google.com/*
// ==/UserScript==

window.addEventListener("load", loader, false);

function loader() {
  var api = typeof unsafeWindow != "undefined" && unsafeWindow.gmonkey ||
    (frames.js ? frames.js.gmonkey : null);
  if (api) api.load("1.0", init);
}

function init(gmail) {
  function viewChanged() {
    var view = gmail.getActiveViewType();
    if ("cv" == view) {
      var div = gmail.getActiveViewElement();
      div.addEventListener("DOMNodeInserted", deBill, false);
    }
  }
  gmail.registerViewChangeCallback(viewChanged);
}

function deBill(event) {
  var font = event.target.getElementsByTagName("font");
  for (var i = 0; i < font.length; i++)
    font[i].removeAttribute("size");
}

Feel very free to base your own hacks on the recipe. I recommend starting out with an action function that does something basic like setting event.target.outline = "1px solid red", to see where it does what it does -- since the callback runs for every conversation view change, including updates to the ad pane on the right, not only when expanding mails you read.

Cooperating Greasemonkey scripts

2008-02-24T23:55:00.007+01:00

I should apologize in advance that most of the scripts I link here use a not-yet-released @require feature of the svn development version of Greasemonkey (the equivalent of #include in C/C++, so if you exert yourself, you can paste together the scripts by hand by inlining the code at the top of the script). Pardon the nuisance; at least it won't ruin this post for you.

I have taken another stab at stigmergy computing, or writing scripts that communicate by changing common environment, here the web page they execute in.

Anyway, last time, I wanted a decent portable web photo album browser; functionality that would follow me around irregardless of where I roam on the web. Something to flip through images with the arrow keys, perhaps with some additional handy features too.

The album browser would consume the very basic microformat <a href="*.image extension">, and turn it into a gallery. This meant that I could use my familiar album browser mostly everywhere without any work at all. It also meant that, when encountering some photo site working differently, I could adapt not my album to the site, but the site to my album. A minimal script that changes a single site only very slightly, is trivial -- adapting old code rarely is. So this little ecology convened around a minimal ad-hoc image album microformat.

This time I have several microformat consumers convening around a microformat I came up with for destructuring pagination into what is popularly (if somewhat incorrectly) called endless scrolling; a page that, when you approach the end of it, loads another pageful or so by means of XMLHttpRequest. Usually an abomination, when some site designer forces that browsing mode for you, but ever so handy at times, if you only got to choose for yourself.

Anyway, I started off basing it on the old <link rel="next"> tag, which is actually standardized, hoping that I would, at least once in a while, not need to produce this bit of the microformat myself, but could ride on data in the web page itself. So far, that has not saved me any work but cluttered the microformat into picking next links in two orthogonal ways, depending on what is available, so I'm inclined to write that off as a bad, yet well-intended, idea (see my first reply below for the messy details of how standards bloat like this typically come about).

I needed three data items: an url to the next page in the sequence, an XPath expression that matches the items on every page, and an (optional) XPath expression that identifies a pagination pane to update with the last page loaded so far. I came up with three meta tags, named "next-xpath", "items-xpath" and "pagination-container". If the web page gave me those, my pagination script could deal only with undoing pagination, and care less about whether the data it actually handles is Flickr photos, vBullentin forum posts, Google or Yahoo search results, or something else entirely.

After doing some crazy limbo (which deserves a post of its own) loading the next page into a hidden iframe (both for letting Greasemonkey scripts producing the microformat run, and to get a correct DOM for the page to run the XPaths -- as it turns out that, amazing as it may sound, browsers can't turn an html string into a DOM without ruining the html, head and body tags; at least not Firefox 2 and Opera 9), the remainder of the script was a walk in the park.

Having written up a few microformat producers for various sites to exercise the microformat and see if it addresses the problems that show up well enough, found and straightened out some minor quirks, I felt the urge to do more with this smallish, but oh, so useful, microformat, telling apart content from non-content in web pages, so I decided to hook up my custom keybindings script with this hack, making it assign "p" and "m" to scrolling up and down among the items of these pages. That came down to a whopping ten new lines of code, to make a dozen, or, factoring in future microformat producers, an unbounded number of sites scrollable in that way.

And -- best saved for last -- both microformat consumer scripts coexist happily, independently and work orthogonally; the unpaginated whatevers that get added to the end of the page are browsable with the new keyboard bindings, so you can work mostly any web page as comfortably as ever your feed, mail or whatnot reader, right there.

If you craft any microformat producer or microformat consumer user scripts, please tag them as such on userscripts.org, and also tag them with the name or names of the microformat(s) they handle too (I picked, somewhat arbitrarily, photo album microformat and pagination microformat for mine), so they form findable microformat ecologies for other users. Any site your microformat gets a producer for, improves the value of all the microformat consumers of that format, and any microformat consumer using the format in some useful way, improves the value of all the producers.

It is very rewarding doing stigmergy / ecology programming; the sense of organics and growth is very palpable. There are some sharp corners to overcome, and Greasemonkey could help out more, but I'll have to get back to that at some later time.

High cpu load web page diagnosis

2008-02-24T00:23:00.003+01:00

For web development, a laptop with a case fan that starts screaming like a jet engine when your cpu load skyrockets is sometimes a Really Good Thing, since it gives you cues to what happens in your browser, whether you set out do do profiling or not when, bad enough things happen. (A bit like how you could debug your programs on the C64 from the overtones that would leak from cpu to audio circuitry, giving distinct audio signatures to the occasional choke point in your code.)

Anyway, a Swedish social site I occasionally frequent used to have this effect for me, and get worse the more tabs I had open to it (and it was bad already with a modest three). I used to think it was a bugged Flash app (which they use for ads), but when I realized I had killed them all with AdBlock and FlashBlock, it occurred to me that it might be run-amok javascript.

Only that javascript that runs amok usually gives you the annoying modal dialog asking whether you want to keep running the cpu hogging script. That, however, only happens when page javascript hogs the cpu from the UI, so the single UI thread gets starved, and you, the user, get annoyed with an unresponsive UI. That can be avoided by running smaller loop fragments that call one another with setTimeout calls, leaving some breathing room for the UI thread in-between.

So how do you debug if that is what is happening? Well, for one thing, at least in Firefox 2, you can call setTimeout yourself two times in a row, some second or so apart, and compare the timeout id:s you get back, because consecutive calls get consecutive id:s. So if the difference is one, no setTimeout call was made in-between.

I crafted a bookmarklet to count setTimeout() calls per second (well, I honestly started with something smaller in the Firebug console, but figured it was useful enough to save, share and reuse later), and sure enough, that site was calling setTimeout 178 times per second. (It's not very interesting to run it on this site at present, but you can always race with yourself and the UI responsivity of your browser, attack clicking it to see how many setTimeout calls you can fire off in a second by hand. I believe it is not one of the performance tests we will see in any side by side browser shootouts any time soon. My personal highscore with Firefox 2 is 3. :-)

To do what? Well, time for another bookmarklet that shows which arguments setTimeout gets called with (it won't run on Internet Explorer; sorry: kindly advise Microsoft to implement the toSource prototype for all javascript native datatypes -- I would want to say it would not be a decade too late, but that would make me a liar :-), and lets you restore the original setTimeout implementation when you are done peeking.

It turned out the site was trying to fetch a node with an advertisement I had AdBlocked away. Over and over again. A tiny Greasemonkey script later, and the laptop fan was happily put to sleep, once again. A toast to audial debugging!

Reducing user scripts to atoms

2008-02-21T17:14:00.002+01:00

Over the past few weeks, I have found myself to prod ever more frequently at two user scripts; a UI reducing one and a UI extending one. Both @include the entire web, but are tailored to sites individually on a site by site basis, as I realize that I miss either feature somewhere, and add it, for my own comfort -- some cruft I rarely want to see here, some action I want bound to a key there.

It was only a question of time, and I just hit that tell-tale moment where these two growing (yet, due to factoring it down into convention based descriptive more than programmatic notation, still fairly maintainable) beasts want to marry one another and ~~finally gain independent sentient thought and a soul~~, uhm, augment features of one another to become greater than the sum of their respective parts.

I'll be back to that in a moment, after summarizing how they came to be and what they do. Each started on the premise of recognizing an improvement I would like to see for web sites mostly everywhere, that would entail approximately the same code for any site I wanted to apply it on, but with minute changes to parameters.

In the first case it was the all-too-popular tendency on the web (if, however, probably mostly also equally unintentional) -- of giving gratuitous focus to visitor voices, at the expense of my taking in the author voice -- that is less popular with me. A few swift DOM cuts, and the excess commentary is folded away.

In the second case, it was recognizing what a great improvement in usability changing web pages to offer comfortable keyboard commands, and more so, comfortable standardized keyboard commands, that I could pick myself, but would work on lots of web sites that I use in a similar fashion. (More on that after this hyperbole. :-)

A few years ago, I was wary of web comics, mostly because I read them seldom and would never remember where I trailed off last time, so I could continue from there, yet I neither wanted to reread lots of them to find my bookmark (and browser bookmarks were too much labour overhead), or miss a strip. One early hack was a bookmarklet that would set a cookie to the present url, or, if invoked from the frontpage, go to the url in that cookie. Instant recollection of last location, and a single bookmarklet for any web site. Excellent.

Eventually, I found and had a love affair with Greasemonkey, and it seems in 2006, or perhaps some year earlier, that bookmarklet, too, was too much to keep track of, and I crafted an automated user script to keep track of last-read for me. The main net benefit with crafting hacks like that is often finding the additional features that are so easy to add, yet prove much more valuable than hard to implement. In this case, it was realizing that this little repository keeping track of all comics I read could link them to a little comics ring that would follow me around, always linking to the next in the ring, and specifically the last one I had read, ready to continue traversing. Better still, that way I wouldn't forget those least frequently updated or read comics, as I often would otherwise. Handy!

And when I started assigning keyboard shortcuts to web sites, adding them for comics was another eye opener -- arrow left and right (or j/k) to head backwards and forwards in the archive, "n" for next site in the ring. Instant gratification, all over again. After I fell in love with Questionable Content, I extended that to also include hotkeys to scroll up/down a panel, p/m (for giving that comfy lazy restful right hand resting on m/k/p, to zoom through its bountiful archive in no time flat), and next thing I knew, my spine expected those handy scrolling keys to work on interesting forums too.

If Jon Udell has a thing for overcoming data friction, I guess one of my bigger things is overcoming user interface friction. With the tiny convention based system I crafted, adding a keyboard binding somewhere was just pasting a hostname, deciding on a regexp for what on-site pages to augment, deciding on keys and crafting a quick xpath expression for picking up which link or page entity to click on with that button, or one to slice out which nodes to snap to for the scroll actions. Done in essentially no time at all, for a geek such as myself. Complete comfort, at essentially no time or work investment on my part.

And then it dawned on me. I want that handy scroll-to-the-next-something on all those sites where I taught that first script how to fold comments. And, as I had stylishly also taught that script how to count comments for each site, it already knew how to find each comment node, so with a quick marriage of these scripts, lots of sites would gain that feature in the blink of an eye.

Except I don't really want to do that. It's conceptually wrong; it makes every script that could use a detail of another clump up into big chunks of things to satisfy the exact preferences of me at the expense of other users that only want, say, the hotkey feature, or perhaps only want it for a single site they care about. It would be much, much better if these scripts could cooperate like in a symbiotic biological environment, each script specializing in one thing and doing that well.

It would reduce each script to a small axiom; one script handling the core aspects of registering keyboard listeners that do not interfere with form input, while assigning key bindings to specific page elements or page actions common throughout the web, from a recipe entailing which keys to bind to what features of pages on a specific site. One script handling the logic of folding entities of some sort, given recipes for the same thing. Lots of little dumb work drone scripts that handle recognizing and marking up page features for other scripts -- comments on this page are here, a link to the next, previous and index pages of this site are here. For the latter, there is even a meta tag microformat which good browsers like Opera handle natively, so these scripts, when possible, should decorate the page with that microformat so other entities which consume the microformat get it for free. And so on; one script to connect microformat with keyboard shortcuts, one to connect comments with folding functionality.

That way, we would get specialization, and we would get decentralization, as anybody on the web can easily, independently, craft a new recognizer targeting sites they use, and I wouldn't even hear about it; they would post their scripts to userscripts.org, and all the Japanese who frequent that site would pick it up, and we basically have a lovely SOA system right there. Each part can evolve and iterate independently, at landslide speeds, and people would find these scripts by looking for scripts catering their sites, not through looking for keyboard shortcut scripts, which no doubt very few do.

I think this could use some catering in Greasemonkey, though, to tie the components together and help users tie together producer (those tiny custom drones) and consumer (the feature implementing agents) with one another, maybe even associate them with one another visually in the Greasemonkey script manager. Signals in the user script header like @produces and @consumes tags, of some sort, seem like one good way of doing it.

I am not sure of exactly which all this plumbing should do, nor how, but I think that there is a lot of good that would come out of it, just like how all the above evolution of three user scripts has changed my web experience to a very luxorous one, compared by the counterpart of only last year. I had no idea how much I would lavish in neither using the mouse nor arrow keys nor page up/down to do my web navigation, but factor in the amount of time you spend with those tools, and it should be quite apparent to anyone.

Your thoughts and ideas are welcome, especially if you blog them after noodling on it for a while; I think we deal in largish ideas that fit the comments field below less than well. Especially as my own site, too, so far, still suffers the "comments dwarf original posts" syndrome when subjected to great visitor attention.

Fun with Firebug: Error Messages!

2008-01-14T11:58:00.000+01:00

When working in the Firebug console, you might have noticed that syntax errors and the like show up in red on executing the command line. Great success! Exceptions thrown by the DOM, however, for some reason don't, so there is no telling whether your document.getElementById() call happily returned undefined or threw an exception on you.

If you go to about:config and type firebug.showChromeErrors into the Filter: field, and double-click that line to read true -- presto! You can suddenly see where you and the DOM disagree, and when you made some XPath syntax error, so you can tell apart cases like $X('//div[@class="post"') and $X('//div[@class="post"]') and the like.

I've filed a ticket on this weird default behaviour, but am glad I finally found this trivial cure. (I really don't want to know how many times I've wrapped a statement on the command line into a try{ ... }catch(e){e} block to figure out just if or what went wrong.)

Unbreak the scrollbar

2008-01-06T16:18:00.002+01:00

This is a post on visitor centric design and usability, which first and foremost applies to popular blogs: don't break the native window scrollbar. It's in a way related to the advice don't reinvent native browser scrollbars, and boils down to this short tip:

If visitor comments on your posts or articles add up to more than a single-digit percentage of your page height, don't show them by default.

To get nice order-of-magnitude measures of page sizes, try this bookmarklet which shows how many pagefuls, at present window dimensions, your document fills; drag it to your bookmark toolbar, if you want to measure other pages than the one you are presently reading (visit the post, if you read this by feed reader, as feeds and feed readers try to save you from badness by stripping away potentially harmful bookmarklets).

If you visit some interesting post which does not heed this tip, perhaps Steve Yegge on size being code's worst enemy, or perhaps the sci-fi writer Charles Stross about why space colonization might not be such a great prospect, you might find that it's 300 pagefuls tall. Without its 825 comments, it comes down to about five pages, making visitor commentary a whopping 98% of the document's size, which by a margin breaks the sub-ten-percent bound.

The reason this is bad is that it breaks two important functions of the document / scrollbar symbiosis: first, it prevents a visitor from the size overview offered by the scrollbar size to scrollbar height ratio -- "How long is this document, and how far into it have I scrolled?" -- and second, it prevents you from quickly scrolling to the bottom of the document using browser native "Page end" bindings, as there is no snapping to where the post ends and comments start.

See Joe Hewitt's blog for an excellent example of showing comments only after having clicked the number of comments hidden. (If you hide comments, by default, it's a very good measure to show how many.)

I've whipped up a quick'n'dirty user script to turn a few blogs I frequently frequent into click-to-unfold-comments reading mode, as I care for the author's view first, and reader reaction only second, if at all. I didn't go the extra mile to come up with a good UI for making it extend to any blog you might encounter, but XPath geeks and Hoodwink'd dialers will probably find themselves right at home to extend it for their own reading habits, and perhaps post their own additions to the script homepage. Enjoy!