Inventive Labs: Weblog

Booki.sh!

2010-10-14T21:12:54Z

We’ve finally readied a “technology preview” of our ereading platform! It’s consumed quite a large portion of my brain for the last few months. There’s plenty more to do, but here at last there’s something for you to enjoy reading.

The platform is called Booki.sh. On the Booki.sh blog, you’ll see a shelf of books and samples you can dive into just by clicking. Nothing to install, works on your phone or your iPad or your Kindle3 or your work computer on your lunchbreak. Based, of course, on Monocle. Try it out and let us know what you think!

A few example Zhooks

2010-07-22T13:58:11Z

There is a storied tradition that file formats should be controversially pronounced. I’m pleased to say that Zhook is no exception. At the Labs, we’ve settled on “chook” as the standard verbalisation.

Which should clear up how chook, the little web app I just built, came by its dubious appellation. Chook provides a very simple Zhook Reading System, so you can now test your Zhook files. Upload them at ochook.org/publish.*

While I was doing that, Virginia was tenderly crafting the world’s second** Zhook, which you can read in chook. It’s Cory Doctorow’s For the Win, with some nice design touches.

You can also view it outside the Reading System, as an Ochook.

Of course we don’t actually recommend chook as an ebook reader — stay tuned for our Monocle-based platform. But if you’re interested in ebook experiments, well, Zhook is ongoing.

* This service is only for testing your Zhooks, and I may in a few weeks or months or years decide to shut it down or clean it up without much warning.

** The world’s first Zhook was beautifully constructed by John of The Golden Grouse. It is Edward Lear’s little ode, The Quangle-Wangle’s Hat.

Zhook, a really simple ebook format

2010-07-19T12:11:31Z

Perhaps it’s just a thought experiment. Perhaps it has a place, like YAML has a place beside XML. It’s not intended a slur against EPUB, or a vainglorious attempt to supplant it. But the more I think about books, the more I believe that the content of them is not special, and is wholly within the capabilities of the HTML and CSS specifications. All we really need is a transmissable package (zip is fine), and a bunch of reasonable mark-up conventions.

That’s what Zhook is.

Read the official specification of Zhook on Ochook.org.

The book-reading platform that we’re building around Monocle will accept Zhook uploads as well as EPUB uploads. We’ll be providing the required documentation for our Zhook support closer to launch.

My long, good-natured article about EPUB 2.1, part 3 of 3

2010-06-21T18:22:13Z

Lurking solicitously on the periphery of this entire discussion is the Apple iOS App Store, at once dark, dangerous and lascivious. The great surviving print institutions have been loudly prophesying their salvation in it. You can understand how EPUB advocates might find that risible.

I don’t think many people actually have grand delusions that EPUB will be an open standard for portable mobile applications, even just for ‘interactive print’ applications (which is an unstable distinction in any case). But it does seem that EPUB finds itself curiously in competition with the iOS SDK, and some parts of the Working Group charter suggest that EPUB’s spoiling for a fight.

It would be a spectacularly bad idea if cudgels were taken up. This fight is for HTML5, not EPUB, and HTML5 already has a fairly sane packaging format. EPUB should eventually incorporate HTML5, just because it offers greater longevity than XHTML 1.1, but it should return to the idea of packaging actual books, first and foremost. There’s a real need for an ebook specification, compact and laser-focussed, and I don’t believe we have anything like that yet.

A dire warning

So my dire warning pertains to the expansive view of EPUB, the view that sees EPUB as the vehicle for the digital future of everything that once rolled off a printing press. To do that drags the whole industry of publishing into an area where they lack expertise, and gives them shoddy tools.

What must happen is that certain sectors of publishing — including some newspapers, magazines, travel guide publishers, technical publishers and so on — must realise that they’re no longer firmly in the publishing game. Today they are software houses, with vital and precious content wings. If they keep making applications-as-books, they’ll just get trampled by real applications (web or closed).

The rest of the publishing industry is engaged in a robust tussle over a finite resource: people’s sustained attention. Books once presided over the entire supply, but (for some time now) the supply is being eroded from several angles: less leisure time, informational databases, movies and television, mainstream gaming, of course the Internet.

For these publishers, the ones that are not software houses in chrysalis, it’s absurd to compete by trying to become their competitors. If you show an interactive novel to a nine year old, she will tell you it’s a pretty crappy computer game. No-one, really, I promise you, is going to prefer a Vook to Youtube. The only sensible thing for these publishers to do is to carry the strength of their convictions and acknowledge the weight of their expertise: which is in the field of picking, refining and publishing books.

These publishers need a specification that makes the reading of ebooks — things for which their very best form is a book — as pleasurable as it can be. In order for that spec to be useful to them and their reading audience, it must declare some things out of scope.

My long, good-natured article about EPUB 2.1, part 2 of 3

2010-06-21T18:31:12Z

Some of what follows may be less than fully informed, because I’m only a casual observer, endeavouring to make a system that is EPUB-friendly if not EPUB-compliant. But it seems to me there are two key problems with the way the EPUB spec is developing.

The first is that it has no reference implementation, and in places it seems to be drawn on gossamer threads around technical obstacles that have confronted a few major implementations. Unlike HTML, none of these privileged EPUB implementations have really gained market traction yet, but as they wax and wane, they leave their detritus in the spec. It appears that EPUB also has a mostly historical habit of gorging on other formats, a habit that might once have prevented it being boxed into a corner of irrelevance, but now creates bloat and dulls the gleam of the spec. There really need not be any provision for an EPUB to contain a PDF. The XML island stuff looks for all the world like an each-way bet — it could be scaled back significantly.

The specification MUST specify

Perhaps it is due to this lack of a canonical or dominant implementation that there comes a preference for the woollier words in RFC 2119: should, recommended, may and optional. Must, required and shall are ventured less frequently than a book designer or reading system developer might hope. And this is not abating. For example, although EPUB has a working metadata solution, the Working Group charter proposes “native support” for a handful of other metadata standards — as if by providing more options, they make the standard more standard. It’s rare to see a spec quite this promiscuous when it comes to other specs, and in my prudish opinion this gets it into trouble.

As a reading system developer, of course I want to see more clarity about what to support. I shouldn’t have to work with the multiple voodoo methods of identifying the (only probable) cover image for a book. I shouldn’t have to wonder why on earth that problem, or questions regarding other obviously bookish entities — like tables of contents or font sizes or footnotes — have gone apparently unaddressed.

But nor should a book designer. I think that’s important. Eventually, and not too distantly, production quality will become a differentiating factor for ebooks in the way that it is for print books now. The muck that Adobe InDesign emits will not be considered good enough, just as exporting your website from Microsoft Word is not good enough.

Then we can hope, if this ebook thing takes off, that an activist group of actual ebook designers emerges and begins pleading for simplicity and consistency. It is the absence of this vital lobby that I think is the second key problem with the manner and rate at which the EPUB spec is evolving.

Actual ebook designers would beg for level ground. For a single recommended way to mark up footnotes. They would campaign against the fetish for strictness in the spec — which among other things tells them to use the grouchy, disowned child that is XHTML 1.1, therefore to robotically obey its concomitant media type edicts (and therefore break so many otherwise workable ebooks, at least in my testing). They would probably push for some level of interactivity, but not as a priority, not as the thing that evolves ebooks into a whole new medium.

This is because, and perhaps I’m being starry-eyed about this mostly-missing bunch of radicals, these actual ebook designers would have some faith in their ability to make reading digital books as books a sufficiently compelling experience. Which neatly brings me to my dire warning. Tomorrow.

My long, good-natured article about EPUB 2.1, part 1 of 3

2010-06-21T18:52:49Z

Last week I wrote a witty essay, incomplete at several thousand words, about why I thought the goals of the EPUB 2.1 Working Group were ‘broadly antithetical’ to a ‘sensible and stable ebook specification’. Today I threw it away. Even after two rewrites, my attempts at wit came across as disparaging, and that’s not what I meant to do.

The people who have undertaken this work and are involved in these discussions are well-intentioned and capable. It seems to me they are earnest and as much in the dark as the rest of us about the future of books in digital form. So I didn’t publish the essay. Good for me. I still have a perspective on this stuff though, and I’m going to try again with a narrower purview.

Today I’m just going to list three concerns I have with the idea of interactivity in EPUB books — which is apparently the foremost consideration on the road to 2.1. Tomorrow I’ll make a couple of observations about the EPUB specification process, as it appears to an inattentive layperson on the outside. On Thursday I’ll close it out, in my bombastic way, with a dire word of warning.

Interactions and infractions

The EPUB 2.1 Working Group charter lists fourteen “main problems” with ebooks as the existing specification caters to them. Some of the problem descriptions are, in my view, cogent and urgent. Others are not. Most are worth thinking about.

I’m specifically interested in the first one, which is about interactivity and rich media in ebooks. The charter makes the example of an interactive crossword puzzle, to be achieved by embedding executable scripts in the book’s content. Indeed, the term “book” is too narrow, because as the example implies, this is intended to attract other formerly printed material (such as newspapers) to the EPUB format. The charter continues:

“These capabilities are necessary for interactive digital textbooks and digital magazines, and more generally to enable eBooks to evolve into a new medium, rather than simply be digital equivalents of paper books.”

In the previous incarnation of this post, it was after this quote that things got snarky. I’m not going to do that now; later I will come back to why I think the ambition expressed above is unhelpful for digital publishing as a distinct and profitable industry.

But here I will just note the premise: that EPUB-lications should become interactive — and the emphasis: that this is the very fabric of the future of digital publishing.

My three counterpoints are somewhat bland and technical. Perhaps set against that big thinking they may be unconvincing or obstructionist or pedantic and irrelevant. Those possibilities were why my initial essay was so ardently but unfairly argued. This time, good-naturedly though.

Security

For my own reasons, as the developer of an EPUB-friendly reading system, security is pretty much the start and end of the discussion regarding script-based interactivity within EPUB packages.

Monocle takes book-like content and embeds it, paginated and with navigation aids, in a webpage. It is, I often say, a way of making books a part of the web, just as videos and audio are now sharable, linkable, embeddable parts of the web.

Monocle must be able to manipulate the contents of parts of the book as it displays them. In doing so, due to the nature of JavaScript security measures, Monocle must expose its execution environment to the components of the book. If these components contain malicious scripts, they could exploit this open access to pinch private data out of cookies or other properties of the page state. Even if one managed to trap EPUBs in that security net, positing some future unidirectional window access or something, that doesn’t stop them annoying users with prompts, pop-unders, et cetera.

Since Monocle is an open solution, we’ll leave this choice in the hands of implementers. Scripts will not be stripped from book components, and component iframes will not use the proposed sandbox attribute. If you use Monocle and you absolutely trust the EPUBs you’re displaying in it (because you created them, for example), you don’t have to disable scripting. But the platform we are quietly building around Monocle to make book embedding easy: this will absolutely strip scripts. Trust is simply of greater importance to us.

Of course, that reason is Monocle-specific. Although for me it is the decisive consideration, we should move on. Next.

Interaction models

When you think about it, there are only two things that a reading system does with EPUB over and above a web browser. One is that it unpacks the somewhat byzantine file into constituent components, serving them up in the correct order. (This feat is perhaps less remarkable than you think.) The other is that it provides an interaction model between the person reading and the HTML being rendered.

What’s an interaction model? It’s the thing that might eventually make reading ebooks a joyful experience. It’s the flappy, wobbly, paper-like experience of turning a page in iBooks, which has probably done as much as anything to introduce less-than-avid readers to the notion of digital books. It includes the very idea of pagination itself, which is absent (or differently meant) in HTML.

Contemporary ebook interaction models include turning pages, scrubbing to different parts of the book, selecting and annotating text, creating bookmarks, magnifying text or adjusting the brightness (which in Stanza, for example, may be controlled with a simple vertical swipe), clicking internal links to advance or to view a footnote, et cetera. The experience isn’t very rich yet and ebook reading systems still have a lot of work to do making these models intuitive and enjoyable. So far the specification has mostly and properly stayed out of it. But the notion of “interactivity” scatters these pigeons like a large cat.

If that crossword puzzle was coded so that a downward swipe displayed the clue for 8-Down, Stanza’s brightness control would not work. The Kobo Reader, to which many people here in ebook-device-starved Australia have flocked, would not do anything, because it has no touchscreen or pointer.

The point is not that there’s an engineering problem, at least not for hobbyist reading system developers like me. With Monocle I could pick and choose which interactions go through to scripts and which are reserved for the system. Given Monocle’s open design, my hope would be to leave as many of those choices with implementers as possible.

The point is the uncertain landscape that emerges for book designers. The device capabilities and interactions models of existing reading systems are wildly divergent; far more divergent than the desktop web browsing landscape, for example. And what we’ve seen over the last decade on the web is that where features have uneven implementations, web designers have generally avoided them. They have fairly and reasonably stuck to the middle path of wide cross-browser support. In other cases they have favoured the dominant implementation — but for ebooks, there is yet no such thing.

That’s my second point, and a good part of my third point too: inherent inconsistency in the implementation will make things very difficult for book designers who attempt significant interactivity.

Implementation minutiae

This is the most bland and technical of all my bland and technical points. But I really think it is the rock against which interactivity in EPUB dashes itself, at least until the (not particularly improbable) day when reading systems are developed by software architects and engineers much cleverer than I.

At this time, more and more reading systems are being built on top of dedicated, drop-in renderers like WebKit. It seems clear that all the momentum for JavaScript-based interactivity comes from WebKit’s capabilities. You can see why: its HTML5 parser and feature-set, the typographic and animation possibilities arising from CSS3 standards and de facto standards as well as the canvas element, and its fast, featureful JavaScript engine. Why wouldn’t you unleash that?

However, even among the handful of reading systems built on top of WebKit of which I have any knowledge, the implementation details of those systems rupture the consistent surface that the renderer appears to provide.

Here is the problem: between loading a component out of an ebook and displaying it, every reading system modifies the shape of the document data (the DOM). Some do a lot of modification, some do as little as possible. The reasons they do it are generally good and entirely implementation-specific. To take a now-retired example, Liza Daly’s trailblazing epubjs goes looking for places to split paragraphs for the purposes of pagination. It creates new paragraph elements as necessary. The technique is ingenious, but since it modifies the DOM tree of the document substantially, it makes for unstable ground for scripters.

Nor do I mean to pick on epubjs: every implementation tweaks the DOM in its own way. Monocle’s only interesting advance on epubjs was to get the browser to do the pagination, using CSS columns. So in that regard, Monocle modifies the DOM very little. But Monocle’s default “page flipper” is configured to display two overlapping iframes, so that as the page you’re reading turns, you see the next page underneath. Even naively, what this means is that you have two execution environments for scripting, which do not share state. Increment an integer in one frame and it is unchanged in the other. It would be very difficult, even for brilliant book designers, to account for that.

My guess, and it is only a guess, is that iBooks renders the current component to an off-screen WebView surface, then essentially blits distinct portals of that content onto a “paged” visual metaphor. If so, this would probably be the purest, least DOM-invasive technique I know of, but even in this case a book designer would need to be aware of many specific implementation details. Similarly for every other reading system, and more so for the very customisable systems like Stanza. Modifications to book CSS, such as Stanza loves to make, will be sufficient to break many scripts.

The troubling outcome here is not that interactivity will be ignored by book designers. What bothers me is that if book designers adopt these features, they will do so for a particular, presently dominant reading system. They will simply have to choose to support the biggest or the luckiest reading system. Their books won’t work, or won’t work as well, on anything else. Innovation will in this way be stifled, not stimulated.

The market and designers will probably do that anyway, but part of the role of the EPUB specification is to avert that.

Which brings us to some observations on how the spec is being specified. Tomorrow.

When books ain't books

2010-03-20T09:32:37Z

An encyclopedia contains many lessons, but the form of the encyclopedia itself is a useful lesson. Its modern shape was drawn in happy innocence at the dawn of the Enlightenment; after two centuries of the primacy of science and inquiry in the West, it had swelled to consume entire bookshelves. It became ponderous and uncompelling, and actually quite hard to use. So, in the early nineties, encyclopedias were among the vanguard of that great gold-rush: multimedia, and specifically, CD-ROMs. Of all the publishing ventures of the CD-ROM age, multimedia encyclopedias (namely Britannica and Encarta) might be the one example of something that approached critical mass.

In the process, their vaunted pages lost something of their power: the very heft of human knowledge you felt pulling a volume from the shelf was not easily translated to dropping a disk into a tray, double-clicking an icon, waiting, and pecking out a search term.

That lost sensation was mourned, but it was not such a bad thing. People came to see an encyclopedia for what it was — not necessarily an intimidating (and quickly dated) academy in printed form, but actually just a database of information about stuff. You could copy and paste a paragraph out of Encarta and into your essay — like a lot of not-so-smart students did in the CD-ROM era and still do now. Of course, smarter kids copied, pasted, then judiciously modified the text in places to cover their tracks. In losing its printed form (and no-one, surely, buys volumes of encyclopedias anymore), it gained a truer form: it left the ivory tower to mingle in the bazaar. In a hop, skip and jump, we had Wikipedia.

How heavy is Wikipedia? How many metres wide is it? Who wrote and edited it? We don't ask these questions. We do ask how reliable it is — but as a kid in the eighties, my family had Britannicas from 1964. According to that arbiter of truth, the moon was a distant and untrammelled body, a virgin of the night sky. This is why Wikipedia is the most reliable encyclopedia ever written.

The point is, the encyclopedia is an idea that was necessarily limited by its form. It had to iterate through a number of forms to arrive at its present (and perhaps not final, but still infinitely better) form: a web application.

Don't think that makes it a special case. Though it's a reference, it is a particular kind of reference — one that suits the model of sitting at a desk and making epistemological leaps of faith. Another kind of reference, the cookbook, has hitherto been poorly suited to a web incarnation, because nobody wants to coat their keyboard and mouse in flour and butter. A book, a book that holds its page when placed on a bench (so a weighty tome and a particular stock), is a much better form. But actually, the clip folder and plastic sheets are a better form again, because the collateral damage of an ordinary kitchen requires some preventative measures. (The internet serves that form very well — with the aid of a printer, of course.)

But technology promises a still better form, because recipes are what a geek might call well-normalised data. Imagine that I have a few items in the fridge that I want to cook up. With some legwork in the indices of various cookbooks, I can locate a recipe for those ingredients. If a cookbook were a database, I could enter those ingredients and quickly see a list of all possible recipes. Or I could look at the date, discover what fresh food is cheap in mid-autumn, and compose a shopping list for one or two specific recipes. Or I could do many other things besides — such as tap to double the quantities, or whatever. The work to do is to bring the technology into the kitchen, and kitchen-proof it, but that's not hard.

If you were developing an ereader, and you had digitised a few novels and made the experience good, you might come to a cookbook and wonder how best to display "that kind of book" in your system. You would probably come up with all these possibilities and more. And your answers might be clever, but the question is wrong. The cookbook is moderately well suited to bound paper, but poorly suited to a Kindle. You aren't inventing the future, you're just inventing a stopgap until someone comes along and does it properly — by giving the collection of recipes and folk wisdom a truer form. Your stopgap is like Encarta, if you're lucky.

I am travelling through San Francisco for a few days, and I have the Lonely Planet application on my phone. The app pays obeisance (too much obeisance) to the bookish origins of the data, but it does at least realise that it isn't a book at all. It's an application. Though I curse the poor implementation of it in the Lonely Planet app, a travel guide is wonderfully well-suited to contextual, database-driven assistance in the form of an app. At any moment I can pull out my phone, and consult the map without looking like a tourist, finding at a tap my precise location, and the directions to the nearest decent bar. If I had the bound paper version, I would get by, but thirstily.

Travel guide publishers, in fits and starts, are getting it. Their asset is not their books, but their commissioned knowledge. And their knowledge, freed from the contraints of a linear page order and clunky, painstaking indexing, is much richer and more valuable in another form.

It's not the same adventure as for encyclopedias, whose patrons are mostly sedentary and browsing on a large screen, but it's the same conclusion. Linearity is a liability. Heft is unhelpful. Books sometimes, not always, but discernably, are not books. For reference publishers, the future shift is much more dramatic than it is for common fiction, opinion and narrative writing. Don't pander to them, don't try to quell their anxiety by pretending their books are still books, you're doing them no service and wasting your time. Concentrate on the things for which the last page must come after the first — that problem is plenty hard enough.

Announcing Monocle

2010-03-08T16:19:41Z

Monocle is an ebook reader. It works in modern web browsers, using standard technologies. It runs in desktop browsers and on mobile devices. It's awesome in Safari and Chrome (the browsers used on iPhone OS and Android devices), increasingly awesome in Firefox, and workable in more standards-fearing browsers like Microsoft's Internet Explorer.

Try it out: http://monocle.inventivelabs.com.au

We've open-sourced it under the MIT licence, which is the most liberal of the mainstream software licences. It can be used by anyone, for private or commercial use, for free (as in speech and as in beer) — as long as they retain the licence information in any copy of the software.

That is, we're giving it away.

Why are we doing this?

We're concerned that the current ebook landscape is dominated by large companies trying to lock publishers, authors and readers into their devices and distribution models. We suspect this might be good for those companies, but no good for everyone else. There's a need to open up these fields to smaller players, and to give the consumer — the person who loves to read books and to 'have' books — better choices.

We expect ebook reading devices to steadily converge with other communication devices. In the past e-readers have been fairly isolated from the network, but as they gain connectivity, consumers will demand better access to the web. Even Amazon's Kindle has stripped-down HTML rendering. Apple's iPad has one of the most advanced browsers around.

The one single platform we expect future e-reading devices to have in common is the web browser. If you want to give your readership the freedom to own (forever) the books they buy from you, the web is where it will happen.

Monocle is unique in providing this functionality. Monocle is also unique in giving publishers and retailers a way to design not only the book's content, but the way it looks and feels too. It's simple at heart, but has enormous flexibility in the way tables of contents, page numbers, running heads, and bookmarks work — even how it feels to 'turn' a page.

Because it's delivered by a web browser, the file format question really doesn't arise. Any text-based format your books' data is in — Monocle can be made to display it. Integrating it with the EPUB standard is straightforward (we've already done it).

Monocle also sidesteps the DRM question, because no files are passed around. If you want to put Monocle behind a paid membership wall, or restrict it to particular devices or IP addresses, that's easy to do. But it's not required by the technology.

If you think Monocle could be an important part of the way you publish digital content, feel free to get some techies exploring Monocle's source code. Or for expert consultation on an e-publishing strategy and workflow with Monocle, drop us a line. You'll find our contact form on the About the Labs page.

The magic behind Castanaut is open source

2010-02-24T12:41:48Z

Two years after we released Castanaut, I've decided to open source the little command-line utility that powers keyboard and mouse input within Castanaut.

It might have want of a better name, but it's called osxautomation because that's what I called it. Here's the github repository for osxautomation, with full compilation and usage instructions in the README file.

Get your Notepods!

2009-09-11T15:55:10Z

Inventive Labs has teamed up with the Jacky Winter Group to produce Notepods - sketchbooks for your app ideas, in a package reminiscent of your favourite smartphone.

It's incredibly fun to come up with an idea in the pub over a few beers and have it produced in the space of ten short days, website and all. Get yours now!.

Peeking at previews in Blueprint

2008-12-26T17:25:49Z

I just uploaded a little Quicktime movie of a recent change to Blueprint: spring-loaded previews.

It's only a small change, but one that has been on my list since, oh, the 4th of December 2006 — when our version control system tells me I started working on the 'drafting' functionality that is at the core of Blueprint.

Formerly, you just clicked the Preview tab to check your changes before saving them. The preview would pop up in the same grey frame you see in the movie. The recent change was to animate the transition — and to allow you to "pull" on the tab to "peek" at the preview.

Like I said, it's a small change, and you could argue it's mere eye candy. But it has stayed on my list of things to do because exactly these sorts of user interactions give an interface its handling — the sense that you're steering it, rather than poking at it. From this, the people using the system develop a confidence and an instinct.

Migrating from Lighthouse to Redmine

2008-12-15T17:46:27Z

We love Lighthouse, but here at the Labs our needs have begun to change and Lighthouse no longer feels like a best fit for us. So we've upped stumps and moved to Redmine, which is less flashy but better complements our workflow (such as it is).

I took a couple of hours to write a rake task that politely exploits the Lighthouse API and imports almost all available data from that system into a Redmine instance.

Here it is:

Download file (5Kb)
lighthouse_import.rake

Steps:

Install the task file into lib/tasks in your Redmine directory.
Add the lighthouse-api ruby library as a plugin. This essentially involves running the following command from Redmine's vendor/plugins directory:
```
$ git clone git://github.com/Caged/lighthouse-api.git .
```
If you haven't already:
- create your database and configure your database.yml
- rake db:migrate
- rake redmine:load_default_data
- Log into Redmine and create the projects that will contain the data you're importing from Lighthouse. Configure them however you like — in order for the import to happen, they must be named exactly as they are in Lighthouse.
- You can similarly create users and assign them to these projects. Any users with matching names to Lighthouse users will be recognised by the importer.
In Lighthouse, set up an API token if you haven't already. This can (and should) be a read-only token.

Now run the script:

$ rake import:lighthouse LH_ACCOUNT=abc LH_TOKEN=xyz...321

Obviously the LH_ACCOUNT and LH_TOKEN environment variables should be entered as per your Lighthouse account. You can also pass in CREATE_USERS=1 to actually create user accounts where they are referenced in the Lighthouse data — they will have dummy names and email addresses, and their passwords will match their login names. If you don't pass in this flag, any unrecognised user references will be assigned to the first admin user.

The script could use a little refactoring, but it worked very well for our needs. Let me know how you go.

A Bloggie for Ms Fits

2008-03-18T15:43:13Z

The fabulous Reasons You Will Hate Me has just been honoured with a Bloggie for Best Australian Weblog. Inventive Labs designed Ms Fits an appropriately sexy new site in the middle of 2007, and built a custom content management system to suit some of her specific requirements - in particular, the now world-famous 'Friday Q&A'.

Congratulations, Ms Fits!

The other things

2008-02-25T10:48:33Z

We do have a habit of saying how busy we are at the Labs, to the extent that it's almost a little obnoxious. We do, in fact, occasionally find time to craft software that is not for particular paying clients, but has a wider audience.

Fridge, our Facebook app, is an example of this. That was a fun project, and the reception has been really rewarding. I worry about the commercial trajectory and privacy implications of Facebook, but in developing Fridge I gained a high regard for their technical platform.

By far our most popular project is Tabulate, released in October last year, for iPhones and iPod Touches. It gives you a slick interface for, well, power-browsing in multiple windows with MobileSafari. Tens of thousands of Apple device owners are using it.

Also for iPhones, released by stealth and only shortly after Tabulate, is Kapow! -- a digital comic book viewer. I think only about three people in the world use it, but they seem to love it.

On the weekend we released another project that's been in (mostly weekend) development for a month or so: Castanaut. In technical terms this is a Ruby DSL for writing executable scripts that control input and application interaction during screencast recording.

In layman's terms it's a way you can take the screencast notes that you're already writing down on paper before recording, double-click them, and have a screencast saved and ready for publishing. It's kinda freaky.

The many tentacles of Inventive Labs

So, we've had a recent proliferation of subdomains here at the Labs. Two of them may interest you:

Gadgets: this is where we put the publicly available goodies we develop.
Wisdom: a more frequently updated stream of mostly technical but occasionally humourous objects that preoccupy us.

Eight days at the Labs

2007-11-27T12:31:21Z

One of the items on our extremely well-populated todo list is to update our Portfolio page. It'll happen shortly, but meantime I thought I'd give you a quick look at three sites we've launched in the last eight days.

Several months in the making, the new Readings website gives this iconic Melbourne business the online stature it deserves. Integrating with their inventory and ordering systems, the site makes fascinating use of "product collections" to recreate the serendipity of browsing in-store. Added to that is a powerful and greatly simplified search, regular postings of events, news, product reviews and downloads, and a shopping basket that is secure, intuitive and remarkably fun.

We're rather proud of the Readings website, which we built from scratch in Rails.

BestQualityCrab is an unusual experiment in the convergence of blogging and geography. We constructed the site in Blueprint for a small cohort of Melbourne academics, who explain it like this:

It starts from the belief that blogs are usually created around time-centred ways of organising thoughts and narratives. BestQualityCrab starts from the idea that blog thinking and narratives can be organised in terms of space and place as well as time.

Blogs are rarely quite this interesting to explore. In fact, we think BQC might herald an entirely new category of weblogs.

Wired Consulting is a new Australian telecommunications consultancy with a remarkable pool of talent. This is our second collaboration with David Koopmans of Mokum Marketing, and once again shows the flexibility of Blueprint.

Three big sites in a little over a week is not bad for a team of two inventors. Obviously our pace isn't always so quick, but since August we've launched 14 Blueprint sites of varying complexity. And you wonder why we love our little CMS so much.

Scheduling periodic tasks in a Rails application

2007-11-08T17:49:28Z

There's a number of solutions around for forking out of the Rails http request cycle and performing intermittent or long-running operations within your Rails application environment. Some of the most well-documented options are summarised on the Rails wiki.

Looking for a way to schedule actions like 'fetch this feed every 15 minutes', 'clear out old sessions from the db', and 'take a snapshot of each site' for Blueprint, I investigated most of these options.

In the past I've used the reliable cron/runner combination, but this requires additional setup -- outside of your app -- on each server where the app is deployed. There are recipes out there for updating the cron table via Capistrano, but this is difficult to do in a way that preserves existing cron configuration. BackgrounDrb and AP4R both look accomplished, but are really intended to solve other (and bigger) problems.

OpenWFEru is a Workflow Engine, which sounds impressively enterprisey. I don't know much about that, but it comes with a pretty good scheduler — check out the API for it here.

This can be used to create a cron-like scheduler within Rails. I wanted the scheduler to run as a separate process alongside the pack of Mongrels that handle web requests, started and stopped by Capistrano. Ruby has a great 'daemons' gem that takes care of daemonising processes for you.

If you need something like this, install these two gems:

$ sudo gem install -y openwferu daemons

Then create script/schedule:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
APP_DIR = File.join(File.dirname(File.expand_path(__FILE__)), '..')

Daemons.run_proc(
  'schedule',
  :dir_mode => :normal, 
  :dir => File.join(APP_DIR, 'log'),
  :multiple => false,
  :backtrace => true,
  :monitor => true,
  :log_output => true
) do

  # Daemonising changes the pwd to /, so we need to switch 
  # back to RAILS_ROOT.
  Dir.chdir(APP_DIR)

  # Load our Rails environment.
  require File.join('config', 'environment')

  begin
    # Initialise the OpenWFE scheduler object.
    require 'openwfe/util/scheduler'
    scheduler = OpenWFE::Scheduler.new
    scheduler.start

    # Now assign jobs to the scheduler (see API). For example:
    scheduler.schedule_every('15m') { Feed.fetch_and_consume! }

    # Tell the scheduler to perform these jobs until the 
    # process is stopped.
    scheduler.join
  rescue => e
    RAILS_DEFAULT_LOGGER.warn "Exception in schedule: #{e.inspect}"
    exit
  end
end

Chmod this file to be executable, then test it:

$ chmod a+x script/schedule
$ script/schedule start

Monitor the file at log/schedule.output -- most errors will be reported here. It will log to log/<environment>.log. Run 'script/schedule' with no arguments to see how to stop and restart it, and do other daemonoid things if you like.

You can then create a task within your Capistrano deployment recipies along the lines of:

task :after_restart do
  run "RAILS_ENV=#{rails_env} #{current_path}/script/schedule stop"
  run "RAILS_ENV=#{rails_env} #{current_path}/script/schedule start"
end

Using Blueprint as a Drafting Tool

2007-09-11T10:50:03Z

A couple of weeks ago I spent a day designing and building simple default templates for each of Blueprint's core page types ('blueprints'), ostensibly to help us demonstrate the system more easily to clients and friends. There are currently ten generic blueprints – Page, News, Links, Search, Events, Contact, Projects, Blog, Gallery and People – and each of them has one or more templates, all mixing a bit of logic with some HTML. There are companion javascript and CSS files, too.

A by-product of completing this task is that Blueprint has gone from being something I started working with after the architecture of a new site was established (most often with pen and paper, followed by laborious and under-appreciated site maps put together in ~~Illustrator~~ Lineform), to being an indispensible tool in the drafting process: I can create a working site prototype in mere seconds, and manipulate it quickly within the drag-n-drop interface, testing hypotheses and seeing roadblocks much sooner than if I were starting from sketches.

Obviously this isn't a substitute for thinking through the more complex IA requirements of larger sites, but for a typical small Blueprint site it's both appropriate and efficient, since whatever is achieved at this stage is carried over seamlessly into the development.

It's also considerably easier for people we're working with to understand, since the results have the dimension and interactivity of a finished site.

Like that but with this: view inheritance in Rails

2007-09-05T16:47:16Z

Yesterday I wrote about a simple method that lets you to place default content in your Rails layouts. This default content can be optionally overridden by your views using the standard-issue content_for.

It's handy enough on its own, but it's also a piece of a somewhat larger puzzle — something I've called view inheritance. The principle is that, just as Ruby classes can extend from parent classes, so ERB templates should be able to extend from parent templates.

Consider this template, a.html.erb:

<html>
<body>
<% default_content_for :title do %>
  <h1>Bonjour</h1>
<% end %>
</html>
</body>

Now b.html.erb, which inherits a.html.erb:

<% inheriting_view 'a' do %>
  <% content_for :title do %>
    <h1>Aloha</h1>
  <% end %>
<% end %>

That says "do everything that the first template does, but make the title say 'Aloha'".

Of course we're not doing much different from what a layout does yet — we've just shifted responsibility for specifying the layout file from the controller (or outer view) to the view itself. Sometimes that's not very useful, and although view inheritance can be wielded as a general replacement for layouts, I'm not really recommending it.

But what about this scenario? You have a model X, and an XController that serves up a form for X. Then you need a model Y, which is like X but with a few extra fields. So you create the Y model:

class Y < X

And perhaps a YController:

class YController < XController

But how do you add the extra fields to the form? Like this:

<% inheriting_view 'x/form' do %>
  <% content_for :additional_fields do %>
    <fieldset>
      <legend>Geography</legend>
      <label>Latitude
        <%= @f.text_field('latitude') %>
      </label>
      <label>Longitude
        <%= @f.text_field('longitude') %>
      </label>
    </fieldset>
  <% end %>
<% end %>

So DRY you'll need a beer when you're done.

Here's the code:

  def inheriting_view(options = {}, &block)
    # We accept a shorthand syntax -- if options is a string, render as a file.
    return inheriting_view({:file => options}, &block) if options.is_a?(String)

    bind = options[:binding]

    # Get our differences and additions to the view we're inheriting.
    if block_given?
      bind ||= block.binding
      instance_variable_set(:@content_for_layout, capture(&block)) 
    end

    raise "Important: inheriting_view() requires a block. " +
      "An empty block (eg, using {}) is suitable." unless bind 

    # If we're inheriting a partial, lend our local context to that partial.
    options[:locals] = eval("local_assigns", bind) if options[:partial]

    # Render our parent view.
    concat(render(options), bind)
  end

(Yes, this works for partials inheriting other partials too.)

Providing default content in Rails layouts

2007-09-04T11:11:10Z

Another quick tip for Rails view developers. We commonly employ the following snippet in our layouts:

<%= (out=yield :sidebar) ? out : render(:partial => 'sidebar') %>

This thorny little one-liner says, if the view we're rendering for this layout has content_for :sidebar, show that content. Otherwise show the default content (which we've tucked away in the 'sidebar' partial).

It's a handy pattern because it lets your views opt to modify some aspect of the layout, without having to — they can go with the default sidebar content if they don't care.

However. Firstly, it's a bit unsightly. Secondly, you must put the default content in a partial (unless it's short enough to just be a string — an approach that is arguably uglier). In most cases, the default content for a block in a layout is best described in that layout, so that you don't have to go hunting down other files to get a picture of what the layout displays. Here's a better way:

<% default_content_for :sidebar do %>
  <p>This is the default sidebar content! (et cetera)</p>
<% end %>

So now, if the view specifies some content_for :sidebar, the contents of the block above (ie, the <p> tag) will not be rendered. If the view doesn't do that, then the layout will render the contents of the block.

If you'd prefer this idiom in your views, drop the following method into application_helper.rb:

def default_content_for(name, &block)
  name = name.kind_of?(Symbol) ? ":#{name}" : name
  out = eval("yield #{name}", block.binding)
  concat(out || capture(&block), block.binding)
end

This method is full of bindings and captures, which you can read up on at your leisure, or feel free to ask questions in the comments.

The writing's on the magnetic wall

2007-09-03T14:47:58Z

Many, many moons ago (thirty-seven, to be exact), I made a little DHTML application called The Fridge. You had a bunch of word tiles that you pushed and shoved into the shape of a poem, much like Magnetic Poetry on fridges all over the world. It was good fun and much loved by many people, but with the passage of time the flocking crowds dwindled down to a core of devotees. And then, as is the way of things, they too found other pastimes.

Last week we took a couple of days out at the Labs to turn The Fridge into a Facebook application. This manifestation is even more fun than the last, because you can now create your own sets of words and share them with others. Also, crafting miniature epics on your friends' walls for posterity is a rather satisfying pursuit.

Here's what it looks like:

If you have a Facebook account, take it for a spin.