Arto Bendiken - Lisp

Rediscovering the Lively Road Not Taken

Arto — 2008-05-27T22:00:00Z

Speaking fondly of an operating system usually misconstrued by a younger generation as merely an ancient text editor older than Stallman's beard, Steve Yegge states that "Emacs is the world's last Lisp Machine. All the rest of them are at garage sales."

While that's indeed the case in a strict sense, in a slightly wider sense the essence of the legendary Lisp machines has managed to survive to the present day in at least one other closely related fork in the road, namely the Squeak project. And now, from Dan Ingalls — the original designer of Squeak and one of the fathers of Smalltalk — comes Lively Kernel, a self-hosting, metacircular prototyping and development environment implemented in JavaScript, a programming language that itself has considerable Lisp ancestry.

Ingalls and the Lively Kernel team aim at nothing less than a bottom-up rethinking of the present-day web development stack. They've recreated, in less than 10 KLOC of JavaScript, a foundation that demonstrates the benefits of systematically and unapologetically examining your assumptions and mercilessly throwing all possible incidental complexity out the window.

Lively Kernel is still at a very early stage, but watching Ingalls's presentation at Google and subsequently playing with the system I experienced much the same "wow factor" as when first encountering Squeak many years ago. Here's one phrase I would like to hear more often: "it's not complicated; it's really easy". No cathedrals being built, here!

True to its intellectual heritage, the Lively Kernel system is "live" in a manner fundamentally unlike even the best mainstream programming environments today. Ditch all existing preconditioned primitive notions of tedious edit-compile-run-crash-debug-rinse-repeat cycles, and jump right in to reprogram the system from the inside out, your changes instantly taking effect as you go along (there is a good demonstration of this in the aforementioned video when Ingalls, on the fly, reverses the time flow in a running clock widget).

As anyone who has used Squeak knows, the immediacy of this way of working is addicting, the unfettered malleability of pure thought stuff giving rise to analogies of, say, sculpting clay &mdash a stark contrast to the all-too-familiar constant friction and resistance of a bureaucratic chain of command founded in long-obsolete metaphors of folders and files, or the often arbitrary and unproductive distinctions between compile time and runtime.

This is powerful stuff, and yet the implementation code is, as Ingalls says, very straightforward indeed. The core of Lively Kernel is a reimplementation of Morphic, John Maloney's ingenious user-interface framework that also formed the basis for both Squeak and an earlier Sun Labs research project, the Self system — the latter which, incidentally, was another major influence on the design of JavaScript. The graphics primitives in Lively Kernel are based on SVG, which means that, as in NeXTSTEP and Mac OS X, all output is inherently scalable to any desired resolution. (There is an in-depth technical overview available for those who want to dig deeper into Morphic and Lively Kernel's architecture.)

One marked difference to Squeak appears to be the abandonment of a pure image-based environment, where the current snapshot of the code base would be stored e.g. in an opaque, system-specific binary format. While serialized "live images" have historically been one of the distinguishing features of Smalltalk systems (and indeed Common Lisp systems), they have not contributed to interoperability with more primitive systems (i.e. Unix) and hence have served as impediments to more widespread adoption.

The near-term roadmap for Lively Kernel seems to include WebDAV-based over-the-wire serialization, which suggests to me that e.g. Subversion (specifically, Apache's mod_dav_svn) could eventually serve as a Lively Kernel repository. If so, this would preserve the considerable advantages of image-based environments while doing away with the major perceived disadvantage.

Of course, given that I'm praising simplicity here, I would be remiss not to mention that WebDAV and DeltaV do suffer from considerable complexity bloat (as I know all too well from painful close-range exposure), and a much simpler REST and JSON-based alternative is not difficult to imagine. However, at some point you do have to interface with the external world in all its decadent glory; I think the ability to piggyback on an existing revision control system might be a worthwhile trade-off in this case.

The Lively Kernel interface at present still feels slightly sluggish, but the project's timing is most opportune. JavaScript is already today one of the most widely-deployed programming languages of all time, with an existing installed base in the hundreds of millions. The imminent next generation of browsers — Firefox 3 and IE 8 — will bring some pretty significant overall JavaScript performance boosts, not to mention that some Firefox version a little further down the line is set to integrate Tamarin, the native-code-compiling JavaScript 2 engine that will run circles around many of the sundry popular "standalone" scripting languages. JavaScript is clearly set to become the lingua franca of the next decade, and Lively Kernel provides an early glimpse of the ride we may be in for.

Yegge remarks "Lisp Machines pretty much had it nailed, as far as I'm concerned", to which I can only say: amen, brother. It seems a peculiar world where our best hope for advancing the state of the art consists of merely attempting to duplicate half-forgotten, decades-old good ideas in newer more-devolved programming languages, but such truly is the myopic, amnesiac reality of "worse is better".

On the other hand, efforts such as Lively Kernel remind us that heretofore passed-by opportunity lies right off the beaten path, on the divergent grassy road just wanting some wear. To paraphrase Ingalls's long-time collaborator, the equally industrious and illustrious Alan Kay: it seems that in an industry stuck on a repeat loop of variations on Greenspun's Tenth, the best way to predict the future may simply be to rediscover it.

A Brief History of Cutting Code in a Universe Three Doors Down

Arto — 2008-03-28T22:00:00Z

Eventually upon a time, in a universe far far away — or three stack frames down the line, to be precise...

Assembly is the ur-knife, known to man from the dawn of time. The flint edge is very sharp, but uneven and brittle. These tools can be fashioned both into knives for cutting and axes for hacking. In either case the edge will wear dull soon. Rarely are these venerated instruments now brought out in the modern day, and when they are, there is great ceremony involved. He who has the fortitude and stamina to hack down a Big Tree using a mere assembly axe, all the while uttering the sacred ancient mantra the meaning of which is now lost, will be much respected among his peers - though perhaps not considered entirely sane.

BASIC is a children's game that remains popular among youngsters. These plastic bricks can be quickly assembled together to form a toy with a certain charmingly crude, and entirely non-lethal, resemblance to an adult's blade. But beware of gravity, little one, as with too long a blade your construction will surely shatter at the midpoint. Some time-honored traditional techniques for counteracting this brittleness are chewing gum and duct tape.

C++ is the leading brand of gas-guzzling, noisy chainsaws. It'll fell the tree quickly all right, but needs serious muscle and experience for correct handling. The attrition rate among woodcutters is high, with many junior apprentices never returning from their first trip to the woods. Even senior loggers are often missing the occasional superfluous limb or two, but no matter: they've developed the forbearance and impressive skill to properly handle these beasts in single-handed combat. As they say, good judgement comes from experience, and experience comes from bad judgement - if you survive it.

Java is now the most widely-used electric kitchen knife in the world. To prevent people accidentally cutting themselves up too badly, the blade is manufactured to a guaranteed consistently dull quality. Indeed this tool was originally developed in response to all the woodcutting accidents taking place with C++, which is perhaps curious considering that its safety features make it somewhat tedious to hack at anything more ambitious than saplings. But, safety first or not, it remains splendid at cutting bread.

Smalltalk is a voodoo stick. Wave it over a carefully constructed model of the real world, and stuff happens - trees fall, timber is cut to spec, and paper flows freely. It has long been a jealously guarded trade secret how this works, exactly, but we do know that the inventor is associated with Lisp - which we'll get to in a minute. Unfortunately, this powerful technology was and remains so far ahead of its time that it languished in obscurity long enough for the patent to expire, allowing some eastern philosophers to popularize a commercially highly successful clone. Catering to slightly more vulgar tastes, this clone may have sacrificed some of the original aesthetics, but not much of the seemingly supernatural power.

Forth self-assembling nanoblades represent the keenest baryonic blade edge currently available. Formed out of femtoscale building blocks that spontaneously stack themselves into arrangements of factorially increasing complexity, these axes will hack through a trunk with a single heave. They will also continuously and autonomously repair themselves so as to maintain the perfect edge down to the molecular level. The best neurosurgeons now refuse to use any other scalpel, and the razor blades made of this material provide for the closest shave you've ever had - just watch out that you don't shave off a little extra.

These are all ingenious inventions of man. But of all the cutting and hacking devices known to us, there is one exalted above all others, and it lacks an earthly origin.

Lisp is, as far as we can determine, magical. Obscure in origin but manifesting itself in our corporeal reality in the form of lightsabers, the crystal nanocircuitry of these devices appears to transcend the ordinary laws of nature as we presently understand them. Infinitely sharper than any scalpel, indestructible under a thousand atmospheres of pressure, these elegant instruments are made of an unknown alloy and keyed to their owner in such a way as to ensure they will never cut him.

Brace yourself when you initially receive yours from your elders. When you grasp the 'saber in your hands for the first time, parentheses and other cuneiform symbols will flicker in and out of your inner vision, and your ears may ring with thunderclaps not unlike the sound of continuous stack overflows. You may indeed have reason to wonder if the red pill you washed down with that lambda-tinted kool-aid at your initiation ceremony wasn't some form of psychedelic drug after all.

Fear not; while research is still ongoing, this is thought to be a harmless side-effect of nanites that have slipped into your bloodstream through the handle and are reconfiguring your brain to accept their base code. The symbols you may see are hypothesized to be a transient interface adapter between the human visual cortex and the alien technology of the 'saber, quickly supplanted by some more direct mechanism that we haven't quite figured out yet.

If you persist in confusion for a moment longer, the hieroglyphics will soon begin to fade into the background, becoming a subconscious part of your inner vision and lending your perception an acuity you didn't even know was missing. You will perceive not merely the apparent hierarchy of nature but penetrate yet deeper to the underlying infinitely more subtle relationships and interconnections between things.

Once the symbiotic link with your 'saber is fully established, you will be able to, at will, slice through the molecular bonds of any Gordian Knot as if they were made out of air. Given that you may will the width of the blade's swath to reconfigure itself from picometers upwards, your 'saber is easily the most precise and capable instrument known to man. The rush and elation, and yes, the pride, of wielding a lightsaber is enough to make anyone giddy. You should understand that with great power comes great responsibility, and you are advised to review the ethical code from time to time.

After further meditation and practice, you may eventually come to understand that it is not the blade that cuts, but your mind that sculps reality according to your imagination. This insight is reported to be the last thing uttered by some of the legendary Lisp grand masters before they, somehow, cut the ties that bound them to our sphere of reality, thus vanishing into thin air.

There is some speculation that, after contemplating year upon year on the grid lines that all Lispers are able to perceive in the night sky, these Lisp masters held infinity in the palm of their hand and found a nearby quantum activation frame, unlocking a suspended gateway continuation that allowed them to ascend to some higher plane of being that, evidently, encompasses and sustains our own.

A competing hypothesis among those of a more hedonistic bent is that perhaps the vanished masters managed to construct a private, entirely self-contained lexical scope, i.e. sub-universe, of their own, a form of personal nirvana obeying their own choice of evaluation rules and containing the promise of limitless futures to explore and indulge in.

There is only one way to find out - apply for a wizard's robe today and begin your own journey from caveman to first-order hacker of the heavenly spheres in this universe and beyond.

Finding Clojure, or How I Learned to Stop Locking and Eval the JVM

Arto — 2008-03-26T18:00:00Z

Ever since the Scheme R⁶RS roadblock, I've been looking to branch out into other Lisps outside the Scheme ecosystem. The good news is that there's no shortage of nascent Lisps and Lisp-wannabes, with many interesting new undertakings being launched month upon month.

There's one in particular, called Clojure (pronounced closure), that was unveiled recently and effortlessly stands out from the bunch. There's much to like about Clojure, but first and foremost, it follows a principle I'm very fond of: code talks, bullshit walks.

That is to say, it's not difficult to imagine any number of interesting directions to take Lisp into, nor is it generally all that much effort to implement proof-of-concepts of such ideas (it's Lisp, after all); but to actually create a full integrated, practical and performant system, that's another matter entirely.

The difference between toy and tool is more than quantitative, and the leap from an SICP metacircular interpreter to something deserving of the hallowed title "Lisp system" is non-trivial, requiring not just deep mastery of the fundamentals but also the discipline and opportunity to take care of all the "boring" stuff so as to ship it. As I know all too well from plenty of practice, this last so-called 20% is where most efforts ultimately wither out to oblivion.

Thus it's a pleasant surprise to discover that while Clojure has no shortage of goodies, it's also remarkably mature and usable right here and now. The author, Rich Hickey, says that he has worked on it for two years, and I think it's admirable that he kept tight-lipped about it for the duration; as a first alpha release, Clojure as it is currently available simply rocks. No Marimba phenomenon, here.

As for the goodies, how does a concurrent, functional Lisp₁ with software transactional memory sound like? To me, it sounds like the future. (And that's just the tagline, there's much more.)

I've thought for a while now that if you took a purely functional subset of Scheme (pre-R⁶RS), sprinkled it with the best of Haskell (without the whole mandatory bondage and discipline, thanks) and built it all on top of an Erlang-like runtime, that might form a pretty compelling basis for a serious attempt at a new practical Lisp dialect. Clojure is perhaps not quite the particular mix that I had in mind, but it's very much in the ballpark.

Indeed Clojure embodies the kind of Lisp I'm interested in, that is, a pragmatic go-getter system actively drawing on the best new computer science research and pushing Lisp ever further out on the frontier of the future, instead of complacently sitting around arguing about how many lambdas can dance on a parenthesis while wondering why the competition is drawing nearer.

Considering how new Clojure is, there is a commendable amount of documentation already available, and the user community is growing in leaps and bounds, drawing both Common Lispers and Schemers as well as a good number of people entirely new to Lisp. The momentum that Clojure has going is tangible.

On the flip side, and these are really pretty serious omissions but may perhaps be intractable on the JVM at present, Clojure lacks tail recursion optimization and continuations. Also, while the concurrency features are impressive, I'm as of yet unsure how well-suited Clojure's JVM-based threading might be to a massively parallel programming style such as what I've been using with the lightweight processes in Gambit and Erlang, both of which will run millions processes without flinching. The author explains that these use cases are kind of supported but perhaps not really Clojure's sweet spot.

Now, given these JVM-imposed constraints, I should admit that I put off looking into Clojure for several months just given that it's based on the JVM; not only have I not recently had any reason to use the JVM, but my antipathy to Java is long-standing and well-documented.

However, in his LispNYC talk, Rich Hickey makes compelling points about why he based Clojure on the JVM, and I can't help but agree with many of them. Similarly, Neal Ford in his superb recent JRuby-related interview on JavaWorld makes a case that the actual positive legacy of Java will be the JVM as an established and widespread high-performance platform and infrastructure for enabling polyglot programming with the already more than 200 languages (including JRuby, Rhino, Jaskell and Jython) that you can run on it. I can buy that argument, too.

So, I'm coming to the unwelcome conclusion that my distaste for Java the language may have blinded me to the potential of the JVM as a platform for hosting other more powerful programming languages. I haven't much followed the JVM's evolution for years now, but intend to keep a closer eye on it in the future - especially the ongoing work around JSR 292 that aims to make the JVM suck less for hosting dynamic languages.

In any case, I'll be taking Clojure out for a spin. It's definitely a singularly unique new Lisp well worth looking at and learning from. I highly recommend Rich Hickey's two-hour LispNYC presentation which I can only describe as delightfully competent. (Also, in case STM is an unfamiliar concept, Simon Peyton-Jones's brief introduction at OSCON last year should be able to fill in the gaps some.)