Clojure/core Blog

Clojure core.async Channels

2013-06-28T00:00:00-07:00

core.async is a new contrib library for Clojure that adds support for asynchronous programming using channels.

Rationale

There comes a time in all good programs when components or subsystems must stop communicating directly with one another. This is often achieved via the introduction of queues between the producers of data and the consumers/processors of that data. This architectural indirection ensures that important decisions can be made with some degree of independence, and leads to systems that are easier to understand, manage, monitor and change, and make better use of computational resources, etc.

On the JVM, the java.util.concurrent package provides some good concurrent blocking queues, and they are a viable and popular choice for Clojure programs. However, in order to use the queues one must dedicate one or more actual threads to their consumption. Per-thread stack allocation and task-switching overheads limit the number of threads that can be used in practice. Another limitation of j.u.c. queues is there is no way to block waiting on a set of alternatives.

On JavaScript engines, there are no threads and no queues.

Thread overheads or lack of threads often cause people to move to systems based upon events/callbacks, in the pursuit of greater efficiency (often under the misnomer 'scalability', which doesn't apply since you can't scale a single machine). Events complect communication and flow of control. While there are various mechanisms to make events/callbacks cleaner (FRP, Rx/Observables) they don't change their fundamental nature, which is that upon an event an arbitrary amount of other code is run, possibly on the same thread, leading to admonitions such as "don't do too much work in your handler", and phrases like "callback hell".

The objectives of core.async are:

To provide facilities for independent threads of activity, communicating via queue-like channels
To support both real threads and shared use of thread pools (in any combination), as well as ClojureScript on JS engines
To build upon the work done on CSP and its derivatives

It is our hope that async channels will greatly simplify efficient server-side Clojure programs, and offer simpler and more robust techniques for front-end programming in ClojureScript.

History

The roots of this style go back at least as far as Hoare's Communicating Sequential Processes (CSP), followed by realizations and extensions in e.g. occam, Java CSP and the Go programming language.

In modern incarnations, the notion of a channel becomes first class, and in doing so provides us the indirection and independence we seek.

A key characteristic of channels is that they are blocking. In the most primitive form, an unbuffered channel acts as a rendezvous, any reader will await a writer and vice-versa. Buffering can be introduced, but unbounded buffering is discouraged, as bounded buffering with blocking can be an important tool coordinating pacing and back pressure, ensuring a system doesn't take on more work than it can achieve.

Details

Just a library

core.async is a library. It doesn't modify Clojure. It is designed to support Clojure 1.5+.

Creating channels

You can create a channel with the chan function. This will return a channel that supports multiple writers and readers. By default, the channel is unbuffered, but you can supply a number to indicate a buffer size, or supply a buffer object created via buffer, dropping-buffer or sliding-buffer.

The fundamental operations on channels are putting and taking values. Both of those operations potentially block, but the nature of the blocking depends on the nature of the thread of control in which the operation is performed. core.async supports two kinds of threads of control - ordinary threads and IOC (inversion of control) 'threads'. Ordinary threads can be created in any manner, but IOC threads are created via go blocks. Because JS does not have threads, only go blocks and IOC threads are supported in ClojureScript.

go blocks and IOC 'threads'

go is a macro that takes its body and examines it for any channel operations. It will turn the body into a state machine. Upon reaching any blocking operation, the state machine will be 'parked' and the actual thread of control will be released. This approach is similar to that used in C# async. When the blocking operation completes, the code will be resumed (on a thread-pool thread, or the sole thread in a JS VM). In this way the inversion of control that normally leaks into the program itself with event/callback systems is encapsulated by the mechanism, and you are left with straightforward sequential code. It will also provide the illusion of threads, and more important, separable sequential subsystems, to ClojureScript.

The primary channel operations within go blocks are >! ('put') and <! ('take'). The go block itself immediately returns a channel, on which it will eventually put the value of the last expression of the body (if non-nil), and then close.

Channel on ordinary threads

There are analogous operations for use on ordinary threads - >!! ('put blocking') and <!! ('take blocking'), which will block the thread on which they are called, until complete. While you can use these operations on threads created with e.g. future, there is also a macro, thread , analogous to go, that will launch a first-class thread and similarly return a channel, and should be preferred over future for channel work.

Mixing modes

You can put on a channel from either flavor of >!/>!! and similarly take with either of <!/<<! in any combination, i.e. the channel is oblivious to the nature of the threads which use it.

alt

It is often desirable to be able to wait for any one (and only one) of a set of channel operations to complete. This powerful facility is made available through the alts! function (for use in go blocks), and the analogous alts!! ('alts blocking'). If more than one operation is available to complete, one can be chosen at random or by priority (i.e. in the order they are supplied). There are corresponding alt! and alt!! macros that combine the choice with conditional evaluation of expressions.

Timeouts

Timeouts are just channels that automatically close after a period of time. You can create one with the timeout function, then just include the timeout in an alt variant. A nice aspect of this is that timeouts can be shared between threads of control, e.g. in order to have a set of activities share a bound.

The value of values

As with STM, the pervasive use of persistent data structures offers particular benefits for CSP-style channels. In particular, it is always safe and efficient to put a Clojure data structure on a channel, without fear of its subsequent use by either the producer or consumer.

Contrasting Go language channels

core.async has obvious similarities to Go channels. Some differences with Go are:

All of the operations are expressions (not statements)
This is a library, not syntax
alts! is a function (and supports a runtime-variable number of operations)
Priority is supported in alt

Finally, Clojure is hosted, i.e. we are bringing these facilities to existing platforms, not needing a custom runtime. The flip-side is we don't have the underpinnings we would with a custom runtime. Reaching existing platforms remains a core Clojure value proposition.

Whither actors?

I remain unenthusiastic about actors. They still couple the producer with the consumer. Yes, one can emulate or implement certain kinds of queues with actors (and, notably, people often do), but since any actor mechanism already incorporates a queue, it seems evident that queues are more primitive. It should be noted that Clojure's mechanisms for concurrent use of state remain viable, and channels are oriented towards the flow aspects of a system.

Deadlocks

Note that, unlike other Clojure concurrency constructs, channels, like all communications, are subject to deadlocks, the simplest being waiting for a message that will never arrive, which must be dealt with manually via timeouts etc. CSP proper is amenable to certain kinds of automated correctness analysis. No work has been done on that front for core.async as yet.

Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein.

Future directions

Networks channels and distribution are interesting areas for attention. We will also being doing performance tuning and refining the APIs.

Team

I'd like to thank the team that helped bring core.async to life:

Timothy Baldridge
Ghadi Shayban
Alex Miller
Alex Redington
Sam Umbach

And once again, Tom Faulhaber for his work on autodoc.

Status

While the library is still in an early state , we are ready for people to start trying it out and giving us feedback. The CLJS port is still work in progress. Please have a look at the examples, which we will expand over time.

It should be noted that the protocols behind the implementation should still be considered an implementation detail for the time being, until we finish our exploratory work around network channels, which might impact their design.

I hope that these async channels will help you build simpler and more robust programs.

Rich

(take 5 anthony-grimes)

2012-06-15T00:00:00-07:00

It seems appropriate in this, my last installment in the (take...) interview series, to go back to the beginning and catch up with one of my early interviewees. Close to two years ago I interviewed a high-school Clojurian by the name of Anthony Grimes -- my favorite of the series. I wanted to catch up with Anthony again to see what life held for him since graduation, and possibly to get a glimpse into what the future holds.

What have you been working on since I last interviewed you?

Oh wow. Lots of stuff!

One significant project is https://www.refheap.com. It's a pastebin site written in Clojure. It uses the wonderful Pygments syntax highlighter for generating syntax highlighted code and aims to be on par with the feature offering of Gist while not using git for storing pastes and sporting a (hopefully) slightly more pleasant interface. There is still a ways to go, of course. We want to support Markdown (among other markup languages) rendering, revisions, diffs of revisions, etc. There is plenty to do and issues for almost all of it, so if anybody reading this is feeling frisky and opensourcy, take a look at them.

To go with refheap, I wrote https://github.com/Raynes/refheap.el for pasting from Emacs, https://github.com/Raynes/refheap.vim for pasting from Vim (uses Ruby), and a little Ruby gem for pasting from the command line: https://github.com/Raynes/refh. And, of course, API client libraries for refheap in Ruby and Clojure with more to come.

Of my projects, I admire refheap in particular because it can attract a wide variety of contributors. For example, William Manire has been contributing significantly to the Javascript portion of the site. As a side effect, he has been teaching me Javascript!

Some other stuff...

There is also Tentacles, my Github API client lib. Lots of people seem to like it. At least one person (hi Phil!) uses it so he doesn't have to use Github's upload file interface.

Also, Conch, a little library for shelling it. It is useful because, while clojure.java.shell is a very useful and well-written library, it (purposely) covers up all of the Java process API. Sometimes you don't want that though, and Conch exists for those times.

Finally, I've been trying to help out with Noir as much as possible. Mostly just reviewing and pulling pull requests and keeping a steady stream of beta releases going. Chris is really busy these days, so he needs all the help he can get with non-Light Table projects.

I think that's enough horn tooting. Check out my Github page if I'm just so irresistible that you must see all my projects.

How is the book coming?

Slowly. I've had a lot on my plate at work and haven't had a lot of time to work on it, unfortunately. I hope to be able to pick things up soon.

Can you explain Flatland for the uninitiated?

It's pretty simple. I work at Geni and a lot of our code is open source. We just adore modularity and we very often recognize patterns and split things out into standalone libraries. Flatland is our open source home. Things like Jiraph, our graph database, Useful, our utility library, and protobuf, our Google Protobuf library and accompanying lein plugin, among other things, are put there. You might note that lazybot, clojail, and irclj are all there as well. We decided to put those there because 1) we use them in our company IRC channels and 2) Alan Malloy and I both work on them and maintain so it makes sense for us both to have administrator access to the projects.

tl;dr: It's Geni's open source Github organization.

What was going through your mind when presenting to ~300 people at the Conj?

It's hard to recall. At first, it was something between "Just do it. Breath. Now talk. Clear your throat. Breath. Talk." and absolutely nothing at all. I actually anticipated being unable to talk and put lots of notes in my slides so that when I got nervous during the talk I could mostly just read them and avoid having to think.

About 5 minutes in it was all fine though. Once everybody had laughed or chuckled at something I said, my hair slicked back and I grew sunglasses and knew I would be okay.

My favorite two moments of the entire conference (not taking away from the talks at all, this is entirely personal) was how, after my talk, Rich told me I was great in person and said I was awesome on twitter. Easily comparable to teenage girls getting a Justin Bieber autograph. Would totally do it again.

Will we see you at the next Conj?

With bells on. I might even submit a talk again. I'd do it under one condition: at some point in my talk, Michael Fogus must come to the stage with a top hat and cane and dance for precisely 45 seconds.

postscript

It's been a wild ride trying to track down interesting Clojure programmers over the past two years, but all good things must come to and end. From the beginning of the (take...) series I've tried to highlight all types of programmers. While it's definitely been wonderful talking to luminaries such as Rich Hickey, William Byrd, Daniel Spiewak and Martin Odersky, I've always tried to highlight the people using Clojure at work or during their free time to create, teach and inform. Undoubtably the computing industry as a whole would be much the poorer without the likes of Hickey and Odersky, but it's community that has allowed them to make a sustained impact. One of Clojure's greatest strengths is its community, and I feel an overwhelming joy to know that its members include amazing people like David Edgar Liebke, Phil Hagelberg, David Nolen, Brenton Ashworth, George Jahad, Justin Balthrop, Carin Meier, Ambrose Bonnaire-Sergeant, Sam Aaron, Arnoldo Jose Muller-Molina, Kevin Lynagh, Baishampayan Ghose, Colin Jones and Jim Crossley. Thanks to them, and all of the other brilliant Clojurians whom I was not fortunate enough to interview, for making the Clojure community one of the language's most awesome innovations.

I'm off to practice my dance steps.

(take 6 jim-crossley)

2012-06-01T00:00:00-07:00

Jim Crossley and his team at Red Hat are attempting to create the programming language polyglot's dream. Their latest creation Immutant is billed as a Clojure application server built on JBoss, but as you'll discover below is aiming to integrate numerous Java.next languages. Immutant is an exciting project and looks destined to become something special.

What is the "elevator pitch" for Immutant?

Immutant is an application server for Clojure built atop JBoss AS7.

Our primary goal is a robust, scalable deployment platform for Clojure applications, with built-in services like messaging, caching, distributed transactions and clustering. We aim to encapsulate the accidental complexity associated with most non-trivial applications and allow the developer to focus on the essential complexity of his problem domain.

Immutant is an integrated stack of services. Instead of provisioning your servers with Jetty for web, RabbitMQ for messaging, and Memcached for caching, for example, you could simply install Immutant.

We currently have projects to support JRuby (TorqueBox) and Clojure (Immutant) apps, and we're looking at exposing the AS7 services to Scala, Python, and JavaScript as well. All of these projects can be overlaid on each other, creating an app server that provides services for multiple languages at once. This allows you to leverage the libraries and skills appropriate to any particular aspect of your application, regardless of the language, e.g. Rails used in the front-end web tier and perhaps Incanter used for back-end report generation.

Were there any barriers to using Clojure in the implementation of Immutant?

Quite the contrary, Clojure has enabled us to remove much of the boilerplate typically associated with JEE services, so we've been able to encapsulate the complex JEE API's and configuration within a small set of Clojure functions.

Clojure also inspired us to support more dynamic applications. It's not necessary to declare all the services required by your app in some sort of "deployment descriptor" for Immutant; those decisions can be deferred until runtime. We also let you expose the application running within Immutant via either Swank or nREPL, enabling a more interactive development experience than what folks typically expect from a JEE container.

So far, the biggest challenge has been working around Clojure's assumption that there is only one Clojure runtime per JVM - we wanted each application deployed within an Immutant to have its own runtime, and even its own Clojure version. To work around this, each application gets its own isolated ClassLoader and we do some classloader munging to give each application its own runtime.

We're aided in this by JBoss AS7's modular classloading system - it does a great job of isolating an application's dependencies from those of other applications or Immutant itself.

Why do you think that there are some who are mistrustful of "Clojure in the enterprise"?

Answering this question requires making broad assumptions about the fears and motivations of others, not to mention an agreed-upon definition of a pretty overloaded term... but what the hell! :)

Fear is the mind-killer.

Seriously, I don't think good developers are mistrustful of Clojure, though some may be justifiably leery of bloated application servers that are traditionally associated with enterprises. JBoss AS7 is not your father's app server: though obviously bigger than any single component of a home-grown stack, it's fast and light enough to not make you "stabby" while coding on your laptop.

But developers are usually only one minority amongst the decision-making stakeholders in an enterprise...

How can Immutant alleviate those doubts?

Oddly, the integrated, java-app-server-ness aspect of Immutant that gives some developers pause is the very thing that IT managers and sys-admins love. Immutant offers a great opportunity to "get Clojure in the enterprise" because it can deploy those Java legacy apps right alongside the newer Clojure/JRuby/etc ones, with minimal impact to the devops guys.

But truthfully, we don't really care about that. We're looking for ways to make development and deployment simple. And by simple, I mean no more complicated than what's required to address the problems within a specific domain.

And though it's true that forces acting on an enterprise often complect applications in ways that don't affect individuals or small teams, we think simpler development and deployment benefits everyone, whether you develop apps for/in an enterprise or not.

What is the state of Clojure in the broader Red Hat corporation?

Well, when we started Immutant almost a year ago, it was pretty much just me and Toby. ;)

We're part of a research team at Red Hat led by our "Director of Polyglot"[1], Bob McWhirter. We're fortunate to have recently added Charlie Nutter and Tom Enebo (the JRuby guys) to our team, and both are very interested in alternative JVM languages (including Clojure) and eager to apply their skills to our JVM-related projects, including TorqueBox, Immutant and OpenJDK, among others.

Red Hat is a very large company of very smart engineers on very long leashes, so I have little doubt others are using Clojure internally. And we have a solid mandate from our leadership to expose the excellent JBoss services through as many JVM-based languages as makes sense.

Do you think that there is a synergy between JRuby and Clojure in the enterprise?

Absolutely! And not just in the enterprise. The proliferation of web frameworks and tools for Ruby make it a great language for developing web UI's and/or services, whereas to this point Clojure has mostly been used to create some pretty incredible data analytics applications.

It's rare to find individual developers with enough expertise in both JRuby and Clojure to effectively combine frameworks and libraries from each. But we believe they're out there, and as the benefits of polyglot programming become more publicized, their numbers will increase. But in the meantime, we expect larger organizations to form teams of experts from both camps, each producing loosely-coupled applications that coordinate to solve domain-specific problems.

We hope Immutant will appeal to both individuals and teams by allowing those seemingly disparate applications to be easily developed and deployed to a single platform.

[1] Best job title ever!

(take 5 colin-jones)

2012-05-22T00:00:00-07:00

Colin Jones is a software craftsman at 8th Light and Clojure veteran. His contributions to the Clojure ecosystem are far and wide, but tend toward helping newcomers to discover the language and provide tools for getting started as quickly as possible. In this installment of the (take ...) series Colin discusses koans, TDD, REPL-ing, and the Clojure Conj.

What are koans in the context of programming?

The basic idea is that they're exercises to help you learn a particular language or other technology. There's a Zen metaphor that ties things together, but the main thing is to have small, bite-sized chunks to guide folks in their learning without being overwhelming. I first heard about the idea in the EdgeCase Ruby Koans, and they in turn credit other sources (including The Little Lisper! with inspiration. So when Aaron Bedra started the Functional Koans project, I was excited to pitch in, and before I knew it, we had a bunch of people having fun working through the Clojure Koans, as their first exposure to Clojure.

Is there a place for TDD in the Clojure ecosystem?

Of course! I think Brian Marick has shown that most publicly with his library Midje, and related screencasts and conference talks, but there are a ton of other great testing tools out there as well: Speclj by my boss, Micah Martin, LazyTest, Expectations, clojure.test, etc.

I have heard TDD and deep thinking put into stark contrast with each other at times, but the people I respect in the TDD world certainly do a lot of deep thinking about simple design, so I think we sometimes get a bit of a false dichotomy.

I personally find a lot of benefits to unit testing - it's a way of making my thinking about problems concrete, and when I'm done, there's an automated system looking for assumptions that have become untrue.

What itch are you attempting to scratch with your latest project REPL-y?

OK, the 5-second version is that I want it to be easier to use a Clojure REPL, unix-style, at the shell. Projects like Russ Olsen's dejour were working towards similar goals, which I found exciting, so I got in touch with Russ to pitch in. As it happened, a lot of my ideas for improvement could be implemented orthogonally to dejour: handling Ctrl-C interruptions without bailing out of the process, basic code completion, in-REPL examples (via clojuredocs), and a truer readline-like interface (because let's face it: JLine 0.9.94 is broken). Hence REPL-y was born.

Down the line a bit, with some encouragement and help from Phil Hagelberg and the Leiningen team, it's now the default REPL frontend for the upcoming Leiningen 2, which is now in preview (nREPL is the backend). So at this point, REPL-y is wiring up of a bunch of great projects: Clojure, JLine2, clojure-complete, nREPL, and clojuredocs-client. There's a shell script to run REPL-y alone, but of course I expect the version in Leiningen to see a lot more use.

The project lives at https://github.com/trptcolin/reply (varying pronunciations are both acceptable and hilarious).

How can Clojure improve, in general?

Language-wise, I don't have too much to say. I know a lot of folks struggle with the various ways of pulling code in from other namespaces, so maybe there are ways to make those easier to use. My main interests have mostly been in helping people get over hurdles - which is of course a cause shared by a lot of people on the IRC channel and beyond. So, needless to say, I was very impressed by the fact that the planning for Clojure 1.4 included error messages, documentation, even potentially a command-line launcher app.

Tools-wise, I think things are moving in the right direction. Of course I mentioned the REPL stuff and Leiningen, and there are dozens of great new things coming out all the time.

I'd love to see clojure.org become a bit friendlier as an introduction to Clojure, from the very first steps. I wonder if it'd be possible to have something like the Java Tutorials that's maintained in an "official" way? There's so much incredibly good information on clojure.org - I refer to it often - but it can be intimidating, and things get out of date from time to time. I also think function findability can be hard in the clojure.core namespace from a REPL if you don't know what you're looking for, like if you want an operation on a particular data structure but don't remember the function name, so maybe a similar function to find-doc could do some nice grouping, like what's laid out on http://clojure.org/data_structures.

First impressions matter, and if we can hook people right at the start without compromising our principles, I think that's a really big win for the language and the community.

You've been to both Clojure conferences; was this years' offering different than the first?

I was pretty bummed about the lack of Clojure temporary tattoos this year, but you can't win them all.

Seriously, though, this conference is amazing. Certainly there were differences: last year's seemed a bit more language-focused, and this year's seemed more product- and library- focused. That's a pretty broad and unscientific impression, but suffice to say, both were awesome.

Some of the bonuses this year: an additional day of talks, bunches of informal sessions that sprung up in the conference space, and lots of great restaurants within walking distance. The talks, like last year, were fantastic. I suspect we'll see even more of these great stories about the things people are building as Clojure gets used in production more and more.

Plus, Overtone? Bad ass.

Anatomy of a Reducer

2012-05-15T00:00:00-07:00

Last time, I blogged about Clojure's new reducers library. This time I'd like to look at the details of what constitutes a reducer, as well as some background about the library.

What's a Reducing Function?

The reducers library is built around transforming reducing functions. A reducing function is simply a binary function, akin to the one you might pass to reduce. While the two arguments might be treated symmetrically by the function, there is an implied semantic that distinguishes the arguments: the first argument is a result or accumulator that is being built up by the reduction, while the second is some new input value from the source being reduced. While reduce works from the 'left', that is neither a property nor promise of the reducing function, but one of reduce itself. So we'll say simply that a reducing fn has the shape:

  
  (f result input) -> new-result

In addition, a reducing fn may be called with no args, and should then return an identity value for its operation.

Transforming Reducing Functions

A function that transforms a reducing fn simply takes one, and returns another one:

  
  (xf reducing-fn) -> reducing-fn

Many of the core collection operations can be expressed in terms of such a transformation. Imagine if we were to define the cores of map, filter and mapcat in this way:

  
  (defn mapping [f]
    (fn [f1]
      (fn [result input]
        (f1 result (f input)))))

  (defn filtering [pred]
    (fn [f1]
      (fn [result input]
        (if (pred input)
          (f1 result input)
          result))))

  (defn mapcatting [f]
    (fn [f1]
      (fn [result input]
        (reduce f1 result (f input)))))

There are a few things to note:

The functions consist only of the core logic of their operations
That logic does not include any notion of collection, nor order
filtering and kin can 'skip' inputs by simply returning the incoming result
mapcatting and kin can produce more than one result per input by simply operating on result more than once

Using these directly is somewhat odd, because we are operating on the reducing operation rather than the collection:

  
  (reduce + 0 (map inc [1 2 3 4]))
  ;;becomes
  (reduce ((mapping inc) +) 0 [1 2 3 4])

Reducers

We expect map/filter etc to take and return logical collections. The premise of the reducers library is that the minimum definition of collection is something that is reducible. reduce ends up using a protocol (CollReduce) to ask the collection to reduce itself, so we can make reducible things by extending that protocol. Thus, given a collection and a reducing function transformer like those above, we can make a reducible with a function like this:

  
  (defn reducer
    ([coll xf]
     (reify
      clojure.core.protocols/CollReduce
      (coll-reduce [_ f1 init]
        (clojure.core.protocols/coll-reduce coll (xf f1) init)))))

Now:

  
  (reduce + 0 (map inc [1 2 3 4]))
  ;;becomes
  (reduce + 0 (reducer [1 2 3 4] (mapping inc)))

That's better. It feels as if we have transformed the collection itself. Note:

reducer ultimately asks the source collection to reduce itself
reducer will work with any reducing function transformer

Another objective of the library is to support reducer-based code with the same shape as our current seq-based code. Getting there is easy:

  
  (defn rmap [f coll]
    (reducer coll (mapping f)))

  (defn rfilter [pred coll]
    (reducer coll (filtering pred)))

  (defn rmapcat [f coll]
    (reducer coll (mapcatting f)))

  (reduce + 0 (rmap inc [1 2 3 4]))
  ;=> 14

  (reduce + 0 (rfilter even? [1 2 3 4]))
  ;=> 6

  (reduce + 0 (rmapcat range [1 2 3 4 5]))
  ;=> 20

From Reducible to (Parallel) Foldable

While it is an interesting exercise to find another fundamental way to define the core collection operations, the end result is not much different, just faster, certainly something a state-of-the-art compilation and type system (had we one) might do for us given sequence code. To stop here would be to completely miss the point of the library. These operations have different, fundamentally simpler semantics than their sequence-based counterparts.

How does one define parallel mapping/filtering/mapcatting etc? We already did! As long as the transformation itself doesn't care about order (e.g. as take does), then a reducer is as foldable as its source. As with reduce, fold bottoms out on a protocol (CollFold), and our reducer can extend that:

  
  (defn folder
    ([coll xf]
       (reify
        ;;extend CollReduce as before

        CollFold
        (coll-fold [_ n combinef reducef]
          (coll-fold coll n combinef (xf reducef))))))

Note that:

folder has the same requirements as reducer - collection + reducing function transformer
when fold is applied to something that can't fold, it devolves to reduce

Thus the real definitions of reducers/map et al use folder (while take uses reducer):

  
  (defn rmap [f coll]
    (folder coll (mapping f)))

  (defn rfilter [pred coll]
    (folder coll (filtering pred)))

  (defn rmapcat [f coll]
    (folder coll (mapcatting f)))

Thus a wide variety of collection transformations can instead be expressed as reducing function transformations, and applied in both sequential and parallel contexts, across a wide variety of data structures.

The library deals with several other details, such as:

the transformers all need a nullary arity that just delegates to the transformed reducing function
the transformers support a ternary arity where 2 inputs are supplied per step, as occurs with reduce-kv and map sources
all of the reducers are curried

These additions are all mechanical, and are handled by macros. It is my hope that the above will help illuminate the core logic underlying the library.

Background

Much prior work highlights the value of fold as a primary mechanism for collection manipulation, superior to iteration, although most of that work was done in the context of recursively defined functions on lists or sequences - i.e. fold implies foldl/foldr, and the results remain inherently sequential.

The two primary motivators for this library were the Haskell Iteratee library and Guy Steele's ICFP '09 talk.

Haskell Iteratees

The Haskell Enumerator/Iteratee library and its antecedents are an inspiring effort to disentangle the source of data and the operations that might apply to it, and one of the first I think to reify the role of the 'iteratee'. An enumerator makes successive calls to the iteratee to supply it items, decoupling the iteratee from the data source. But the iteratee is still driving in some sense, as it is in charge of signaling Done, and, it returns on each step the next iteratee to use, effectively dictating a single thread of control. One benefit is that even operations like take can be defined functionally, as they can encode their internal state in the 'next' iteratee returned. OTOH, and unlike reducers, the design wraps the result being built up in a new iteratee each step, with potential allocation overhead.

Being an automaton in a state, an iteratee is like a reified left fold, and thus inherently serial. So, while they form quite a nice substrate for the design of, e.g. parsers, iteratees are unsuitable for defining things like map/filter etc if one intends to be able to parallelize them.

Guy Steele's ICFP '09 talk

Organizing Functional Code for Parallel Execution or, foldl and foldr Considered Slightly Harmful

This talk boils down to - stop programming with streams, lists, generators etc if you intend to exploit parallelism, as does the reducers library.

Where reducers diverges from that talk is in the structure of the fork/join parallel computation. Rather than map+reduce, reducers uses reduce+combine. This reflects 2 considerations:

It is accepted fork/join practice that at some point you stop splitting in half and handle the leaves 'sequentially'
- if the best way to do that at the top is reduce, why not at the bottom as well?
map forces a result per input

You can see the awkwardness of the latter in the map/reduce-oriented definition of parallel filter in the talk, which must 'listify' items or return empty lists, creating a bunch of concatenation busy-work for the reducing step. Many other collection algorithms suffer similarly in their map/reduce-oriented implementations, having greater internal complexity and wrapping the results in collection representations, with corresponding creation of more garbage and reduction busy-work etc vs the reducing function transformer versions of same.

It is interesting that the accumulator style is not completely absent from the reducers design, in fact it is important to the characteristics just described. What has been abandoned are the single initial value and serial execution promises of foldl/r.

Summary

I hope this makes reducers easier to understand, use and define.

Rich

Reducers - A Library and Model for Collection Processing

2012-05-08T00:00:00-07:00

I'm happy to have pushed today the beginnings of a new Clojure library for higher-order manipulation of collections, based upon reduce and fold. Of course, Clojure already has Lisp's reduce, which corresponds to the traditional foldl of functional programming. reduce is based upon sequences, as are many of the core functions of Clojure, like map, filter etc. So, what could be better? It's a long story, so I'll give you the ending first:

There is a new namespace: clojure.core.reducers
It contains new versions of map, filter etc based upon transforming reducing functions - reducers
It contains a new function, fold, which is a parallel reduce+combine
fold uses fork/join when working with (the existing!) Clojure vectors and maps
Your new parallel code has exactly the same shape as your existing seq-based code
The reducers are composable
Reducer implementations are primarily functional - no iterators
The model uses regular data structures, not 'parallel collections' or other OO malarkey
It's fast, and can become faster still
This is work-in-progress

Basics

The story starts best at the bottom.

Clojure and other functional languages have a function called map that takes a function and a collection/list.

What does it mean to map a function on a collection?
What are the common signatures?
Do they complect what to do with how to do it?

The classic recursive functional definition of map is to apply f to the first thing in the collection, then cons the result onto the result of mapping f on the rest of the collection. This definition includes plenty of 'how':

How: mechanism - recursion
How: order - sequentially
How: laziness - (often) lazily
How: representation - making a list/seq, or other concrete collection

Newer OO frameworks will often remove some of these problems by having map be a function of fn * Coll -> Coll for any type of Coll, removing the sequentiality but also losing the laziness, and they still specify a concrete collection result.

Semantically, and minimally, map means "apply-to-all" e.g. (map inc coll) means give me a (logical) collection where every item is one greater than it was in coll. But, map doesn't know how to navigate around every collection - the use of seqs/lists/iterators/streams etc forces a shared known representation. Nor does inc (or any function) know how to apply itself to every collection representation, else we could just say (inc coll).

The only thing that knows how to apply a function to a collection is the collection itself.

What is the generic gateway to a collection applying things to itself? In Clojure, it is (internal) reduce.

We now have a new super-generalized and minimal abstraction for collections - a collection is some set of things that, when given a function to apply to its contents, can do so and give you the result, i.e. a collection is (at minimum) reducible. In other words, you can call reduce on it.

Thus, core.reducers/map is a function of fn * reducible -> reducible. (Whereas core/map is a fn of fn * seqable -> seqable)

Now, how? If someone is going to ask the result of (map inc coll) to reduce itself with some function f, map must ultimately ask coll to do the job. Rather than pass coll f, map passes coll a new, transformed, reducing function that takes what coll supplies, calls inc on it, and then calls f on that.

  
  (reduce + (r/map inc [1 2 3])) === (reduce (fn [ret x] (+ ret (inc x))) (+) [1 2 3])

i.e. the core work of map f looks like this:

  
  (fn [f1]
    (fn [ret v]
      (f1 ret (f v))))

It takes a reducing function f1, and returns a new reducing function that calls f1 after applying f to its input.

Thus you can define map as a function of fn * reducible -> reducible by merely transforming the reducing function. Mapping is semantically a function of the function of one step of a reduction. This transformation is decomplected from both representation and order. We call functions such as this map, that take a reducible, and in turn return something reducible via transformation of the reducing function, reducers.

Now let's revisit the hows above...

How: mechanism - functional transformation of reducing function
How: order - doesn't know
How: laziness - doesn't know
How: representation - doesn't build anything

It is important to note that now, when (map f coll) is called nothing happens except the creation of a recipe for a new collection, a recipe that is itself reducible. No work is done yet to the contained elements and no concrete collection is produced.

The beautiful thing is that this 'transformation of reducing function' mechanism also works for many of the traditional seq functions, like filter, take, flatten etc. Note the fact that filter is (potentially) contractive, and flatten is (potentially) expansive per step - the mechanism is general and not limited to 1:1 transformations. And other reducer definitions are as pretty as map's - none of the imperativeness of iterators, or generators with yield.

Ok, So Where's My Cake?

If map doesn't do the work of mapping, but merely creates a recipe, when does the work get done? When you reduce its result:

  
    (require '[clojure.core.reducers :as r])
    (reduce + (r/filter even? (r/map inc [1 1 1 2])))
    ;=> 6

That should look familiar - it's the same named functions, applied in the same order, with the same arguments, producing the same result as the Clojure's seq-based fns. The difference is that, reduce being eager, and these reducers fns being out of the seq game, there's no per-step allocation overhead, so it's faster. Laziness is great when you need it, but when you don't you shouldn't have to pay for it.

The reducer fns are curried, and they can be easily composed:

  
  ;;red is a reducer awaiting a collection
  (def red (comp (r/filter even?) (r/map inc)))
  (reduce + (red [1 1 1 2]))
  ;=> 6

Thus reduction 'recipes' (reducers) are first class.

What if we want a collection result? It's good to know that into uses reduce:

  
  (into [] (r/filter even? (r/map inc [1 1 1 2])))
  ;=> [2 2 2]

Note there are no intermediate collections produced.

And, of course, you don't always want a result of the same collection type:

  
  (into #{} (r/filter even? (r/map inc [1 1 1 2])))
  ;=> #{2}

Simplicity is Opportunity

Decomplecting the core operations from representation and laziness has given us some speed, but what about the elimination of order? It should open the door to parallelism, but we are stuck with the semantics of reduce being foldl, i.e. it uses an accumulator and is fundamentally serial. We can parallelize reduction by using independent sub-reductions and combining their results, and the library defines a function that does just that: fold.

The primary signature of fold takes a combining function, a reducing function, and a collection and returns the result of combining the results of reducing subsegments of the collection, potentially in parallel. Obviously if the work is to occur in parallel, the functions must be associative, but they need not be commutative - fold preserves order. Note that there is no initial 'seed' or 'accumulator' value, as there may be with reduce and foldl. But, since the subsegments are themselves reduced (with reduce), it raises the question as to what supplies the seed values for those reductions?

The combining function (an associative binary fn) must have some 'identity' value, a value that, when combined with some X, yields X. 0 is an identity value for +, as is 1 for *. The combining fn must supply an identity value when called with no arguments (as do + and *). It will be called with no arguments to supply a seed for each leaf reduction. There is a fn (called monoid, shh!) to help you build such combining functions.

If no combining fn is supplied, the reducing fn is used. Simple folds look like reduces:

  
  (r/fold + [1 2 3 4])
  ;=> 10

But by promising less (i.e. not promising stepwise reduction from left or right) fold can do more - run in parallel. It does this when the collection is amenable to parallel subdivision. Ideal candidates are data structures built from trees. Clojure vectors and maps are trees, and have parallel implementations of fold based upon the ForkJoin framework.

What if the underlying collection is not amenable (e.g. is a sequence)? fold just devolves into reduce, producing the same semantic, if not physical, result.

There's a tremendous amount you can accomplish with this reduce+combine strategy, especially when you consider that the map, filter etc reducers will not constitute independent layers of parallel jobs - they just transform the reducing fn working on the leaves.

You can have a look at the cat function included in the library for an interesting example of a combining fn. cat quickly gathers up the fold results, forming a binary tree with the reductions as leaves. It returns a highly abstract, yet now quite useful 'collection' that is just counted, reducible, foldable and seqable.

Oh yeah, perf. Don't be surprised to see things become 2-3X faster, or more with more cores.

More Opportunity (i.e. Work)

As much fun as this is, there's still more fun to be had by those so inclined:

There are more seq fns that could become reducer fns
Given multiple iterable sources, we should be able to build a multi-reducible, recovering the multi-input capabilities of map.
Arrays, arraylists, strings etc are all amenable to parallel fold.
- fork/join-based vector fold is 14 lines, so these are not difficult.
Those IFn.LLL, DDD etc primitive-taking function interfaces can now spring to life.
- We should be able to build primitive-transmitting reducer function pipelines.
- We'd then need to look for and use them in the reductions of arrays and vectors of primitives
Internal reduce solves the lazily dangling open resource problem, a problem solved similarly by Haskell's enumerators and iteratees. (Note that unlike iteratees, reducers do not allocate wrappers per step)
- We need reducible I/O sources.

Summary

By adopting an alternative view of collections as reducible, rather than seqable things, we can get a complementary set of fundamental operations that tradeoff laziness for parallelism, while retaining the same high-level, functional programming model. Because the two models retain the same shape, we can easily choose whichever is appropriate for the task at hand.

Follow Up

See the follow up blog post for more details about what constitutes a reducer, as well as some background about the library.

Rich

Clojure/core Wants to Visit You This Summer

2012-05-07T00:00:00-07:00

We're hitting the road! Clojure/core members have been delivering Clojure training since the language was in its infancy. Historically we've delivered this as an open registration option in partnership with The Pragmatic Studio in Reston, VA. Occasionally we'd venture out to other locations when there was a major event like Clojure/West or Clojure/conj. Now we'd like to find the cities most excited to host this training and bring it right to you!

We are continually updating our training offering to keep up with the latest changes in the Clojure world. ClojureScript is a great example of this, which we added to the training shortly after its release to the world. Our passion, both personally and professionally, is to share this language with others. The hands-on and interactive nature of the class gives you a chance to dive into the topics that interest you most.

We've launched the Clojure/core Roadshow registration site for you to tell us where we should deliver our three day training program this year. If you're looking to level up in your Clojure game, this is the perfect opportunity to do so with us. Head on over to the site and register your interest today. As an added bonus, we are going to be delivering an extra day of training specifically on Datomic. This day of training will be free and open to the public, but by attending the paid training, you will have a seat reserved for you at the Datomic training as well.

If you are an organization interested in partnering with us to host an open registration training in your city, we would love to talk with you. Contact us and let's make that happen.

If you are interested in arranging a private training event, we do that as well! We can tailor the course to address the needs of your organization. Head over to Relevance to see the details around private training.

(take 5 daniel-spiewak)

2012-04-19T00:00:00-07:00

Daniel Spiewak is a force of nature. As a highly respected member of the Scala programming language community and an overall thoughtful polyglot he seemed a natural fit as an interesting speaker for the 2011 Clojure/Conj. Daniel's talk entitled *Extreme Cleverness: Functional Data Structures in Scala was highly energetic and astoundingly informative. Daniel's open source contributions are not to be forgotten however. In addition to the important anti-xml Scala library, Daniel is attempting a bit of text-based collaboration magic with his Common Colaborative Coding Protocol project.

In this interview we talk about Clojure and its community, the conj function and Java.next languages learning from each other and their mutual struggle for mindshare.

What are your thoughts on the Clojure community?

Absolutely awesome! I've had the supreme pleasure of spending a fair amount of time amongst Clojurists (Clojurers??), and it's always a blast. Thanks in no small part to Rich Hickey, the Clojure community is smart, creative and absolutely intolerant of intolerance. The very fact that I was asked, as a Scala speaker, to come and present at Clojure/conj is a testament to how welcoming this community really is. Communities of this sort tend to be incredibly fertile soil for new ideas and profound advancement of the state of the art. I eagerly anticipate pillaging future innovations!

Are there any lessons that Clojure could learn from Scala?

I think there are a few lessons, the most important being that rich and uniform collections are extremely valuable. One of the things that drives me up the wall in Clojure is the following:

  
    (conj [1 2 3] 4)     ; => [1 2 3 4]
    (conj '(1 2 3) 4)    ; => (4 1 2 3)

Thus, the behavior of conj is a little bit unclear, since it depends on the input type. The object-oriented analogue to this would be if we defined two implementations of an interface, each defining the same method in opposite ways. I can't even begin to imagine what Liskov would say to that.

It should be noted that although Daniel is absolutely correct that conj depends on the input type, it's idea is that it will do the most efficient action given its input type. For lists the most effiecient action is to place an item at the front, for vectors elements are put onto the back.

Even more annoying is the following:

  
  (drop 1 [1 2 3 4])    ; => (2 3 4)

So, I drop the first element of a vector and get a list?! That seems, well, weird. Combining this with the conj issue, we can get something really bizarre:

  
    (conj (drop 1 [1 2 3 4]) 1)        ; => (1 2 3 4)
    (conj [2 3 4] 1)                   ; => [2 3 4 1]

In other words, Clojure's sequence functions complect behavior with input type.

(my understanding is that a lot of these issues are resolved in ClojureScript, but I haven't had a chance to really look yet)

Clojure's sequence abstraction is a neat idea in theory, but the practice leaves something to be desired. The root of the problem here, incidentally, is that Clojure's sequence abstraction is a little bit of subtyping embedded within an otherwise functional language. This problem could be resolved by playing the same trick that Scala's collections do, putting factory accessors on each collection to allow functions to build a collection of a dynamically-determined type.

For anyone who might not know, Scala's collections solve this in a really nice way, so everything falls out essentially the way you would expect:

  
    List(1, 2, 3) drop 1      // => List(2, 3)
    Vector(1, 2, 3) drop 1    // => Vector(2, 3)

    0 +: List(1, 2, 3)        // => List(0, 1, 2, 3)
    0 +: Vector(1, 2, 3)      // => Vector(0, 1, 2, 3)

We can even do fancier things, like working with data structures that can only contain certain data types:

  
    BitSet(1, 2, 3) map { _ * 2 }       // => BitSet(2, 4, 6)
    BitSet(1, 2, 3) map { _.toString }  // => Set("1", "2", "3")

It's all very nice and extremely simple (though certainly not simple at all if you want to actually write a new collection, but I digress...). Granted, it has taken Scala four full collections rewrites to get to this point, but it's nice now that we're here! Some of this is using the magic of Scala's type system (like the BitSet thing), but almost all of it could be applied to Clojure.

Aside from collections, I would say that there are a few things that Scala does better than Clojure, but I'm not entirely sure I want Clojure to go down those roads. Polymorphic modules are super-useful, but they bring with them a giant raft of complexity. Virtual dispatch is a gateway to a lot of syntactic power that Clojure really cannot achieve without dodgy macros (for example, see Scala's parser combinator DSL), but again, lots of complexity and not really a good fit for the rest of the Clojure language.

Is there room for both Scala and Clojure?

Absolutely! So, here's the thing: neither Scala nor Clojure are going to displace Java. It's just not going to happen. Both Scala and Clojure are substantially simpler than Java, but vastly harder. Java has achieved what I like to call the "boat anchor" phase of a language lifecycle, where it is so ubiquitous that it has become impossible to unseat without a massive paradigm shift. An example of such a shift would be if quantum computing came around and no one ported the JVM to run on that architecture (or perhaps if it were to run substantially worse than other VMs).

In the meantime, we have a whole bunch of alternative JVM languages who's adoption is likely to remain rounding error for quite some time. Paradoxically, this is very good for the relationship between Scala and Clojure. Our biggest hurdle is overcoming Java's inertia, and that hurdle is so large that petty little power struggles between alternative languages are rendered insignificant.

In other words, it's not a zero sum game between Scala and Clojure. When a new developer picks up Scala or Clojure, they are choosing to bridge a gigantic chasm from the "mainstream" languages to something a little more on the fringe. Yes, this is a win for whatever specific language they choose, but it's an even bigger win for alternative languages in general. People need to realize that they have options, and in particular that they have options on the JVM. Every developer learning or being exposed to an alternative language is a win for everyone, and right now that is the lion's share of the battle.

In the long term, as Clojure and Scala grow in adoption, we may find that the languages are starting to be in competition for the same problem spaces. However, if and when this happens, I suspect the languages will be sufficiently divergent as to have generally disjoint niches. Even today, if you know both Scala and Clojure, there is rarely any ambiguity as to when you should apply one rather than the other. This is likely to continue even as alternative languages steal more and more of Java's developer share.

An idea that has a cult following in Clojure is that of optional (and pluggable) type systems.

What are your thoughts on this perspective?

That's an interesting question. Right off the bat, I think it's important to point out that a "pluggable" type system immediately implies an optional type system. There is also a weaker implication that such a type system could be split out from the normal compilation process and run separately. This is certainly something that I've heard Rich talk about on several occasions, and I think it's an interesting idea. However, there are a couple immediate things to point out.

For starters, I'm not entirely sure that a separate, pluggable static analysis phase is in fact a type system at all, particularly if you aren't using it to definitively reject programs. In a formal context, type systems are an intrinsic and (generally) inseparable part of the language. If you have a type system that is separable from the language, that type system is not a type system at all but merely static analysis. Now, there's absolutely nothing wrong with that! I think static analysis is a very useful tool and one which can answer much deeper and (often) more revealing questions than a baked-in static type checker. However, I think it's a definitional hair that's worth splitting because it has some fairly profound implications (such as the strength of your guarantees and how much you have to worry about composability).

Addressing the idea in general: I think it's a good one. The more questions we can answer about our code before it sits in front of customers, the better we are. We live in the information age, where an incredible amount of research and effort is being put into developing tools that allow us to make sense of truly astronomical amounts of data. Why shouldn't we be applying those tools and techniques to programming? I've seen some code bases which could qualify as astronomical in size, particularly once you consider the information density of most programming languages. Why shouldn't we be using rich, statistical tools to ask semantically deep questions about our code base? This seems like a natural evolution of the modern development process, at least to me. Pluggable static analyses make it possible to apply these sorts of techniques without being tied to the compilation cycle.

There are some very deep traps to beware though. For example, a "pluggable" type system is really not very useful unless you can compose multiple type systems into a coherant whole, and this is where problems arise. Anyone who has studied type theory in a formal context and run a few soundness proofs will understand that seemly innocuous and self-contained type rules will almost always interact in surprising ways. A good example of this is extending the simply-typed lambda calculus with reference values. The moment you do this, your type system explodes in complexity, despite the conceptually tiny nature of the change.

The fact is that you can't just tease type systems apart into composable atoms. Their features are intertwined; they are the very soul of complexity. (note: I'm not saying that they are complex to use or even to understand, but they are certainly complex to design and build) So, while I think that we're going to see an up-tick in the richness and proliferation of static analysis tools, I do not think you're going to see them really replacing type systems. I see these two concepts and orthogonal and complementary.

Is there anything in Clojure that you wish that Scala provided?

Oh, wow... I think the biggest thing is that Clojure has a much stronger focus on functional programming in that it carrots developers (nearly forcibly) to control their state. Anyone reading this who hasn't already watched Rich's talk on concurrency needs to go out and do that right now (then watch it again). The notion of epochal time is quite central to Clojure, and its benefits are manifest. Scala on the other hand is a little bit more laissez-faire with respect to state. Well-written Scala code is going to keep state on a tight leash and will end up looking a great deal like well-written Clojure code, at least in terms of where state is and how it is respected. However, Scala provides comparatively few incentives for developers to do the Right Thing. Unlike Java, it doesn't provide disincentives, but it also doesn't bias the coin in the right direction as Clojure does.

On another note, Clojure is a much simpler language than Scala, and that makes it very nice for a lot of things. Clojure code has an aesthetic which really appeals to me. That's not to say that Scala is overly complex or ugly, I just happen to really like a lot of what I've seen from Clojure. I certainly think there are syntactic corners of Scala that could be substantially smoothed. Maybe not really taking inspiration from Clojure, but certainly improving things in an area where Clojure is quite strong.

By and large though, I think that most of the awesome bullet-points that Clojure hits have already been stolen whole-sale by Scala. :-) A few examples: vector, map/set, agents (in Akka), STM (in Scala STM), SLIME (see ENSIME), etc. The Scala community is very actively watching the Clojure community. Y'all are a very fertile source of inspiration!

(take 5 baishampayan-ghose)

2012-03-16T00:00:00-07:00

To celebrate the first day of the Clojure West conference, this entry of the (take...) series forcuses on the inimitable Baishampayan Ghose. As Clojure grows and gains mindshare, more and more companies are betting their success on the language, and Baishampayan's company Infinitely Beta is quite unique. In this installment we'll focus on finding Clojure, Clojure in India, and Clojure's past conferences.

How did you discover Clojure?

I have been a Lisp aficionado for as long as I can remember. I studied Scheme when I was in college and my first job out of college was a travel startup where I had the privilege of building an air search & ticketing platform in Common Lisp.

I first read about Clojure on comp.lang.lisp in early 2008 but I was quick to dismiss it in my mind because I was not too sure about its utility and honestly, I abhorred the `J' word :-)

One day, we incurred a loss of a million Rupees at my company because of a bug that got triggered by my (apparently pure) code due to our usage of mutable datastructures in Common Lisp. That made me grow a distaste for any language that had mutable datastructures by default and I started looking at Clojure in earnest.

I watched Rich's videos, read anything that talked about Clojure and started writing little bits of code in Clojure to get a feel of the language. It took me some time to understand Clojure's state management semantics and its relationship with the JVM but when I did, I was truly enlightened.

How are you using Clojure in your business?

We are two year old startup and we've been a Clojure company from day #1.

We've built two products so far, the first one being a equities research and portfolio management application for the Indian stock market trader. We used Clojure to build the backend that would do all the data-processing and expose a REST API for the frontend to consume.

The new product that we are building is a social CRM platform for enterprises. This product is built completely in Clojure from front to back. We are expected to launch before the end of this year and I will make announcement when we go live.

The only thing left for us is to move to ClojureScript and then our circle will be complete.

What motivated you to start Planet Clojure?

I was just scratching my own itch. That was mid-2009 -- we were busy building our first product and I wanted to keep a tab on all the Clojure related blogs in one place so that I could keep myself abreast with the latest happenings in the Clojure world. We have come a long way since then as we now track more than 300 Clojure blogs! A big shout-out goes to Alex Ott who has been instrumental in growing and managing Planet Clojure.

You've been to both Clojure conferences (and now the third!); what were the differences between the latest and the first?

My first reaction when I heard about the Clojure/conj 2010 was "Wow! Clojure is going mainstream". When I got to the conference I was amazed to meet and interact with the stars of the Clojure community and people whom I had known only from their blogs/tweets. I was fascinated to peek into the minds of people using Clojure to solve real-world problems, I felt that my decision to bet my startup on Clojure not all wrong -- I felt vindicated.

To me, the first conference was about Clojure asserting itself as a first-class programming language that could be used to solve all kinds of problems elegantly; it was about firmly establishing the very ethos of the language.

But the last one took the cake. While the first one was about Clojure being able to do certain things, the last one was about showing the world how it's done.

Since the first Clojure/conj the community has been able to come up with some absolutely mind-blowing things that address some really tough problems, problems that people otherwise dare to attempt solving.

Here we have David Nolen going into the hard-core land of Logic Programming, Pattern Matching and Predicate Dispatch, giving us tools to tackle really hard problems with equal amounts of ease and elegance. With the blessings of none other than Professor Friedman himself, I am sure David (and Ambrose) will take us to even greater heights.

We have Sam Aaron, with his amazing creation Overtone showing us how we can transplant Clojure's mind-blowing abstractions into a completely different domain, and thereby augmenting the whole creative process of making music.

And of course, there was ClojureScript, yet another gift to the world from Rich Hickey proving once again that simplicity, power and focus, when combined can help us solve some really tough challenges.

In a way, while Clojure/conj 2010 was about signalling the arrival of Clojure, Clojure/conj 2011 was about taking Clojure to completely uncharted territories, where no other programming language has gone before.

What is the state of Clojure in India?

As far as I know, we are probably the only Indian product company that does Clojure full-time. There are a tonne of people who are using or learning Clojure at a personal level but I guess there is still time before it's adopted by more companies. Most software companies in India do services exclusively so many times they can't bid for Clojure projects because they don't have any Clojure expertise and since they don't get those projects they don't find it worthwhile to invest in developing any expertise in Clojure. So it's kind of a chicken-and-egg problem. Having said that, India has a growing startup ecosystem and I think if Indian startup founders adopt Clojure in some way it will not only be a great competitive advantage for them, it will also help them in attracting talent -- it has certainly been that way for us.

I do a lot of Clojure evangelism in India by speaking at events, user-group meetings, etc. In the past I had conducted a 2 a day, completely gratis Clojure course and I would like to conduct one again, given sufficient interest.

(take 5 kevin-lynagh)

2012-02-28T00:00:00-08:00

Kevin's company Keming Labs was one of the earliest adopters of ClojureScript and his talk at the second Clojure Conj focused on how they use it, how it helps them succeed, and some of its painful aspects. Specifically, he focused on the use of the D3 JavaScript data visualization library by way of ClojureScript interop. Kevin has contributed to ClojureScript itself by helping to spread the word about its advantages and to contribute fixes and libraries.

How did you discover ClojureScript?

ClojureScript came to my attention either through Hacker News or my Github activity feed. I used Clojure for the first time in 2010 with Ring to build the backend for a document annotation service. Most my work now is clientside data visualization, so when ClojureScript came out I jumped on it.

How are you using ClojureScript in your business

Right now the ClojureScript pilot project at Keming Labs is a hotel analytics dashboard. We've wrapped D3, Mike Bostock's data-driven JavaScript DOM manipulation library, and use it from ClojureScript to build statistical graphics in the browser.

Hotel operators get occupancy and pricing data every week as Excel spreadsheets; we scrape those data using JVM Clojure, throw it into PostgreSQL, and then serve it to the browser using Chris Granger's awesome Pinot + Noir projects. Pinot's "remotes" in particular are pretty cool---they let you make server side calls which return Clojure maps and vectors to the client. So far we haven't needed to muck around with JSON at all.

We have a lot of technical freedom on our projects: clients come to us with handfuls of data and just want a report or some dashboards for their iPad or something. Much of the time they could care less about the backend details. Most of us have mathematical backgrounds (I studied physics in university), and that community has historically had good relationships with functional, REPL-happy languages like R and Mathematica. Since so much of our work is data-oriented, Clojure and ClojureScript have been a great fit.

What are your general feelings about the languages that target JavaScript?

I think they're a good idea.

JavaScript has a lot of reach, but the language itself is half-baked.

Developing large applications and reusable components in JavaScript is very difficult. It takes a lot of discipline and experience to know all the tricks for encapsulating state, namespacing, dealing with multi-arity dispatch, &c. Either you spend a lot of time and mental energy learning this trivia, or you use a language where these semantics are baked-in rather than sprinkled-on "good parts".

An iOS and web developer friend of mine hates these kind of languages and says "you should just develop natively". I'm not sure what that even means, though. It's not like we're running on "JavaScript machines". There are probably a dozen layers between the code I write and what actually gets executed as little nibbles of god knows what on the CPU registers. There is a cost to the additional abstraction of languages, sure, but depending on the complexity of the project and your appetite for risk, the benefits can really outweigh raw JavaScript.

Things are new, definitely; it's the bleeding edge. Sometimes shit breaks, the abstractions leak, and you've got to dive into the JavaScript to really see what's going on. Ultimately I think many of these problems are just bleeding edge issues; fundamentally there isn't anything inherently wrong with building up abstractions. I've never had to dive into C++ to figure out a bug in my JavaScript DOM-manipulation code. Hopefully in a few years I'll never need to look at the JavaScript that gets emitted from my ClojureScript.

The biggest challenges with these kind of languages isn't technical, it's social. For many people, "frontend" means HTML+CSS+JavaScript, end of story. Plenty of folks don't even have a frontend build chain for concatenating and minifying assets, so it can be hard to even talk about languages that compile into JavaScript.

We've been going back and forth with a large scientific journal publisher on this very issue.

So far, things have boiled down to:

We deliver minified JavaScript and you have to rely on us for all maintenance.
We deliver source code in some crazy hipster language none of your people have heard of, and then train them to use some elaborate build process.

The biggest ClojureScript challenge is the same as that of JVM Clojure, or any new language. You have to be very upfront about the risks and benefits and figure out what best addresses the client's needs.

Did you try any of the other languages before settling on ClojureScript?

I moved over to CoffeeScript the summer of 2010 when I realized there are parts of JavaScript I just never want to know about (e.g. 0 vs. "" vs. undefined vs. null). I think CoffeeScript is the best thing JavaScript can be, as a general purpose dynamic language. CoffeeScript's syntax is very clean, and it basically files down JavaScript's rough edges while retaining the semantics. You can train a JavaScript developer to use CoffeeScript in a day or two, and automatically his or her code is just a ton better because the language keeps you from forgetting "var" or otherwise shooting yourself in the foot.

CoffeeScript is a great local maximum. Especially now that it's bundled into Rails, I think CoffeeScript will suffice for most developers on most of today's projects As people start building richer clientside applications, though, they will start feeling the limits JavaScript/CoffeeScript's semantics. I'm not sure how people are going to react to these limits. We're betting on Clojure, but I can easily imagine most developers dealing with JavaScript's language limitations by just building up more libraries and frameworks.

How/where can ClojureScript improve?

Right now stuff is pretty raw. You have to know a fair bit about Clojure and JavaScript tooling to use ClojureScript effectively. When ClojureScript first came out I killed most of a Saturday just trying to get it into my JVM classpath and have it spit out a console.log("hello world");

At the time there wasn't exactly a ton of example projects on Github I could rip off infrastructure code from.

(The complete state of our ClojureScript infrastructure understanding is in the cljs-d3 README.)

Right now all of our ClojureScript projects access the compiler via git submodules, which feels gross. Lein and Cake are awesome at managing dependencies, and it'd be great if there were an official ClojureScript build that integrates into that system (i.e., something you can just add to project.clj).

The browser REPL looks promising, but it is just different enough than the usual SLIME/swank workflow that I haven't really used it much. So far we've been delegating to D3 for DOM manipulation, which means most of our ClojureScript code is statistical / data transformation in nature, so I've been cheating a bit and just using the SLIME. I'm thinking about building a declarative visualization library like D3 in pure Clojure(Script), but before I get really deep into that it might be worth spending some time polishing up a browser REPL starter kit with nice Emacs integration.

ClojureScript the compiler itself is remarkable given its age; pretty much everything I've written that works in Clojure works in ClojureScript. What I'm most excited about is seeing more open source ClojureScript code. Clojure the language is just really elegant, and the community is full of freakishly bright early adopters who aren't afraid of the academic literature. I've learned a ton reading Clojure code, so I'm interested to see how Clojure folks handle the clientside---user interaction, asynchronous processing, &c. Callbacks get gnarly pretty quickly, and the Clojure community and language are powerful enough to explore and implement other approaches like functional reactive programming and state machines.

I'm just a guy who makes charts and graphs, so I'm expecting my mind to get blown as the Clojure community starts tackling the web.