Talks

This year I was unable to attend all of the conference due to some scheduling problems, so I can’t give an in depth report on talks. I used some of the time that I might normally have spent in talks to catch up with people that I haven’t seen in a while. I was able to attend a good number of the talks in the Hadoop track. The track was larger than last year’s track (due in part to a larger room), but I felt that last year’s track was stronger. It might also be that I’ve become a bit more familiar with Hadoop, making it harder to make a bing impression. It’s definitely the case that there was a lot of interest in Hadoop, and I expect that to continue.

Unfortunately, I missed the NoSQL meetup during the Apache BarCamp. I think that there could/should have been an entire NoSQL track, especially given the fact that Cassandra and CouchDB are both frequently mentioned NoSQL technologies, and both are housed at Apache.

One talk that surprised me was Ross Gardler’s talk Teaching and Learning about Open Development, originally I didn’t think that I would have time to stay for that talk slot, but a rearrangement of my return flight loosened my schedule so that I could stick around. Ross is the chairman of the newly created Community Development PMC at Apache. This is a new effort aimed at improving the experience of contributors and new committers. Some of the people on the PMC have been heavily involved in the ASF’s Google Summer of Code outreach, and will be bringing their experiences over with them. It seems like this PMC will also be a good place for people concerned about diversity issues to dig in and help in a concrete fashion.

Celebration

This year’s ApacheCon’s have been a celebration of the 10 year anniversary of the founding of the Apache Software Foundation. At Oakland, there was a cake, a proclamation from the Mayor of Oakland, and (I didn’t get to see this) a letter of congratulations from the Governor of California. Rather than try and describe the festivities in prose, I’ll leave you with some photos:

The entire set of photos is up on Flickr.

November is just around the corner, which means that once again it’s time for ApacheCon US. This year is a special year for the Apache Software Foundation – its 10 year anniversary. Since I got involved with Apache just a few months after the foundation was created, it is also my 10 year anniversary of being involved in open source software.

This year I am going to be speaking twice. On Wednesday I’ll be speaking on the Apache Pioneers Panel, and on Thursday I’ll be giving a talk titled How 10 years of Apache has changed my life. I owe a huge professional debt to the ASF and its members and committers, so in my talk I’ll be interweaving important events in the life of the foundation with my own personal experiences and lessons learned.

Unfortunately, I’m not going to be there for all of the conference this year – I’ll be arriving Tuesday afternoon and flying out on Thursday evening. If you want to meet up, I’m in the ApacheCon Crowdvine, and I’ll be around with camera in hand (and on the LumaLoop).

The following week at DjangoCon, I lost the strap portion of my Upstrap quick release strap. I liked the Upstrap, but it wasn’t ideal. The Upstrap was great because of the non stick rubber pad that they use – it really won’t move. But like most other camera straps, I found that I was constantly getting it fouled in my arms or something, especially between landscape and portrait modes.

Duncan had promised me one of the early prototypes of the LumaLoop, so I put the official black and neon yellow strap on the D3 and waited patiently. Yesterday, my LumaLoop arrived, and I quickly installed it in place of the Nikon strap. The LumaLoop is a “sling strap” similar to the Black Rapid R-Straps that have become popular recently. The Black Rapid straps screw into the tripod socket on your camera, which is a problem if you have any kind of heavy duty tripod plate mounted on your camera, or if you shoot vertically a lot (this is even more of a problem if you have small hands and a camera with a battery grip). The LumaLoop attaches to one of the regular strap mounts on your camera, and once attached, you can slide the camera up and down the strap. The mounting loop is attached with a quick release clip, so swapping cameras/straps is easy as well. Duncan has a series of blog posts that detail the reasoning behind the design:

• Hacking the R-Strap
• Camera Strap Update 1
• Camera Strap Update 2

Here’s a quick snapshot of mine:

You can see the loop part that goes on the camera, as well as the quick release between the loop and the rest of the strap. It’s a bit harder see the padded non-slip shoulder pad.

The LumaLoop is going to be available from Luma Labs sometime very soon (Duncan gave me perimission to talk about the LumaLoop in advance of its general availability). You can follow Luma Labs on Twitter to keep up with all of the news and the official announcement. I’m excited to have a strap that both holds my camera securely and stays out of my way when the action gets going.

I made the statement

From where I sit, this is all about exploiting multicore hardware

because I’ve seen a pile of actor and other concurrency libraries which have not taken parallel execution of the concurrent program seriously. If I am going to go to the trouble of writing a concurrent program, then I want that execution to be parallel, especially in a multicore world.

Simon Marlow from the GHC team said that if programming multicore machines is the only goal we ought to be looking at parallelism first and concurrency only as a last resort. Haskell has some nice features for taking advantage of parallelism. However, I explicitly stated that I was not as interested in highly regular or data parallel computations, which is what Haskell’s parallelism tools are aimed at. These are fine ways to get parallelism, but I am interested in problems which are genuinely concurrent, not just parallel. In a Van Roy hierarchy, these are the problems with observable nondeterminism. I also specifically called out reduction of latency as one of my goals, something which Marlow says is a possible benefit of concurrency. The GHC team is interested in a different mix of problems than I am.

Van Roy in short

I also forgot to mention Peter Van Roy’s paper Programming Paradigms for Dummies: What Every Programmer Should Know, which includes an overview of his stratification of concurrency and parallelism (and other stuff). If you don’t have time to read his book, the paper is shorter and more digestible.

The Cambrian

The Cambrian period was marked by a rapid diversification of lifeforms. I think that we are in a similar situation with concurrency today. Although many of the ideas that are being tossed around for concurrency have been around for some time, I don’t think that we really have a broad body of experience with any of them. So I’m less optimistic than Tim and Bruce Tate, at least on time frame. I think that we have a lot of interesting languages, embodying a number of interesting ideas for working with concurrency. I think that some of those languages have gained enough interest/adoption that we are now in a position to get a credible amount of experience so that we can start evaluating these ideas on their merits. But I think that the window for doing that is pretty large, on the order of 5 to 10 years.   

What kinds of problems

The kinds of problems I am interested in are general purpose programming problems. I’m specifically not interested in scientific, numeric, highly regular kinds of computations or data parallel computations. Unlike Tim, I do think that web systems are a valid problem domain. I see this being driven by the need to drive down latency to provide good user response time, not to provide additional scalability (although it probably will).

It’s not like Java

Erik Engbrecht, one of Tim’s commenters said:

To get Java, you basically take Smalltalk and remove all of the powerful concepts from it while leaving in the benign ones that everyday developers use.

I think there’s something to be learned from that.

This presupposes that you know what all the good concepts are and what the benign ones are. It doesn’t seem like we are at that point. When Java was created, both Lisp and Smalltalk had existed for quite sometime and it was possible to do this kind of surgery. I don’t have a clear sense of what actually works well, much less what is powerful or benign.

The hardware made me do it

From where I sit, this is all about exploiting multicore hardware, and when I say this I mean machines with more than 4 or 8 hardware threads (I say threads, not cores – actual parallelism is what is important). The Sun T5440 is a 256 thread box. Intel’s Nehalem EX will let you build a 128 thread box later this year. Those are multicore boxes. If you look at experiments, you see that systems that seem to work well at 2 or 4 threads don’t’ work well at 16 or 64 threads. Since there’s not a huge amount of that kind of hardware around yet, it’s hard for people to run experiments at larger sizes. Experiments being run on 2 thread MacBook Pro’s are probably not good indicators of what happens at even 8 threads.. This is partially because dealing with more hardware threads requires more administrative overhead, and as the functional programming people found out, that overhead is very non-trivial. The point is, you have to run on actual hardware to have believable numbers.   This makes it hard for me to take certain kinds of systems seriously, like concurrency solutions running on language implementations with Global Interpreter Locks. See David Beazley’s presentation on Python’s Global Interpreter Lock, for an example.

Comments on specific languages

At this point I am more interested in paradigms and constructs as opposed to particular languages. However, the only way to get real data on those is for them to be realized in language designs and implementations.

• Haskell – Functional Laziness aside, the big concurrency thing in Haskell is Software Transactional Memory (STM). There are other features in Haskell, but STM is the big one. STM is an active research field in computer science, and I’ve read a decent number of papers trying to make heads from tails. Among the stack that I have read, it seems to be running about even between the papers touting the benefits of STM and the the papers saying that STM cannot scale and will not work in practice. The jury is very much out on this one, at least in my mind.
• Erlang – I like Erlang. It’s been in production use for a long time, and real systems have been built using it. In addition to writing some small programs and reviewing some papers by Erlang’s designers, I spent a few days at the Erlang Factory earlier this year trying to get a better sense of what was really happening in the Erlang community. While there’s lots of cool stuff happening in Erlang, I observed two things. First, the biggest Erlang systems I heard described (outside of Facebook’s) are pretty small compared to a big system today. Second, and more importantly, SMP support in Erlang is still relatively new. Ulf Wiger’s DAMP09 presentation has a lot of useful information in it. On the other hand, BEAM, the Erlang VM is architected specifically for the Erlang process/actor model. This feels important to me, but we need some experimental evidence.
• Clojure – Clojure as a ton of interesting ideas in it. Rich Hickey has really done his homework, and I have a lot of respect for the work that he is doing. Still it’s the early days for Clojure, and I want to see more data. I know Rich has run some stuff on one of those multiple hundred core Azul boxes, but as far as I know, there’s not a lot of other data.
• Scala – The big thing in Scala for concurrency is Actors, but if you compare to Erlang, Actors are the equivalent of Erlang processes. A lot of the leverage that you get in Erlang comes from OTP, and to get that in Scala, you need to look at Jonas Boner’s highly interesting Akka Actor Kernel project. Akka also includes an implementation of dataflow variables, so Akka would give you a system with Actors, supervision, STM, and Dataflow (when it’s done).   
• libdispatch/Grand Central Dispatch – Several of Tim’s commenters brought up Apple’s Grand Central Dispatch, now open sourced as libdispatch. This is a key technology for taking advantage of multicore in Snow Leopard. GCD relies on programmers to create dispatch queues which are then managed by the operating system. Programmers can send computations to these queues via blocks (closures), which are a new extension to Objective-C. When I look at Apple’s guide to migrating to GCD from threads, I do see a model that I prefer to threads, but it is not as high level as some of the others. Also, the design seems oriented towards very loosely coupled computations.   It will be several years before we can really know how well GCD is working. I am typing this post on a 16 thread Nehalem Mac Pro, and I rarely see even half of the CPU meters really light up, even when I am running multiple compute intensive tasks. Clearly more software needs to take advantage of this technology before we have verdict on its effectiveness in production.
• .Net stuff like F#/Axum, etc – There is some concurrency work happening over on the CLR, most notably in F# and Axum. I spent some time at Lang.NET earlier this year, and got a chance to learn a bit about these two technologies. If you look at paradigms, the concurrency stuff looks very much like Erlang or Scala, with the notable exception of join patterns, which are on Martin Odersky’s list for Scala. I will admit to not being very up to speed on these, mostly for lack of Windows and the appropriate tools.

Other thoughts

Jacob’s take away from my talk at OSCON was “we’re screwed”. That’s not what I wanted to convey. I don’t see a clear winner at the moment, and we have a lot of careful experimentation and measuring to do. We are quite firmly in the Cambrian, and I’m not in a hurry to get out – these things need to bake a bit longer, as well as having some more experimentation.

In addition to my talk, and Tim’s wiki page, if you are really interested in this space, I think that you should read Concepts, Techniques, and Models of Computer Programming by Peter van Roy and Seif Haridi. No book can be up to date with the absolute latest developments, but this book has the best treatment that I’ve seen in terms of trying to stratify the expressiveness of sequential and concurrent programming models.

Talk Highlights

Here are some of the talks that I found notable:

Shawn Rider and Nowell Strite of PBS gave a talk titled: Pluggable, Reusable Django Apps: A Use Case and Proposed Solution. I think that very few people in the Django community had any idea that Django was being used extensively in PBS. That definitely falls into the category of pleasant surprises.    One of the strengths of Django is the focus on building small single purpose applications which can be used to build up larger applications. Doing this is harder than it sounds, and Shawn and Nowell described some of the problems that they ran into, as well as showing some ways of dealing with those issues. There was some crosstalk between the PBS guys and the Pinax developers, who are also doing a lot with reusable apps. I hope that these folks will work to share and combine their knowledge and then disseminate that to the broader Django community.

There were a number of talks which followed the theme of “how not to use parts of Django”. It’s interesting because people like Django, and even if they don’t like some parts, they want to use the rest, and are willing to work to make that possible. You would expect people to just walk away from the framework in cases like these.

Eric Ocean, one of the developers of SproutCore gave what I considered to be a pretty interesting talk. Unfortunately, his talk didn’t have much of a connection to Django, other than to suggest some things that Django could do to support SproutCore better. I know from watching the IRC and Twitter back talk, that people were put off by the style of presentation (a little too like a commercial), and the weak connection to Django. SproutCore is interesting to me because it’s at a different level than most of the Javascript frameworks. It’s at a higher level, which I think will be necessary as browser based applications become more sophisticated. I know that I am going to be taking a closer look at SproutCore, and I hope that a useful Django/Sproutcore collaboration will emerge from the sprints.

Simon Willison gave a keynote about Cowboy programming. The big piece for me was his description of how the Guardian built an application to help the public scrutinize the expenses of British MP’s. There’s something about these situations that appeals to me, against my knowledge and better judgment of “sound software engineering practices”. I guess it’s a guilty pleasure of sorts.

Ian Bicking gave a keynote which might be described as “a free software programmer’s midlife crisis”. Ian was very philosophical and reminded us that free software (as opposed to open source software) was rooted in a set of moral (not economic or process) imperatives. It was a very thoughtful speech, and I think that its worth several reads of his text (something which is hard to do on a train ride with an iPhone) and some additional ponderings.

Avi Bryant’s keynote took its root in his experiences building Trendly. As one might expect, Avi started building Trendly using Seaside. But by the time he finished, he noticed that very little of Seaside was actually being used. He attributed this to the fact that Trendly’s architecture involves loading a single HTML, with a ton of Javascript. That Javascript then manages all of the interaction with the server, which consists of snippets of JSON data. This range true to me because we used a similar architecture for Chandler Hub, the web based version of Chandler (our interaction with the server was based on atom and atompub, not JSON), and it’s the kind of architecture that GMail is based on. Avi also treated us to a demonstration of Clamato, his Smalltalk dialect that compiles to Javascript. Again, another attempt to deal with the challenges of engineering large Javascript applications in a web browser.

There were plenty of other good talks, and many of the slides are already available.

My keynote

I’m afraid that I am not equal to the task of writing out my presentation text in full as Ian and Jacob have done, so you will have to settle for the highlights and wait until the video appears.

My keynote was organized around two major sections.

The first section was a look at what I see in the Django community at present. This includes a look at some pseudo statistics around job postings and a poll of web frameworks being used by startups in an effort to get some view into whether and how much adoption of Django is happening. The short answer is that things look promising, but there is still plenty of room to grow. On the technology side, I pointed out the emphasis on combining applications and the work of the Pinax and PBS folks. The other major technology thing that I called out was GeoDjango, which is undoubtedly the most sophisticated GIS functionality in any web framework in any language or platform. This is going to be very attractive to people building location aware mobile apps, and I showed two examples of augmented reality applications as illustrations. This section ends with some observations about the Django community, using the PyCon sprints as an example. Ok, there are also some lighthearted slides about Django’s mascot, the djangopony.

The remainder of the talk was about the ways that web applications are changing and how Django might adapt to them. There are (at least) three groups of people that will be impacted by these changes. From the view point of users, the two big things are richer, more interactive applications, and access from location enable devices. Developers are going to need help in dealing with these new requirements, and the people who operate web applications need much more support than they currently have.   

I see several technologies that will be important in facilitating these changes. The first of these is some Rich Internet Application technology. The second is API’s to web applications. A digression on this point. When the iPhone was introduced, the only way to develop applications was using web technologies. This made a lot of people very angry, and Apple followed up with the ability to build native platform applications. It should be possible to build rich web interfaces on the iPhone. My observation is that given the choice beween a rich web interface and a native iPhone application, users pick the native application. Look no further than the furor over the native Google Voice application. The native applications are talking to the servers using API’s. Those API’s are not just cool Web 2.0 frosting. The last technology is cloud computing, which started out as a deployment/operations technology and is now moving up to impact application development at many levels.

In light of this, what are framework developers to do? I did a quick survey of several web frameworks which have interesting ideas or approaches in them, so that the Django folks could see what their “competitors” are up to. The frameworks that I included are Rails, Lift (Scala), Webmachine (Erlang), Nitrogen (Erlang), CouchApps (CouchDB + JavaScript – this isn’t quite a framework in the traditional sense, but it met the spirit of my criteria), and Javascript. In the case of Javascript, the observation is that the rapid increase in Javascript performance coupled with a good Javascript framework leads to something which is economically attractive (same technology in the server and client).

The talk finishes with a set of proposals for “science projects” that might be attempted in the context of Django. Some of what I outlined is emerging, and in some cases speculative. Django doesn’t need to blow itself up and start over. Instead, what’s needed is for people with Django sensibilities to look some of these problems and see if a Django flavored solution can be found. Here’s the list of projects:

1. Asynchronous Messaging – if there’s any use of messaging, it’s typically to do jobs in the background. What would happen if we made the use of messaging pervasive throughout the framework?
2. Comet – I think that the Django+orbited approach to Comet is limited in comparison to what you see in Lift or Nitrogen. Can Django do Comet support at the same level (or better) than these frameworks? What would happen if the Comet stuff were hooked up directly to the messaging stuff I just described? Imagine the equivalent of urls.py that routed Comet requests to messaging.
3. REST – There are several good packages for dealing with REST in Django. It would be nice to have this all packaged up neatly and made available for people.
4. Deployment – This is really a mess. Are there changes that could be made to Django to make it easier to deploy, or to work better with tools like Puppet, Chef, Fabric, etc?
5. Monitoring – Typically frameworks provide very little monitoring information. It seems like there is a lot that could be done here.
6. Analytics – Once you have raw monitoring information, the next step is to do some analytics on it. Django is famous for creating admin UI’s with a very small amount of effort. What if we applied that same thinking to analytics?
7. Cloud – If you add up the first 6 items, you are well on your way to what might be a cloud friendly framework. There is still some work to do in terms of making applications on the framework adapt to elastic deployment scenarios, but it would be a good step.
8. Stacks – A very basic step towards cloud stuff would be to build a preconfigured stack of software to run/develop a Django app. This is a controversial idea, because everyone has their own idea of what software should be in such a stack, and how all the configuration switches should be set. I still that having one (or more) such stack would help more than it would hurt. In my ideal world this stack would be delivered as a virtual machine image that could also be uploaded to cloud providers.

All of the talks were videotaped so those of you who were unable to attend will be able to catch up soon.
I had a great time hanging out with the Djangonauts. My thanks to Leah Culver and Robert Lofthouse for inviting me to speak.

1. What sucks, now, about web development?
2. How will we fix it?
3. Can we fix it with Python?

There’s some good stuff in here, and it’s definitely a worthwhile read.

Jacob is in South America, which means he won’t be at DjangoCon next week. I was disappointed about that before I saw his talk, and I’m even more disappointed now. I’ll be giving the last keynote at DjangoCon, and I’ll be discussing some thoughts and ideas for where Django (and other web frameworks) might go next. It would have been a great opportunity to carry on the conversation in person. I guess we’ll be doing by blog instead.

One of the topics that Jacob covered in his talk was concurrency, and he pointed to my OSCON talk on concurrency constructs as something that has influenced his thinking. I do think that he got the wrong idea from my conclusion. At the moment I don’t see a clear solution for concurrency, but I don’t believe that the situation is hopeless, either. I think that we are looking at a period where we have a lot of experimentation, and I think that’s a good thing. It is way premature to say that we have a solution, and I’d rather people keep experimenting.

I have some more thoughts on some of Jacob’s points, but those are in the keynote, so I’ll save those until after I’ve actually given the presentation at DjangoCon on Thursday. I suspect that the other keynote speakers, Avi Bryant, and Ian Bicking to have some thoughts in this general direction as well.    I think it would be great to have an open space on these topics sometime on Thursday.

Is organization the problem?

On the other hand, I think that I’m slightly more optimistic about the situation than Chris is. Chris holds up the idea that there ought to be a design dictator, who drives the design and preserves the unity of the design. I’d point out that there are some open source communities where there are such people. Perhaps the best example that I can come up with are the programming languages. A good language is very hard to design. It needs to have the kind of unity that one expects to find in a good design. In some of the language communities, these designers have titles such as “Benevolent Dictator for Life”, and the community as a whole has recognized their giftedness and given them the ability to make final binding decisions about design issues. This isn’t end user facing desktop or web software, but it’s also not bunches of libraries, or implementations of JSR’s, IETF RFC’s, W3C recommendations or POSIX standards. These situations are very delicate mixes and their success is highly dependent on the particular individuals who are involved, so they tend to be rare. Nonetheless, I do think that its possible for communities to work even if there is a chief designer.

I also don’t think that there needs to be a single chief designer. Chris cited Luke Wroblewski’s description of the design process at Facebook. Very early in that post you read:

The Facebook design team works on product design, marketing, UI patterns, branding, and front-end code. The team consists of 15 product designers, 5 user interface engineers, 5 user experience researchers, 4 communication designers, and 1 content strategist. 25 designers in a company of 1,000.

Design can be done by teams. I think that we all know that, but in many of the discussions that I’ve had on this topic, the focus seems to be on the need for a dictator. The dictator model works, but so does a team model.

I think that the organizational challenges of design (dictator vs team) can be dealt with. If you bootstrap a community with a DNA that is committed to good design, and to the value of a good designer, and if the designer has won the respect of the community, then I can see a path forward on that front.   

The problems that I see

In my mind the problems are:

How do you find a designer or designers who want to work in this kind of environment? We know that not all developers are well suited to distributed development. I’d venture that the same is true for designers. It’s much easier for coders to self select into a project than it is for all other types of contributors, including designers.

How can a non-coding designer win the respect of a (likely) predominantly coding oriented community? If you believe that open source projects should be organized around some notion of merit, then what are the merit metrics for designers? Who evaluates the designers on these metrics? Are the evaluators even qualified to do so? In my examples of communities with designers, those designers are all coders as well.

Can we actually do design using the commonly accepted tools of e-mail, version control, wiki’s and bug trackers? The design process relies very heavily on visual communications. The code (including design and architecture of code) process is predominantly a text based process. It is very difficult to do design efficiently in a distributed setting using the existing stable of tools. This is going to be a challenge not just for designers but for many other problem domains that could benefit from commons-based peer production.

What’s with you and that long word?

I prefer to use Yochai Benker’s term “Commons Based Peer Production” instead of the term open source. The problem with the term open source is that everyone means something different when they use it. Some people just mean licensing. Some people think of a particular community’s set of practices. Others think that it means some kind of fuzzy democracy and mob rule.   
One of the reasons that I went to work at OSAF was to see if it was possible to design a good end-user application via some kind of community oriented design process. As far as I am concerned the jury is still out.

The conference

Every other OSCON that I’ve been to (since 2003) has been in Portland, and in some ways the two have become synonymous for me. I’m not taking the move to San Jose very well. There are a variety of little things, like the fact that you could end up walking 1/4 of a mile to get from one talk to another only to end up reversing the trip for the next session. At the end of Thursday, I bagged going to a talk because I was tired of walking back and forth. I had a bad experience (much worse than usual) with the WiFi connection in the hotel where I was staying, something that I don’t tolerate very well. The fact that the hotel acknowledged the problem and then offered drink vouchers as an apology didn’t help any. I had to ask the checkout agent to remove the charges for the days that I got under 20kb/s. If you take the view (which I do) that OSCON really starts at 6pm and ends at 3am, then downtown San Jose doesn’t really hold a candle to downtown Portland. My understanding is that OSCON only has a one year contract for San Jose, so maybe we’ll get something else next year. I hope so.

Another thing about OSCON relates to the attendees themselves. I was unsurprised to hear that attendance was down. The combination of the economy and the move away from Portland could explain some of that. The lunch hall seemed pretty full (and the food was very good for a conference lunch – maybe the best I’ve ever had), and it seemed a decent size to me. What I noticed was something else. Normally when I show up at OSCON, even on the first day of tutorials, it is pretty hard to go very far before I run into someone that I know. This year that was not the case, and I don’t feel that it improved that much once the conference proper began. In combination with the move to San Jose, this had a pretty major impact on the value that I got out of the conference.

The talks

This year I wound up all over the map session wise. I took in some sessions on tools: The SD distributed bug tracker, and Theo Schlossnagle’s talk on his new monitoring system, Reconnoiter. I also attended Tom Preston-Warner’s talk on github. His talk ended up being much more about git in general. I was hoping that he would have more to say on the social/community behaviors that they’ve observed on projects on GitHub. There’s not a lot of data on how the use of DVCS’s is impacting the social/community dynamics of open source projects, and the folks at github are in a unique position to observe some of this. Maybe next year.

I also continued to gather more information on things related to cloud computing. In this case there was some storage stuff in the form of Neo4J and Cassandra. Adam Jacob’s s talk on Chef was well attended despite being in the last session block of the conference, and people stayed well past the ending time for the Q&A. Reconnoiter also falls into the cloud tools space. I attended Kirrily Robert, Yoz Grahame, and Jason Douglas’ talk titled “Forking Encouraged: Folk Programming, Open Source, and Social Software Development“, hoping to glean some insight or data into “fork oriented” open source. That wasn’t really what I got. The talk was fairly philosophical for a while. The most interesting (and surprising) part of the presentation was a brief demonstration of Metaweb’s new Freebaseapps.com, which is a development environment for Freebase which embodies some of the principles discussed in the philosophical portion of the talk. From my cloud computing oriented point of view, it looks to me like an “IDE for the cloud”. I need to dig into this a bit more.

One topic which was brand new to me this year was R, which is a functional language for statistical computing and graphics. I’d been hearing a little bit of buzz on R via Twitter, and I was just invited to join the advisory board for REvolution Computing, a startup that is working to foster the R community and to support those users that want a more commercialized offering of R. Since I didn’t know much about R, I found Micheal Driscoll’s talk “Open Source Analytics: Visualization and Predictive Modeling of Big Data with the R Programming Language“. Analytics of all kinds are going to be much more important as the amount of data in web applications grows. If you are interested in big data, and don’t know about R, that seems like a problem. I know that I am going rectify my own personal lack of knowledge.

My talk

As in previous years, I gave a talk at the conference. One of the presentations that I’ve done in several places has a large section about the problem of programming concurrent systems, motivated by the arrival of multicore processors. For OSCON, I took that section of the talk and expanded it into a session of its own. Despite two one hour out loud run throughs, I still got the pacing a little bit wrong and had to rush at the end to get all the content in. If I’m not careful this is going to wind up turning into a three hour tutorial. I’ve embedded the slideshare version for those of you that are interested.

Photography

OSCON is a significant event for me photographically, since OSCON 2005 happened days after I got my first digital SLR. It’s also one of the times that I usually see my friend James Duncan Davidson, who has been one of the people that has helped me along my photographic journey.   

This year things were a little different. Regular readers will know that I am getting a little burned out on conference photographs. I’ve been to a lot of conferences and shot a lot of pictures. After a while, they start to look and feel the same. It’s hard for me to both concentrate fully on the conference, the talks, the hallway track, etc, as well as concentrating on doing stuff that would be interesting photographically. All of which is a long way of saying, “I shot less. A lot less”.

One other reason that I don’t feel bad about shooting less at OSCON is that Duncan is there. Or normally he is. This year, he was absent because he got the nod to be the main stage photographer for TED Global 2009. Those of you who follow Duncan will know that when he needs a second camera he turns to Pinar Ozger. They’ve been working together for a while, but I’ve never met Pinar in person, because I don’t usually end up at the two camera events, and the one time that she, Duncan, and I were all in the same place, we just never ended up meeting. So this was the year that I got to meet Pinar – we bumped into each other at the OSCON speaker’s party and had a great chat. This was also my first time to really get a sense for her eye. When you second shoot for someone, you try to follow the lead of the main photographer. So I am the most familiar with Pinar’s work when she’s working with Duncan. Since she was the lead this year, I (and everybody else) got to see her eye at work. There are some wonderfully artistic shots in her coverage of the show.

One person that you’ll see in Pinar’s set is photographer Julian Cash, the head of the Human Creativity Project. I first met Julian at ApacheCon in San Diego back in 2005. At the time, I didn’t really know much about photography, and I didn’t really get to see much of what he had done with his light painting portraits. Today, I have have much better appreciation for his light paintings. He did one of me at the MySQL conference earlier this year, and he did a bunch at OSCON too.

The photographic tradition at OSCON is going strong.

I was happy to see that there were a number of people involved in organizing the event. Those folks were easily distinguishable by their red and yellow propellor beanies.

The Bar Camp / Foo Camp structure for a conference is pretty liberating when compared to the usual pre-planned, eyes forward conference. However, it’s not enough to guarantee a good event. I’ve attended a number of these kinds of events, and in my view, who actually turns up is just as important as how the event is structured. Bar Camp Seattle had a lot of people who were interested in talking about various social media related topics. I have no idea if any of those sessions were any good, because I didn’t end up going to any of them. I ran into and met some developer type people, but not as many as I hoped to. Despite the use of Pathable’s cool registration / attendee matchmaking system, I didn’t find it that easy to make use of the information printed on my badge. I would have loved some clever mixer based on the badge labels or something along those lines.

Brian Rice and I (but really mostly Brian) ran a session a session for people interested in programming languages. I was pretty happy because the session was very interactive, but the session length of 30 minutes made it hard to get very far during the allotted time.

The best session that I attended was a session that was literally and figuratively off the grid. Brian Dorsey put up a session for sitting outside under the nearby bridge. The weather in Seattle was really beautiful last Saturday, so it really begged for being outside. There were about 9 or 10 of us who wandered out and sat talking about a wide range of topics. I enjoyed the sense of flowing from topic to topic, shifting naturally with the flow of conversation, the changing of roles of various participants as the topics changed, and so forth.

I think that I am at a crossroads as to the value of these generic, unstructured events. I helped organize the first Seattle Mind Camp, and I am glad to see that there is now a Bar Camp in Seattle as well. At the same time, I’ve frequently left these events feeling unsatisfied. Beforehand I am filled with excitement at the possibility of meeting new people from other tribes / fields and somehow stirring the pot of creative juices. Most of the time, I end up leaving without that stirring having occurred. If I look back over the last four or five years worth of conferences that I’ve attended, only a handful of “unstructured” events really stand out. Those events were Foo Camp and the Scala Liftoff, and in both cases, I would say that the particular sets of people involved made a huge difference.   

MacBook Pros

The laptop refresh was a surprise to me. I wasn’t expecting anything until Intel’s Nehalem based laptop CPU’s and chipsets hit the market late summer or early fall. The basics of the machines haven’t improved that much, and won’t until that happens. I’m wary of the unibody built in battery – I had to have my MacBook Pro batteries replaced recently, and the built-in battery would make that a lot harder. As a photographer, I like the wider color gamut of the LCD, but I don’t like the glossy finish. I also find there replacement of the ExpressCard slot with an SD card slot odd. It would have been more “Pro” to at least use a Compact Flash slot.   

In any case, I’m not in the market for a new laptop, so the minor changes and the nice price reduction don’t mean much to me at the moment.

Snow Leopard

Snow Leopard, on the other hand, is of great interest to me now that my primary box is a Mac Pro. I’m eager to have OS X taking better advantage of the all the hardware threads in the box. I was disappointed that there wasn’t more discussion of this in the keynote, but I also understand that having more than 2 cores is still a bit out there. I’m also disappointed that there was no mention of ZFS in either the workstation or server editions of Snow Leopard.

I guess that Snow Leopard is not as ready as many people (including me) thought. It won’t be shipping until September. Apple has taken a very reasonable approach to pricing the upgrade. The biggest issue for me is that I’ve been having problems with 10.5.7. I uninstalled it from the MacPro, and my work laptop wigged out on me last week during JavaOne, and I am very suspicious that the problems are 10.5.7 related. Jeffrey Zeldman is chronicling his own set of problems with the update. It’s going to be a long time between now and September if Apple doesn’t sort this out.

iPhone 3.0

The iPhone 3.0 stuff was pretty much a rehash of what was previewed back in March. The only surprise was the “Find My iPhone” feature, which really ought to be a standard feature. I’m not sure if I’m going to buy MobileMe just to get this ability. Everybody is going to get an upgrade to this version of the software so there’s nothing but happiness all around.

What’s not so happy is that some of the features will be unavailable because AT&T isn’t ready to support them: MMS and Tethering. I’m not really sure that I would actually use the MMS. I do most of my picture sharing via Twitter or Facebook. I am pretty sure that I would use tethering, either when riding the ferry or when traveling for work. However, if AT&T adds another $30 a month for the privilege, I probably won’t do it. I can get a Boingo account form $10 a month. True that it won’t work everywhere, but it will work on the ferry and in major airports. Does AT&T really think that we don’t know how to comparison shop?

iPhone 3GS

The iPhone 3GS is a nice upgrade. I’d be happy with the speed, but I’m going to get a speed increase (supposedly) from the iPhone 3.0 software. Faster 3G data would also be nice. The battery life improvements don’t cover the 3G radio usage, which is how I pound my iPhone.   

There are two features which really stand out to me: the compass and the camera.

I travel a lot, and I get mixed up a lot. Having the compass to help decipher directions would really be a help to me. I can think of several occasions in the last 6 months, where I could have saved some aggravation if I knew what direction I was pointed in.

The improvements to the camera look really good. Chase Jarvis is calling it the photographer’s iPhone, which is pretty much a no brainer. There was no mention of speeding up the amount of time it takes to get the camera to come on, which is one of my biggest gripes with it. Is it really a decisive moment camera? No way. But it looks like it is a much better camera than what we have now. I could probably justify $199 to upgrade my 16G iPhone 3G – it’d be a lot cheaper than a camera.

Unfortunately, I’m not going to get to do that. At least not until December 2009, due to the subsidized pricing of the iPhone. Lots of people are complaining about this, but that’s the way that the carriers have always worked. It’s not something new, in fact, its a sign that AT&T has a little more pull on Apple that we thought. So I’ll be waiting at least until December. The problem is that if I wait till December, I’m only 6 months away from the next iPhone product launch (if they keep to the current schedule), and as TechCrunch points out, if Apple lets its exclusive contract with AT&T expire in 2010, then you’d actually have carrier choice. That would be a good thing, and since getting onto Verizon’s huge network can only help iPhone sales, I’d bet that the iPhone is on Verizon in 2010. That’s not an impossible thing. Verizon made its first appearance ever at JavaOne this year, a sign that things are starting to change over there. I guess I’m going to wait and see how AT&T treats me between now and then. But they should be painfully aware that people are buying the iPhone, not the carrier.

CommunityOne

For me, the big theme at CommunityOne was cloud computing. Sun itself was emphasizing cloud stuff and the latest release of OpenSolaris, 2009.06, which were the main topics of the CommunityOne general session. The Sun Cloud is due out sometime this summer, so much of the cloud part of the session was having partners come up and tell about their experiences working with our cloud. The OpenSolaris team has done a huge amount of work in 2009.06. The feature that stuck out to me the most is “Crossbow“, which is a completely rewritten networking stack. Solaris already had CPU virtualization technology built into it via the zones feature. Crossbow makes it possible to virtualize networking configurations. This means that you could run an instance of OpenSolaris on your laptop (either natively or via VirtualBox, VMWare or whatever) and actually have a virtualized data center configuration running right there. That’s pretty interesting stuff.

I went to several cloud sessions, and I’d have to say that the current state of cloud computing is pretty rough. At least that’s true at the Infrastructure as a Service level where Amazon and the Sun Cloud are. As an example, I went to a good presentation by fellow Sun employees on cloud computing patterns. I happened to be sitting with James Governor and Stephen O’Grady of Redmonk, and I turned to Stephen and said “these patterns are all at a level that I never want to have to worry about”. The patterns themselves were fine, but I personally don’t want to have to deal with things at that level in a cloud platform. There is lots of room for improvement and innovation in this space.

My CommunityOne talk was called “Programming Languages for the Cloud”.

The talk is based on my experience as a language guy who has been asked to work on cloud computing stuff. As such, I’m really trying to raise questions (for which I don’t yet have answers) about places where work on programming languages might usefully intersect with cloud computing. I figured that this would be a niche kind of talk, so I was very surprised to find myself in one of the larger rooms at Moscone, complete with a live video feed. I was even more surprised to see that the room was pretty full. After the presentation, one of the Salesforce.com engineers working on Apex (their domain specific language for the cloud) came up to the front. We ended up having lunch and I learned a bunch interesting stuff about their experience withe Apex. This sort of thing is what makes conferences worthwhile.

JavaOne

I spent the first day of JavaOne prepping for my presentation, “Seeding the Cloud”, which was about some ways that tools could help developers who choose to build applications in the cloud.

The CommunityOne Party

The CommunityOne Party

The CommunityOne Party

The CommunityOne Party

The CommunityOne Party

Today at CommunityOne I’ll be giving a talk called “Programming Languages for the Cloud“. This is an exploration of areas where programming languages might intersect cloud computing.

Tuesday afternoon at JavaOne Ashwin Rao and i I will be giving a talk called “Seeding the Cloud“, where we’ll be giving a developer centric view of cloud computing and cloud computing issues. We’ll also have demos of some of the cloud developer stuff we’ve been working on.

If you are around and want to meetup, you can always get a hold of me via Twitter.

For some time now, I’ve been wondering when someone would start to use systems like Twitter as a way to deliver information between programs. A few weeks ago, Todd Fast, a colleague at Sun gave me a preview of what is now the Twitter Data proposal. Todd and Jiri Kopsa have done all the heavy lifting on this, so if you have substantive comments or requests, they are really the people you should be dealing with. They were kind enough to recognize me as a reviewer of their work, but the initial idea is theirs.

Twitter Data is a bit different than what I was envisioning. I was thinking more along the lines of jamming JSON or XML data into a Twitter message as a starting point for program level data exchange. That would allow us to leverage existing tools and libraries and make the entire thing straight forward. The interesting part, then, would be in the distribution network that arose from programs following other programs. This could also be embedded into a person’s Twitter feed by allowing clients to ignore tweet payloads that were structured data.

Twitter Data proposes a way to annotate the data oriented parts of a regular Tweet in order to make it easier for machines to extract the data. Some people think this is a good idea, and some people think it’s a terrible idea. It’s easy to see the arguments on both sides. Pro, is that you could turn your Tweet stream into a way to deliver information about you to programs, and that Twitter Data would make it that much easier to do. The Cons (that I’ve seen so far) are that people don’t want to have this kind of data exchange mixed into their Twitter stream, or that parsing the natural language that appears in the 140 characters of a tweet shouldn’t be that hard.

So we have two dimensions (at least) to the problem that Twitter Data is trying to address:

1. Is it a useful thing to have structured or semi structured information about a person included in their Twitter feed?
2. If so, should that data be out of band, mixed in, or extracted (natural language processing)?

Independent of the merits of the specific Twittter Data proposal (and I definitely think that there are merits), I think that these two questions are worth some discussion and pondering.

I ended up getting an 8 core machine, because these are the machines that can be expanded to an outrageous amount of memory, something which is a necessity for systems doing a lot of Photoshop. Due to the benchmarking controversy, I got the 2.66GHz processors, so that single threaded programs wouldn’t suffer as much. Here’s a quick rundown on my experience after having the machine for a few weeks.

Hardware

All of my hardware moved over without a hiccup, except for my Logitech Z-5500 speakers. I needed a TOSLINK to TOSLINK cable, which was rectified by a trip to Radio Shack (yes, we have one on Bainbridge Island. It’s not Fry’s but once a year or so they save my bacon.). The machine is much quieter than I expected. The last desktop machine that I owned was a homebuilt Windows box, and that thing was really loud. The Mac Pro is quieter than some of the external FireWire drives that are plugged into it. Heat would be a different story. My office is already several degrees warmer than the rest of the house, and now it’s probably another several degrees warmer. I’m having to be very careful about leaving my office doors open in order for things to cool down. Figuring out how this works in the summer is going to be interesting.

Performance wise I am pretty happy. Things are definitely snappier than my Sun supplied 2.6GHz MacBook Pro. I moved some external disks off of Firewire and into the Mac Pro’s internal SATA drive bays, and I am sure that the change in interface made a big contribution to the improved speed. The machine has 12GB of Other World Computing RAM in it, so it basically doesn’t page unless I am doing something big in Photoshop or have several VirtualBox VMs open at the same time.

There are some things that I miss:

We don’t have TV, so we do a lot of NetFlix and other DVD’s. This happened mostly on the MacBook Pro via Front Row and the Apple Remote. The Mac Pro doesn’t talk to the Apple remote, and I miss that. If people have suggestions for controlling Front Row on a Mac Pro, please leave them in the comments.

I got used to having the laptop hooked up to the LCD display, and using the laptop LCD as my “communications display” for IM, IRC, Twitter and so forth. Now I’m back down to a single display and missing it. I’m also missing it in Lightroom.

The Mac Pro came with an Apple keyboard, and the keyboard I was using was a Microsoft Natural Keyboard from 2000, and some of the keys were starting to get hard to push. So I figured that I would try the Apple keyboard. So far I don’t mind it, but keys are in different places, and the new keyboard has 9 years of muscle memory working against it. But that would be true of just about any keyboard.

Software

Any time I get a new machine I update my Macintosh Tips and Tricks page. I definitely have some updates that I could make, and I might make some of them after JavaOne. The rumor mill is suggesting that MacOS 10.6 Snow Leopard is going to ship this summer, so I might just wait until that happens, since I expect a lot of things to need updating, rearranging, etc.

I did have a problem when I tried to update the machine to 10.5.7. Things were behaving very oddly, so I restored the machine back to 10.5.6 with Time Machine. Time Machine backups on an internal SATA drive take less time (and make less noise) than on an external FireWire drive. I’m going to give this another try after JavaOne. And for prospective commenters, yes, I repaired permissions and used the Combo Updater.

Photoshop occasionally makes use of the additional cores, but it’s the large amount of RAM that is really making the difference at the moment. The same is true for Lightroom. Perhaps the next editions of these programs, coupled with 10.6, will do a better job of keeping multiple cores busy. In the meantime, my Lightroom to Photoshop batch jobs are definitely running quite a bit faster than before.

On the whole

On the whole, I am happy with the machine, and I expect to be a lot happier when 10.6 ships this summer.

Why did I go?

I’ve written about Erlang in this space before. Erlang is having a major influence on other languages, such as Scala on the JVM side and Axum on the CLR side. In addition every language seems to have several implementation of Erlang style “actors” (despite the fact that this is historically incorrect). Erlang has been around for a long time, and has seen industrial usage in demanding telecom applications. As a dynamically typed functional language with good support for concurrency and distribution, it is (if nothing else) a source of interesting ideas. Earlier this year, my boss asked me to start doing some thinking about cloud computing in addition to the stuff that I was already doing around dynamic languages — another good match for Erlang. This was the first large scale gathering of Erlang people in the US (at least that I am aware of), so I wanted to drop in and see what was going on, what the community is like, and so on.

Talks

The program at the Erlang Factory was very strong. In many of the slot sessions, there were 3 excellent talks to choose from. Every single talk that I went to was of very high quality. It was so bad that I wasn’t able to explore all the areas that I wanted to. Fortunately, the sessions were videotaped and are supposed to be made available on the web. Also, there was a decent amount of twittering going on, so a Twitter search for #erlangfactory will turn up some useful information.

I attended a number of “experience” talks by companies / individuals. There were experience talks from Facebook, SAP, Orbitz, and Kreditor (the fastest growing company in Sweden). I made it to the Facebook talk and the Kreditor talk. Facebook’s usage/deployment is on the order of 100 machines, which provide the chat facility for Facebook. Erlang is doing all the heavy lifting, and PHP is doing the web UI part. There was a lot of this kind of architecture floating around the conference. It seemed like the most popular combination was Ruby/Erlang, but there was definitely Python and PHP as well. The Kreditor talk was interesting because their site has been running for 3 years with very small amounts of downtime. Unfortunately, their entire deployment is probably less than 10 machines, so that blunts the impressiveness of what they have done. Still it was interesting to hear how they accomplished this using features of Erlang. In addition to the talks, I spoke with many attendees who are using Erlang in their companies. One such person was eBay founder Pierre Omidyar, who is running Ginx, a web based Twitter client. Pierre is doing the coding and deployment of the site, and was well versed in the Erlang way of doing things. An interesting data point.

The Erlang community (like all communities) has it’s old guard. These are folks who have worked with Erlang for years, before its recent burst of interest. There were a pair of keynotes by Erlang long-timers Robert Virding (The Erlang Rationale) and Ulf Wiger (Mulitcore Programming in Erlang). Both of these talks shared a common trait — the speakers were pretty honest about what was good about Erlang, and where there were problems. Given how prone the computing business is to fashion, I found this to be refreshing. Virding talked about the reasons why Erlang is designed the way it is. He accepted the blame for inconstencies in the libraries, talked about the need to avoid the process dictionary, and agreed that “a char type is probably not wrong”. Wiger’s talk was about why parallelizing code is hard (even with Erlang). He used the example of parallelizing map to demonstrate this, and showed the use of the QuickCheck testing tool to aid in finding parallelism bugs. The Erlang version of QuickCheck was inspired by the Haskell version of QuickCheck, and it’s a very very useful tool. The adaptations for parallelism look very nice. It’s a shame that the Erlang version is commercial software. I don’t grudge the authors the right to charge money for their software, but I do think that this will hold back adoption of this important tool.

There were many talks on what I would describe as “cloud problems”. For example, Ezra Zygmutowicz’s “You got your Erlang in my Ruby” was really about how he built a self assembling cluster of Ruby daemon’s (Nanite), Dave Fayram and Abhay Kumar’s “Building Reliable Distributed Heterogenous Services with Katamari/Fuzed“, and Lennart Ohman’s “A service fail over and take-over system for Erlang/OTP”. Like PyCon, there was a lot of interested in eventually consistent databases/key-value stores/non-relational databases. Cliff Moon’s talk on dynomite (a clone of Amazon’s Dynamo system), was particularly encouraging because he was reaching out to other people in the audience (and there were a decent number of them) to try an consolidate all their efforts into a single project. From what I could tell, people seemed receptive to that idea.

CouchDB also fits into that last category of non-relational databases, but it gets it’s own paragraph. One reason is that I helped mentor the project through the Apache Incubator (and chauffeured those CouchDB committers who were present). Another is that CouchDB creator Damien Katz got a keynote. Third is that there was basically a CouchDB track on the second day of the conference. There was a lot of interest in CouchDB, and a lot of activity as well. I was told that some of the people who took the CouchDB training during the training days had actually submitted patches on the project already. Damien’s talk was not about the technical details of CouchDB, but about his personal journey to CouchDB, which included selling his house and living off his savings in order to see CouchDB come to life.

Activity has really picked up in the Erlang web framework space. In addition to Erlang Web, and Yariv Sadan’s Erlyweb, there is also Rusty Klophaus’ Nitrogen. Nitrogen focuses more on the UI side of the web framework, omitting any kind of data storage. It’s very easy to create an AJAX based user interface using Nitrogen, and there is nice support for Comet. As part of his presentation, Rusty showed his slides on a Nitrogen based webcast reflector. You specify the UI using Erlang terms, which then causes HTML/Javascript/etc to be generated, which caused a stir in part of the Twitter peanut gallery. I was mostly happy to see people focusing on solving the current generation of problems. My favorite web space talk was probably Justin Sheehy’s talk on Webmachine. I think that I prefer the description of WebMachine as a REST or HTTP toolkit. Webmachine gives you what you need to implement any HTTP method correctly, and then provided a set of callback functions that can be implemented to customize that processing to do actual work. One of the coolest things about Webmachine is it’s ability to visually show you that path taken in processing a particular HTTP request, and being able to inspect/dump data at various points in the diagram. It makes for a very nice demo.

There were not that many “language geek” talks. This contrasts with the early years of PyCon (at least for as long as I have attended) where there were quite a number. I missed Robert Virding’s talk on Lisp Flavored Erlang (but I saw some example usage in a CouchDB talk), because it overlapped the dynomite talk. I was able to attend Tony Arcieri’s talk on “Building Languages on Erlang (and an introduction to Reia)”. During the first part of his talk, Tony showed how to construct an Erlang module on the fly in the Erlang shell. He then discussed some tools which are useful to people trying to build languages on top of BEAM, the Erlang virtual machine:

• Robert Virding has written leex, a lexical analyzer generator
• yecc, a Yacc style parser generator is included in the Erlang distribution
• the erl_syntax_lib library aids in constructing Erlang abstract syntax trees, which can then be compiled to Erlang bytecode.
• Erlyweb contains the smerl (simple metaprogramming) library for creating and manipulating Erlang modules at runtime.

After that, he launched into a description of REiA. I’m not sure that I agree with some of the choices that he has made, but I am happy to see people experimenting with languages on top of BEAM, and in keeping with Erlang’s process model and the OTP infrastructure. One of the things that Tony mentioned was abandoning indentation based syntax. He wrote an entire postmortem on that experience in his blog. Python’s indentation based syntax has won me over and made me a fan, and I am sad to see that indentation syntax, blocks/closures, and expression orientation continue to be at odds.

Coda

It looks like Erlang is starting to find a home. Companies are using it in production. There are books starting to be written about it. Many (not all) of the things which make Erlang seem odd to “mainstream” programmers also appear in languages like Scala, Haskell, and F#. At the same time, Erlang has a long history of industrial deployment, albeit in a single (large) market segment. Many of the problems which we now face in large web systems (and the cloud): concurrency, distribution, high availability, and scalability are strengths for Erlang. Indeed, many of the people that I heard from or talked to basically said that they couldn’t solve their problem with any other technology, or that their solutions were dramatically simpler than the technologies that they already knew. Will that be enough to propel Erlang into the mainstream? I don’t know. I also don’t know if our current state of mainstreamness is going to remain. More and more I’m seeing an attitude of “let’s use the best tool for the job”, not only in languages, but in all parts of (web) applications.

There’s also the issue of the Erlang community itself.   Around 120 people showed up for the conference. As I mentioned previously, there are the folks who have been doing Erlang for years. Then there are the relative newcomers, who are web oriented/web savvy, and solving problems in very different domains than the original problem domain of Erlang and it’s inventors. Thus far, the two segments seem to be getting along fine. I hope that will continue — success or the potential for success has a tendency to bend relationships.

There were a lot of people using hashtags in their PyCon tweets, and Jacob Kaplan-Moss showed me Twitterfall, which made it easy to keep track of uses of the tag. That made it *much* easier to find the virtual twitter stream for PyCon. This was also true at Lang.NET, the DSL DevCon, and the MySQL conference. This week(end) I’ll be using hashtags to track the progress of JSConf.   From now on I’ll always use hashtags when I’m at a conference or event.

One reason that it’s taken me so long to get the hash tag thing is that I use Twitter primarily via rich desktop (or iPhone) clients. Until recently I wasn’t using clients that could do searching. I had tried TweetDeck, and it never stayed with me. When Nambu came along, I was pretty enthusiastic because it was a native TweetDeck. Unfortunately, I had crashing problems with it at Lang.Net (since fixed, I think), and I put it aside when I realized that Syrinx 2.0 had searches. While Syrinx doesn’t save searches across restarts, its memory use is tolerable enough to leave it running all the time, so it’s not a big problem, and I am hopeful that MRR will include saved searches in a future version. Commenters: yes, I tried Tweetie for Mac, and I didn’t like it. I love Tweetie for iPhone, though. Go figure.

There is a sizeable ecosystem around MySQL. There are extension patches from Google and Percona, which were mentioned in many talks that I was in. There’s MariaDB, Monty’s community oriented fork of MySQL. There’s the Drizzle project, which looks really interesting. There’s lots going on, and I got the feeling that there’s lots of innovation happening in various parts of the ecosystem. It feels energetic and fun, and what I would expect of a big open source community, despite it being a long way from Apache or Python.

I attended all kinds of talks. I went to a number of talks about analyzing performance and monitoring, including 3 talks on DTrace. Sadly, these talks were sparsely attended, which is a symptom of some of the problems that Solaris/OpenSolaris has been having. What was interesting was that all of these talks were given by former MySQL employees, and all of them were genuinely enthusiastic about DTrace. The best of these talks was Domas Mituzas’ Deep-inspecting MySQL with DTrace, where he showed some very cool MySQL specific DTrace scripts. If DTrace got ported to Linux as a result of the Oracle/Sun acquisition, that would be a good outcome for the world.

I also went to several cloud computing talks, where the topics was how to run MySQL in the cloud. These were pretty interesting because it turns out that there is a bunch of stuff that you need to do and be aware of when running the current versions of MySQL in a cloud environment. I hope that the Drizzle folks are aware of some of these issues and are able to solve some of these problems so that running in the cloud us pretty simple.

Here are my 3 favorite talks:

• Don MacAskill’s The SmugMug Tale – I’m a photo guy, but not a SmugMug customer. Don’s been tweeting his experiences using Amazon Web Services to build SmugMug, and he’s been blogging his experiences with ZFS, the Sun Storage 7000, and so forth. I’ve been following his stuff for a while, so this was mostly a chance to see an in person rendering of an on line personality.
• One talk that I didn’t expect to enjoy was Mark Madden’s Using Open Source BI in the Real World. I’m not really a Business Intelligence guy per se, but the world of blogging and twittering and so forth starts to make you attuned to the usefulness of various kinds of analytics. Anyone building any kind of non-trivial web software need analytics capabilities, so having open source solutions for this is good. It probably also didn’t hurt that I talked to several BI vendors on the expo floor the night before. What I really enjoyed about the talk was the beginning sections on how to be an analyst, think about and project the future. I’m given to a bit of that now and then, so I found this part of the talk pretty interesting.
• The best talk that I went to was Yoshinori Matsunobu’s Mastering the Art of Indexing. The speaker pretty much covered all the kinds of indexing in MySQL, which indexes work best in which conditions (both for selecting and inserting — there were some interesting surprises for insert), and even tested the differences between hard disks and solid state drives. Maybe I loved this talk because it brought back all the research that I did in query optimization back in graduate school. But that wouldn’t explain all the other people in the room, which was standing room only.

Based on what I saw this week, I’m not in any way worried about the future of MySQL.

Reporting

I’m going to call out four talks that really stood out for me. There were more than just these four, but it was either these four or all of them, and all of them is too much work.

• Guillaume LaForge’s talk on Groovy DSL’s was important because he not only showed how to build DSL’s using Groovy, but he’s actually working with real customers, like Mutual of Omaha, who are using those DSL’s in production.   

• I was happy to hear Markus Voelter’s talk Textual DSL’s and Code Generation with Eclipse Tools because a lot of the noise that I’ve heard on the DSL front has been coming from the Ruby and .NET side of the world. One thing that got my attention at the DevCon was the importance of tooling, so it was good to see that there are some tooling efforts in the Java space. It’s too bad that no one from JetBrains was there to present on MPS.

• Brad Cross and Ted Neward did a talk entitled “Functional vs. Dynamic DSLs: The Smackdown”. I came away from this talk wanting more, and not in a good way. Brad and Ted really needed about 2 hours in order to give all the relevant background a chance to settle in. During the talk they presented a set of things which differentiated the functional programming and dynamic language styles of creating “Internal” (I really dislike the Internal/External terms) DSLs. Unfortunately, there wasn’t enough time to really dig in and explore the meat of what they said. I think that a deep addressing of the points that they made would be a very important contribution to the DSL topic. Maybe we’ll get to see a series of blog posts, developerWorks articles, or even an academic paper of some kind.

• I view Intentional Software as one of those grand computer science projects. Having worked on Chandler, I have an appreciation for the perils of large, grand efforts. This is the first time that I had a chance to see a presentation by anyone from Intentional Software, and it is just as well that it was a demo of their just shipped product. I took note when the Intentional Software project was started back in 2002, but I’ve not heard a lot about their progress since then. What we saw was a demonstration of a production version of their “Domain Workbench” which is a system for allowing domain experts and programmers to work together to build a system which domain experts can then use to write software. Instead of writing programs, the programmers write the generator which takes the domain language (which can be visual) and then generates code. The system represents the domain information in a way that allows multiple, editable, “projections” (views). The demonstrations that we saw included an actuarial workbench, complete with mathematical notation, and an electronics workbench, expressed as circuit diagrams. If you are interested, your best bet is to watch the video when the videos are posted.

  I am pretty impressed with what I saw, but there are lots of questions. How many domains can this actually work for? How hard is it to write generators? What’s the business model for domain workbenches? It seems pretty clear to me that for the domains and organizations where this can work, this approach is going to have a pretty sizable impact. Perhaps not this year, but within the next 5 years. I have to hand it to the Intentional Software guys. Their presentation was pretty low key, and they are going out of their way to not hype their stuff. They plan to work with a small number of customers to gradually prove out their approach. In an area which is highly susceptible to hype, it was refreshing to see people trying to keep expectations to a reasonable level.

The DevCon (and Lang.NET) were also my chance to meet two people who I’ve followed for sometime from afar: Ted Neward, and Larry O’Brien. Ted is well known and I’ve been following his blog for some time. He’s local to the Puget Sound area, and it’s probably just bad timing that we never met before this week. Larry O’Brien has been a commenter on my blog, as well as a responder on Twitter. I’ve appreciated his blog as well as the columns that he’s written over the years. It was great fun to run to the back of the room after each talk and see what the Twitter cabal (which included Larry) had to say about the material we had just seen.

Analysis

I think that DSL’s are inevitable. It’s remarkable to me how prescient Abelson and Sussman were when they defined three categories of abstraction: control abstraction, data abstraction, and metalinguistic abstraction. If you look at some of the recent frenzies in languages, you’ll see that we are mostly improving the ability of various languages to perform various kinds of abstraction. These concepts are not new, but they are appearing in languages which are approachable by today’s practitioners. Object oriented programming? Data abstraction. Closures? Control Abstraction. Pattern Matching/Algebraic datatypes? Data and control abstraction. DSLs and the capabilities needed to enable them? Metalinguistic abstraction.

Language as an abstraction is very powerful, and requires support from the underlying language as well as the tools. These two topics (as well as specific examples of domain specific languages) were the focus of the DevCon. The audience makeup appeared to be mostly language and compiler geeks. There were a few people (mostly consultants as far as I could tell) who write business applications, but this group was pretty small. This is important because most of the DSL’s presented were aimed at very computer science kinds of domains. If DSL’s are to have a broader impact, then it would be great to see more business people at events like this.

One thing which was not addressed at all was the process end of this. In order to build DSL’s for non computer domains, there has to be a collaboration between developers and domain experts. The Intentional Software guys recognize this via some “groupware” to facilitate this process. However, tooling alone is not enough to bridge this gap. I hope that we’ll be hearing reports on the process of collaboration between developers and domain experts as more and more people build DSL’s.

This is an interesting space, from a technical point of view. There is lots of cool language design and compiler stuff, some of my favorite topics.   On the business end, it seems like there are some decent sized opportunities here, and that tooling is going to play a very large role — language support for DSL’s will be important, but may be overshadowed by the importance of good tools.

Update: the videos are now avaiable

Hopscotch

Gilad Bracha talked about Hopscotch, the IDE for Newspeak. Newspeak is one of those things that’s on my very long list of things to look at. For a Smalltalk or Lisp developer, much of the talk was familiar — an illustration of the power of writing the IDE for a dynamic language in a dynamic language. In fact Bracha asserted that the true power of Smalltalk is the dynamism in the IDE itself. I think that’s probably true today, since almost all popular IDE’s are written in a static language. The other interesting feature of Hopscotch is the pervasive use of a web browser style UI in the IDE. This looks promising, and gives me hope that a web browser based IDE might be feasible.

Newspeak is implemented atop the Squeak VM, and when questioned about this choice, Bracha said that it would be much harder to implement Newspeak on either the CLR or JVM (and he’d be a position to know about the JVM in particular). He was hopeful that John Rose’s work on dynamic extensions for the JVM would take out some (but not all) of the pain.

Powershell

At the JVM Language Summit, Rich Hickey gave a 15 minute talk that left me wishing he had been given 30 minutes. Jeffrey Snover’s 15 minute talk on Powershell left me feeling the same way. Powershell may not be a monad toting concurrency friendly uber functional language on the CLR, but it was compelling to me because of it’s sheer practicality and usefulness. I’ve been following Powershell via blog posts since it was announced, but seeing it demoed by its designer is another thing altogether. The Powershell folks have done some nice work to present shell users with a very consistent user interface. A side effect of this work makes it easy to make GUI’s that can output Powershell commands, and apparently in future versions of Windows, the GUI admin tools will do this. In addition to rationalizing the user visible behavior of the shell, Powershell is able to do some very impressive stuff in terms of remote execution, limited/secure execution, single machine transactions, and more.

I spent some time talking with Snover later in the day, and he seemed to think that UNIX shells could gain a fair amount of PowerShell’s capabilities by recognizing that pipes ship bytestreams, adopting a data format (like JSON or XML) for those byte streams, and proceeding from there. That might be true technically, but that would be a huge cultural change for that community.   

Lars Bak on V8

Lars Bak talked about his work on V8. V8 is the nth VM that Bak has worked on, and my main takeaway from his talk was that the V8 team has tried to do a lot of careful measurement of real (and anticipated future) Javascript programs. They’ve then turned around and used those measurements to guide them in selecting from the techniques developed for Self, StrongTalk, and Hotspot. None of those techniques would be a mystery to people familiar with the literature. But the insight into this all applies to Javascript was pretty interesting.

Since this is all about performance, it’s interesting to note that V8 has just about doubled its performance during its one year life. Bak predicted that by 3-4 years in perrformance will have quadupled. There was a funny moment where someone asked about comparison’s with Firefox’s TraceMonkey tracing JIT. Bak’s reply was “let’s run the (Google V8) benchmarks“, which he then proceeded to do (V8 won). Bak is clearly competitive about this, which can only mean good things for Javascript users.

F#

F# is something that’s been at the edge of my radar because of asynchronous workflows. Lang.NET was a chance for me to get a quick education on what that’s all about. It turns out that async workflows are an application of computation expressions, which are a way of doing monadically inspired language extension. Async workflows end up being a way of using CPS style tasking but with a nice syntax. Most of this became clear during Tomas Petricek’s talk on Reactive pattern matching for F#. The code that Petricek showed was very evocative of actor style code and the use of F# pattern matching was evocative of Erlang’s use of the same. One interesting point was the use of join patterns, something that doesn’t have a direct analog in either Erlang or typical actor implementations.

Amanda Laucher’s talk Concurrency for F# was a discussion of a client engagement that she did using F# and the concurrency features of the language. The application in question was an insurance application, and the speedup numbers looked pretty interesting. This is notable because most of the concurrency examples that you can find are outside the domain of “business programming”. Having this kind of data is really useful for people interested in the concurrency space. Amanda’s work was done as a Microsoft case study, and published version will be available at some point.

Erik Meijer

Erik pretty much gets his own section because he’s that entertaining/interesting. This time he was talking about the Livelabs Reactive Framework, where he described a way to use LINQ for distributed computations, using the specific example of an AJAX dictionary based suggestion function. Erik started by using (throwing) coins to illustrate the IEnumerable and IObservable interfaces. He then used mindless symbol pushing to demonstrate that these two interfaces were duals of each other, and that they obeyed the rules for monads. That being true, he then showed how you could define observable collections which could be used with LINQ.

Tidbits

Herman Venter talked about a Common Compiler Infrastructure for the CLR. There’s some nice stuff in here if you are implementing a language on the CLR. One user of this infrastructure is the Code Canvas, a “spatial development environment”. It’s good to see that people are stretching the notion of what a development environment should be. The thing that stuck out to me most from this talk was not the technology (which is good), but Venter’s call to action at the end. The CCI is being open sourced, and Venter’s call for participation and contribution was indistinguishable from the same end of talk call at any open source conference. A sign of changes underway.

Joshua Goodman did a product rundown on What’s new in CLR V4 for Languages. There’s several years worth of changes to the CLR queued up for V4, which will be a big update. Most interesting to me was the inclusion of a Hill Climbing algorithm for allocating / managing threads. This strikes me as a kind of policy decision, something which you might want to plug into the CLR as opposed to having it installed under the hood. It will be interesting to see how this ends up working out in production settings.

Deja Vu

In a previous life, I did some work on database programming languages, and I experienced deja vu twice during the conference. The first was during Philip Wadler’s Links: Web programming without tiers. The goal of Links is enable front end and back end web programming using a single language. Wadler didn’t discuss the front end part during his talk. He focused on the back end, particularly on the ability to translate programming language code into SQL, in a fashion slightly beyond LINQ. This is done using a combination of the monadic techniques used for LINQ, as well as a type/effect system which is used to help determine whether a piece of code is SQLizable or not. The work was eerily familiar, and when Wadler quoted Limsoon Wong and Leonid Libkin’s work on Kleisi, I knew why. We looked at their work on structural recursion (and comprehension notation) for our work on list and tree queries.

The last talk(s) of Lang.NET was a double header talk on Oslo/M by Paul Vick and David Langworthy. David and I were officemates while we were graduate students at Brown. Even more deja vu was induced when I saw David demoing how the M toolchain could generate SQL schemas and queries. As he put it to me afterwards – “this is all that stuff we were talking about 10 years ago”. Well, closer to 20 now, but who’s counting. M is in its very early days – I think it will be more interesting when it can generate CLR code to match the SQL.

Last bits

Thanks to Harry Pierson for making sure that I got an invitation to Lang.Net. People were very welcoming, even though I had to ask questions about various bits of Microsoft alphabet soup. I suspect that this is the only conference I will go to all year where Macs are the minority. It’s probably good for me to get out of the bubble every once in a while…   

Overall, I found the topics to be a little more broad ranging than the JVM Summit. There were quite a few talks on tools, while there were comparatively fewer at the JVM summit, and those tools discussed at the JVM summit were really aimed at language implementors. One thing that worked well with the JVM Summit was allowing for some open space sessions so that attendees could get together and discuss topics of mutual interest, often based on what had been presented during the sessions. I think that some similar time slots would have enhanced the experience.

I definitely came away impressed by some of the work going on in the CLR ecosystem, and I hope that the exchange between the JVM and CLR communities will continue and expand.

Update: Videos of the talks are now available.