Fraser-Green

Meteor: a recipe for the perfect project launch

2012-04-11T23:56:00+02:00

Today was the day Meteor went public and I haven’t been so excited by a web framework since Rails. As I’m writing this, the Hacker News announcement has scored 1126 points. A friend mailed me about it this morning (he’s never informed me about a new web framework before), I told a colleague via IM and, just as I was doing so, he overheard another conversation in the office as one guy said “Have you heard about the new Meteor framework?”. There’s clearly a lot of buzz. I expect a lot will be published in the next few days discussing the technical merits or pitfalls of Meteor and I’ve barely even played with it yet so I’m not going to cover that topic in much detail here. What fascinates me though is how the project managed to conjure such a wave of enthusiasm and that’s going to be the subject of this post.

Now, Meteor isn’t the only web framework news item this week. Yesod, “a Haskell web framework for productive development of type-safe, RESTful, high performance web applications”, 1.0 was also announced on Monday. From my searches, it appears to be highest scoring web framework announcement in 12 months, yet it only made 141 points on Hacker News. So what’s the difference? There are a lot of web frameworks, so why did Meteor stand out? It comes down to the age old tried-and-tested formula used by any product which manages to break through in a crowded market: one killer feature plus great marketing. We’ve seen it succeed in great products such as the iPod and great services such as Heroku but it’s surprisingly rare to see it in open source projects.

Killer feature

I said I wouldn’t delve much into the technical details but I think it’s worth just clarifying what Meteor’s killer feature actually is. There are 9 points listed on the home page but one thing stands out as particularly fascinating and unique. It’s how it completely blurs the distinction between client and server and does so from the client’s point-of-view.

Removing the client-server boundary can be immensely helpful as there are less things to worry about: one less language, fewer APIs, one less paradigm, one less deployable etc. Numerous frameworks do this already but they’re all server-oriented and their expressiveness on the client-side is normally very limited compared to just writing good ol' JavaScript. When you also consider that for a growing class of webapps, the amount of logic on the client-side can be far greater than that which would typically run on the server, this creates an impedance mismatch.

With Meteor, you can put some world-class front-end JavaScript developers in a room, let them do what they’re best at and create the best looking app you’ve ever seen, and the very same developers can wire it up to the database with real-time collaboration thrown in for good measure. Job done.

What’s also interesting is that while Meteor has a killer feature, it currently lacks some pretty basic essential ones! Security is non-existant, it can’t serve static contant, it’s Mongo-only etc. Yet Meteor 0.3.2 is a fine example of what the Lean Startup crowd would call a Minimum Viable Product. So long as you can kind of write a webapp and Meteor really does deliver on this new and awesome data synchronization with “latency compensation”, people will come flocking.

As killer features go, I think it’s a pretty good one. But then again, almost every web framework had some killer feature in mind in the first place otherwise their authors wouldn’t have been motivated to write them. A lot of frameworks, mind you, don’t do anything more unique than to port existing ideas into a new language. (The aforementioned Yesod smells a little of this but shoot me if I’m wrong as I’ve never even tried it.) Many frameworks, libraries and tools of all kinds though really do possess some killer feature yet still they languish in the graveyard of GitHub. That’s because for a project, or indeed a product of any kind, it takes more to gain traction than just being good at something.

Marketing

When I first looked at the Meteor site today, I had a flashback to 2005. A tingly feeling inside, my head started nodding, and a bit of my brain telling the rest of me “This is going to do exactly what you want.” And I hadn’t even read the front page properly yet. What’s that all about? Marketing, pure and simple.

Thanks to the magicians at the Wayback Machine, we can go back to 2005 and see how the Rails site looked just after its public release. Sure, the flow diagram looks a bit garish nowadays, but all the elements which got us going in the first place are still there and if we compare it alongside the Meteor page, we see the old formula still holds true. First, a bold statement about what it is (check), then comes the killer feature - for Rails it was Convention over Configuration and Don’t Repeat Yourself - (check), then a screencast (check), then more features (check). The only significant difference with the Meteor page, aside from looking a lot more fashionable, is that it places greater emphasis on contacts and getting involved. It’s a crying shame that in seven years (I seem to remember the look of the Rails site being something of a revolution at the time) there aren’t a lot more open source projects adopting this strategy. It’s easy, mind, to overlook the importance of design and Meteor’s done a wonderful job for which they should be heartily commended.

The goodness doesn’t stop at the front page, and I think the following could be used almost as a recipe for creating great project sites:

Provide multiple, simple examples. Most project provide a single example, perhaps intentionally to make it appear more simple, but the concept becomes much more potent when there’s an example fairly close to your use case. Meteor has roughly one example per use case and I can pick the one which is closest to the project I'm planning on using it for.
Low barrier to adoption. Everything’s a one-liner. Installing, running, testing, getting examples etc. This tells me that I'm going to waste at most 2 minutes if I try out the project and it's a dud.
Talk to users. Following the Hacker News post, the Meteor developers were busy answering comments. While chatting on IRC. While providing answers on Stack Overflow. Users prefer to discuss on different channels. The whole Meteor team had a busy day covering all of them.
Provide great documentation from the start. There’s never time to write the documentation later so the only way it to build it up as you go along. Meteor has all the technical documentation on a single page with a navigation menu down the side so I can print it, use Ctrl-F to find keywords etc.

All this has a profound effect on our first impression of the project. We’re so used to investigating new projects that most of us probably only spend a minute or two before we have a feeling for whether the project is a good one or just another reject. We have different ways of making that assessment which is why it’s important to try to cover as many bases as possible. What happens if our main criteria are fulfilled (particularly if they are done so exceedingly well), is that we become convinced that this is a “good” project. These guys know what they’re doing. Whatever shortcoming there may be, they’ll fix it. We’re sold.

Conclusion

If you're hacking on a project which you think will be the next best thing, and getting traction and more people involved is important to you, I urge you to take a couple of leaves from Meteor's book. You must have your killer feature in place before anyone's going to take notice, but as little of the rest that you can actually get away with. When you're ready, make sure you have the documentation, examples, screencasts in place and make it blindingly simple to get startet. Next, tell the world!

Managing seed data with Google Docs

2011-08-23T15:15:00+02:00

When you're building a website what should you do with all that rarely (yet not never) changing seed data? I'm talking about things like Localization strings, choices in some drop down menus and so on. This is a situation I've faced a few times and I've gone through a variety of solutions to the problem. A couple of projects back, we tried putting everything in flat files - in the case of localization strings, this tends to be encouraged by the framework - but then you end up having to teach people Git just so they can fix a few typos. Then when they're done (and you've cleaned up the mess they made in Git) they complain:

Foreign marketing dude: 'elp, somezing eez wrong! I'm about to do a demo and I do not see zee changes I made to zee site.

me: Oh, yeah. I was just notified by Jenkins that the last build failed. I think I can fix it quickly, then I'll redploy.

Foreign marketing dude: Re-de-huh?! And 'oo eez zis Jenkins?

Too technical. So then we figured everything had to go in the database and we'd make a user-friendly front-end. Every table needed its own CRUD admin pages to maintain the data. Some grew huge and even needed a search box - AJAX naturally. As far as the end-user website was concerned, the project turned out to be a bit of a flop but the admin site rocked! At least I thought it did til one day someone accidentally deleted loads of localization strings. He didn't even seem that concerned:

Foreign marketing dude: Zis Git, 'eez taken care of zee backups, no?

me: Erm...

So to my current project and I started pondering what a reasonable amount of development time was to spend on seed data maintenance and in the end, I came to the conclusion: not a lot. At the same time, we'd started using Google Docs for pretty much anything we could so I figured we could use Google Docs spreadsheets to maintain the seed data. They have some convenient properties:

Anyone you give access to can edit the data. We share the documents "People at Our Organization who have the link can edit". We're only four but you can share with individuals if you prefer.
It has version control. If someone messes it up you can see the whole history, click on individual revisions to see what changed, and revert to a previous version.
Everyone understands spreadsheets (at least better than git). In fact our previous attempt at making a seed data admin interface began looking more and more like a spreadsheet since all the data's tabular by nature and it's convenient seeing all the values in place and being able to edit in situ.
There's a well-documented API which provides CSV export. Authenticating is slightly tricky but thereafter it's just one call to download a CSV-export.

So how does it work?

Each class of data (what would normally be a table in a database) has its own spreadsheet, and a config file in the app declares all the links.

When the app launches, it authenticates to Google Docs using a dedicated apps account and, for each document it hasn't already cached as a file, it saves the CSV export as a file in the cache directory, and loads the data. The cache means that if the app is restarted and Google Docs is having a bad day, the app can start fine using the previous run's data. Also, if someone is in the middle of editing when the app happens to restart, you don't want the half-baked changes appearing.

There's a single seed data admin page where site admins with links to edit the spreadsheet and reload the data.

The cached file is replaced and the app begins using the new data straight away. Note, when you're re-reading the data, don't just empty the data structure holding the data and load it up again because if any requests arrive while that's happening, they'll see partially loaded data. Instead, load it into a new structure and make the actual structure point to the new one. This also helps cope with bad data since if the actual import crashes, the app can carry on using the old data.

Details, details, details

Our app is built using Lift so we could use Google's Java API to access Docs. We also used JavaCSV to parse the CSV data. Of course CSV is quite a trivial format so it's tempting just to do it yourself, but there are a few corner cases like data containing commas and new-lines which 3rd party libraries handle. Each set of seed data is handled by a model class holding the dataset in an immutable Map.

Each of the model classes extends one of two generalizations. One deals with multi-column data such as in the first screenshot above. In these cases, lookups are typically made for a key to retrieve a class with properties mapped to the columns in the spreadsheet. The other generalization deals with data such as localization where a column is picked first (e.g. the locale) and a key lookup is made to fetch a single element.

Conclusions

So how's worked out? Pretty good. We've used this in production now for six months and so far there haven't been any hiccoughs. You wouldn't want to use this for large data sets since everything's held in memory but in each case we've used it, that's worked to our advantage. Doing round trips to the database to look up localization strings when there could be 50 on a page, for example, would be very expensive.

If you try it yourself, let me know how it goes.