Traditionally, message queues have been implemented in enterprise systems as large-scale “message queue middle tier” deployments. Most of these are provided by the big boys (read big investment) such as IBM, Tibco, etc. and usually require a dedicated team of engineers to integrate (though this has changed lately). However, there weren’t really any independent or open source solutions ready for production use until the emergence of RabbitMQ - an Open Source Enterprise Messaging Broker.

RabbitMQ is built on the foundations of Erlang and implements AMQP - an emerging standard of high performance messaging. It is based on a proven platform, is interoperable and offers low latency and good throughput. I’ve been hearing good things lately about it, specifically about the ease-of-setup (packaged EC2 server configs), performance and scalability, numerous protocol bindings and integration frameworks, etc.

This Google Tech Talk video (embedded below) explains the goals of RabbitMQ, their internal architecture (and decisions behind their design) and their work so far:

Here are the slides:

• Part 1: http://www.rabbitmq.com/resources/google-tech-talk-final/alexis-google-rabbitmq-talk.pdf
• Part 2: http://www.rabbitmq.com/resources/google-tech-talk-final/google.html

What seems most interesting is that RabbitMQ seems to be the first real step towards open sourcing the messaging middleware, which traditionally has suffered from vendor lock-in. Since it’s a $1 billion industry, I don’t quite expect the vendors to take this lying down. So either they will compete by lowering client costs, provide better support, or offer more managed integrations, or open source products will win the critical mass game by getting most small business/startups to adopt rapidly (similar to MySQL). In either case, it should be interesting to watch what RabbitMQ has started.

A natural extension message queues is where many processes consume a message from the queue (i.e. Publish/Subscribe pattern). PubSub is great for broadcasting messages to a variety of consuming processes. E.g. In the e-commerce order placement example above, once the credit card processing is complete, the message could be broadcasted to a variety of processes down the pipeline (order fulfillment, inventory update, shipping tracking, etc.). Fortunately, LShift has implemented a PubSub layer on top of RabbitMQ.

With the growing use of cloud computing services like Amazon Web Services, web developers are going to have start thinking more enterprisy (if that’s a word?). If you’re a web developer, what are your thoughts? If you’re a web developer who is already thinking in this direction (good for you!), how has your design/architecture process changed from the early days of web development?

July 6, 2009 Update: Thanks to @andrewgleave who linked to this presentation which goes more in-depth about AMQP, PubSub and RabbitMQ.

Richard Jones (from Last.fm) has compiled an useful comparison of the various distributed key-values stores out there including MemcacheDB, Cassandra and CouchDB. He seems most interested in Scalaris and Project Voldemort.

redis seems to be a very interesting key-value store. It behaves like memcached but supports strings, lists and sets as values. It also includes atomic operations like increments, push/pop, etc. They even wrote a Twitter clone using redis + PHP.

Bret Taylor from Friendfeed goes in-depth about how they built a ’schema-less’ MYSQL schema which attempts to solve the problem of extensibility and maintainability in most large RDBMS solutions. It doesn’t hurt improving performance as a result.

As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.

Leonard Lin also has some notes on distributed key-value stores. He compares some current tools out there, but finding none viable, creates his own. He has some words of wisdom:

• The distributed stores out there is currently pretty half-baked at best right now. Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.
• If you’re dealing w/ a reasonable number of items (<50M), Tokyo Tyrant is crazy fast. If you’re looking for a known, MySQL is probably an acceptable solution.
• Don’t believe the hype. There’s a lot of talk, but I didn’t find any public project that came close to the (implied?) promise of tossing nodes in and having it figure things out.
• Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is.

The popular consensus seems to be that even though numerous solutions exist out there, none seem to be widely adopted and production-proven yet.

July 1st, 2009 Update: Think Vitamin has a good article on reasons for switching to a non-relational database and also compares the tradeoffs between the different data storage options.

There’s no denying that Quantum of Solace is stellar. It helps further the new direction that the Bond franchise is moving in. It’s a thrill-ride (yes, it’s cliche) for the die-hard Bond fan as well as casual action-seeking movie goer. The opening scene is breathtaking, the story progresses well, and even though the running time is only 1 hour 46 minutes, you never feel like you didn’t get your money’s worth. The direction is reminiscent of the Bourne series, if a bit theatrical at times. Quantum of Solace was expected to show James Bond get in touch with his feelings, which he did and it was handled expertly, not going overboard with emotion. The movie may not be as good as Casino Royale, but it stands far apart from previous Brosnan’s exploits like Die Another Day and The World is Not Enough. So go watch it.

With that being said, the rest of this review is for the die-hard fans. These are the small details that make a Bond movie special or fall flat for the fans.

First off, the opening credits scene was a bit weak. As you’ve come to expect with any Bond movie, the credits are surreal, full of visual trickery and have a memorable theme song. The theme song was not catchy, the visual trickery was not as good as Casino Royale. The trademark body silhouettes just weren’t evocative enough.

Quantum of Solace is full of chase sequences. Bond is chased through mountains, lakes, in cars, boats, planes, anything else you can think of. It is so full of chases, that you almost wish it would stop. There are too many, some overly theatrical. Most Bond movies (including Casino Royale) intertwine a bit of fun, glamour, romance (if you can call it that) in with the captivating action. But this movie seems to take itself too seriously. There are few moments that give you a chance to reflect, even fewer that show Bond having a bit of fun or mentally jabbing with the villain (like the poker scene in Casino Royale). At times, the movie was so intense, you almost wished that they would show 5 minutes of Bond sleeping.

The only other gripe I have is that there are too many location changes in the movie. Usually Bond does his share of world traveling, but being a short movie, it felt very fragmented each time he kept switching continents every 10 minutes. You’ll also notice that (for some odd reason), they start referring to M as Maum (Mom?). They alluded to this in one short scene in Casino Royale, but she is addressed by this name exclusively in Quantum of Solace. The villain is a bit weak, and there are no classic henchman like Jaws, Oddjob, etc. There are almost no gadgets in this movie, which isn’t a bad thing, but it moves away slightly from the classic Bond theme.

This Bond movie, in my opinion, felt like the second part in a three part trilogy. This is the penultimate one that furthers the story line set forth in the prequel, and sets the stage for the final one. The Bond franchise seems to be making a deliberately effort to change the Bond legacy to a new format - one of realism and humanness. Ironically, it almost starts to feel Bourne-like.

With all that being said, this movie is definitely worth a watch. The added bonus: after watching the movie, you find out the real meaning of “Quantum of Solace”.

Enter FriendFeed. With the launch of their new feed publishing feature, you can post to Twitter any or all services that are linked to your account. Here are a few reasons why it’s better to do it this way:

• FriendFeed allows linking multiple blogs/services.
• FriendFeed enables your readers to leave comments.
• FriendFeed polls for updates more frequently than TwitterFeed.

Here’s how you can use FriendFeed to publish your blog feed to Twitter:

1. Log in or create a new account on FriendFeed
2. Add your blog as a new service on FriendFeed
3. Enable Feed publishing to Twitter in the Account settings on FriendFeed
4. Enter in your Twitter account info.

That’s all you need to get started.

The question often asked is: if you were asked to choose from two candidates - a specialist and a generalist - which would you choose? Like most things in life, it’s not a binary answer. There isn’t a right or wrong answer. There are, however, right or wrong decisions on hiring the right candidate for the right job. Each type of candidate has their strengths and weaknesses. It’s best to make the decision after careful evaluation of how closely aligned these types of candidates are to the job requirements and to the company at large.

This post is intended to be an introduction to a series of posts where I hope to give valuable tips on becoming an effective generalist. Before we begin though, I’d like to clarify a couple of things. By offering tips on becoming a better generalist, I don’t mean to suggest in any way that a generalist is better than a specialist - it’s just that I have more experience as a generalist as I have spent my career being one. And more specifically, I will tell you how to be a better generalist. How you use my tips in practice to improve yourself will be an exercise left up to your talent, dedication and passion. With that out of the way, let’s begin with the first in the series.

Become a specialist at being a generalist

This may sound almost like an oxymoron, but hear me out. Often, I’ve seen too many startup-headed, caffeine-infused, entry-level engineers who are ready to conquer the world. They dive into their projects with vigor and passion, trying to excel and outshine other members of the team. This is great and is the right attitude to have, and if you’re one of these individuals, you’re on the right track.

But commonly, what develops is this insatiable thirst for doing too much. It starts slow at first. Inconspicuously over time, you may find your behavior leading to a need for constant exploration of new fields of knowledge. In a startup company, it’s the engineers in the team who have the “i-can-do-everything” attitude. You start to not only code, but help out with tech support tickets. A marketing colleague drops by and you promise to deliver a new undocumented feature that will look good on your resume. Your geek buddies suddenly start talking about Python, and you want to learn it right away and test it out on a project. A new framework hits the web, you need to rewrite that last web application using the new framework. This weekend. Sound familiar?

What is important to note is that the above attitude isn’t bad - instead it’s possibly detrimental to your development as an outstanding generalist. In fact, the attitude is the focal point of being a great generalist. However, being caught in a never-ending spiral of exploration is the crux of someone who could have been a great generalist. That’s why it’s important to specialize in being a generalist. What does this mean? Specialization in any field means knowing the ins and outs. It means being ahead of the curve, leading rather than following innovation in the field. It also means doing what you do best, and better than the rest. And this is exactly what differentiates an effective and outstanding generalist from the rest.

It means to know what to explore. It means knowing how to keep your skills up to date. It means critically thinking about every new technology and what it means to you. It means visualizing the results before jumping into a new project.

So, I hear you say, that’s easier said than done. Actually, you’ll be amazed at how easy it is to accomplish with a little discipline and a few tips. Here are a few:

• Stay up-to-date with your area of generalization. This is crucial to the art of being a generalist. You must be able to consume large amounts of information each and every day to stay current on trends in your field. Over time, get into the habit of refining your methodologies. Always adapt as tools evolve and new ones come up. Become a master at picking good information out from the bad. Try to read sources of information that you trust, and have helped you in the past. Don’t get caught in the ‘chatter’ - opinions are plenty and every blogger has more than a few, learn to separate them from the real information.
• Know what to explore and what to ignore. This is the second step after you’ve found the right information. Read and learn more than what you actually explore and implement. Remember that time is very precious for every generalist. Don’t waste it by exploring a ‘cool’ website, learning a new language that all the cool geeks are talking about, etc. without knowing exactly how and why it will help you. Almost 50% of what you decide to explore will be of no use to you. The higher you manage to push this success rate, the more successful you’ll be.
• Be critical of new technologies. Mankind will always continue to invent and innovate. And when there’s not enough innovation, there’s recycling. If you’ve been around long enough, you’ll be able to pick out what’s recycled and what’s innovative. Learn this skill and get better at it. Think critically of a new technology that claims to solve a problem. Can this problem be solved with what you already know? Was this ever a problem or is this a problem that’s made up just to invent a new technology that solves it?
• Visualize the results of all new pursuits and endeavors. It’s easy to start every new project with an unknown factor. It’s the thought that whatever work you do will be worth it and at the end, there will be a lesson learned. Though this is a very healthy attitude to have, it only works well if you have a lot of time on your hands. But when time is short, it’s better to visualize the rewards from such a project. Is it going to increase your skill set in a particular language? Or is it going to help you gain understanding of a core technology across all languages? Spend time on it only if the rewards qualify the effort. Otherwise, be patient and spend time on something else that will give you bigger rewards.
• Don’t over-generalize. No potential employer is going to look through 30 different buzzwords and be excited that you tried and explored each. In most cases, an employer has already decided and probably spent a great amount of resources on a few key technologies. They need someone that fits into one or more of the technologies they use - not the other way around. The key is to explore many technologies but concentrate and gain expertise in a select few.

These tips should help you direct yourself on the path to becoming a more effective generalist. Stay tuned for more articles on specific tips to help you along the way.

Always remember that being a generalist helps you get your resume through the door, specializing at being a generalist gets you the job.

Imagine if the attempts by startups in the genome space like 23andMe were to succeed at cataloging everyone’s DNA. Would this piece of DNA become the ultimate representation of who and what you are? What would be the effects of having all the things that make you uniquely human be reduced to a string of characters?

It’s interesting to wonder if there would exist social networks around your DNA. Imagine a Facebook-style profile page that is uniquely identified by your DNA. Others could browse for the abilities that make you unique, your inherent strengths (and possibly your weaknesses), and possibly “friend” you based on your DNA. Maybe your DNA will be your OpenID or MicroID?

Here are a few interesting thoughts on how different applications could/would use your DNA:

• Dating sites - Matching people based on their DNA would outperform any complex dating algorithm out there right now (or so I believe). Imagine choosing your partner not just based on an algorithm’s recommendation, but based on specific scientific properties - maybe someone that complements your recessive genes?
• Recruiting - What if companies started asking for your genome in addition to your resume? It’s not that far fetched really - some companies already do extensive drug tests. Could companies accept/reject you solely based on your genes? Imagine if a company could pass on hiring a graphic designer just because he/she didn’t have the desired ‘creative’ gene?
• Insurance companies - Your DNA could easily identify the diseases that you’re most susceptible to. Could life insurance companies reject you based on that? Would that be fair?

If you notice, all the above have a common theme of solving the problem of selection (the irony of natural selection). A way to solve it is to carefully quantify the selection criteria and score individual candidates against each criteria. Fortunately, that’s not mathematically and/or scientifically possible right now (that’s why we have approximation models). But that may all change once each selection criteria can be mapped to a specific gene. The selection problem will not only be solved, but the answer may be very accurate - too accurate.

Do you think its a good idea to decode the entire human genome? Will companies like 23andMe have unfortunate and undesired side-effects? Should there be a fair-use clause for DNA samples?

The upgrade process was a snap thanks to this guide. The whole process took about 20 mins (backup, downloading the new version, overwriting, setting up plugins, and tweaking the template). Luckily, none of the plugins I use had any problems (including the Actionstreams plugin).

Here’s what I like about this release:

• Visual Editor: The interface is less cluttered and slick. The full-screen version rocks.
• Tags: Even though they were introduced as a core functionality in a previous release, they weren’t really usable until the new AJAXy version came along in this release.
• Widget-based dashboard: Now plugins can use the Dashboard screen real-estate for administrative information like the WP Stats widget.
• Lightweight UI: The UI looks more current than the previous ‘99 style. It looks more customizable, so I’m hoping for a outbreak of WP Admin themes (like the useful Fluency theme).
• Related section: In most admin pages, there is a nifty new section titled simply “Related”. It has a list of links that are related to the current page (writing a post, editing categories, managing comments, etc.)
• Multi-file uploading (with progress bar): It’s very similar to the Flickr uploader. Its intuitive, easy and just works.
• Built-in galleries: Though I’m not sure if I’ll ever use this, its good to know that I could roll out my own gallery instead of having to install another app. Amazingly, you only need to include the text in your post to turn it into a gallery (complete with navigation).

I highly recommend watching the 4-min screencast by Matt Mullenweg about the new Dashboard and Editor features.

For the curious, no additional tweaks were needed for the Grid Focus theme (the one used on my blog) to work in WP 2.5.

Download Wordpress 2.5 here.

Comparing FriendFeed to SocialThing is not like comparing gelato to ice cream (regular, fatty kind). Rather, they are really two sides of the same coin. FriendFeed aggregates your activities. SocialThing aggregates your friends’ activities.

Why people get into flame wars over this is beyond me. But clearly, people feel strongly either way.

Unfortunately, during this heated school yard brawl, I fear a very important question is going overlooked. Regardless of how aggregating services collate all the information, what is their strategy for propagating conversations back to the original source? FriendFeed’s strategy is to start a completely new thread of discussion. SocialThing instead attempts to back-propagate to the original source (at least in the case of Twitter).

Setting aside the emotions and hype, which model is better? Is it better to have parallel threads of discussion? Or should all conversations be funneled back to the source? Which model do you prefer?

Update: Lifestream Blog has an interesting take on this. Mark argues that instead of blog authors and their visitors discussing in a vacuum, peers (in Twitter/FriendFeed) get instant gratification by being able to discuss these same issues within a close circle of friends. But those seem to be two mutually exclusive use cases. Bloggers will want to have conversations with their visitors, and others should be free to discuss elsewhere too.

Wouldn’t you as a blog author want to stay informed of all the peripheral discussions about your posts? This just doesn’t seem possible if FriendFeed keeps comments limited to its own networks without propagating back to the original source/blog.

Update: Coincidentally, Jason at Blog Herald brought up the exact same point today. The ironic thing is that I found out about his post from a discussion outside his blog.

Update (Mar 24, 2008): FriendFeed seems to have been listening. They now feed comments back to Twitter (ala SocialThing style). More coverage here.

Download:

• Download Actionstreams Wordpress Plugin from Stephen’s blog
• Download from DiSo (latest)

Installation steps:

1. Download and unzip the file into /wp-content/plugins
2. Activate the “Actionstream” plugin through the WP Plugins admin page
3. Go to Users > Action Stream and set up your services with the needed info
4. As you add services, you should start to see actions in the Preview section
5. Add <?php actionstream_render(); ?> where ever you want to display it in your template

Updates:

• Mar 9, 2008: Mark Krynsky over at Lifestream Blog had the scoop even before I could finish hacking it up.
• Mar 10, 2008: Stephen updated the code (bugfixes and more services now supported). Download v0.11
• Mar 11, 2008: Stephen blogged about the plugin. Download here.
• Mar 18, 2008: Stephen released v0.2 of the plugin. Download here.

There have been some good recaps of the conference by Andrew Chin, ReadWriteWeb, and MyBlogLog. So I won’t do another. Instead, I thought I’d share some of my notes from the conference. These are not intended to be complete, and may not accurately represent what the speakers/panelists intended. They are some of my interpretations, predictions and are based purely on my visual, aural and note-taking capabilities.

• Social networks/cloud:
  • General consensus is that the social networks of today have got to stop behaving like ‘walled gardens’; more cross pollination and data sharing
  • Some sort of social graph + personal identity federation breakthrough is sorely needed
  • There’s a need for implicit relationships (often inferred from real life relationships) eg: Neighbors are friends but often neighbors aren’t your friends on Facebook
  • Prediction: The major players (Yahoo!, Google, Microsoft, AOL) will be the ones that breakthrough these problems
  • Business model for social apps is largely unclear; they are unable to monetize beyond ad impressions/click thrus
  • Prediction: Each person/user will have their own “personal CPM”
  • Social networks will have to compete to have the best experience for the high-influence individuals
  • Prediction: By 2013, social networks will become ubiquitous and taken for granted, they will be embedded everywhere; Question is how do we get there? What is the catalyst for wide adoption?
• Lifesteaming/Event streams:
  • Lifestreaming allows for checking updates across networks in one central location, is suggested to be a more ‘valuable’ feed as it is ‘filtered’ by the social graph
  • Need: Permissions are an unsolved problem (public vs. private events, shared among friends vs. everyone); need more fine-grained control without any surprises
  • Prediction: Filtering/value addition is the next big thing
• Facebook apps:
  • ~650 new apps a month (unconfirmed statistic)
  • app usage has flattened and may be trending down
  • most verticals are saturated
  • top 1% of the apps are used by 78% of facebook users (unconfirmed statistic)
  • tendency towards winner-take-all; either apps make it or not
• Myspace app platform:
  • publicly launching 3rd-party apps built on the OpenSocial developer platform by end of March
  • hyper-targeting is based on user profiling, machine learning and has shown up to 300% increase in click-thru rates
  • self-serve advertising system is intended for the long-tail of the advertisers
  • monetization models will only make sense once the problem of engagement is solved
  • engagement of the users in a social context will depend on the value the app/network provides
• Open Questions:
  • Is OpenSocial really open or is it directed/largely influenced by Google?
  • How can people reliably and easily port their data across networks? (Someone stated that Microsoft named it correctly as the Passport but implemented it completely wrong)
  • Is advertising the only monetization model for social apps/networks?
  • Is OpenID the solution to personal ID federation?
  • Is OAuth the solution to authentication?
  • Are social networks only limited to casual networks? Can they be extended to tradition enterprise software (Oracle, Salesforce, etc.)?
  • System-centric networks or citizen-centric networks? Hubs are social networks themselves or are the users the hubs?

Any thoughts on the open questions?