soabloke

Amazon Shareholder Letter

2011-09-06T07:51:52Z

This is a few months old, but Werner Vogels republishes the Amazon.com Shareholder Letter. A summary of how the quintessential 21st century company views its competitive advantage:

Service Oriented Architecture
Distributed state management
Decision management

Regarding Service Orientation:

Our technologies are almost exclusively implemented as services: bits of logic that encapsulate the data they operate on and provide hardened interfaces as the only way to access their functionality. This approach reduces side effects and allows services to evolve at their own pace without impacting the other components of the overall system. Service-oriented architecture — or SOA — is the fundamental building abstraction for Amazon technologies. Thanks to a thoughtful and far-sighted team of engineers and architects, this approach was applied at Amazon long before SOA became a buzzword in the industry. Our e-commerce platform is composed of a federation of hundreds of software services that work in concert to deliver functionality ranging from recommendations to order fulfillment to inventory tracking. For example, to construct a product detail page for a customer visiting Amazon.com, our software calls on between 200 and 300 services to present a highly personalized experience for that customer.

Tweet This Post Post to Delicious Post to Facebook

Crockford on Javascript

2011-08-12T08:01:22Z

A couple of years ago I read a book – “Javascript: The Good Parts” – which changed my mind about a language that I’d struggled with back in the nineties. I realise now that much of my struggle then was with the DOM and horrible browser incompatibility. Nevertheless the scarring was bad enough to keep me on the server side for the next decade.

I recently found a series of videos featuring Doug Crockford expounding on Javascript and its place in history. This series provides an interesting and entertaining perspective on computers and languages beyond just what runs in the browser.

The whole lot is a bit of a marathon but you can sample some of the main themes in a couple of episodes. I especially recommend episodes 1 and 4. Episode 6 stands well on its own as a compressed version of the whole series. Episodes 2 and 3 are very specific to Javascript the language if that’s what you’re after.

I like episode 1 because it has nothing whatsoever to do with Javascript but is an interesting tour through the history of computing, describing how we got to the languages and architectures that we work with today. This is essential context for any working programmers today. If you view only one episode, I recommend this one. The historical perspective is reduxed in Episode IV where Ajax is discussed in the context of HTML and its origins.

The important take away from these two talks is how the history of computing is anything but linear and deterministic. We got to the place we are via a series of steps, mis-steps, loops, accidents and fashions. And the random-walk continues. Crockford says:

“…important new innovations are received with contempt & horror and are accepted very slowly – if ever.

Often it is necessary for the previous generation of technologists to “die off” before major progress is made – especially in programming languages.

A key problem is that it usually requires many years of hindsight to determine which innovations really are useful and innovative.

Tweet This Post Post to Delicious Post to Facebook

No related posts.

Any Lessons from the AWS Outage?

2011-05-02T12:06:23Z

You’ve probably heard that Amazon AWS had some problems recently. A question on Stackoverflow recently pointed out a detailed summary of the problem posted on the AWS message board.

Obviously every distributed system is different and every outage is unique so it is difficult to generalise. Some takeways I have are:

Outages happen to even the best guys on the block…so you better plan for yours.
Building distributed systems is hard…so you need experience and experienced friends.
Manual changes are a common cause…not said explicitly in the AWS writeup, but strongly implied.
Outages are often “emergent” phenomena whereby a simple error causes many systems to interact in a way which grows exponentially. The AWS writeup refers to this as a “storm” and I have witnessed similar “storms” in large distributed systems. The degree of coupling and simple aspects like backoff parameters can make the difference between a disturbance that grows exponentially or decays exponentially. Think of the Tacoma Narrows bridge – perhaps the analogy is a stretch, but tuning of a few simple parameters can avoid destructive resonances.
One of the responses pointed to the Netflix Chaos Monkey as being vindicated by the outage. The “Lean” guys have taught us that if something is difficult (like testing or deployment) then you should do it often until it aint difficult any more. Perhaps system failure/resilience is the next frontier for this approach.

Tweet This Post Post to Delicious Post to Facebook

No related posts.

Ruby, Rails and Windows

2011-02-21T09:56:21Z

A few notes (mostly to self) about running the latest ruby 1.9.2 and rails 3.0.3 environments on Windows (yeah I know all the kool kidz have Mac OSX):

Ruby Installer for Windows is great: http://rubyinstaller.org/

Even better is the DevKit (available on http://rubyinstaller.org/downloads/) which gives you an MinGW compile environment for building native gems.

If you want to run in debug mode in NetBeans, then check out this patch: http://noteslog.com/post/netbeans-6-9-1-ruby-1-9-2-rails-3-0-0-debugging/

And finally if you get this error with sqlite:

no such file to load — sqlite3/sqlite3_native

then make sure you’ve got sqlite3.dll in your ruby/bin directory.

Tweet This Post Post to Delicious Post to Facebook

No related posts.

Who Cares About Technology?

2011-01-16T07:05:22Z

Steve Jones reminds us that the business doesn’t care about technology – so stop harping about it and using it as an excuse for underperformance.

I totally agree that this is a key reason behind the endemic business/IT culture divide that is the root of many problems.

However this poses an obvious question – who does care about the technology?. The trick is not to over-engineer, but to engineer to just the right level to deliver business value now and into the future.

Somebody has to care about the technology (products, tools, methodologies), because otherwise you lose control and foster a legacy of technical debt which ultimately erodes business value.

I guess this is an axiom of Enterprise Architecture – that lack of governance leads to chaos and inefficiencies. Some would argue with this assertion, but I have never seen a counter-example. And of course the inverse statement is not necessarily true either.

So if the business doesn’t care about technology then who does? And if that is “nobody” then what happens?

Tweet This Post Post to Delicious Post to Facebook

Is This CEP?

2010-11-30T06:52:25Z

A simple use-case:

1. Unhappy customer publishes Tweet containing negative sentiment.

2. Correlate Tweeter with frequent flyer membership.

3. Eight hours later, Tweeter receives feedback offer from vendor.

Is this automated marketing reaction, or is it just a coincidence?

If the former, it’s a simple example of Event Processing – a single event detected, classified (sentiment), correlated with standing data and a response dispatched.

Did they need to operate in realtime? Probably not…same day is good enough (but no longer!)

Did they need to use a CEP engine for this? Probably not (perhaps they did it in PHP).

Did they get the right result?

Tweet This Post Post to Delicious Post to Facebook

No related posts.

Whither CEP?

2010-11-30T06:35:03Z

An interesting post by Colin Clark lamenting the inability of CEP to live up to earlier growth expectations. The article is definitely worth reading in full, but if I can pull out a few cogent points, I believe Colin ascribes lack of CEP growth to:

CEP is mostly a marketing phrase.
Vendors have focussed tightly on High Frequency Trading and neglected other areas of opportunity.
Vendors have diss’ed developers by forcing them to learn new and arcane language(s).
Vendors have neglected business users by neglecting visualization requirements.

In broad terms I agree – although I’m not sure languages are an impediment given the explosion of new language interests in the mainstream. I think the fundamental problems are two-fold:

CEP products haven’t yet crossed the chasm from toolbox to platform. They are still very technical and incomplete. Most CEP products concentrate on the “engine” and neglect two really important areas – visualization (as pointed out by Colin) and context. Event processing relies on understanding events within the broader business context which requires no barriers between the event stream and other systems of record or operational data stores. This is an extremely challenging technical problem – how to marry real-time data streams with large volumes of data “at rest”.

The business value for CEP is not always obvious. Unless you’re involved in a high stakes, low latency arms race moving at ever more relativistic velocities, batch will usually work for you. Most organizations don’t yet operate in realtime. Those outside of HFT that do or plan to operate in real-time are doing some work with CEP (e.g. Telcos, Utilities and Logistics) but there the challenges are around my first point – integrating existing/legacy network optimization applications with the event stream. In such situations, it’s the optimization technology that drives the implementation, not the event processing technology.

So whither CEP?

Ultimately CEP has three pre-requisites: the business need to operate in real-time, the IT infrastructure to support this and the ability to analyse events within the context of all relevant data assets. The CEP “product” comes at the end of a long line of dependencies.

Tweet This Post Post to Delicious Post to Facebook

No related posts.

Coupling as Inertia

2010-11-20T12:34:20Z

In systems architecture, there are rarely any right answers – mostly just trade offs between one solution or another. In such cases it helps to bear in mind some fundamental principles as a guideline. One principle I often use is cost vs. benefit. Another useful principle is to minimize coupling between systems. Coupling is pervasive and leads to a kind of inertia in enterprise systems. Newton discovered that inertia prevents change and if there is one thing that enterprises struggle most with, it’s change.

Tweet This Post Post to Delicious Post to Facebook

AWS PubSub to a WebHook

2010-10-19T10:43:42Z

In my last post I showed how to send a Simple Notification Service (SNS) message to an email endpoint. Now I show how to easily add a WebHook endpoint. WebHooks are a design pattern using an HTTP POST to send a notification to a URL which the “subscriber” has registered with the service. WebHooks are being used in an increasing number of web APIs and there is an interesting interview with Jeff Lindsay on this topic at IT Conversations.

A useful test platform for WebHooks is PostBin.org. Simply click on the “Make a PostBin” button and you will be presented with a new URL for your notification messages – something like “http://www.postbin.org/1hf0jlo“. This is the URL you register with SNS.

Turning to the SNS dashboard, add a new subscription to a topic that you’ve already configured in SNS. Specify protocol “HTTP” and enter the PostBin URL as the endpoint. SNS will post a confirmation message to this URL before you can send through messages.

Go back to your PostBin URL and you should see the confirmation message.

Buried in the message is the SubscribeURL which you need to hit in order to confirm the subscription. I pasted it into notepad and edited “cleaned up” the URL before pasting it into a browser. This confirms the subscription with SNS.

Now back in the SNS Dashboard you can send a new message. In my case, since I still have my email endpoint, the same message is sent to both the email and the WebHook endpoints…thus:

…and thus:

Tweet This Post Post to Delicious Post to Facebook

Swarm Intelligence

2010-10-20T08:23:18Z

From the book “Honeybee Democracy” here are some cogent lessons on effective group decision making:

Compose the decision making of individuals with shared interests and mutual respect.
Minimise the leader’s influence on group thinking.
Seek diverse solutions to the problem.
Aggregate the group’s knowledge through debate.
Use quorum responses for cohesion, accuracy and speed.

Tweet This Post Post to Delicious Post to Facebook

No related posts.