Distributed Maze

Building Brand Value with Web 2.0 and SOA

noreply@blogger.com (Snehal) — Fri, 10 Oct 2008 21:12:00 +0000

I recently found a presentation online that I did last year at IBM Impact called Building Brand Value with Web 2.0 and SOA.

It goes through the history of some of the work that I've been doing over the last few years with a services oriented architecture and how we've been able to leverage the SOA towards building integrated websites with rich content.

The article doesn't read as well on paper as the presentation did at one point because it has been paraphrased from a transcript. Either way, hope you enjoy reading how one architectural paradigm can lead to other successful projects.

SOA and Web Service Versioning

noreply@blogger.com (Snehal) — Mon, 29 Sep 2008 02:49:00 +0000

One of the most important you can do when developing a new Web Service is to devise a way to implement versioning. The worst thing you can do is paint yourself into a corner by not doing so. If you don't, your clients will be too tightly coupled with your service and they'll be forced to update their client code and every time you update yours.

There are some simple ways to do so.

1. Use a new URL for new versions of your service and deprecate older URLs.
2. Use optional data types in your schemas as you add new data types over time.
3. Use a coarse grained request mechanism where the client can specify which data types to return. Over time, you can add more data types without affecting your older clients' deprecated data types. This mechanism is especially useful if your service is a content driven service like a destination content service or a mapping service.

Number of Reducers in Hadoop Distributed Mode

noreply@blogger.com (Snehal) — Tue, 05 Aug 2008 22:30:00 +0000

So I may be a complete idiot here and may be missing something completely... But I couldn't figure out for the life of me why my Map Reduce program was only returning partial files in pseudo-distributed mode. After putting in some log4j messages into my code and fishing through the log files for each individual task, I noticed that if I had 2 reducers running in distributed mode, that I would only get a small portion of the file that I was expecting.

Then I tried decreasing the number of reducers down to only 1 and magically, the framework started collecting everything into 1 file as opposed to rewriting the file that was already there and yielding a small portion of the file. Now onto more complicated things...

Simple AND Elegant

noreply@blogger.com (Snehal) — Tue, 01 Jul 2008 01:42:00 +0000

Simple AND Elegant. Two adjectives you use all the time to describe algorithms like MapReduce, but you know you have stumbled upon something special when you can also say the same about a framework that implements that algorithm. I've been following Hadoop, the open source implementation of Google's MapReduce algorithm, for some time now and until recently, I haven't had the urge to really look into how it works and what it can do until I read a blog called Hadoop by a New York Times developer, Derek Gottfrid. The incredible power of the framework and Amazon Web Services can easily be seen by what Derek Gottfrid did for the NY Times Archive.

So welcome to Distributed Maze, my foray into the distributed computing world and a blog post to help others who want to do the same. Sure, the name isn't creative, but at least it's original as of today's date! Hope it helps someone do some amazing things one day.