2009 have been a very busy and an eventful year. My wife graduated and thus ended our 2 year stay at the university residences at UTM. All though a small townhouse (albeit with 4 bedrooms - no idea how they fit them all in that tiny space) it was nestled in a lovely location. There was greenery all around and our townhouse backed into a wooded lot where there was deer with plenty of raccoons and the occasional rabbit. The campus had clearly paid a lot of attention to maintaining a healthy balance with nature, so there was plenty of tress around the area. I am not really a shutter bug, but we did take a few pictures here and there. Sadly I can’t seem to find them, so I’m linking with a few photos (pic1, pic2, pic3) I found on the net taken by one QueenCindy. My kid loved this place as there was plenty of room to run around and play.

After my wife graduated we had to moved into another place and after a very exhaustive search we found our selves back in the northern edge of Brampton,ON. House hunting was no fun and we spent countless hours running around looking for a place that fits our whims and fancies and also within our budget. Once we found the house it was a mad rush to find a mortgage at a suitable rate as we had only a few days. At the time the rates were very attractive, but the lending institutions were very thorough with so many requirements for various kinds of documents. I just about managed to get all the paper work done in time, but not without a last minute scare where for reason only known to them the bank refused to work with my lawyer. So I found another one at the eleventh hour that had a good working relationship with the bank which ensured we closed on time. All this happened while my wife was doing her final exams and I had to work on site with a customer that had a fairly aggressive schedule. My interactions with customers were mostly via email or phone calls, so being onsite was a new experience, all though I have to admit it was fairly stressful as well. I have blogged on my experience about it here. By the time the gig came to an end, I had lost quite a few hair.

The university gave us a one month grace period as there was nobody waiting to lease our townhouse. It allowed us some flexibility in the closing schedule. We have moved 5 times during the last 6 years, where one move was from Ottawa to Brampton (in Greater Toronto Area) !!!. So the very thought of moving again itself was a dreadful experience. I managed the last 5 moves on my own by renting a UHaul trunk. It wasn’t fun and I had enough, so this time I hired a mover. It took us quite a bit of time to get everything unpacked and settled in.

I started playing cricket in earnest during the last season (2009) for the Toronto and District Cricket League. It was an up and down season ending with me breaking my finger pretty badly during the last match of the regular season. The finger was dislocated and had a volar plate chip fracture. My finger is still in a pretty bad shape with a real possibility of it not attaining at least a 70% of what it used to be. One highlight of the season was facing Navin Stewart from Trinidad (who came to Canada to play club cricket during his off season). He plays for Trinidad and Tobago and featured in the Stanford Super Series and the esteemed Champions League in India. The bastard bowled two bouncers the second of which hit my glove, ricocheted off my helmet before being caught at the slips. He had decent pace and it was difficult to pick the red ball against the fading light. I hope I had better luck this coming season, but not sure if I am able to play at all due to the finger injury.

With all the madness happening around I had managed to post a grand total of 3 blog entries for 2009. I am certainly hoping to improve on that in the coming year. Again I wish everybody the very best for 2010 !!!

From the read me the following features are available,
1. WCF service model programming using one way contracts
2. WCF channel model programming using IInputChannel and IOutputChannel based factories
3. Programmatic access to AMQP message properties on WCF messages
4. AMQP version 0-10 (as provided by the Qpid C++ native client library)
5. Shared connections for multiple channels based on binding parameters
6. WCF to WCF applications (using SOAP message encoders)
7. WCF to non-WCF applications (using raw content encoders)
8. Rudimentary AMQP type support for headers (Int and String)
9. Channel functional tests using NUnit
10. Programming samples

There are several opportunities for folks to collaborate in this area. Cliff Jansen who worked with Qpid on the code drop identified the following areas as potential areas for contributions.

From qpid/wcf/ReadMe.txt:

2. Planned features (not yet available)
=======================================

1. Full AMQP type support, including maps and arrays
2. System.Transactions integration (local and distributed with dynamic escalation)
3. Prefetch window for inbound messages
4. Shared sessions
5. Connection failover with AMQP broker clusters
6. Temporary queues
7. Broker management
8. System logging and tracing
9. CMake build system support
10. Transport and message based security

One item that I would like to add to the above list is documentation for the WCF client and it’s examples.

If you are interested in contributing please join us by sending an email to dev-subscribe@qpid.apache.org
You are most welcomed to contribute by way of code, testing, providing feedback or documentation.

In any case it is important that software engineers develop necessary skills to handle such situations effectively. Some of these skills will help when communicating within your organization and some for life in general.
Following are some tips that may help you to make your engagement a productive and a pleasant one.

1. . Pay attention to the subtleties when communicating over email.
   It is often easy for people to misunderstand you when communicating over email as opposed to phone or in person. The tone could be easily misunderstood as being aggressive or rude as the person reading is unable to gauge your emotions or see your body language. This is especially important if you haven’t met them in person or spoken to them over the phone. Generally once a person can put a face to a name it becomes a bit more easy.

   I have had several instances where this has happened and I now pay close attention to any email I send out to ensure that I communicate exactly what I intended.

   Also it is important to state the obvious at least for completeness sake. What is obvious to you may not be to others. This especially important when using emails as there is no opportunity for instant feedback/questions. Otherwise it will result in lengthy email threads.




2. Respect the people you are dealing with and try to build good working relationships
   Telling a customer to RTFM is probably not a good idea. Even if you are frustrated due to repeated questions that you have answered a 100 times to several people during the last few days, it is best to answer it one more time. Even when you have no choice but to tell them to RTFM, do so in a respectful manner which doesn’t make people feel bad. Nobody in this world wants to be belittled or disrespected. When egos are hurt it creates a very unpleasant work environment.

   In fact good healthy relationship can go a long way in making sure the project runs smoothly and successfully.




3. Your code vs my code - avoid the customary finger pointing
   Most people are very passionate about the code they write. And often when there is a problem we tend to take a defensive stand. Anybody who has seen a conversation between a programmer and QA engineer would know what I am talking about.
   But when you are dealing with a customer, you need to be extra careful about how you look at a problem. Even if the issue is not in your product you need to control your urge to point it in the same emphatic way you do so with your QA folks. You need to be more diplomatic in how you point out the bug on their side and try to avoid using words like “your code” or “our code”.




4. We can’t reproduce in our lab environment is not a good answer.
   Certain problems may not be possible to reproduce (in the exact form), outside the customers environment as it is difficult to replicate the exact environment due to differences in hardware and not having access to their application binaries due to security, legal and other issues. . But it is important to identify the issues at a more deeper level. Perhaps if you understand the root cause, you maybe able to simulate the issue in a different way.
   Take the time to understand the issue and try to talk to all people involved to get a complete and accurate picture. From experience I have seen that sometimes you want get all relevant information the first time you talk to people. People often forget unless you ask probing questions. Most times the first reactions from people capture the symptoms rather than the root cause. So a careful analysis maybe need to nail the real issue.




5. Encourage the use of proper support procedures right from the beginning. Never take short cuts.
   I have been burned several times by not following this. When an issue happens it is important that you encourage your customers to follow the correct procedure. For example the use of a bug tracking tool instead of a series of adhoc emails. It is often tempting to ask customers to email log files, or if you are on site to walk over to their desk and try to investigate.
   But if you encourage the use of a bug reporting tool it will have several benefits. It will force the person reporting the issue to think through in a methodical way and capture all the right information while it is still fresh in their mind. For example most bug reporting tools have fields to capture version information, a problem description, a component, steps to reproduce (if any), expected and observed result and provisions to attach log files ..etc. Once information is captured in a central location it is easy to share that information with your engineering team rather than digging through a series of adhoc emails.




6. Proactively educate the people you work with
   By experience I have found it is helpful to have a meeting as soon as possible with all involved parties and educate them on the following. Some of these may have already being communicated to them. But most may have forgotten or in some cases not being given to them at all.
   • Provide links to documentation, training slides ..etc
   • Educate them on the proper procedure to follow when reporting issues (can’t stress this enough)
   • Based on the most frequent questions, try to provide an faq that is tailored to their environment
   • Once you identify the full system setup provide a basic trouble shooting guide for them to isolate if the issue is related to your organizations product or not. This I found reduced a lot of pain and non issues being reported.
   • Provide useful links to other skills involved in the project. For example I sometimes encountered folks who work mostly in a windows environment, but now dragged into a product being deployed in a unix environment. So links on basic unix skills would make it easy on everybody in such a situation.




7. Don’t babysit the people you work with
   Initially you may get into the habit of spoon feeding information and babysitting them. This may seem like an easy option rather than trying to get them involved in the process. But it is in everybody’s best interest to get the customer to gain as much confidence in getting to know how to install, debug and use your product. This will reduce a load of pain later on in the process. Also try to,
   • Avoid getting sucked into debugging the customers code.
   • Avoid the temptation to debug or fix an issue on your own. Try as much as possible to get them involved in the process.
   • Document trouble shooting attempts and consolidate them into a trouble shooting guide. Most products have official trouble shooting guides, but a supplementary trouble shooting guide tailored to their environment and using terminology familiar to them will make them more comfortable.




8. It is helpful if you have multiple programming language skills
   Since you write code, the customer will expect you to answer questions at the code level. Sometimes you may have to review code segments and provide an opinion. However if your product is providing clients in different languages, these questions may be in a language that is outside of your core competencies. Therefore it is important that you have a at least some knowledge using those languages.
   In general the ability to write code in several languages is always an added bonus as you will be a valuable asset to the company.




9. Know thy unix tools
   Also if you are in the unix world it is nice to have some skills in basic unix tools like top, vmstat, pmap, netstat, sar, lsof ..etc and text processing languages like perl, awk, sed etc. These tools can help you diagnose issues and obtain important information about the env. The text processing tools will be invaluable when you need to make sense out of massive log files or other text processing tasks. These skills will make you more productive by allowing you to extract information quickly and efficiently than just using grep or crunching numbers using a spreadsheet.




10. Keep your conversations professional and never take things personal
    You always need to remember that you are representing your company/product and the team you work with. Therefore the people you work with at a customer’s site may directly vent their frustration at you. This may manifest in the form of rude comments, lack of cooperation and in rare situations a bit of yelling (Fortunately I’ve only seen only one such incident that happened a few years back) . Never take them personally or get into any sort of argument. It is best not to take anything personal or get emotionally involved. When the stakes are high and push comes to shove, humans will react and we need to understand that and continue to behave professionally. Sometimes all they expect is for you to keep quiet, listen to them and empathize with their situation. And something simple as “I hear your point/understand your frustration. We are doing all we can to help” is all it takes.

    However if you are finding it difficult to cope, it is best to bring it to your managers attention and discuss possible measures to address the situation. It’s a bad idea to just keep quiet and bottle them inside, as you run the risk of an emotional outburst. Let your manager help you by dealing with the management on the customers side. All though the customer is the king, it is not acceptable for anybody to be treated badly.

Apache Qpid (http://qpid.apache.org) is a cross platform enterprise messaging solution which implements the Advanced Message Queueing Protocol (http://www.amqp.org).
It provides brokers written in Java and C++ and clients in C++, Java (including a JMS implementation), .Net, Python, and Ruby.

New features included in this release are:

• .NET, WCF and excel support for AMQP 0-10
• SSL added for C++ broker and all clients
• Windows port for C++ client & broker
• C++ Broker
  • ACL
  • Active-Active clustering
  • Federation, push bridges & dynamic routes
  • RDMA for C++ broker & C++ client (70-80us, yes us max latency on a well setup machines)
  • Support for message TTL
  • Queue options
    • added RING/ STRICT ring
    • LVQ
  • Exchange options
    • LVE
    • message sequencing
  • XQuery based XML Exchange now as plugin
• Performance work
• Management for AMQP 0-10
  • QMF C updates (console)
    • Python
    • C++
  • QMF Agent
    • C++
  • QMan JMX bridge for QMF
  • Alerts/ logger for QMF events
• JMSXUserId
• Java broker
  • Message Priority
  • bug fixes
  • some prep work for AMQP 0-10

It is available to download from:
http://www.apache.org/dist/qpid/M4/

Complete release notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310520&styleName=Html&version=12313279

Spin bowling is more art than science. The principles of spin bowling are very simple. One could learn to bowl spin by learning these principles, but mastering them is an art. A closer look at spinners who were successful at their trade, shows that each made an art out of these principles. They had something unusual to offer that set them apart from the rest of the pack.

Looking at some of the greatest spinners, for example Bhagwat Chandrasekhar had an unusually whippy action due to a withered arm which was a result of childhood polio. It is said that he delivered googlies, topspinners and leg breaks at near medium pace and eventually producing an unplayable delivery. Abdul Qadir had a fancy run up, the temperament of a fast bowler and good variety to back it up. His googly was probably the best. Muttiah Muralitharan was a vicious spinner of the ball on any type of surface. Saqlain Mushtaq was probably the first to use the “doosra” effectively. Shane Warne undoubtedly the best leg spinner we have seen so far, has quite a bit of variation. Then finally you have Ajantha Mendis who uses variation as his main weapon. All these folks mixed their unique talent (and their deformity in the case of Mural and Bhagawat) with these principles to create an art form that is almost magical. (For the die hard fans, the above is not an exhaustive list of great spinners. So if your favorite is not listed there please do not think I have thought less about your favorite spinner).

As a bowler you need to consistently beat a variety of batsmen to become a successful bowler. Sure you may get a few freebies when the batsmen eventually makes a mistake (every human is going to make a mistake at some point) or due to pressure at the other end. But if you need to improve your wickets column and become a champion spinner, then you have to ensure that the batsmen makes those mistakes while you are bowling.

There are several ways to beat a batsmen and more often than not a combination of those techniques is what buys you a wicket. Most of these techniques are intertwined. One technique is to try and out think a batsmen. For example Shane Warne always tried to anticipate a batsman’s move and bowled accordingly. Creating doubts in a batsman’s mind is another way. Ajantha Mendis is a master at this as he bowls very accurately with subtle changes in the direction where the ball spins. Bhagwat Chandrasekhar didn’t even know what would happens when he bowled and never planned his deliveries. When a batsman is not picking up the variations he begins to doubt and his confidence suffers, effectively stifling the scoring and increasing the chances of making a mistake. Deceiving a batsmen is another way. You could deceive a batsmen through changes in the amount of spin imparted, pace, line, length , flight or direction of spin by cleverly disguising your variations off the hand. For example a spinner could bowl a quick top spinner, a cleverly disguised googly or an arm ball. The batsmen depending on his ability and concentration level may not pick the subtle change in your action, the pace or the direction it takes off the pitch, Yet another way is to setup and lead a batsmen into a trap. For example a leg spinner or an off spinner could use their stock ball on a given line and length and then stealthily produce a googly or a doosra on the same line and length inviting the batsmen to make a mistake. Or the trap could come in the way of a cleverly placed field setting.

If we look closely most of the great spinners, have devised their own way of executing those techniques. The rest are ordinary bowlers who approach spin bowling mechanically as a science and most with a bit of coaching can get there. But the legendary spinners used what ever skill they had to transform spin bowling into an art of it’s own. Great spinners are not made, they are born.

One of the engineers from Microsoft Anandeep Pannu, has written very excitedly about participating in Qpid.

In July Sam Ramji announced a sponsorship for the Apache Software Foundation. It kind of signaled about their intent to get their feet wet with playing an active role in open source. It seems that most large vendors have realized the importance of participating and promoting open source on one hand and also harnessing the power of open source on the other hand. IBM for instance played this strategy very well over the last decade or so. This is very good PR for the Qpid project and hopefully attract more participants and end users to the project. Definitely looking forward to working them in Qpid and the AMQP working group.

Without any further do here are the reasons in no particular order.

1. Difficulty in identifying and dealing with failures.
When communicating between processes failures can happen at many levels. Dealing with them is not trivial. Of course you rely on frameworks based on technologies like RMI, CORBA, COM, SOAP, AMQP, REST(is an architectural style not a standard) etc to handle these. But the fact remains that you still need to clearly think about these cases and handle these situations properly.

For example if we consider a simple interaction between two processes on different computers, the following failures can happen.

• Failures that occur within the process that initiates the communication (sending the message or invoking the RPC call).
• Failures between the time the process hands over the request to the OS and the OS writing it to the network.
• Network failures between the time it takes to transmit the packets from one computer to the other.
• Failures between the time the OS on the receiving end receives the packets and then handing it over to the recipient process.
• Failures that occur when the recipient process tried to process the request/message.

Sometimes the framework you use, is unable to/may not report all these error cases. Sometimes when the error is reported, it may not contain enough information to figure out at which level the error occurred.
Did it reach the remote computer? if so how far up the stack did it go?. If the receiving process got the request or message did the error occur before or after the request/message was processed?
In some cases where idempotency is built into the the receiving application or the framework/protocol (ex a message client that detects duplicate messages, or doing an HTTP GET) a simple retry maybe ok. In some cases Idempotency and retrying maybe expensive or difficult to implement. In such cases careful thought needs to be given on how these different errors are identified and handled.

2. Achieving consistency in data across processes.
One of the hardest problems in programming distributed systems is achieving a consistent view of data across the processes. When one processes updates some data, you need to replicate them across the other processes, so if any other process decides to operate on the same set of data, then it is doing so on the most current copy.
Lets look at two examples.

Assume a global banking application for ABC bank. A customer goes to a branch in New York, US and deposits money to an account. A few moments later his relative in London, UK does a withdraw on that account. Due to latency there is obviously a time lag before the process in London, UK sees the updated amount in the account.

In an online trading system, a user in NY places an item for sale. The transaction is updated on the closest data center which is in Boston. A few moments later another user in LA is searching for the exact same item and is served off a data center in Phoenix. The user in LA may or may not see the item due to the latency involved in replicating the data across

For example 1 strong consistency is required, while for example 2, you could get away with weak consistency, for example by setting an SLA that says data is valid within a 5 min time window.
This is not an easy problem to solve and this area itself is a subject on its own. Wener Vogels wrote a nice peice on this called Eventually Consistent which is worth reading.
Of course there are specialized frameworks/libraries that can handle this for you. But still there is no escape for you and you pretty much need to have an understanding of the pros and cons of various approaches, failure modes etc.

3. Heterogeneous nature of the components involved in the system.
A distributed system may contain components written in a variety of languages deployed across machines with different architectures and operating systems. Needless to say that this poses certain challenges (especially integration, interoperability issues) when implementing the system. A whole range of standards/technologies were presented to solve these issues, including but not limited to CORBA, SOAP, AMQP, REST (is an architectural style not a standard) and RPC based frameworks like ICE, Thrift, Etch etc. Anyone who has worked with these technologies knows that neither of these are trivial to use nor provide a complete solution in every situation.

If anybody has read the recent posts by Steve Vinoski and the discussions around it would realize the issues/challenges surrounding RPC. The following paper discuss the impedance mismatch problems when working with IDL based systems. The issues with type systems and data formats are not limited to RPC only. When using a message oriented approach like SOAP (doc lit style) or AMQP you will end up tunneling data thats not supported by the protocol as a string or a sequence of bytes. When using REST you would need to represent your resource in a format the requesting application understands/supports, which maybe quite different from the native format.

Again not an easy issue to deal with no matter what technology or framework is used. As an architect/developer you need to understand these issues and deal with them accordingly.

4. Testing a distributed system is quite difficult.
This is arguably one of the hardest aspects of developing a distributed system. Verification of the behavior and impact of your code in the system is not easy.
There are many aspects that needs to be tested, and doing so before every checkin is not a fun task at all. Running some of these tests before every checkin is not practical. But its a good idea to run them nightly and some tests during the weekend. Here are some of the areas that needs to be tested (I plan to write another blog entry elaborating on the testing aspects).

• Functionality testing (can be covered with well written unit testing)
• Integration testing - you need to test the distributed system as a whole with all the components involved
• Interoperability testing - this is crucial when heterogeneous components (different languages, OS) are involved, and is quite different from integration testing
• TCK compliance - If your system is based on standards/specifications, then you need to ensure that you haven’t broken anything w.r.t compliance
• Performance testing - to ensure that your changes haven’t accidentally caused a degradation in performance
• Stress testing - to ensure that your checkin hasn’t accidentally caused any stability issues - ex increased chance of deadlocks when the load increases
• Soak testing - to ensure that your checkin hasn’t caused any longevity issues - ex a memory leak thats manifested after a couple hours, days

Most often than not developers cut corners in their testing as running these tests are tedious and time consuming. Also these tests need to be run regularly to catch issues in a timely manner and the best way to tackle this issue is to automate as much testing as possible. There many options with continuous build systems like cruisecontrol or using a plain old cron job.
Functionality testing, TCK compliance, certain types of integration and interoperability tests can be run periodically.
In most organizations test machines are just lying around doing nothing during the night (unless around the clock testing is done with development centers in different time zones.). Instead of wasting computing cycles, you could automate test suites to run during the night. More time consuming integration and interoperability tests, performance, stress and soak testing can be done nightly, while more longer duration soak testing can be scheduled to run during the weekends.

While testing is a tough issue for any type of system, distributed systems have a lot more failure points which adds to the complexity.
Getting these tests right to cover these failure points and executing them needs a lot of careful thought and planning.

5. The technologies involved in distributed systems are not easy to understand .
Distributed system are not easy to understand. Neither are the myriad of technologies used in developing these systems.
Most folks find it difficult to grasp the concepts behind these technologies. If you look into the discussions and misconceptions surrounding REST you can understand what I am trying to get at. CORBA was not an easy spec to understand, so is WS-* or AMQP. While it is true that you don’t need to understand everything to develop using them, you still need at least a reasonable understanding to figure how to tackle some of the above mentioned issues. Frameworks based on these technologies are touted as the cure for these problems. Sure they could help, but it still does not shift the burden away from you.
To compound the issue all sorts of vendors keep touting their technology/framework as the next silver bullet. No matter what vendor you use, at the end of the day you are still responsible for getting it right. And it is not an easy task. You need to face the reality that distributed systems are hard and that you cannot hide every complexity behind some framework.

Watch the WSO2 blog space or the Axis2/C website for a tutorial/blog post from Danushka.