Life of a Software Engineer

Apache Storm Message Broker Spouts

2014-04-18T16:55:00.001-04:00

I was working with Apache Storm for sometime now. I wanted to get the messages from different message brokers and and process them inside Storm. The brokers I'm interested are RabbitMQ, JMS Based brokers like ActiveMQ, Appollo, Kestrel and Kafka. The existing spouts were lacking some of the functionality I was searching for.. The most important things for me was total control of the message and ability to read from multiple queues from a single spout. So I thought of writing these spouts for various brokers and I have written RabbitMQ and JMS spouts. The source code can be found in github

https://github.com/supunkamburugamuva/storm-broker-connectors

The software is under Apache License Version 2. So you are free to use it anyway you like.

Starting Hadoop, setting Datanode directory file permissions

2013-04-03T10:19:00.000-04:00

I was trying to start Apache Hadoop in my Ubuntu machine with the data node and name node on the same machine. I've changed the hdfs-site.xml as following.

Loading ....

Then when the datanode starts it said the permission of the data directory is incorrect.

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/supun/dev/apache/hadoop-data/data, expected: rwxr-xr-x, while actual: rwxrwxr-x

So I had to change the permission of the data dir as following and it worked.

chmod 755 hadoop-data/data

Graphene supercapacitors, can be the biggest invention in 21st century

2013-03-25T14:23:00.001-04:00

It seems finally humanity has found a way to create more efficient batteries.

There is a long way to go. But the future seems promising. This can be one of the biggest inventions that changed the human history. Looking forward to a day where we can charge our cars within minutes, cell phones and other devices instantly.

Scripting with perl

2013-03-25T11:22:00.002-04:00

Here is a tutorial about some basics of perl that can be very handy.. The tutorial talks about how to connect to a database, reading a config file, replace a string in a file etc.

Java Regular expression for getting sentences with a given word

2013-03-08T12:00:00.000-05:00

Here is a simple regular expression for finding a sentence with a given word.

Loading ....

A Myth about Threads

2013-02-07T16:02:00.001-05:00

We were given an assignment in one of our classes to write a port scanner. This was a great assignment and we've learn't a lot from it. The assignment was pretty huge with lot of functions. For those of you who don't know what a port scanner is, a port scanner is a piece of software that can be used to determine the status of network ports in a given machine. A port scanner can also be used to determine what are the software running on open ports etc. Ports are specific to TCP and UDP and a port scanners functionality is border than TCP and UDP and can be used to determine the protocols that are running in machines etc. A good place to look at for more details is here. For making our discussion simple lets assume we are only going to scan TCP ports.

One of the key things about a port scanner is the ability to scan ports in parallel. Usually there are 2 to the power 16 ports in a given machine. But usually we want to scan a subset of these ports and even this subset can be huge. So if we scan each port one by one it will take a lot of time. To avoid this we have to send requests in parallel to the range of ports that we scan.

A port scanning request for the most part is an IP packet with a TCP or UDP payload. In the TCP case we don't want to create a full TCP connection in order to determine weather the port is open. We can simply send a TCP SYN packet and if we get a response with SYN + ACK we can conclude that the port is open. There are many other techniques for determining the status of a port and non of them require us to create a connection. To send these TCP SYN packets we need to use RAW Sockets and we cannot use regular stream sockets.

The important summery of the above is that we create a TCP SYN packet for each port that we are going to scan and send them in parallel to the destination IP using a RAW Socket.

Since these are RAW sockets there is no TCP Flow Control or TCP Congestion control. Once we call the send with the IP packet, the kernel will send the packet as fast as it can. First it will copy the buffer to the kernel space and hand over this to the IO Subsystem to send. The important thing is the call to send the packet is non blocking on IO as we are not waiting for the success or failure of this packets to be delivered to the destination or out of the host machine. But since send involves a system call it can be bit slow compared to other method calls. Also note that these are very small packets, with byte sizes no more that 50 bytes for the IP packet and there is no data in the TCP packet.

The assignment asked us to create threads for making the sending process parallel. The expectation is that as the number of threads increases the sending speed will increase. But is this really true? Will threads increase the sending performance linearly?

If we run the program on a multi-core machine definitely the increase of threads should increase the sending rate until the number of threads is somewhat greater than the available processors. But will this increase the performance if we increase the number of threads to 100 and available processes is 8? My gut feeling is it NO. But people tend to think that increasing the number of threads will increase the rate of sending. But since the operations are CPU bound increasing the number of threads beyond the number of available cores is not going to give any performance. On the contrary increasing the threads should decrease the performance of the system.

This is a conclusion I came from theoretical knowledge and past experiences. No performance test was done. may be I should do a performance test or may be I should not :) Lets see what others have to say.

REST vs SOAP Adoption - A Perspective

2012-08-31T16:04:00.000-04:00

I worked for three year in a company called WSO2, building Web Services based middle-ware systems. Five years ago we believed Web Services is going to take over the world by becoming the de-facto standard in communication. Web Services is such a nice and well thought technology. It has a metadata model, a rich set of tooling, a built in security model; which are essential parts of enterprise grade communication. Also WS is backed by industry giants like Microsoft, IBM, Oracle etc. No matter what this background is WS lacks mass adoption. It seems only the big companies with lot of money and resources invest on Web Service adoption. For sometime I was bothered by this strange situation. Why such a robust and well thought out messaging paradigm backed by industry giants lacks the adoption?

On the other hand REST doesn't have any of the nice things about Web Services. It doesn't have a well thought metadata model, comprehensive tooling support or a standard security model. But against all odds REST seems to be gaining traction and becoming the cool thing in communication. Everyone wants to build a cool REST Api.

So what is wrong with SOAP and what is so appealing about REST?

In my perspective this all lies in the developer mind set. Developers love when the sky is the limit for their programming. They want to create unique designs that no one else has done. They want to explore the unknown.

REST as a framework gives the developers exactly that. The freedom to innovate. Developers love that. As a framework REST introduces very few rules. Most of them are not even rules. Rather they are soft guidelines. So building a proper powerful REST API is an Art and a challenging task. In my opinion this makes it cool.

This is completely opposite to SOAP. SOAP gives the developer all the rules. First thing about SOAP is you get a SOAP envelope where you have to put your data according to some rules. So from start to end the system is pretty well defined. There are standards for doing even the simplest things. Top of all this SOAP is very complex to understand. So in my opinion SOAP doesn't give user the nuts and bolts to innovate. Instead it gives a half baked solution that user has to obey. I guess not all developers like to bog down by rules and they don't like to learn that kind of rules as well.

There are many aspects to REST vs SOAP than what I have mentioned above. Someone can argue SOAP is better than REST. But I just wanted to express my perspective on what is happening in the world.

Install Google Chrome on Ubuntu 11

2012-07-31T10:52:00.000-04:00

Download the Google Chrome from the internet.

Go to the download folder and execute the command

sudo dpkg -i google-chrome.deb

For example:
sudo dpkg -i google-chrome-stable_current_i386.deb

If it fails due to un-satisfied dependencies

Use the following command

sudo apt-get -f install

This will download the dependencies and install Google-Chrome.

Enjoy!

Counting the code lines in a Java program

2012-05-18T16:49:00.001-04:00

To count the code lines in a java program use the command

find . -name "*.java" -not -iwholename '*target*' -not -iwholename "*.svn*" | xargs wc -l

This will search for .java files and ignore the files in target and .svn directories.

Password protect a file in Linux

2012-03-27T11:18:00.002-04:00

We can use the gpg utility to password protect a file in Unix based systems. The command

gpg -c filename

will ask for a password and create a new file with the extention .gpg. Now the new file is a password protected file.

To decrypt the file, just use

gpg "protectedfilename"

It will ask for the password and decrypt the file.

Streaming HTTP Client using Apache HTTPCore-NIO

2012-03-21T22:28:00.002-04:00

Recently I'm working on a sensor project where we need to transfer large amount of sensor data. These data includes Video streams, Kinetic data etc. Due to the high volume of data it is necessary to have a client that can transfer data using a constant amount of memory.

Java NIO technology is a best fit for writing such a client because it gives the total control over how the data is handled. Also we have used the HTTP as the wire protocol for transferring the data. HTTP with chunked transfer encoding can be used to stream data. For this project we are using Apache HTTP Core NIO for writing the client. Apache HTTP Core NIO is a set of libraries for writing efficient HTTP clients and servers. The code for the client can be found in github at the location

https://github.com/iotcloud/IoTCloud/blob/master/modules/streaming-server/src/main/java/cgl/iotcloud/streaming/http/client/core/HttpCoreClient.java

The send method of the written client takes the data as a input stream. The nice thing about this implementation is that the input stream is backed by a constant memory buffer. So when the buffer is full the write to the input stream is blocked until more space is available. All this is achieved with the Java NIO technology and I'm not going to go in to the technical details. Still we haven't done lot of testing on the client but we are hoping to do so soon.

Command for SVN Move directories selectively based on a search criteria

2012-03-21T09:41:00.000-04:00

I wanted to search and move particular directories in a svn project by replacing part of there name. Here is the command I've used.

find . -depth -type d -name "*to_change_directory*" -print | while read ; do svn mv $REPLY `echo $REPLY | sed s/to_change_directory/changed_directory/` ; done

Here I'm moving the directories by replacing the part "to_change_directory" to "changed_directory". For example

TESTto_change_directoryTEST will be moved to TESTchanged_directoryTEST.

POST file using Curl with chunked encoding

2012-03-16T12:21:00.002-04:00

Curl is a very useful tool for communicating with HTTP servers. Here is a command for sending a large file with chunked transfer encoding.

curl -v -X POST --header "Transfer-Encoding: chunked" -d @Test.txt "http://localhost:5000/"

Note: we have to set the header Transfer-Encoding: chunked.

K-Means algorithm implementation in Java

2012-03-05T18:49:00.001-05:00

We have given a homework assignment in Data-Mining class to implement the k-means algorithm and it was a pretty exciting and fun home work assignment to work on. We could get our hands dirty with a real data set and a real data mining algorithm. So I thought of putting the work in github so that everyone can use the algorithm.

K-Means is a simple yet very powerful algorithm for clustering data. It is a predictive algorithm for determine the clusters. There is a whole lot of research done around K-Means because it provides a fast and reliable solutions for most practical applications. The idea is to put the data points in to a cluster with the smallest distance from the clusters mean to the data point. We do this iteratively until we hit some threshold. These threshold can be very simple like constant number of iterations or it can be very complex based on the variance and the quality of the data in clusters.

The algorithm I have developed is pretty generic one and we've used the Wisconsin breast cancer data as the sample data for clustering.Bulk of the Code is pretty generic and can be used for clustering other data sets as well.

One of the best things about the k-means is that there are so many parameters to tune it. This means depending on the nature of the data set user has to do some investigations to pick the correct set of parameters. In the given implementation I have made the following parameters configurable. I have included the default implementation of these configurations but user can extend the code to integrate their own as well.

Number of Centroids - This is the number of centroids to be used, and is the k of the algorithm

Centroid Initialization - User has the choice of two method.

Data is put in to k partitions randomly and the mean of these partitions are taken as the initial centroids
k data points are selected randomly and used as the centroids

Splitting - If a partition becomes empty we are going to split another partition in to two so that we can maintain the k number of partitions. User has the choice of selecting two different splitting methods.

Get the largest partition and split it in to two equal parts randomly
Get the partition with the largest variance and get the data point that is at a maximum distance from the mean of that partition

Distance algorithm - The default implementation has 3 built in algorithms for calculating the distances.

Euclidean distance
Euclidean squared distance
Manhattan distance

Stopping threshold - Here also user has two choices

Algorithm can be stopped at a constant number of iterations
Or algorithm is stopped when we get the same set of centroids repeatedly. Here to compare the two centroids a comparison error can be specified

At the end of clustering we calculated the PPV value of the clusters. Also we did a 10 fold cross validation of the data as well.

The source code can be found in github and the url is https://github.com/supunkamburugamuva/Kmeans. The source is distributed under Apache License Version 2 and you are welcome to contribute and use the code without any restrictions.

Is the innovation in movie industry diminishing?

2011-09-25T23:47:00.001-04:00

Here are the top 10 box office movies of the year 2011 so far.

Rank	Title	Studio	Worldwide	US and Canada	UK	Australia
1	Harry Potter and the Deathly Hallows – Part 2	Warner Bros.	$1,325,066,054	$378,366,054	$114,820,753	$54,319,291
2	Transformers: Dark of the Moon	Paramount Pictures	$1,117,601,953	$351,458,166	$45,470,012	$38,820,321
3	Pirates of the Caribbean: On Stranger Tides	Walt Disney Pictures	$1,039,529,902	$241,011,325	$54,187,325	$29,006,640
4	Kung Fu Panda 2	DreamWorks Animation	$659,146,003	$165,146,003	$27,062,099	$20,219,645
5	Fast Five	Universal Pictures	$609,484,033	$209,837,675	$30,243,825	$26,794,607
6	The Hangover Part II	Warner Bros.	$581,464,305	$254,464,305	$53,496,869	$35,371,018
7	Cars 2	Walt Disney Pictures / Pixar	$550,079,088	$189,679,088	$24,894,721	$20,811,602
8	Rio	20th Century Fox	$483,886,518	$143,619,809	$21,536,048	$17,230,382
9	The Smurfs	Columbia Pictures	$483,360,857	$137,759,100	$24,454,426	$3,634,929
10	Thor	Paramount Pictures	$448,512,824	$181,030,624	$22,521,717	$20,135,378

Out of the 10, first seven movies are attempts to create the same glory as they did with the very successful first movie of the sequel. But the ratings of all these seven movies are pretty low and their success is mainly due to the fact that they are part of sequel with a very successful first movie.

Here are the ratings given to these movies by yahoo users. The critics ratings are worst than that.

1. Harry potter   A-
2. Transformers B
3. Pirates           B
4. Panda            A-
5. Fast Five        A-
6. Hangover       B-
7. Cars 2            B+

Some of the movies like Transformers and Pirates are very lame attempts to create a movie out of thin air and they are more like putting together some random scenes.

So what does this mean? Aren't there any ideas left that can be turned in to a great movie? Or we have seen so many movies that any new idea is just a new way of looking at an old idea.

After a long time

2011-09-09T00:12:00.001-04:00

It is being a long time since I've posted my last blog entry. It was a busy time with a lot happening. To make it short I've started my higher studies in Indiana University Bloomington as a Compute Science Graduate student under Prof. Geoffrey Fox. I did resign from WSO2. I had a great time at WSO2 and would like to thank all the friends and specially Dr. Sanjiva Weerawarana for the great time I had in WSO2.

Approaching the Tipping point of Web Services

2010-09-23T12:05:00.001-04:00

Every technology that is meant to last has similar adoption patterns. Here is a graph describing the time vs users.

At the initial stages only the early adopters dare to use a fresh technology. Early adopters has advantages as well as disadvantages.

Biggest disadvantage or risk comes for an early adopter because the implementations of the technology are not matured. A product implementing a fresh technology may seize to work in a critical moment. But early adopters do get lots of advantages as well. They get the best support and special attention from the vendors as well as they get to influence the implementation of the technology. Also early adopters has the competitive advantage over their competitors because of the adoption of advanced technologies. In these stages the usage of the products is a random and there is no increase of decrease to be seen in terms of users.

After the early adopters the technology is open to more general users. In these stages the usage of the technology or the products slowly increases. This is where general users start to think about adopting the new technologies. But majority of the users still not comfortable to use the new products and they are waiting for their confidence to build up to the right level.

This means during this adoption period where gradual increase of usage happens there is a very large group of people building up, who are hoping to use the products but the attributes are not at the correct level for them. These are the normal users who uses a technology for every day usages. They just wait until the correct moment.

When we reach the state where normal people start using the technology we can see a sudden dramatic increase in a technology because of this large group of people who were waiting. This is what we call a “Tipping Point”.

Web Services are now in the adoption stage where there is a gradual smooth increase. Web services was founded around six seven years ago. But the adoption was slow due to its complexity. Although it is complex web services provides the only solution agreed across all the key player for a very real problem. Problem is to connect the various computing resources. So web services as a technology is bound to be used by the majority.

Right now it is used mostly by the high tech companies with a long vision and a pile of money to spend. But I believe in a few years time we will reach the tipping point of web services. It will at least take another year and a half to reach that point. But we will hit that tipping point for sure.

Reaching the tipping point will be great from the technological point of view. But it will bring greater challenges to the companies creating web services middle ware as well. The biggest challenge will be the diverse use of web services. We will see people using it for all kinds of crazy things. Best way to succeed in this kind of a wide customer base is to have technologically sound products. Companies with the most technically sound products will prosper and others will gradually fade away.

Refactoring Java Singletons to regular classes

2010-09-15T02:15:00.001-04:00

One of the really bad things about a Java singleton is when you use it, you never think about the correct place to create it. Even if a class is implemented as a singleton there is always a logical place for a singleton. This is really important in getting your design right. But with singletons we tend to forget about this and design the things incorrectly.

When the time comes to get rid of the singletons it becomes very hard due to the design constraints.

Difference between service chaining and callouts in WSO2 ESB

2010-06-07T10:43:00.001-04:00

In a typical integration there can be a single service exposed supported by two implementation services.

Client ---------> ESB ------> Service 1
------> Service 2
Scenario 1:
When the request comes to the ESB it should be validated by sending to the Service 1. If the request is valid it should be sent to service 2

Scenario 2:
Request only has few information. In order to access the service 2, there is some information that need to be retrieved from service 1. After retrieving this information request must be transformed. In this case before sending the request to service 2, request must be enhanced with information from service 1.

Obviously Scenario 1 fits Service chaining because it is chained invocation. You invoke service 1 then invoke service 2. This kind of invocation should be done using the send mediator instead of callout mediator of WSO2 ESB.

Scenario 2 is bit different. Before proceeding with the request message we need to enhance it with information from service 1. So by looking at the problem we can see it requires a blocking call to the service 1 to get some information. This kind of invocation can be done using the send mediator. But my belief is it fits more with the callout mediator which does blocking calls.

For more info please refer the configuration langauge guide of WSO2 ESB.

Worshiped Sri Pada (Adam's Peak)

2010-03-29T08:41:00.001-04:00

We went to Sri Pada during the last week end (27th & 28th March). It was a family trip with my family and my mother's sister's family.

We went through the Hatton Road. The most beautiful road to Sri Pada is through Ratnapura Kuruwita. But it is the hardest as well. We didn't take that road because it could be very hard for my mother and aunt.

We started climbing around 8 p.m. Then it was raining very hard. We climbed very slowly. We often climbed 50 steps and took a rest for 15 mins.

Week end was very busy for Sri Pada because people were pouring in to visit this great mountain.

When we were at Mahagiridambe it was really crowded and there was a long line to the peak. It was a very hard journey through Mahagiridambe because we had to stand waiting in a line wasn't moving very fast.

We went to the peak around 5 p.m in the morning. We worshiped the Sri Pada pathula and stayed in the peak for around 30 mins. It wasn't cold as I thought it would be on the peak.

Then we started descending. Unfortunately the Eastern Sky was full with rainy clouds. We coudn't see the Ira Savaya.

Descending was harder than climbing. But we managed it around 11 a.m in the morning. After that we took a little break and came straight home.

Few friends and myself were lucky enough to clime Sri Pada in the off season last year. I must say climbing Sri Pada in off-season was ten times beautiful, peaceful and adventurous than the journey we took this year in the regular Sri Pada season.

WSO2 ESB Reffering to Resources over a HTTP proxy connection

2010-03-19T04:43:00.001-04:00

With the 3.0.0 upcoming release WSO2 ESB can retrieve resources through a HTTP proxy server. Usually these resources are WSDLs.

A WSDL may be in a remote server and WSO2 ESB can refer that. But in order to reach the WSDL ESB may have to go through a Proxy Server.

To configure the proxy settings user has to update the synapse.properties file in the webapps/ROOT/WEB-INF/classes directory. Here is the configuration for enabling the proxy.

# address of the proxy server
#synapse.http.proxy.host=
# Proxy server port
#synapse.http.proxy.port=
# Proxy server user name, this is used for HTTP basic authentication
#synapse.http.proxy.user=
# Proxy server user password, this is used for HTTP basic authentication
#synapse.http.proxy.password=
# list of host address excluded from going through the proxy, list is comma separated
#synapse.http.proxy.excluded.hosts=localhost, 127.0.0.1

All the configurations are self explanatory. If user name and password is set ESB will use HTTP basic authentication.

HTTP Basic Authentication over a proxy server with WSO2 ESB

2010-03-19T01:42:00.001-04:00

ESB may be behind a firewall and it may want to talk to a server through a proxy server. A proxy server may require HTTP basic authentication. Here is the scenario I'm talking about. This full scenario is only available with upcoming WSO2 ESB 3.0.0 release.

Client ----> ESB ----> HTTP Proxy Server -----> Service Provider

There are several configurations that needs to be done in-order to configure ESB to handle this.

First you need to tell ESB that is is behind a Proxy Server. To do this edit the axis2.xml file in the repository/conf directory.

You need to edit the transportSender configuration of the http transport in axis2.xml. The configurations are self explanatory.

<transportSender name="http" class="org.apache.synapse.transport.nhttp.HttpCoreNIOSender">
        <parameter name="non-blocking" locked="false">true</parameter>
        <parameter name="http.proxyHost" locked="false">localhost</parameter>
        <parameter name="http.proxyPort" locked="false">8080</parameter>
</transportSender>

Then you need to set the following two properties in the synapse configuration before doing the send through the proxy server.

<syn:property name="Proxy-Authorization" expression="fn:concat('Basic ', base64Encode('userName:password'))" scope="transport"/>
<syn:property name="POST_TO_URI" value="true" scope="axis2"/>

The first property set the Proxy-Authorization HTTP transport header with the base64 encoded user name and password as expected by the HTTP basic authentication.

Second property make the out-going url a complete URL understandable by the proxy server.

For example in normal scenarios we send the outgoing request as following
POST /services/SimpleStockQuoteService HTTP/1.1

But for proxy server we need to send the actual URL:
POST http://localhost:9000/services/SimpleStockQuoteService HTTP/1.1

Here is a Simple sample Proxy service with these two properties

<syn:proxy name="StockQuoteProxy" startOnLoad="true">
    <syn:target>
        <syn:inSequence>
            <syn:property name="Proxy-Authorization" expression="fn:concat('Basic ', base64Encode('supun:test123'))" scope="transport"/>
            <syn:property name="POST_TO_URI" value="true" scope="axis2"/>
            <syn:send>
                <syn:endpoint name="endpoint_urn_uuid_379C485AD3CB65037F10216600509076498395882">
                    <syn:address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
                </syn:endpoint>
            </syn:send>
        </syn:inSequence>
        <syn:outSequence>
            <syn:send/>
        </syn:outSequence>
    </syn:target>
    <syn:publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
</syn:proxy>

My thoughts on WSO2 Carbon Registry Seperation

2010-03-07T09:36:00.001-05:00

For the upcoming Carbon release (3.0.0) we are going to introduce a new registry separation model for all the carbon based applications. I feel there is something fundamentally wrong with this registry separation. But still I'm not clear why I got that feeling. So I thought of writing down my thoughts. This is not a structured post. I'm just writing down my thoughts.

Under the new Registry model there are three registries.

1. Local registry
2. Configuration registry
3. Governance registry

First lets look at the problem we are trying to solve using these three registries.

Every program needs configurations to run. These configurations can be coming from a file or can be coming from a UI. Also every program has a state. When we change some configurations or deploy a new component or enable something program state can be changed. A program usually saves its state transparently to the user. For example when we run a program like IntelliJ Idea or ecclipse we can specify some configurations like temp directory, using a configuration file . Some configurations like editing options are saved interanally by these applications and users doesn't know the format of the stored configuration information.

There is another type of information that a program require. They are application specific configurations. For example an IntelliJ Idea project may be using a JSP Tag Library. IntelliJ Idea has to know the location of this tag library in order to property render the JSP tags. An ESB may require a XSLT file for message transformation. This file has to be stored somewhere accessible from the ESB.

With a tool like intellij Idea configuration storage is always the file system. But with carbon configuraion storage is the WSO2 Governance Registry.

Carbon is another program that has all sorts of configurations. Also a carbon program can run in multiple topologies like Standalone mode with a remote registry. Clustered mode with a remote registry etc. From the carbon perspective it shouldn't matter weather it is running against a remote registry or a embedded registry. Standalone mode and clustering mode are very important.

There is a very very important concept when it comes to storage requirement of a program. There are only two types of storage requirements to a program.

There is state information internal to a program that shouldn't be exposed to the user as it is.
There are configurations specified by the user

If we agree to the above fundamental principle there can be two configuration areas to a program.

Internal storage space used to store the state information
Storage space to put user specified configurations

But Carbon introduces three configuration areas. But I think there should be only two configuration areas. A local registry that can be used to store internal stuff to carbon that shouldn't be exposed to the user and a general registry that can contain all sorts of user configurations.

Different components behave differently in different topologies. One of the ideas behind carbon is when a application runs in standalone mode it should be able to run in clustering mode without any changes. That is one of the fundamental reasons behind registry separation. But clustering and standalone modes are vary different and trying to design a application to run a standalone app in cluster mode can be a bad design decision.

Also carbon wants to start two clusters and share information among the two clusters using the registry. That is why we have the governance registry. But this is a feature that we could have simply achieve at the carbon level without over complicating the registry model.

For me registry is just a storage. It is like a file system or a database. A database provide a generic programming model so does the registry. It is the application level programmers responsibility to use the database properly. Also it is the carbon's responsibility to use the registry correctly to suite its needs. Changing the registry model to suite carbon's need is not a good idea.

I may be completely wrong. Every technical decision has its pros and cons.

Amazon Prime, Amazing service

2010-03-06T23:01:00.001-05:00

I've subscribed to Amazon prime shipping. It is free for the first month. I did not want to subscribe to it any longer than the first month because I'm not living in USA. But I forgot to cancel the prime membership after the first month. So Amazon continued with my prime membership and charged my credit card for the first year membership.

Amazing thing is I could cancel the prime membership with the total refund. The only condition is I shouldn't have bought anything after I pay for the prime membership. Isn't it a wonderful service?

I just did the cancellation and waiting for Amazon refund. Hope it will arrive as they said.

Writing a Load Balance algorithm for WSO2 ESB

2010-02-28T12:49:00.000-05:00

Load balance algorithms are very interesting. They need to be fast and efficient. Otherwise we will spend most of our time trying to find the next route rather than doing the actual message sending.

Also load balancing algorithms involve lots of synchronization operations as well. So they should be coded with care.

The load balance algorithm I was interested in should have the benefits of Round Robin, Weighted and Least connection based algorithms. So it is a combination of all those three algorithms.

Round Robin is a pretty powerful algorithm mainly due to its simplicity. One assumption it made is all the servers are identical and has the exact same capacity. But in practice this may not be the case always.

Static Weighted algorithms assume machines have different capacities and based on those they are configured statically. Combined with round robin algorithms these can be pretty useful in some scenarios. But both these algorithms neglects the run-time behavior of servers.

Some times due to various reasons these servers can get slow or un-responsive. If we don't take these situations in to account, a Load balancer can effectively kill some of the already slow servers. This happens because the load balancer pumps the messages at the same rate even if the server gets slow.

Least connection can be used to some what overcome this problem. Least connection algorithms take the current number of connections in to account. More active connections than usual indicates a server saturation. At this point if the load balancer can reduce the number of messages send to this server, it can save the server from being killed.

So my goal was to write an algorithm that take all these three factors. I wrote the algorithm but still couldn't do enough testing. Also there are improvements in core ESB as well as the algorithm for this to work.

More on these later...