Life of a Software Engineer

Is the innovation in movie industry diminishing?

2011-09-25T20:47:00.001-07:00

Here are the top 10 box office movies of the year 2011 so far.

Rank	Title	Studio	Worldwide	US and Canada	UK	Australia
1	Harry Potter and the Deathly Hallows – Part 2	Warner Bros.	$1,325,066,054	$378,366,054	$114,820,753	$54,319,291
2	Transformers: Dark of the Moon	Paramount Pictures	$1,117,601,953	$351,458,166	$45,470,012	$38,820,321
3	Pirates of the Caribbean: On Stranger Tides	Walt Disney Pictures	$1,039,529,902	$241,011,325	$54,187,325	$29,006,640
4	Kung Fu Panda 2	DreamWorks Animation	$659,146,003	$165,146,003	$27,062,099	$20,219,645
5	Fast Five	Universal Pictures	$609,484,033	$209,837,675	$30,243,825	$26,794,607
6	The Hangover Part II	Warner Bros.	$581,464,305	$254,464,305	$53,496,869	$35,371,018
7	Cars 2	Walt Disney Pictures / Pixar	$550,079,088	$189,679,088	$24,894,721	$20,811,602
8	Rio	20th Century Fox	$483,886,518	$143,619,809	$21,536,048	$17,230,382
9	The Smurfs	Columbia Pictures	$483,360,857	$137,759,100	$24,454,426	$3,634,929
10	Thor	Paramount Pictures	$448,512,824	$181,030,624	$22,521,717	$20,135,378

Out of the 10, first seven movies are attempts to create the same glory as they did with the very successful first movie of the sequel. But the ratings of all these seven movies are pretty low and their success is mainly due to the fact that they are part of sequel with a very successful first movie.

Here are the ratings given to these movies by yahoo users. The critics ratings are worst than that.

1. Harry potter   A-
2. Transformers B
3. Pirates           B
4. Panda            A-
5. Fast Five        A-
6. Hangover       B-
7. Cars 2            B+

Some of the movies like Transformers and Pirates are very lame attempts to create a movie out of thin air and they are more like putting together some random scenes.

So what does this mean? Aren't there any ideas left that can be turned in to a great movie? Or we have seen so many movies that any new idea is just a new way of looking at an old idea.

After a long time

2011-09-08T21:12:00.001-07:00

It is being a long time since I've posted my last blog entry. It was a busy time with a lot happening. To make it short I've started my higher studies in Indiana University Bloomington as a Compute Science Graduate student under Prof. Geoffrey Fox. I did resign from WSO2. I had a great time at WSO2 and would like to thank all the friends and specially Dr. Sanjiva Weerawarana for the great time I had in WSO2.

Approaching the Tipping point of Web Services

2010-09-23T09:05:00.001-07:00

Every technology that is meant to last has similar adoption patterns. Here is a graph describing the time vs users.

At the initial stages only the early adopters dare to use a fresh technology. Early adopters has advantages as well as disadvantages.

Biggest disadvantage or risk comes for an early adopter because the implementations of the technology are not matured. A product implementing a fresh technology may seize to work in a critical moment. But early adopters do get lots of advantages as well. They get the best support and special attention from the vendors as well as they get to influence the implementation of the technology. Also early adopters has the competitive advantage over their competitors because of the adoption of advanced technologies. In these stages the usage of the products is a random and there is no increase of decrease to be seen in terms of users.

After the early adopters the technology is open to more general users. In these stages the usage of the technology or the products slowly increases. This is where general users start to think about adopting the new technologies. But majority of the users still not comfortable to use the new products and they are waiting for their confidence to build up to the right level.

This means during this adoption period where gradual increase of usage happens there is a very large group of people building up, who are hoping to use the products but the attributes are not at the correct level for them. These are the normal users who uses a technology for every day usages. They just wait until the correct moment.

When we reach the state where normal people start using the technology we can see a sudden dramatic increase in a technology because of this large group of people who were waiting. This is what we call a “Tipping Point”.

Web Services are now in the adoption stage where there is a gradual smooth increase. Web services was founded around six seven years ago. But the adoption was slow due to its complexity. Although it is complex web services provides the only solution agreed across all the key player for a very real problem. Problem is to connect the various computing resources. So web services as a technology is bound to be used by the majority.

Right now it is used mostly by the high tech companies with a long vision and a pile of money to spend. But I believe in a few years time we will reach the tipping point of web services. It will at least take another year and a half to reach that point. But we will hit that tipping point for sure.

Reaching the tipping point will be great from the technological point of view. But it will bring greater challenges to the companies creating web services middle ware as well. The biggest challenge will be the diverse use of web services. We will see people using it for all kinds of crazy things. Best way to succeed in this kind of a wide customer base is to have technologically sound products. Companies with the most technically sound products will prosper and others will gradually fade away.

Refactoring Java Singletons to regular classes

2010-09-14T23:15:00.001-07:00

One of the really bad things about a Java singleton is when you use it, you never think about the correct place to create it. Even if a class is implemented as a singleton there is always a logical place for a singleton. This is really important in getting your design right. But with singletons we tend to forget about this and design the things incorrectly.

When the time comes to get rid of the singletons it becomes very hard due to the design constraints.

Difference between service chaining and callouts in WSO2 ESB

2010-06-07T07:43:00.001-07:00

In a typical integration there can be a single service exposed supported by two implementation services.

Client ---------> ESB ------> Service 1
------> Service 2
Scenario 1:
When the request comes to the ESB it should be validated by sending to the Service 1. If the request is valid it should be sent to service 2

Scenario 2:
Request only has few information. In order to access the service 2, there is some information that need to be retrieved from service 1. After retrieving this information request must be transformed. In this case before sending the request to service 2, request must be enhanced with information from service 1.

Obviously Scenario 1 fits Service chaining because it is chained invocation. You invoke service 1 then invoke service 2. This kind of invocation should be done using the send mediator instead of callout mediator of WSO2 ESB.

Scenario 2 is bit different. Before proceeding with the request message we need to enhance it with information from service 1. So by looking at the problem we can see it requires a blocking call to the service 1 to get some information. This kind of invocation can be done using the send mediator. But my belief is it fits more with the callout mediator which does blocking calls.

For more info please refer the configuration langauge guide of WSO2 ESB.

Worshiped Sri Pada (Adam's Peak)

2010-03-29T05:41:00.001-07:00

We went to Sri Pada during the last week end (27th & 28th March). It was a family trip with my family and my mother's sister's family.

We went through the Hatton Road. The most beautiful road to Sri Pada is through Ratnapura Kuruwita. But it is the hardest as well. We didn't take that road because it could be very hard for my mother and aunt.

We started climbing around 8 p.m. Then it was raining very hard. We climbed very slowly. We often climbed 50 steps and took a rest for 15 mins.

Week end was very busy for Sri Pada because people were pouring in to visit this great mountain.

When we were at Mahagiridambe it was really crowded and there was a long line to the peak. It was a very hard journey through Mahagiridambe because we had to stand waiting in a line wasn't moving very fast.

We went to the peak around 5 p.m in the morning. We worshiped the Sri Pada pathula and stayed in the peak for around 30 mins. It wasn't cold as I thought it would be on the peak.

Then we started descending. Unfortunately the Eastern Sky was full with rainy clouds. We coudn't see the Ira Savaya.

Descending was harder than climbing. But we managed it around 11 a.m in the morning. After that we took a little break and came straight home.

Few friends and myself were lucky enough to clime Sri Pada in the off season last year. I must say climbing Sri Pada in off-season was ten times beautiful, peaceful and adventurous than the journey we took this year in the regular Sri Pada season.

WSO2 ESB Reffering to Resources over a HTTP proxy connection

2010-03-19T01:43:00.001-07:00

With the 3.0.0 upcoming release WSO2 ESB can retrieve resources through a HTTP proxy server. Usually these resources are WSDLs.

A WSDL may be in a remote server and WSO2 ESB can refer that. But in order to reach the WSDL ESB may have to go through a Proxy Server.

To configure the proxy settings user has to update the synapse.properties file in the webapps/ROOT/WEB-INF/classes directory. Here is the configuration for enabling the proxy.

# address of the proxy server
#synapse.http.proxy.host=
# Proxy server port
#synapse.http.proxy.port=
# Proxy server user name, this is used for HTTP basic authentication
#synapse.http.proxy.user=
# Proxy server user password, this is used for HTTP basic authentication
#synapse.http.proxy.password=
# list of host address excluded from going through the proxy, list is comma separated
#synapse.http.proxy.excluded.hosts=localhost, 127.0.0.1

All the configurations are self explanatory. If user name and password is set ESB will use HTTP basic authentication.

HTTP Basic Authentication over a proxy server with WSO2 ESB

2010-03-18T22:42:00.001-07:00

ESB may be behind a firewall and it may want to talk to a server through a proxy server. A proxy server may require HTTP basic authentication. Here is the scenario I'm talking about. This full scenario is only available with upcoming WSO2 ESB 3.0.0 release.

Client ----> ESB ----> HTTP Proxy Server -----> Service Provider

There are several configurations that needs to be done in-order to configure ESB to handle this.

First you need to tell ESB that is is behind a Proxy Server. To do this edit the axis2.xml file in the repository/conf directory.

You need to edit the transportSender configuration of the http transport in axis2.xml. The configurations are self explanatory.

<transportSender name="http" class="org.apache.synapse.transport.nhttp.HttpCoreNIOSender">
        <parameter name="non-blocking" locked="false">true</parameter>
        <parameter name="http.proxyHost" locked="false">localhost</parameter>
        <parameter name="http.proxyPort" locked="false">8080</parameter>
</transportSender>

Then you need to set the following two properties in the synapse configuration before doing the send through the proxy server.

<syn:property name="Proxy-Authorization" expression="fn:concat('Basic ', base64Encode('userName:password'))" scope="transport"/>
<syn:property name="POST_TO_URI" value="true" scope="axis2"/>

The first property set the Proxy-Authorization HTTP transport header with the base64 encoded user name and password as expected by the HTTP basic authentication.

Second property make the out-going url a complete URL understandable by the proxy server.

For example in normal scenarios we send the outgoing request as following
POST /services/SimpleStockQuoteService HTTP/1.1

But for proxy server we need to send the actual URL:
POST http://localhost:9000/services/SimpleStockQuoteService HTTP/1.1

Here is a Simple sample Proxy service with these two properties

<syn:proxy name="StockQuoteProxy" startOnLoad="true">
    <syn:target>
        <syn:inSequence>
            <syn:property name="Proxy-Authorization" expression="fn:concat('Basic ', base64Encode('supun:test123'))" scope="transport"/>
            <syn:property name="POST_TO_URI" value="true" scope="axis2"/>
            <syn:send>
                <syn:endpoint name="endpoint_urn_uuid_379C485AD3CB65037F10216600509076498395882">
                    <syn:address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
                </syn:endpoint>
            </syn:send>
        </syn:inSequence>
        <syn:outSequence>
            <syn:send/>
        </syn:outSequence>
    </syn:target>
    <syn:publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
</syn:proxy>

My thoughts on WSO2 Carbon Registry Seperation

2010-03-07T06:36:00.001-08:00

For the upcoming Carbon release (3.0.0) we are going to introduce a new registry separation model for all the carbon based applications. I feel there is something fundamentally wrong with this registry separation. But still I'm not clear why I got that feeling. So I thought of writing down my thoughts. This is not a structured post. I'm just writing down my thoughts.

Under the new Registry model there are three registries.

1. Local registry
2. Configuration registry
3. Governance registry

First lets look at the problem we are trying to solve using these three registries.

Every program needs configurations to run. These configurations can be coming from a file or can be coming from a UI. Also every program has a state. When we change some configurations or deploy a new component or enable something program state can be changed. A program usually saves its state transparently to the user. For example when we run a program like IntelliJ Idea or ecclipse we can specify some configurations like temp directory, using a configuration file . Some configurations like editing options are saved interanally by these applications and users doesn't know the format of the stored configuration information.

There is another type of information that a program require. They are application specific configurations. For example an IntelliJ Idea project may be using a JSP Tag Library. IntelliJ Idea has to know the location of this tag library in order to property render the JSP tags. An ESB may require a XSLT file for message transformation. This file has to be stored somewhere accessible from the ESB.

With a tool like intellij Idea configuration storage is always the file system. But with carbon configuraion storage is the WSO2 Governance Registry.

Carbon is another program that has all sorts of configurations. Also a carbon program can run in multiple topologies like Standalone mode with a remote registry. Clustered mode with a remote registry etc. From the carbon perspective it shouldn't matter weather it is running against a remote registry or a embedded registry. Standalone mode and clustering mode are very important.

There is a very very important concept when it comes to storage requirement of a program. There are only two types of storage requirements to a program.

There is state information internal to a program that shouldn't be exposed to the user as it is.
There are configurations specified by the user

If we agree to the above fundamental principle there can be two configuration areas to a program.

Internal storage space used to store the state information
Storage space to put user specified configurations

But Carbon introduces three configuration areas. But I think there should be only two configuration areas. A local registry that can be used to store internal stuff to carbon that shouldn't be exposed to the user and a general registry that can contain all sorts of user configurations.

Different components behave differently in different topologies. One of the ideas behind carbon is when a application runs in standalone mode it should be able to run in clustering mode without any changes. That is one of the fundamental reasons behind registry separation. But clustering and standalone modes are vary different and trying to design a application to run a standalone app in cluster mode can be a bad design decision.

Also carbon wants to start two clusters and share information among the two clusters using the registry. That is why we have the governance registry. But this is a feature that we could have simply achieve at the carbon level without over complicating the registry model.

For me registry is just a storage. It is like a file system or a database. A database provide a generic programming model so does the registry. It is the application level programmers responsibility to use the database properly. Also it is the carbon's responsibility to use the registry correctly to suite its needs. Changing the registry model to suite carbon's need is not a good idea.

I may be completely wrong. Every technical decision has its pros and cons.

Amazon Prime, Amazing service

2010-03-06T20:01:00.001-08:00

I've subscribed to Amazon prime shipping. It is free for the first month. I did not want to subscribe to it any longer than the first month because I'm not living in USA. But I forgot to cancel the prime membership after the first month. So Amazon continued with my prime membership and charged my credit card for the first year membership.

Amazing thing is I could cancel the prime membership with the total refund. The only condition is I shouldn't have bought anything after I pay for the prime membership. Isn't it a wonderful service?

I just did the cancellation and waiting for Amazon refund. Hope it will arrive as they said.

Writing a Load Balance algorithm for WSO2 ESB

2010-02-28T09:49:00.000-08:00

Load balance algorithms are very interesting. They need to be fast and efficient. Otherwise we will spend most of our time trying to find the next route rather than doing the actual message sending.

Also load balancing algorithms involve lots of synchronization operations as well. So they should be coded with care.

The load balance algorithm I was interested in should have the benefits of Round Robin, Weighted and Least connection based algorithms. So it is a combination of all those three algorithms.

Round Robin is a pretty powerful algorithm mainly due to its simplicity. One assumption it made is all the servers are identical and has the exact same capacity. But in practice this may not be the case always.

Static Weighted algorithms assume machines have different capacities and based on those they are configured statically. Combined with round robin algorithms these can be pretty useful in some scenarios. But both these algorithms neglects the run-time behavior of servers.

Some times due to various reasons these servers can get slow or un-responsive. If we don't take these situations in to account, a Load balancer can effectively kill some of the already slow servers. This happens because the load balancer pumps the messages at the same rate even if the server gets slow.

Least connection can be used to some what overcome this problem. Least connection algorithms take the current number of connections in to account. More active connections than usual indicates a server saturation. At this point if the load balancer can reduce the number of messages send to this server, it can save the server from being killed.

So my goal was to write an algorithm that take all these three factors. I wrote the algorithm but still couldn't do enough testing. Also there are improvements in core ESB as well as the algorithm for this to work.

More on these later...

Priority Based Mediation for WSO2 ESB - Part 2

2010-02-23T01:53:00.000-08:00

In the previous post I was talking about the requirements for priority based mediation. Now I'll explain how we have solved this within the WSO2 ESB.

We have implemented the priority based mediation in two levels. They are HTTP transport level and message mediation level. From the users perspective, key to any priority mediation is to determine the priority of an incoming message.

At the Message mediation layer this can be done using content filters. Tis means the full power of ESB configuration language is available to the user for determining the priority of a given message. For example a message may contain an element called priority and depending on its value the priority can be determined.

At the HTTP layer, user has access to HTTP headers, HTTP parameters and URL values. By looking at these values, user can determine the priority of a given message. We have come up with a configuration language to determine the priority at the HTTP level using these different HTTP param values.

Now lets go in to the actual details of how priority based mediation is implemented. The approach is based on Queues and ThreadPoolExecutors.

We are using the ThreadPoolExecutor provide by Java. ThreadPoolExecutor accepts a BlockingQueue implementation. We have implemented a custom blocking queue that can be used to order the jobs based on priority. ThreadPoolExecutor starts queuing only when the all the core threads are busy. So this suites our model perfectly. We believe, every message should get equal priority until all the core threads are used. This is a fair assumption in most cases.

Internally our custom BlockingQueue uses multiple queues for accepting jobs with different priorities. Once jobs are put in to the queue, it uses an pluggable algorithm for choosing the next job. The default algorithm chooses the jobs based on a priority based round robin algorithm. For example lets say we have two priorities, 10 and 1. This algorithm tries to fetch 10 items with priority 10 and then 1 item with the priority 1.

We have put the priority implementation in two levels due to following reasons. Some users would like to use the ESB as a pure router. In those cases it is better to determine the priority at the HTTP level.

Some users use ESB for heavy processing like XSLT and XQuery. For them it is better to have this at the mediation level.

I'll explain the configuration model in the next post.

Priority Based Mediation for WSO2 ESB

2010-02-14T05:59:00.000-08:00

An ESB can be deployed as a front end for various services with a very high load passing through it. The traffic passing through it may be of different priorities and can have different load charasterisitcs. Also in some situations load of different types of messages can vary dramatically over time.

The goal is to serve high priority traffic when there are resources available in the system. The serving of different priority messages should be independent as much as possible. For example an ESB can be bombarded with very high volume of low priority traffic. Even in this case ESB should obey its contract to serve high priority traffic.

Usually a person deploying an ESB expects a certain guarantee of message deliverance from the ESB. ESB should be able to honor that.

Here is a possible scenario with a high volume of traffic.

Lets assume ESB is configured to serve two types of messages with different priorities. Message type m1 has the priority p1 and message type m2 has priority p2. p1 is about ten times the priority of p2. In a very high volume traffic scenario the services can afford to loose some amount of m2 messages. But it cannot afford to loose any of m1 messages.

In normal operation mode, volume of m2 is about twice as m1. But in some situations m1 can be higher and m2 can be low.

The expectation is to serve the m1 traffic no matter what the volume of m2 traffic is.

The problem is very interesting. I'll explain how we have solved this problem with next major release of WSO2 ESB (3.0) in the next post.

Books

2009-12-07T14:50:00.000-08:00

Recently I got the chance to read The Mythical Man Month by Fred Brooks Code Complete Second edition by Steve McConnell. I would recommend these two books to anyone interested software engineering. They are truly excellent books.

WSO2 ESB error handling in Endpoints

2009-11-28T11:25:00.001-08:00

I’ve written an article, WSO2 Enterprise Service Bus - Endpoint Error Handling, to the WSO2 Oxygen Tank. It is written aiming the advanced users who are already familiar with WSO2 ESB and want to do real deployments. It covers lot of details in depth. So a beginner can quickly grasp some of the internal workings by reading the article as well.

A Rock Solid Release of WSO2 ESB 2.1.1

2009-10-13T11:34:00.001-07:00

We did a very stable release of WSO2 ESB 2.1 some time ago. Now we have a more robust version of it available. This version has many bug fixes for app server deployment. Also quite a number of general bug fixes. We have changed the registry to support transactions. All the ESB features are very stable and ready to go in to production.

You can grab the latest release from

http://wso2.org/downloads/esb

Java Double checked locking – don’t use it!!

2009-08-31T01:23:00.001-07:00

First lets look at what is double checked locking. Double checked locking has 4 steps

Check a variable
Synchronize
Check the variable again
Change the variable

Here is a scenario where this can be used theoretically. There is a object shared by multiple threads. Creation of this object happens at the runtime. Multiple threads can see the need for creating this object. Even if multiple threads want to create the object we only want a single thread to create the object.

Here is a simple demonstration.

public class Test {
    // this is the shared variable accessed by many threads
    private MyObject myObject = null;

    /**
     * This method is invoked by many threads.
     */
    public void doWork() {
        if (myObject == null) {
            synchronized (this) {
                if (myObject == null) {
                    myObject = new MyObject("Supun", 24);
                }
            }
        }
        // do our work
    }
}

class MyObject {
    private String name;    
    private int age;
    MyObject(String name, int age) {
        this.name = name;
        this.age = age;
    }
}

You might think this will work. Theoretically it should work. But in practice it won’t work. Specially in production servers with multiple CPU’s and shared memory there is a high chance for this to fail.

Let me explain why it fails. Here is a demonstration of how java may be implementing the call to new. This order can be different in different compilers. But there is no guarantee.

    public void doWork() {
        if (myObject == null) {
            synchronized (this) {
                if (myObject == null) {
                    // here is how the java code may be actually executed by the JVM
                    myObject = allocateMemory;
                    myObject_constructor();
                }
            }
        }
        // do our work
    }

It first allocates memory and assign it to the reference. At this point myObject is not null. But the myObject is not fully initialized yet. At this point a thread can come to the first null check and see it is not null and can try to access the object fields. This will produce unexpected behavior.

You can solve this problem by using volatile keyword for myObject reference. But volatile is not guaranteed to work across JVMs and the most of the implementations of volatile are slower than synchronization.

So avoid using double checked locking in Java.

Fancy cars, Beautiful houses, Bank loans & SRI LANKA

2009-08-19T22:55:00.001-07:00

Everybody loves to have a fancy car and a peaceful beautiful place to live in. This is the dream of Sri Lankans. They all want to by a car. They all want to build the biggest and the most beautiful house possible. All this is fine until they do it with their own money. But they don’t. They do it by getting the biggest loan from some bank.

Still Sri Lanka is a developing country. Every year it goes to World Bank or IMF for getting loans for running the country. So as Sri Lankans we are in a great debt to the world. Actually not to the whole world but to few powerful countries like USA, Japan etc. So obviously a big portion of every loan a Sri Lankan get from a local bank comes from a loan Sri Lanka took from one of these global organizations.

But the problem lies with the way we spend this money. We spend it for building a house that has no capability of generating revenue. It is like every time we build a house we are burying some of the money we took from some other country. Also to build these houses we get raw material from the same countries that we took the loan itself. So we are doubling our debts.

Situation is worse with Cars. From the car itself to spares parts to fuel we are importing from other countries. So we get loans from other countries, we buy cars from these same countries from the loans we got and then we get fuel from these countries as well. So we are entitled to an eternal debt.

There is another sad side to these loans. Most of these loans are taken by people who don’t do work worth of what they get as their salary. This happens mostly in government sector. Still government sector plays a big role in Sri Lanka. 90 percent of the government workers don’t do work worth of what they get. So ultimately some poor guy has to pay dearly for what these people do.

WSO2 ESB 2.1 Places for putting custom mediators

2009-08-18T23:15:00.001-07:00

There are three places for putting a mediator. They are

<esb_carbon_home>\repository\components\mediators
<esb_carbon_home>\repository\components\dropins
<esb_carbon_home>\repository\components\lib

mediators directory

User can put a regular non OSGI mediator jar in to this directory. The system will convert the mediator jar in to an OSGI jar and deploy it in to the server. This way is easy and simple to use. In this way user cannot specifically use the OSGI features for the mediator. For example user cannot make certain packages private or import specific versions of packages. Other than that as I said earlier this is pretty easy. Also user can put mediators that are class mediator as well as mediators that have its own xml configuration.

dropins directory

User need to build an OSGI bundle of the mediator and put it in to the dropins directory. This requires a basic knowledge about the OSGI and maven bundle plug-in. So it can be little bit difficult for an programmer who doesn't aware of the OSGI technologies. Since user is creating the OSGI bundle he can use the OSGI features as he wish. When creating the OSGI bundle a mediator with a XML configuration should be a fragment of the synapse-core. If it is a class mediator it can be a normal bundle.

lib directory

User can only put class mediator jars in to this directory. A regular jar can be put in to this directory and it will be automatically converted to an OSGI bundle by the system.
The recommended way is to put the jar in to the mediators directory.

Note: in all these cases you need to restart the server after putting the mediator..

The WAY I like to see future Carbon architecture

2009-08-15T01:52:00.001-07:00

WSo2 Carbon provides a SOA Framework with Configuration + Monitoring. It has the concept of Front end and Back end. Front end and back end communicates through SOAP. Front end is JSP driven. So the actual configuration is done through the browser. Then this configuration is transferred to JSP layer with HTTPS. JSP layer uses SOAP to communicate with the back end. So back end has a series of Services deployed in an Axi2 instance for configuring and monitoring the server.

At the moment users deploy there services to the same Axis2 instance that hosts the Admin services. This has few drawbacks. Axis2 doesn’t have the concept of user. Also a service has access to the whole Axis2 instance. This means a service can virtually do anything in Axis2. So if several user deploy services in carbon they can access services deploy buy others. Also this service has access to all the admin services as well. But at the moment we consider this type of situations as user errors. System doesn’t enforce any restrictions on the user. But if we want to use Carbon to provide services in a multitenant way(As in gmail or Amazon ec2) this model won’t work.

Here is how I would like to see carbon architecture in the future that will solve all the above issues.

Here most important thing is Administrative services run in a separate Axis2 instance and user services run in their own Axis2 instances. This will clearly separate the administrative configuration from the service hosting environment providing better security and reliability. Also by nature this architecture is multitenant.

Different user groups or realms can have their own Axis2 instance as their hosting environment. Mapping the user to their respective Axis2 instance can be done by the management layer. These Axis2 instances should have their own modules, services, and transports.

Although it is simple to describe in a high level, this can be very hard to implement. But writing software is not an easy task anyway.

Most painful task in Web Services Development

2009-08-03T01:45:00.001-07:00

it is Testing!!

I have being writing web services related technologies for almost three years now. Testing is the most boring, time consuming task in Web Service Stack Development. I’m not talking about developing just Services. I’m talking about developing the tools needed for these services. Stuff like web service engines, ESBs etc..

It takes few weeks to write the actual code. Then it takes months to test it properly and get it to a stable enterprise ready application. If you look at the code behind web services tools they are pretty simple and doesn’t have lots of very hard to understand algorithms. But these applications takes years to develop and mature and surprisingly there are not much good web service engines around.

Here are some of the points I’ve noticed

When a bug is found we need to compile the classes, deploy it in the server and then test it. This implies shutting down the server copy the compiled files and then starting it again. Also to go to the actual place where error is found we need to configure the server as well. If you made another mistake you have to repeat the cycle which can lead to a vicious cycle. Normally shutting and starting up the server can take about 1 min.

Web Services are distributed in nature. Typically to test something we need a client + server. If you write a server component you need to write a client as well. Every time a bug is found we need to run the server + client. For products like ESBs we need client + ESB + web services server to test a full scenario.

Since we are writing the client, it can be implemented according to the server implementation we have. This only implies our server and client can work together. It doesn’t imply our implementation is correct.

Web Services invocations are not tangible. They get messages in and produce messages. So if we want to find a bug we need to analyze the messages. This is a hard thing to do as well as automate.

When an UI is involved with the these tools things get much more complicated.

WSO2 ESB – A High Level Architectural Perspective

2009-07-19T01:05:00.001-07:00

Enterprises are inherently complex, comprising of hundreds of applications with completely different semantics. Some of these applications are custom built, some are acquired from third parties and some can be a combination of both and they can be operating in different system environments.

Integration among these heterogeneous applications is vital to the enterprise. But conventional integration mechanisms can lead to a maintenance nightmare by coupling different components too tightly.

SOA is today’s integration technology. But SOA alone will not solve all the problems at hand. Service interfaces can change. Different services may be using different data formats and communication protocols. Physical locations of services can change arbitrarily. All these constraints means your applications are still tightly coupled together.

An ESB can be used for loosen up these couplings between different services and service consumers.

WSO2 ESB is a full fledged enterprise ready ESB. WSO2 ESB is built around Apache Synapse project. Apache Synapse is built using the Apache Axis2 Project.

Below is a diagram describing WSO2 ESB architecture from a messaging perspective. There are lot of other areas like Tasks, Events which are not describe in this diagram. This article focus on how messaging is handled in the ESB.

Figure: High Level Architecture of WSO2 ESB

Note: Various components of the pipes doesn’t imply any order in this diagram.

An application sends a message to the ESB.

The message is picked up by the ESB transport.

Transport sends the message through a message pipe. Quality of service aspects like Security and reliable messaging of the message is taken care in this pipe. Internally this pipe is the in-flow and out-flow of Axis2. ESB can operate in two modes. Message mediation or proxy services. In case of message mediation a single pipe is used. In case of proxy services we can think of separate pipes connecting transport to different proxy services.

Message transformation + routing can be seen as a single unit. As the diagram specifies there is no clear separation between message transformation components and routing components. WSO2 ESB call this the mediation framework. Some transformations happens before routing decision has taken. Some transformations happens after the the routing decision has taken. This part is the Synapse implementation.

After this message is injected to the separate pipes depending on the destinations. Here again quality of service aspects of the messages is determined.

At the end there is a transport layer. This transport layer takes care of the transport protocol transformations required by the ESB.

I have explained how a request propagates to its actual endpoint through ESB using its architecture. Response handling is the reverse of this operation.

All these components cab be managed and monitored through WSO2 ESB management console.

This is a birds eye view of the ESB architecture. Definitely Architecture can be seen in a more detailed view. Possibly in a few more blogs I’ll explain each and every aspect in a more detailed manner.

Writing a WSO2 ESB mediator

2009-07-16T02:17:00.001-07:00

Mediators provide a easy way of extending the ESB functionalities. WSO2 ESB is based on WSO2 Carbon platform which uses OSGI as the underlying technology. This implies everything that runs inside the WSO2 ESB to be OSGI bundles. No exception for mediators.

There are two ways of writing a ESB mediator.

Using Class mediator, this doesn’t allow mediator specific XML configurations
Writing the mediator with Factory and Serialize methods which allows it to have its own XML configuration

Writing the mediator is straightforward in any of these cases. Easiest way to write the mediator is to extend your mediator class from org.apache.synapse.mediators.AbstractMediator class. I’m not going to go in to details of how to write the mediator. Here are two good articles about how to write a mediator in each of the above cases.

After you write the mediator you need to build it with WSO2 ESB. This is the tricky part. You have to make your mediator an OSGI bundle in order to make it work with ESB.

Easier Way

You just need to create a regular jar which links to the Synpase core jar and put it in to the repository/component/mediators directory. The platform will automatically make it an OSGI bundle and deploy it to the server.

Harder Way

But if you want the full control of how your mediator should be created as an OSGI bundle you have to write the following POM files. I have given two maven pom files that can be used to build the mediator in each of the above cases. This way you can export the packages you want, you can import packages you want etc. This comes handy if your mediator is using some versioned jar files.

Important

Both pom files are created for the ESB 2.1 release. If you want to use them for another release make sure you change the versions accordingly.

Here is a POM file if you use class mediator to create the mediator.

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">    

    <modelVersion>4.0.0</modelVersion>
    <groupId>org.test</groupId>
    <artifactId>org.test</artifactId>
    <version>1.0.0</version>
    <packaging>bundle</packaging>
    <name>My Samples - Test mediator</name>
    <url>http://www.test.com</url>

    <repositories>
        <repository>
            <id>wso2-maven2-repository</id>
            <url>http://dist.wso2.org/maven2</url>
        </repository>
        <repository>
            <id>apache-Incubating-repo</id>
            <name>Maven Incubating Repository</name>
            <url>http://people.apache.org/repo/m2-incubating-repository</url>
        </repository>
        <repository>
            <id>apache-maven2-repo</id>
            <name>Apache Maven2 Repository</name>
            <url>http://repo1.maven.org/maven2/</url>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0</version>
                <configuration>
                    <source>1.5</source>
                    <target>1.5</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>1.4.0</version>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Bundle-SymbolicName>org.test</Bundle-SymbolicName>
                        <Bundle-Name>org.test</Bundle-Name>
                        <Export-Package>
                            org.test.mediator.*,
                        </Export-Package>
                        <Import-Package>                                                                                    
                            *; resolution:=optional
                        </Import-Package>                        
                    </instructions>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>        
        <dependency>
            <groupId>org.apache.synapse</groupId>
            <artifactId>synapse-core</artifactId>
            <version>1.3.0.wso2v1</version>
        </dependency>        
    </dependencies>
</project>

You can see we are using the maven bundle plugin for creating the OSGI bundle. Make sure you export the correct package which contains the mediator code. Unless you do this your mediator won’t work.

Here is a POM file if you use Serialize and Factory classes to create the mediator with it’s own XML configuration

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">    

    <modelVersion>4.0.0</modelVersion>
    <groupId>org.test</groupId>
    <artifactId>org.test</artifactId>
    <version>1.0.0</version>
    <packaging>bundle</packaging>
    <name>My Samples - Test mediator</name>
    <url>http://www.test.com</url>

    <repositories>
        <repository>
            <id>wso2-maven2-repository</id>
            <url>http://dist.wso2.org/maven2</url>
        </repository>
        <repository>
            <id>apache-Incubating-repo</id>
            <name>Maven Incubating Repository</name>
            <url>http://people.apache.org/repo/m2-incubating-repository</url>
        </repository>
        <repository>
            <id>apache-maven2-repo</id>
            <name>Apache Maven2 Repository</name>
            <url>http://repo1.maven.org/maven2/</url>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0</version>
                <configuration>
                    <source>1.5</source>
                    <target>1.5</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>1.4.0</version>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Bundle-SymbolicName>org.test</Bundle-SymbolicName>
                        <Bundle-Name>org.test</Bundle-Name>
                        <Export-Package>
                            org.test.mediator.*,
                        </Export-Package>
                        <Import-Package>                                                                                    
                            *; resolution:=optional
                        </Import-Package>                        
                        <Fragment-Host>synapse-core</Fragment-Host>
                    </instructions>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>        
        <dependency>
            <groupId>org.apache.synapse</groupId>
            <artifactId>synapse-core</artifactId>
            <version>1.3.0.wso2v1</version>
        </dependency>        
    </dependencies>
</project>

In this case we have to make the mediator an OSGI fragment of the synapse-core bundler. To achieve this we are using the <Fragment-Host>synapse-core</Fragment-Host>.

After you create the mediator drop the jar file in to the {ESB HOME}\repository\components\dropins folder.

Why you need WSO2 ESB 2.1 in your SOA?

2009-07-11T00:09:00.001-07:00

ESB is a key component in any SOA based solution. WSO2 has released it’s latest version of ESB 2.1.0. Here are some of the key features of WSO2 ESB that you are expecting to see in your ESB.

Functionality	Description
Invocation	Support for synchronous and asynchronous transport protocols like http, jms, mail, vfs, udp, fix and service mapping (locating and binding) through proxy services
Routing	addressability, static/deterministic routing, content-based routing, rules-based routing
Mediation	protocol transformation, service mapping
Messaging	message-processing, message transformation and message enhancement through mediation framework
Service orchestration	coordination of multiple implementation services exposed as a single, aggregate service
Quality of service	security (encryption and signing), reliable delivery, transaction management, throttling, caching
Management	monitoring, audit, logging, metering, admin console
Component architecture	built on the revolutionary WSO2 carbon architecture which allows you to build your own ESB that suites your enterprise
Clustering	Running in a cluster for better load handling
Load balancing	ESB can load balance the requests to a cluster

You can see WSO2 ESB offers a comprehensive list of functionalities which can be seen only in enterprise grade ESBs.

Hmm quite a lot of features… Now you must be wondering what is the price for all of this? You won’t believe this. It is ZERO. Also the code is available for free download and it is Apache 2.0 license. You know what you can do with Apache 2.0 license. If you are looking for commercial support, WSO2 offers it for may be quarter of what you are paying for other commercial ESBs.

Bottom line is WSO2 ESB is an enterprise grade ESB, free of charge. I don’t see any reason why you shouldn’t have it!

WSO2 ESB 2.1 Released

2009-07-10T01:09:00.001-07:00

http://wso2.org/downloads/esb

It is more stable, more user friendly and has lots of new features. You must give it a try……