One thing that particularly jived well with me is that it is pretty funny despite the fact that it is liberally sprayed with technical insights.  One would have thought that technology is almost an antipathy to anything in the nature of humor. But this rant proves one wrong.

Another reflection is that it is pretty “forthright” – if I can be excused for using an understatement. Many a time I get tempted to take a scalpel in hand and tear down the management of one or the other of the companies that I worked for but could never muster the courage nor the bank balance to make up for such an outright assault. So all in all it gives me some amount of sadistic pleasure to see another passionate guy giving vent and in the process possibly taking another step towards his “google”ian grave if that is the word I want.

Which brings me to another comment that I am seeing a lot about this rant. People were profuse in their praise for Google for condescending to leave the post on in Google+. But I maintain that it would seem incredibly bad form to remove it just because someone did not resound with the views of the powers that be. The lone voice of discord may well take your company out of the trouble that sometimes you are prone to get into as you flex your muscles and start to grow.

All in all I would heartily recommend a read.

No related posts.

I am enclosing the deck that I had used in the meeting here.
Ecommerce & Java

Take a look!

Related posts:

1. IOC, AOP – 101
2. Java & Performance
3. Multi Column Sorting in Java

But how do I go thru this morass of XML configuration and make sense of it? We need a parser that is flexible and lends itself to this kind of thing. That is when I discovered the commons digester.

Apache Commons digester is an offshoot of the earliest Struts project. It was chiefly used to parse the struts.xml file at that time but was found to be more generic and hence a new project was created within the apache realm to publish the framework in its own right.

The first thing that strikes us as we navigate the framework is the fact that it uses a stack. Coming to think of it, an xml itself is a perfect example of a LIFO construction – the last tag inserted is the first one that is closed followed by the tag that precedes it and so on. Hence it is most amenable to be stored as a stack structure. This small but brilliant observation motivated the creation of a framework that replicates the stack to store various configuration elements. I typically use a value object which I call a descriptor to store various configurations from the XML. Let us illustrate this with an example:

Consider the following XML from the state transition machine (STM) :

Without going too much into the detail of the framework itself, the objective is to parse this xml and  keep information about each flow. We can quickly invent a “FlowDescriptor”, “StateDescriptor” and TransitionDescriptor  value objects  that contains information from various levels of granularity. Here are the quick Java definitions.

Parsing the XML

The XML above can be parsed through the digester by passing a stream that contains this XML through the digester. But before that we need to ensure that the digester is set up correctly.

The whole idea is to set up the digester stack and add elements to it as we navigate thru the XML so that the correct value objects are created and added. Let us talk about the plan in detail.

The plan is to first add a FlowsContainer object to the top of the stack before we even begin to parse the XML. Then as we encounter individual flow tags we should create new FlowDescriptor objects and add them to the FlowContainer object. Then we need to do the same for the StateDescriptor and TransitionDescriptor objects. Here is a snippet of code that accomplishes this:

As we see, the built in rules of the digester make XML parsing trivial. The stack based parsing and processing makes the entire process trivial and keeps it encapsulated well.

Digester Extensions

It is possible to add our own rules to the digester to achieve very specific behavior. We just need to extend the Rule base class.

No related posts.

A Segue – Enterprise Architecture & Systems of Records

Let us consider a different problem. Let us think about an enterprise with a large number of applications.

As the enterprise started fanning out to envelope multiple enterprise applications, it becomes imperative to reckon with a large number of different databases of information often called as systems of records.

For instance, let us take the example of a large enterprise that has thousands of employees.

There would be an employee system of record that stores details about the employees. There would be multiple applications that the enterprise might have to use to discharge its day to day activities. These applications might all require access to the employee database for both validation and other purposes. Ex: The accounting application might want to add an employee as an “owner” to each account.

SOA & The Enterprise

How do these applications access employee information? One option is to use SOA and hide the employees data behind a service. The Employee Service is responsible for maintaining employees and facilitating updates to the central Employee System of Record. This approach is great but can have a few drawbacks:

• If there is a batch process (such as a scheduled job) that iterates over multiple records in the database and for each record needs to obtain employee information, then this might suffer from the (n+1) selects problem that is mentioned elsewhere in this blog.
• It is hard to enforce reference integrity constraints between an entity in a local database and an employee since the employee information is only available via SOA. Ex: How do we ensure that all accounts owners are valid employees?

Replicated Data & ETL

EDA & Real time Incremental ETL

Related posts:

1. The n+1 selects problem..

Code, like good wine, takes time to mature. The earlier you write the code, the more opportunity it gets to evolve. The same arguments apply for a development team. A development team has no sense of belonging to the project in the absence of code. They sit nonplussed by the surfeit of documentation that surrounds them. Give them an IDE, a proper module structure and some code to write and voila! watch the difference.They start getting involved in the project.

As human beings, I think, we need the ability to mentally slot things. We cannot work with intimidating wholes. We need to break them down into their constituent parts. That is only possible if we are given the slots to put the parts into. A module structure gives the developers just that ability. It tells them that the problem at hand needs to be broken down into smaller consumable units which can then be pigeon holed into individual modules. As the modules have been set up in the IDE, the developers are able to translate the requirements into stuff that they can relate to.

This makes all the difference to a project. As the developers and their code start “soaking in” , the project gets more mature. In a month or so, the developers become adept at the code and the environment. The code starts becoming more stable as it starts getting exposed to more testing very early. The inception stage in a project is a tension free time. ( as indicated in the diagram below) . Hence extra effort expended at this point is sustainable. As the release deadline approaches, the tension starts to mount. But if the team has been ramped up adequately, the stability of the code reinforces confidence to the team about the feasibility of meeting the project release deadline. 

IMO, the diagram below represents a good recipe for a project. The initial “tension free zone” needs to be utilized to ramp up both the team and the source code. The source code must have been modularized and set up in individual developers’ IDE by the time the actual coding phase begins. By that time, the team should have got ramped up and must be performing at its optimal effort level as indicated by the purple portion of the diagram. There would be a spurt in effort required as the current release approaches culmination. However, that effort spurt is minimal and is not even close to the spurt that was originally expended during ramp up.  We all know that tension and high effort levels lead to quick project fatigue resulting in considerable productivity loss.  Hence it is imperative that we ensure that the team is comfortable as we approach the release date. 

Also, observe that even the spurts do not take the effort level anywhere close to the saturation point.

It is important that we ensure sustainability for the team by keeping their effort as much possible at the optimal level. Sub optimal level effort is wastage of resources. Spurts over the optimal effort level are necessary but cannot be protracted over a long period of time.

All failed projects were characterized by super demands on effort level during the period preceding the project release. This lead to a considerable loss of productivity and increased the risk of attrition.

In my projects, I tend to encourage a “war room” kind of atmosphere during inception so that the team quickly gets off the mark!

Related posts:

1. Project Automation
2. Bug Driven Development
3. SEPG

At first blush, I am instinctively uncomfortable about anything that requires a high degree of predictability in software development since that is going to be violated if you are ever intending to produce software that could be deemed useful by the ultimate consumers!

It is exactly like doing the interiors of a house which is an activity that I was partaking of recently. I was engaging some interiors contractors to do the work. Their estimation and invoice depended on a simplification. They merely told me that it would cost me a certain amount based on the fact that I am finishing a wall that is “y” sq.ft in area. The cost of one sq.ft is x. Hence the total cost is x*y of course exclusive of applicable taxes, surcharges and all the rest of the appurtenances and caveats you would find in an estimate. Now it is possible that I might decide to finish the whole wall or just a small fraction of it. But the estimate (and in this case the ultimate invoice as well) does not change. At the end of the day I felt that the dice was loaded against me since I had to pay for the whole wall irrespective of what parts of it, I decide to furnish.

But I realized that this is the only way an estimate ever works even in software development. I decide on the unit cost based on previous experience. I just estimate the “approximate sq feet” I need to build and multiply it by the unit cost to arrive at the total cost. However, if I were to take a more agile approach, I would exactly pay for the amount of materials I use, the labour charges etc. which would be more accurate. It might or might not work out cheaper. But at least, I know that I get what I paid for and that no one gets “shafted” in the process. Both the parties get a fair deal.

The trouble with software is that there is no raw material to build on. Sure you need application servers, package solutions, products to enhance,languages, operating systems etc. But these are not raw material for the software. These are more like edifices on which the software development rests.  So it is all labour charges so to speak. Hence estimation (and ultimately the cost) tends to be time based and the cost is determined by the total time and the unit cost per time for the “labourers”. In short, the numberof people is directly proportional to the cost of the project. This simplification is essential for estimation but is at variance with experience. I don’t get the exact same kind of people. They don’t cost the same.  More importantly, the quality of the artifacts that they produce is not the same.

On the one hand you are comfortable with the certitude that comes with knowing what is the exact cost upfront.  On the other hand, you want the flexibility of a dynamic solution that is geared towards changing business needs. It is hard to reconcile these two inherently conflicting forces.  Hence, all estimates are approximate at best.  At worst,they lead to law suits, high attrition and all kinds of unpleasant things bordering on eternal perdition.

This harks back to a comment by Robert Martin in his great treatise titled “Agile Software Development – Principles, Patterns and Practices” which is a book I heartily recommend. You can find a quick and unhelpful review of mine in the Amazon web site here.

Robert Martin says that people are not “plug compatible programming units”. Most estimation processes assume that people are exactly that. They assume that people can be shoved into a software building assembly line and over a period of time – the software comes out the other side! Hence, IMO, the estimation process is at loggerheads with agility.

That is why most great softwares are not produced by SOWs. Software development needs trust. It needs a certain relationshipto be built over time among the participants. It requires passion. Most importantly, it requires excitement from the individuals which can only be obtained if they perceive career growth. Treating them insouciantly as “plug and play resources” devalues them and makes them feel sub-optimally used.

It is not for nothing that Alistair Cockburn opined that:

Process and technology are a second order effect on the outcome of a project. The first order effect is the people.

Related posts:

1. SEPG

The problems involved different IoC containers – one of them being Spring and the other being ATG Nucleus. Both stated the same thing namely that the application is taking too long to look up components from the IoC containers. Ex: In Spring, the SpringBeanFactory.getBean(String name) call was taking too long. Now, if you were an application doctor who got commissioned to solve this case, your natural temptation is to prescribe the following:

Thou shalt not useth an IoC container to look up components. Use it instead to inject one component into the other during start up.

But before you scribble these words of profound wisdom  with an illegible hand, as doctors are wont to do, you would do well to take a small pause and consider the situation more holistically.  Why is a component forced to “look up” other components from an IoC container? In other words, what are the situations that would warrant one component being forced to look up another component? Components need to look up other components if they cannot benefit from injection. The only reason that such a thing happens is if the two components are in different scopes. In the case of a Spring application, one component might be a singleton whilst its dependency maybe a “prototype”. Hence the singleton component has to “look up” the  prototype component. It can do so either by either using BeanFactoryAware interfaces or method injection. (Please see here for a detailed description for both these artifices in Spring)

But the point I am making is to figure out “why” we need to have components in different scopes. In general, I would look askance at any attempt to use a component as a non singleton.  My reasoning is as follows: If you use a component as a non singleton, then it implies that you are storing some kind of state in that component.  This state may be instance specific , request specific, session specific etc. This means that the component is no more stateless. This design violates my favorite class dichotomy doctrine as advocated here. So why do we need to store state in a component?  I think that the following use cases warrant them:

• We may be using a framework that requires stateful components. Ex: All actions in Struts 2 are considered stateful since the framework injects request specific parameters (stuff that got submitted by the user) into the component.
• We maybe using value objects to store components. Consider a situation where a value object contains a method that invokes a strategy to act on it. Ex: We may have an XmlRenderer component that renders other objects to XML. A value object may have a render() method that invokes this renderer to render itself to XML.

In the above situations, the class in question acts as both a carrier of state as well as something that accomplishes some work. This kind of mixup of responsibilities leads to a situation where a more ephemeral object (ex: the value object or the struts 2 action) needs to look up a singleton component that is naturally contained in an IoC container. A ton of post processors (using the Spring parlance) come into play in this lookup which can lead to a substantial slowness of this call.

Note that in the above discussion, we did not discuss another candidate use case that requires a component lookup from an IoC container. This happens in writing frameworks. Let us say that we write a framework that requires a component to be created by the client of the framework. (such as for instance an MVC framework such as Struts) This component might require us to look up actions that are best created by  an IoC container. In this case, we need to look up the component when the request to be serviced by the component arrives. (in Struts case, the request hits a URL that is mapped to a struts action). At that point in time, we can either look up the component from the IoC container or instead use a cached reference to the object that was created during application start up. The latter approach is preferred since it does not incur the penalty of the look up with each request.

These kind of problems are best avoided by the judicious separation of data containers from the strategies in an application. I am well aware that this may tilt the design excessively towards an anemic domain model which is frowned upon by the purists of object design. My point is that the term Anemic Data Model has itself been coined with prejudice. You can always have strategies that leverage inheritance rather than becoming transaction scripts.

Just my point of view!

Related posts:

1. KPI – Key Performance Indicators
2. Identity Crisis
3. The Law Of Demeter

I know I haven’t been posting too much and that has been breaking a few hearts. So to heal the breach and reward the faithful for waiting this long, I have published a new open source framework called STM which is a representation of the State Transition Machine. Please check it out in the following pages and let me know what you think.

1. Maven web site

2. Main page that describes the project.

3. Sourceforge site (contains SVN code, jars etc )

Related posts:

1. Value Object Wizard (VOW)
2. Project Automation

Let us say we have a table called rank1 that stores two columns – an Id and a value. The ID is non-unique in that an ID can have multiple values. I want to compute the average and the 90th percentile of the values for each ID. Example: Consider the following data:

What I want is the following:

Here is a sql that accomplishes this:

The gist is as follows. We first compute the average, count and the number at which the percentile figure occurs in the inner most select.
We use this information to join back all the Ids and values along with the grouped information in the intermediary select. This intermediate SQL also computes the rownumber IDs and groups them.
The if clause in the @row computation works to group the row numbers by ID. The @id is required to achieve this since it is a variable that monitors the change in ID from 1 to 2.

The percentile is computed by finding the row number that corresponds to the number that was computed in the inner most select.
Shoot me a question if you are having trouble following the logic.

Related posts:

1. Bug Trends and some interesting SQL stuff

I had given a keynote in a conference sometime ago, about the Spring framework. It constituted a presentation on the patterns that necessitated the creation of  Spring in the first place. I am attaching a slide deck on this. I would add some more material here as we go to elaborate on the content of the deck.

Here is the deck.DIAOP

Related posts:

1. Ecommerce & Java