Insight: Anoop's techologue

Preamble for developing distributed systems

noreply@blogger.com (Anoop) — Wed, 22 Jan 2014 06:51:00 +0000

Over the past few years most companies have found the fallacy in trying to maintain a monolithic enterprise level application. The much talked about silver bullet has been designing distributed systems. A common mistake we make in the race to make our systems distributed is ignoring the different set of challenges it could come with. By all means for most enterprise application systems providing multiple services going the distributed way is the most effective in the long run. I wont be discussing the details for choosing one over the other here, and assume the reader has basic exposure to SOA and creating multi agent systems that are then orchestrated in a fashion so as to serve the end user a variety of services. The main points I would like to bring out to people who are presently at the doorstep to moving from a monolithic system to a distributed architecture are the following considerations for a more successful implementation

Decomposing/Identifying sub systems
Message, protocol and network
Security
Debugging issues
Monitoring and Management
Fault tolerance, recovery and handling

Decomposing/Identifying sub systems

The first order before designing a distributed system is to identify the various sub systems that would form part of the application eco system. This requires knowing how to decompose an application into sub systems. There are different ways to decomposition. The ones I would like to discuss are subject based decomposition and aspect based decomposition. The former is defining a sub system that is based on a functional need and the latter based on a technical need. Examples of former would be systems that provide functional services like a shipping service, a billing system etc while the latter would be systems that are cross cutting in nature like authentication service, auditing service a scheduling service etc. Keep in mind to give this section the most importance so that you define effective subsystems that could mean a world of difference in maintainability for the future.

Message, protocol and network

Once you have defined your subsystems, the next task should be on deciding on how the sub systems talk to each other. This includes defining the grammar, verbs, nouns etc of the conversation. Be mindful to define the attributes of messaging carefully with a good amount of documentation as part of governance for future maintenance. Also make sure to add an attribute like correlation id that can uniquely identify a message across systems. The other key thing to understand and account for is that with distributed systems usually come distributed teams and some time varying requirements for different subsystems based on if that service is being used by multiple other services. This means that these sub systems could have independently evolving vocabulary of message structure. However this independent evolution should not be disruptive to the existing sub systems using this service. In simple sense, version your message vocabulary and have backward compatibility so that you could add, modify the message structure and at the same time service requests with older version number. This leaves the depending sub systems to evolve at their own pace.

Having the message defined itself is just one third the task here. Next you need to decide what is the expression of the message delivery, would it be SOAP/REST or JSON/XML/POJO? Along with it also the medium of message delivery i.e JMS/HTTP/RPC/File etc.

Security
Obviously leans to how each service can be secured and more so if exposed over the internet. This however depends on the the mode of transport and your network deployment topology. There are various ways to secure services, some of them are:
1. Digest of User/Password based access to services
2. Encrypt payload. Use this with caution, as more secure algorithms can reduce throughput.
3. Use secure network resources like https where possible.

Debugging Issues
No matter how good our team is in building software, we have to admit there are always going to be situations after production deployment where you might be called to debug an elusive issue. There are more than ways to make sure that debugging is not a very stressful experience. Detailing some below:
Logging:
Not many realize, but logging the right information at the right log level can help alleviate many situations before being declared a "code red" situation. Take time to review all log statements before releasing to make sure it has all the information needed by the support team. I am sure many argue that the we log all information but still get calls at mid night :). My advice, would be to try reading the log statements as if you were in the shoes of a support team member and if you understand the logs and know what to do next, you have hit the bulls eye. Apart from this, when logging an exception and or error scenario, try to give additional information to the reader as to if there is anything he/she could do to correct the situation. Use error codes effectively with a detailed pre mapping of how to correct such situations. An example log statement for the readers to review and vote:
"java.lang.NullPointerException: Unable to service request at the moment"
or
"S110X: The lat/log values received seems to be null, please check if geocoding is enabled for the triangulation"

Searching:
In a distributed system, since the log files could reside on multiple servers, trying to trace a message correlation across systems could be a pain. Here is where some tools like splunk or logstash could help. Personally I believe logstash (http://logstash.net/) has come a good way and still has further to go. That said it is a very good open source alternative to splunk and give quite a bit of insight into our collective eco system. Basically it can aggregate the logs over multiple files and provide a very interactive dashboard called Kibana to query for log statements instead of doing the same think across multiple terminal windows and vi.

Monitoring and Management
While debugging is a reactive way to understand an issue, many issues could be proactively traced and managed if the health of the entire distributed eco system is exposed as metrics. In the case of Java application, JMX could be used to expose important information regarding the health of the system. What constitutes the health of the system is dependent on the nature of the application, however some pointers are given below.
Metrics:
1. Current throughput of the system over x period in time
2. Average throughput since system uptime
3. Sub System start time
Operations:
1. Circuit Break number: A number of failed responses beyond which your application would stop servicing requests so that the issue could be fixed by manual intervention.
2. Health Check test message : Ability to run a set of sample messages to confirm all components are up an well but does not contribute to any auditing or service persistence
3. Log Level: Ability to change Log level during diagnostics without having to restart the application
4. Thresholds for alerts: Set thresholds to enable JMX notifications.
5. Active startup/shutdown: Ability to startup/shutdown an already active JVM hotspot without affecting the other components when needed for debugging so that messages are not consumed.

Using a framework like jolokia (http://www.jolokia.org/), you could expose the JMX features through REST and hence control via web ui. Another tool to look at might be hawt.io (http://hawt.io/) to put a UI to your metrics so that the support team has all the tools to effectively monitor as well as manage.

Fault Tolerance, recovery and handling
Usually in monolithic systems, you have all the information you need to handle an error and decide what to do next; however in a largely distributed eco system, some of the context is lost during the various conversations and this means extra effort needs to be invested on how each component handles error. In the case of synchronous communications like in a web service, the service provider needs to clearly publish the semantics of an error response like an error code, error description and the correlation id of the message (though redundant, it helps if logged on the client component). When done as SOAP response, using the soap fault header to do the same. If the mode is HTTP REST, you could use the HTTP Status codes as response with custom messages. All this goes out the window if your mode of transport is asynchronous by nature like a JMS message broker. In such cases, you could propagate the error replies by using an JMS reply channel check out this url for more details (http://www.eaipatterns.com/RequestReplyJmsExample.html)

So we talked about the responding with an error message, but to truly handle it would depend on the needs of the specific component and the nature of exception. You could broadly describe exceptions of two types: Transient Exceptions and Non Transient ones. Transient exceptions are usually ones that are not in control of the application and may self correct eg: Network exception, database locks etc whereas non transient ones are anything that as expected non transient like a business exception or a logical exception etc. You may need to have the capability to identify the transient ones and retry the operation hoping the transient nature of error self heals to a maximum retries. For non transient ones, some of them may be safely ignored based on the requirements of the component whereas others may not, so you should have a configuration to skip certain exceptions (maybe with a max limit to the number of skips) or fail on others.

I hope the above information helps the reader in some small measure to improve their design. This by no means is exhaustive or a silver bullet for all situations, so please evaluate them to your needs, improve on them and give me feedback if anything in this blog could be improved.

The way to operational excellence - DevOps teams!

noreply@blogger.com (Anoop) — Sat, 07 May 2011 19:59:00 +0000

Nowadays, most software written is complex, both in its intent and existence. How many times have we in our own employment written "quick fixes" in the interest of time and business? We would have argued that once we have the patch ready, we could devote quality time to understand and fix the issue by design. However, it usually is deja vu when 2 years down the lane we still hear from the operations team that Jimmy's process, isnt running properly? Jimmy's process? Well that how, we have the nomenclature running for the little quirks and fixes we create, the creator's name and then the fixed component :). The fix would have been intended for subset of the business processes at that time, which we could eventually have expanded across impact many business processes in the future. The point being, what we seem to quick fix for keeping our business process running could come down hard on us later.

Lets try doing an impartial analysis of why a Jimmy's process (please bear with me for picking on Jimmy :)) would come into existence in the first place:

1. There was an issue in the processing of some software component.

2. If the issue was not fixed, business would end up spending a lot of time trying to fix it manully or it meant a revenue loss or it just meant loss of face for the IT team.

3. The business and/or the project manager wanted a quick way to fix it so that the issue is resolved for now, as analysis on the root cause could mean more time spent while the issue remains unfixed in production.

4. One of the most important factors: QA never tested out the scenario because of unavailability of production like infrastructure or data volumes.

5. It could have been because we had a churn in the dev team and the new comer decided it was easier to write a quick fix instead of spending quality time (which is always in short supply!) to get to the right fix.

6. Well if it isnt all the above, it was because a solution was implemented by the software development team without involving the operations team in any of the meetings who could probably have brought out the gap.

It goes without saying that it usually is the operations team who face the heat on a day today basis governing the use of the software processes and in that they end up being the most constructive critics of the system. It also is a fact that they are usually the first line of defense for all exception scenarios while the dev team is probably the last. Seems right to say they probably have seen an issue like the one before and hence on hindsight may be able to suggest scenarios where a solution could fail.

I was reading about the adoption of the principle of DevOps and it seemed to appeal to me in companies where processes are usually written and rewritten on the fly. I do accept there are critics to the argument and hell, tell me one thing that hasnt ? So if you think you are in a position to address issues like the ones written above. Please do read up on http://en.wikipedia.org/wiki/DevOps

I was thinking I would summarize on the same and had written up my piece and somewhere in that process found a blog by Damon Edwards which illustrated it pretty succinctly. So decide to link to his blog rather than reinvent the wheel: http://dev2ops.org/blog/2010/2/22/what-is-devops.html

Spring AOP - Under the hood.

noreply@blogger.com (Anoop) — Sun, 30 Jan 2011 18:51:00 +0000

Let me start by hoping all readers had a very enjoyable christmas and wishing you and your families a very happy new year 2011! Today, on popular demand, I am going to blog on how Spring AOP works under the hood. Again reiterating that there are other implementations of AOP which may differ from how Spring AOP works.

Broadly, we could classify AOP implementations is two ways:
1. Compile time aspect weaving : This method involves compiling the source code using a weaving compiler like AspectJ weaver and produces a compiled class which contains the aspects also integrated into the compiled class. So as a user, we need only worry about the aspects while designing the systems and the compiler would take care of integrating it into the code. Now there are 3 types of compile time weaving viz.
   a) The aspects are integrated during compilation i.e from source code into binary classes
   b) The aspects are integrated into already compiled binary classes and is called post compilation weaving
   c) The aspects are weaved when the class loader loads the class into the JVM and is called Load time weaving.

2. Run time aspect weaving : This form of aspect implementation involves creating proxies for a class that is to be advised. Spring AOP makes use of this type of aspect implementation.

Now that we have a primer on what aspect weaving is, lets discuss more about the run time weaving as it is going to be the focus of this blog. The key to understanding this, is in understanding what a proxy pattern is. A proxy is in its generic form is a class functioning as an interface to something else (in this case another class). The advantage of having this proxy is that you could put in behavior into the proxy class for common needs (cross cutting concerns like logging, exception handling etc) and hence externalize them from your actual code. This if used wisely would help reduce code scattering, tangling and duplications in applications. Also makes the developer focus only on the business needs for the application classes.

For more details on the proxy pattern itself, please refer : http://en.wikipedia.org/wiki/Proxy_pattern

This is exactly how the Spring AOP works, by creating proxy classes for your aspect enabled classes. The proxy class methods are called first which would then call the necessary advices as configured like logging etc. Now how does this proxy class get generated,? Spring aop has two options, the jdk dynamic proxying feature which works on interfaces and the cglib library which works on classes.

Now, what all this mean? Let me explain a little more, imagine you have created aspect advices and configured your business spring beans with the advices in your context or via annotations. The Spring container would then create proxy classes on the fly for your spring beans and call the advice methods before (or after) calling your spring bean methods. All this is transparent to the developer as all he is involved in is in writing the spring beans and the aspect advices. Spring AOP makes use of two ways to generate proxy classes.

The first option is to use the default jdk dynamic proxy. But the limitation here is that the class for which the advice is being written has to implement an interface. The proxy would be created based on the interface.

The second option is to use cglib library which would again help in creating proxies and this time it is not necessary for the advised class (your bean) to implement an interface.

Both these options have some drawbacks as they would not be able to apply advices to internal private methods and that the beans have to initialized by Spring (i.e it wont work if you create the class objects using the new operator). These proxies are not serializable and hence not applicable for distributed applications.

All of these can be mitigated by weaving the aspects at compile time using aspectj which gives the user a lot more power to the concept of AOP but at a slightly higher learning curve. That said, Spring AOP is a great way to start wading into the waters of AOP, once you have the grasp of it and have a need, you can jump onto the AspectJ bandwagon!

Hope this has been useful as intended for the folks who requested it and the other avid readers. Please let me know if you need to know specifics and I'll try to answer to the best of my knowledge.

Demystifying the Cloud - A perspective

noreply@blogger.com (Anoop) — Mon, 13 Dec 2010 02:51:00 +0000

Its been a long time again since my last blog. Bringing up a 3 month old baby I understand now isn't exactly a breeze :). All the same, would like to put the blame more on my own laziness than my bundle of joy. Now getting to the matter of this blog. I am sure everyone on our side of the trade would have heard of the term "The cloud". So without further ado, here goes:

The term "cloud" seems to have permeated every section of the technology space from the development arena to the database hosting to usual technology chit chats. It has become as ubiquitous as another term that has probably given the biggest boost to the internet interaction ..Web 2.0. What I will attempt to do in this blog is try to explain how it has the potential to affect us as developers, administrators and technologists in general. Also lets discuss the various terms being used at the present in lieu of the Cloud.

So lets begin at the top..what does the cloud mean? As per google, one of the definitions goes as "Computing where services and storage is provided over the internet", and this is a pretty close definition. In essence it means that resources are shared and provided to consumers/computers on demand. The shared resource could range from computing power to storage to software. The details are abstracted from the user and hence the user could scale or reduce his usage as required or demanded by his own business need. It has been a natural evolution from virtualization. Lets try to understand both of these terms. Virtualization as wikipedia puts it is the creation of a virtual (rather than actual) version of something, such as an operating system, a server, a storage device or network resources. The cloud has become the business model over virtualization. We can as of now virtualize anything from networks to memory to storage to software and hardware. I could provide multi tenancy (ability to host multiple tenants or users) in a single resource by abstracting out the usage of a large resource or aggregating multiple smaller resources to masquerade as a single resource to the user.

Lets talk in examples. I am an yearning entrepreneur with a great business idea which as I reckon would need enough computing power and storage and being as I mentioned a startup don't want to be expending my precious dollars on equipment and people as of now. The equipments are prone to be outdated and people are prone to churn in employment. So, I decide to call up one of the cloud service providers for lets say computing power and storage. One of the available ones being Amazon which has EC2 (Elastic Cloud) to provide for computing power and Amazon S3 for storage. I can then decide to host all my artifacts for the applications in the S3 node (I could initially take a smaller size of data storage) and ask my provider to scale up to a larger size as my application would need more storage. Similarly, with regard to the computing power, I could again ask Amazon to scale my EC2 instances and providing load balancing for say my web application.

Thats in short what a cloud does. But what does it mean for the developers? Ideally, nothing in our application code would change except that we would have to be cognizant of the how seamless the cloud is i.e how reliable it actually is? We would need the operations team to keep tab of the uptime of our cloud instances. Our deployment strategy would differ for each of the cloud computing platform we would use. If we intend to use our own built on platforms like Gigaspaces for example, it would require some change in the deployment artifact to include configuration changes like using the JavaSpaces api to access data in the grid (instead of the database) and partitioning data as well as to co-locate (deploy) applications in each partition. Apart from these, how reliable our cloud is in terms of security? I'll try and follow up on each of these concepts in later blogs.

As a special mention, the example here was just a shot to show one of the facets of the use of the cloud and its utility certainly is not limited to the scenario mentioned. Also, I am neither directly or indirectly involved with either Amazon or Gigaspaces. I will be giving a list of available cloud platforms/vendors to my knowledge along with my next blog on this topic.

Aspect Oriented Programming: The Elixir to Java? - Part 3

noreply@blogger.com (Anoop) — Thu, 02 Sep 2010 13:34:00 +0000

Its been a busy few weeks and the transition from an expectant dad to becoming dad, comes with a thrill that just didn't get me to concentrate on the blogs.But anyways, since I am back blogging, lets get down to business. In this blog I am going to discuss about my design on how we could handle exceptions using AOP and potentially provide for post processing and analysis of your exceptions. This could provide you with more than enough data to mine and make informed decisions and analysis of errors and maybe self correct. Beyond that it could give you the tool to provide for numbers to your management to questions like "So did your bug really fix the exception we were getting earlier?", "How many exception scenarios do we get on a weekly basis?", "Did we have any exceptions today?"...by going beyond your usual log scraping.

Again like in my last post I am assuming here that the user has an understanding about Spring and IoC. In case you don't, not to worry, the spring website has enough and more tutorials on the same. I would suggest the reader to go through the Spring Core manual. So, lets begin by trying to understand a scenario where one of my java swing applications need to connect to the database. Knowing my network isn't always reliable (is it deja vu for you too?), I need to retry getting my connection with a time interval of say 10 seconds and a maximum of 10 retries. How would our regular programming direct us? We would put our getConnection method within a try catch block and increment a counter and then do a recursive call on my wrapper method with a Thread.sleep.

Lets now see how we could do this a little differently using AOP and probably make this an application wide choice without having to repetitively code this, or making this a concrete implementation without a choice. And as a bonus for this blog, lets use annotations to define the behaviour. First thing, lets define an application exception class say ApplicationException as a wrapper for all Exceptions I intend to handle like so:

package com.cquent.spring.exception;

public class ApplicationException extends Exception{

public ApplicationException() {

super();

}

public ApplicationException(String message) {

super(message);

}

Now, lets define a default ExceptionHandler which implements a behaviour expressed in IApplicationExceptionHandler interface.

package com.cquent.spring.exception.handler;
import com.cquent.spring.exception.ApplicationException;

public interface IApplicationExceptionHandler {
public void handleException(Exception ex);
}

-----------------------------------------------

package com.cquent.spring.exception.handler;

import org.apache.log4j.Logger;

public class ApplicationExceptionHandler implements IApplicationExceptionHandler{

Logger log = Logger.getLogger(getClass());

public void handleException(Exception ex) {

log.debug("Handled exception : " + ex.getMessage());

}

Now for the meat of the behavior, lets define a Java annotation. But before doing that , lets just do a checklist of behaviors that are desirable for me :

1. We should be able to handle an exception by being able to pass the class.

2. We should have a retry mechanism if required to rerun the method to give it a chance of redemption :).

3. Be able to define the maximum retries for the method in case of exceptions.

4. Be able to define if the entire JVM needs to be exited on reaching maximum retries, or just continue after the exception has been handled by the handler.

The annotation definition would look like this after setting these behaviors:

package com.cquent.spring.aop.annotations;

import java.lang.annotation.ElementType;

import java.lang.annotation.Retention;

import java.lang.annotation.RetentionPolicy;

import java.lang.annotation.Target;

import com.cquent.spring.exception.handler.ApplicationExceptionHandler;

@Retention (RetentionPolicy.RUNTIME)

@Target (ElementType.METHOD)

public @interface HandleException {

Class exceptionHandlerClass() default ApplicationExceptionHandler.class;

int sleepTime() default 1000;

int maxRetry() default 0;

boolean exitOnMax() default false;

}

The retention time mentions when this annotation is active and in our case it would be Runtime. The target gives the scope of responsibility of this annotation called "HandleException" which in this case would be at the method level. I will in my later blogs explain how to write your annotations and their uses, but for now, I believe an application first approach would probably interest the reader to the benefits of annotations. So much, so that the reader would either read up on annotations on his own or wait for my blog ;).

We have the behavior template for handling an exception and what remains is the implementation of the behavior. For the first part ,we need to define an entry point to our aspect through our SystemArchitecture super aspect like we did for the logging behavior in the last blog:

package com.cquent.spring.utils;

import org.aspectj.lang.annotation.Aspect;

import org.aspectj.lang.annotation.Pointcut;

@Aspect

public class SystemArchitecture {

@Pointcut(this(com.cquent.spring.aop.markers.ILoggable)")

public void inLoggableMethod() {}

@Pointcut("@annotation(com.cquent.spring.aop.annotations.HandleException))")

public void inException() {}

}

So what have I done here? Defined a point cut so that whenever it encounters a method that has a HandleException annotation , it would invoke the inException method

Now, we need to define an aspect! So here goes and this is how our lil genie of exception handling for the retry mechanism would look:

package com.cquent.spring.aop.aspects;

import java.lang.reflect.Method;

import org.apache.log4j.Logger;

import org.aspectj.lang.JoinPoint;

import org.aspectj.lang.ProceedingJoinPoint;

import org.aspectj.lang.annotation.AfterThrowing;

import org.aspectj.lang.annotation.Around;

import org.aspectj.lang.annotation.Aspect;

import com.cquent.spring.aop.annotations.HandleException;

import com.cquent.spring.exception.handler.ApplicationExceptionHandler;

@Aspect

public class DefaultExceptionHandlerAspect {

private Logger log = Logger.getLogger(getClass());

@Around("com.cquent.spring.utils.SystemArchitecture.inException()&& @annotation(handlerInfo)")

public Object retryOnException(ProceedingJoinPoint pjp, HandleException handlerInfo) throws Throwable {

int numAttempts = 0;

int maxRetries = handlerInfo.maxRetry();

int sleepTime = handlerInfo.sleepTime();

do {

numAttempts++;

try {

return pjp.proceed();

} catch(Exception e) {

log.error("There was an exception for retry attempt = " + numAttempts);

log.error(e);

Thread.sleep(sleepTime);

}

while(numAttempts < maxRetries);

if(handlerInfo.exitOnMax()) {

log.info("Exiting process as could not proceed with execution of method " + pjp.getSignature().getName() + " after " + maxRetries + " retries" );

System.exit(1);

}

throw new Exception("Reached Maximum retries for this method");

}

Notice, I have used a new annotation called @Around for the inException method, basically this gives us control during the exception of the method unlike the @Before and @After annotations that gives us access only before or after execution of the method. Also the @annotation(handlerInfo) would decorate the handlerInfo argument with the annotation instance details from the method..Since we would not know at what point an exception might be thrown, we resort to calling the method within our shell and then expect the exceptions to happen in the process. Also notice the argument is a ProceedingJoinPoint instead of a JoinPoint and an additional argument to represent the annotation HandleException for the method which is retrieved at run time. The ProceedingJoinPoint would enable us to invoke the target method. After this, all I do is retrieve the details of the maximum retries defined and with the proceed() method on the ProceedingJoinPoint would invoke the target method till the maximum method retries is reached and in the process log the retry number.

Now lets test this out. We need to include aspects in our spring configuration file:

<?xml version="1.0" encoding="UTF-8"?>

 <beans xmlns="http://www.springframework.org/schema/beans"

 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xmlns:aop="http://www.springframework.org/schema/aop"

 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd

  http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">



 <aop:aspectj-autoproxy />



 <!-- For AOP Aspects of Logging, ExceptionHandling -->

 <bean id="systemArchitecture"

class="com.cquent.spring.utils.SystemArchitecture">

 </bean>

 <bean id="loggableAspect"

class="com.cquent.spring.aop.aspects.DefaultLoggingAspect">

 </bean>

 <!-- For AOP Aspects of Logging, ExceptionHandling -->

<bean id="exceptionAspect"

 class="com.cquent.spring.aop.aspects.DefaultExceptionHandlerAspect"&gt

</bean&gt

</beans>

As discussed for case material, lets now create a getConnection() method in a class that returns a connection to a database:

public class DBUtils {
@HandleException(sleepTime=2000,maxRetry=5, exceptionHandlerClass=ApplicationExceptionHandler.class, exitOnMax=true)
   public Connection getConnection() {
   dataSource.getConnection();
   }
}

This method when invoked and if throws an exception would be handled by the ApplicationExceptionHandler class and would be retried 5 times before exiting. Pretty simple right? Now imagine, I could use the handler class to probably use a runnable instance to write data to the database (obviously for non-db connection issues) or to a separate log file so that they could later be mined for information.

public class ApplicationExceptionHandler implements IApplicationExceptionHandler{

Logger log = Logger.getLogger(getClass());

public void handleException(Exception ex) {

log.debug("Handled exception : " + ex.getMessage());

new Runnable() {

public void run() {

writeExceptionToDb(ex);

}

};

recoverIfPossible(ex);

}

We could use it with forward chaining expert systems like Jess,Drools etc to take corrective action if required. Would have loved to give more information on this, just that I am probably running a lot ahead in terms of ideas than actions, so would be take a more pragmatic approach and introduce possibilities and blog the implementations later. These should be helpful for those in the design and architecture and should help if you are starting a new project using Spring and would give you a multitude of choices to implement later when required.

Stay tuned for further techologues!

The Misunderstood "kill" command

noreply@blogger.com (Anoop) — Mon, 02 Aug 2010 07:28:00 +0000

A friend of mine recently had an issue where his running java code was getting stuck at a point and not moving forward. I had faced this issue many times before and much more so as I have had to work on batch processes for a while now. But thankfully, I had a secret weapon which has been a great defence for me in debugging the cause. Yes, as the name suggests it was the kill command and no, I did not terminate and restart the process like we reboot our windows systems as a resort of all kinds and for all problems.

Well the kill command actually is a wrapper over the kill() system call and can be used further than just the termination of a process. The default signal it sends to the process is to terminate the same, however kill can be used to send a SIGTERM or a defined constant to a running process and the process can explicitly handle the same to do useful functions like cleanup before termination etc. However the scenario in question here is that you have a java process, web or otherwise taking up too much heap or not moving forward in its processing but is taking a large slice of your CPU processing. In such cases, the first thought that might occur to you is to debug through the code and find the cause of this delay. But, most often than not, this might happen in your production system and you would be in limbo to enable debugging.

Now, one of the solutions I am going to discuss here is the kill -3 command. The JVM hotspot has a handler for this SIGTERM and it immediately spurts out thread dump of all the threads running in the java process to the standard error stream. So you need to have the java process redirect its output to a file like so :

java -jar test.jar >> console.out

This would redirect all output from the java system console and err to the console.out file. Lets now try an example with an innocent looking code:

public class TestMain {

String[] fruits = {"Orange","Apple","Mango"};

private int findFruit(String fruit) {

int index = 0;

boolean found = false;

while ( index < fruits.length) {

if(fruits[index] == fruit) {

found = true;

break;

}

if(!found) {

index = -1;

}

return index;

}

public static void main(String args[]) {

TestMain test = new TestMain();

test.findFruit("Apple");

}

The code is pretty straightforward in its intent to find the index of a fruit in an array of fruits. The only issue being that the loop that runs over the array of fruits does not increment the index variable and the program goes into an infinite loop. If you were to check your cpu usage with the "top" command, you will see that the java program has bumped the cpu traffic to 95%. or greater as shown below:

So, now in the best interest of my server laptop not crashing, I intend to investigate the reason for this anomaly in otherwise perfect code (I know its far from it :)). The next thing I do is run the command to get the process id of this java process:

ps -ef | grep TestMain

Then I run the kill -3 on the same. Since I have been running my program in eclipse, the output would show up in its console, on the otherhand in production systems the thread dump would show in the console file (if redirected as shown earlier). My output looked like this:

2010-08-01 23:51:50

Full thread dump Java HotSpot(TM) Client VM (16.3-b01 mixed mode, sharing):

"Low Memory Detector" daemon prio=10 tid=0x099a6800 nid=0x15ff runnable [0x00000000]

java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x099a4800 nid=0x15fe waiting on condition [0x00000000]

java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x099a3000 nid=0x15fd waiting on condition [0x00000000]

java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0999d400 nid=0x15fc in Object.wait() [0xb4a5c000]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <0x7f2f0b00> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

- locked <0x7f2f0b00> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x09998800 nid=0x15fb in Object.wait() [0xb4aad000]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <0x7f2f0a08> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:485)

at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <0x7f2f0a08> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x09974000 nid=0x15f8 runnable [0xb6db2000]

java.lang.Thread.State: RUNNABLE

at TestMain.findFruit(TestMain.java:9)

at TestMain.main(TestMain.java:22)

"VM Thread" prio=10 tid=0x09997000 nid=0x15fa runnable

"VM Periodic Task Thread" prio=10 tid=0x099a8c00 nid=0x1600 waiting on condition

JNI global references: 665

Heap

def new generation total 4800K, used 273K [0x7f2f0000, 0x7f820000, 0x846f0000)

eden space 4288K, 6% used [0x7f2f0000, 0x7f334598, 0x7f720000)

from space 512K, 0% used [0x7f720000, 0x7f720000, 0x7f7a0000)

to space 512K, 0% used [0x7f7a0000, 0x7f7a0000, 0x7f820000)

tenured generation total 10752K, used 0K [0x846f0000, 0x85170000, 0x8eef0000)

the space 10752K, 0% used [0x846f0000, 0x846f0000, 0x846f0200, 0x85170000)

compacting perm gen total 12288K, used 26K [0x8eef0000, 0x8faf0000, 0x92ef0000)

the space 12288K, 0% used [0x8eef0000, 0x8eef6820, 0x8eef6a00, 0x8faf0000)

ro space 10240K, 61% used [0x92ef0000, 0x93516298, 0x93516400, 0x938f0000)

rw space 12288K, 60% used [0x938f0000, 0x94026cb8, 0x94026e00, 0x944f0000)

The output has given the dumps of all the threads that were running at that point of time by the JVM for the TestMain java process. We see 8 threads running as follows:

1. Low Memory Detector

2. Compiler Thread

3. Signal Dispatcher

4. Finalizer

5. Reference Handler

6. main

7. VM Thread

8. VM Periodic Task Thread

Following this there is also the stats for the heap space usage split among the various temporal heaps.

Each thread is followed by whether the thread is a daemon thread and then by the priority of the thread. What follows below that line is the stack for the thread.The thread I am concerned with is the main thread which shows also the call hierarchy of the stack. You can see that the process has been stuck at the findFruit method and at line 9 of the program. This may not be too much for such a small program, but for large production systems, it can aid a lot in isolating the pain method calls, to say the least , other than also showing deadlocks among threads and if the thread is waiting for an object reference to be released. I wish I had a more comprehensive example to show, but may be for later. From this dumped reference, I Have narrowed down my search to an individual method potentially causing the issue, and if I am a reasonable programmer, I should have found the root cause. I hope, this exercise would help you too in finding causes and can greatly improve your time to resolution of runaway cpu problems and others. It goes without saying that the thread dumps are not a replacement for your regular profilers and debuggers, but they are effective in giving you the starting place. The newer versions of Java have heap/vm analysers like jhat and jstat that will probably make this redundant, but its a good way to get your hands dirty.

Aspect Oriented Programming: The Elixir to Java? - Part 2

noreply@blogger.com (Anoop) — Sat, 31 Jul 2010 08:23:00 +0000

In last week's blog, we did the ground work on AOP , lets now try and do some practical examples of sorts. The assumption that I make here though, is that the reader is knowledgeable in the Spring framework and Java annotations.

Lets think of a situation where you already have some code that takes care of your business logic, now we need to inject our cross cutting concerns like logging and exception handling, something lets say, you try to keep consistent enterprise wide.

First things first, we start off with defining a SystemArchitecture class that would act as an aggregator Aspect and aggregate all point cuts for the application as below:

import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
@Aspect
public class SystemArchitecture {

@Pointcut("this(com.cquent.spring.aop.markers.Loggable)")
public void inLoggableMethod() {}
}

So, what have I done here? I have a class called System Architecture which is annotated as an Aspect (and acts like a super aspect). Then I have an empty method called inLoggable which is annotated with a point cut. Here the point cut is expressed as an instance of type Loggable. Note the this used for describing how the AOP interceptor should intercept any instance which is of type Loggable. Loggable can be a marker interface so that you could implement multiple classes with the same and have it log all its methods.

package com.cquent.spring.aop.markers;
public interface Loggable {
}

So, now our requirement is to log all entry points and exit points from the methods in a class that implement Loggable. And to this effect, we need to write a Logging Advice which would be something as follows:

package com.cquent.spring.aop.aspects;
import org.apache.log4j.Logger;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.After;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
@Aspect
public class DefaultLoggingAspect {
private Logger log = null;

@Before("com.cquent.spring.utils.SystemArchitecture.inLoggableMethod()")
public void beforeLoggingAdvice(JoinPoint jp) {
String method = jp.getSignature().getName();
Class clazz = jp.getTarget().getClass();
log = Logger.getLogger(clazz);
log.debug("Entering method " + clazz.getName() + "." + method);
}

@After("com.cquent.spring.utils.SystemArchitecture.inLoggableMethod()")
public void afterLoggingAdvice(JoinPoint jp) {
String method = jp.getSignature().getName();
String clazz = jp.getTarget().getClass().getName();
log.debug("Exiting method " + clazz + "." + method);
}
}

But for all this to work, you need to configure the application context with the details like below:

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xmlns:aop="http://www.springframework.org/schema/aop"

 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd

  http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">



 <aop:aspectj-autoproxy />



 <!-- For AOP Aspects of Logging, ExceptionHandling -->

 <bean id="systemArchitecture" class="com.cquent.spring.utils.SystemArchitecture">

 </bean>

 <bean id="loggableAspect" class="com.cquent.spring.aop.aspects.DefaultLoggingAspect">

 </bean>

 <!-- For AOP Aspects of Logging, ExceptionHandling -->

</beans>

We need to declare the xml schema definitions for spring aop in the applicationContext.xml as shown above. The aop:aspectj-autoproxy prepares the spring container for loading the subsequently declared aspects and also introspect the classes that have been "advised" or rather annotated for the loggable aspect.

Let me try to explain what happens here, when a method of a class that implements Loggable interface is invoked, it would using the AOP interceptor first invoke the inLoggable method in the SystemArchitecture class. Now before that is invoked, since we have a advice for the method, the control in the JVM is passed to the DefaultLoggingAdvice class and specifically to the beforeLoggingAdvice method and the JoinPoint instance is passed as argument. If you were to look up at the Spring AOP javadocs, we can derive the name of the class and the method which was originally being invoked and using that detail, we could log the same. The same process is repeated before exiting the method by calling the afterLoggingAdvice and thereby logging the exit of the method. You could do more..just read about on how to use the "Around" advice like the "After" and "Before" we just did to get more control on logging what happens in the method.

I have left out some of the details in the blog like how is advice actually implemented in the AOP space etc. But then it would have made this blog very bloated and hard to follow. For the avid readers, google about on
"Runtime weaving in AOP" and "Bytecode weaving in AOP". In the next blog, lets discuss on how we can handle runtime exceptions without writing any catch blocks for them in the actual business code.

Aspect Oriented Programming: The Elixir to Java? - Part 1

noreply@blogger.com (Anoop) — Sat, 24 Jul 2010 17:33:00 +0000

This is my third and hopefully successful attempt at getting myself to start to blog. Anyways, the topic for todays discussion is "Aspect Oriented Programming" and its usefulness in Java. Its been an awed technology for quiet some time and biggest boost it ever got is with the advent and adoption of the Spring framework in the Java world.

To begin with, lets discuss the concepts that make this technology awed, interesting and possibly (extremely) useful with a few anecdotes in the form of some questions.

1) How many times have you had to extend and abstract class for some common behavior across the application?

2) Had to laboriously remember to log when entering a method and exiting the same?

3) Forgotten to handle runtime exceptions in classes where there are chances. And if you did, forgotten to log the details of the arguments.

4) Seen connection objects being opened but forgotten to be closed after use? Same so for transaction commits/rollbacks?

5) Have wanted to see how the hits to the methods go (to analyze at production runtime)without having to change code?

Well if your answer to this list (on the smaller side) of questions is yes, then probably it is valuable for you to consider AOP. To think, all the scenarios I mentioned above can be classified into "cross-cutting" concerns in AOP terminology. In simple words, these are all concerns that do not directly affect/impact your business logic, but nevertheless, they are required to be addressed for the smooth running of your code.

The philosophy here is that while developing a business application, the developer should be concerned only about how the business scenarios need to be handled, all the cross cutting concerns can and should be handled out of it. With AOP, you can write "Advices" (in java terms, some functionality like may be logging), that can be triggered at a "join-point" (a particular execution point in the code). These advices are external to your business logic. Did I loose you guys? hmm...lets make it a lil more bare bones.

"Advices" : Can be another set of classes. Lets say a class, whose sole purpose is to log when a particular set of methods (I'll explain later how you can define which) are being executed. i.e it should log when a method is entered and when it is exited. So you dont have to remember to write your :

logger.debug("Entering method xyz()-----") and logger.debug("Exiting method xyz()-----") in every method you write.

"Join Point" : When your business code is executed, you would want the logging advice to be triggered at certain execution points, like for instance when you enter the methods of your DAO class. The execution point at run time when an advice is triggered is called a Join Point.

Now, how do we tell the system when the join-point has reached? For this we use an expression language for the "point-cut" which can be interpreted by the run time engine as a call to the advice. For example execution(com.xyz.dao.AppDao.execute*(..)) is a point cut expression which means that all methods in the com.xyz.dao.AppDao class that start with "execute" are potential join-points. In short an advice runs at any join point that is matched by the point cut.

I am going to take it a lil easy and stop here, so that you ponder on what else is possible with AOP. Will follow up in later blogs with specific example and what happens under the hood.