Praveen Manvi's Technical Diary

2011-09-30T07:29:00.000-07:00

Backward and Forward compatibility with Java serialization protocol:-

Java offers a nice protocol for storing and communicating the persisted information. In a distributed deployment environment the server code has to support both the new and old information when its anticipated the control over the deployment of different versions of the servers are not available. The code has to deal with both backward as well as forward compatibility. i.e. VERSION N should be able to read both VERSION N-1 & VERSION N+1 object.

Maintaining the backward compatibility is easy. i.e. Version N code can read Version N-1 code by maintaining the version attribute. The version attribute in the read() API will make sure that the reading of new information is skipped. This is quite straight forward to achieve through “java.io.Externalizable” interface provided java where we have the full control of the context information on what we need to save. But it becomes tricky when we try to read the information created by future versions. The problem with forward compatibility is that the Java de-serialization it has to know what the new stuff that has been added & also should know where the information has been added.

The solution explained below tries to solve the problem in a pure java with less overhead & suitable for systems that have already established using Java serialization with ”java.io.Externalizable” interface.

Here the top down approach is followed for the explanation, it starts with how the usage of API should look like VERSION 0,1 & 2 then show how that can be achieved through code.
The challenge here is to read VERSION-1 (N) data from VERSION-0 (N-1) class. The current solution targets supporting only N+1 version & not the future versions which is the only important part of 24 X 7 support systems in the event of release back out.
Let's take an example of class "Test" having 2 attribute in VERSION-0, VERSION-1 introduces a new attribute called "versionData1" & VERSION-2 introduces the one more new attribute "versionData2". To simulate the case the data is inserted in between the data. In real-time scenario we will have lots of classes with readExternal()/writeExternal() methods spread across all over the code, but usually controlled with single mother object.
SKIP_START & SKIP_END are the marker interfaces used to identify the new information and ObjectInputWrapper is the new class used to move the cursor to skip the details when we read new information from the old class.
"Test" is Externalizable class having 3 attributes. We will include a new attribute in the second version and see how a "Test" object with VERSION -0 (N) reads VERSION -1 (N-1) object.

This plugin architecture is quite scalable. ObjectInputWrapper can be injected with different serialization protocols like Google proto buffer, Oracle Coherence's POF, etc... using a using factory pattern like ObjectInputOutputProviderFactory that provides different implementation including custom logic to efficiently store & retrieve the information

VERSION = 0;

    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       out.writeObject(name);
    }
    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        name = (String) in.readObject();
    }
"Test{VERSION=0, number=10, name=praveen}"

VERSION = 1

This class has new String attribute called versionData1 with the value "versionData1"
    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       // Added for version1 in between
       out.writeObject(new SKIP_START());
       out.writeObject(version1Data);
       out.writeObject(new SKIP_END());
       out.writeObject(name);
    }
A marker interface class has been included to identify the new information that the older version can safely ignore.

    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        if(VERSION>=1) {
             in.readObject(); // we can safely ignore skip signals
            version1Data = (String)in.readObject();
            in.readObject();
        }
        name = (String) in.readObject();
    }

"Test{VERSION=1, number=10, versionData1=versionData1, name=praveen}"

VERSION = 2

    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       out.writeObject(version1Data);
       // Added for version2 data in between
       out.writeObject(new SKIP_START());
       out.writeObject(version2Data);
       out.writeObject(new SKIP_END());
       out.writeObject(name);
    }
    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        if(VERSION>=1) {
           if(VERSION==1){
                  in.readObject();
           }
           version1Data = (String)in.readObject();
           if(VERSION==1)
       in.readObject();
           }
        }
        if(VERSION>=2) {
            in.readObject();
            version2Data = (String)in.readObject();
            in.readObject();
        }
          name = (String) in.readObject();
    }
"Test{VERSION=2, number=10, versionData1=versionData1, versionData2=versionData2, name=praveen}"
The skip data version can safely be ignored in future versions. It will be removed in write first and also from read on subsequent versions.

The 2 key classes used to implement are explained here.

The ObjectInputWrapper class will have read method implementations for each basic data type. The sample below shows just two methods reading and readObject for illustration. For real implementations, we need to create final ObjectInputReaderTemplate classes for each basic data type and reuse them in the read methods of ObjectInputWrapper & they can be cached to avoid lots of object creations. The profiling of this code showed that these light weight are pretty cheap to handle, JVM will make sure that there is overhead in terms of memory or the cpu by in lining them automatically.

This class makes use of generics and anonymous inner classes to move the cursor to skip the new information that version N was un-aware while reading.

One sample code having all the classes published in github

2011-08-20T02:02:00.000-07:00

Caching Exceptions in server applications -

In web applications it’s common that response page of certain types can be cached for given http request, similar approach could be used to cache the exceptions in the server side applications as well. If the reason for any exception known upfront that it will be applicable for given amount of time, there is a opportunity to cache these to reduce the load on resources. This could be huge gain in cases where a particular piece of code is doing heavy operations like cpu intensive tasks, making remote calls, Database/File manipulation etc..

Here is a solution making use of Java generics.

For Ex:

public static int mockHeavyOperation(String someData) throws FileNotFoundException,Exception{

Thread.sleep(3000);//simulating some heavy operation

if(true) throw new FileNotFoundException();

return 10;

}

Can be converted into

public static int mockHeavyOperationWithCacheExceptionHandler(String someData) throws FileNotFoundException{

return new ExceptionCacheTemplate<Integer,FileNotFoundException>(){

@Override

public Integer handle() throws Exception {

Thread.sleep(3000); //simulating some heavy operation

if(true) throw new FileNotFoundException("Checked Exception");

return 10;

}

}.runIn(new ExceptionCacheTemplate.ExceptionKey(someData,40000));

}

As we can see from above code extra 4 lines are doing the trick.
If we run the first operation 10 times it takes 30000 millis where as 2nd example takes only 3000 millis, that's a huge gain I suppose.
1. The input information that resulted in exception can be stored as part of Exception Key class. It can be any class including String as shown above as long as it implements equals/hashCode(). runIn method works directly with String as well. If any Exception re-appears within the interval 40000 as shown above the Exception will be thrown from cache.
2. Only the exceptions that are included as part of the generic exceptions,
3. A custom map can be provided to manage the expiring of cached exceptions, Default implementation will be managing through the value provided while creating cache key.
"public Map getCache()" can be overridden to provide different implementation
For example Google's Guava library provides awesome fluent interface to manage expiry of keys.

Map<Key,Graph> graphs = new MapMaker()
       .concurrencyLevel(4)
       .weakKeys()
       .maximumSize(10000)
       .expireAfterWrite(10, TimeUnit.MINUTES)
       .makeComputingMap(
           new Function() {
             public Graph apply(Key key) {
               return createExpensiveGraph(key);
             }
           });

It can be handled at the framework level making it completely transparent to the applications.

ExceptionTamplate is available in github

2011-07-17T10:13:00.000-07:00

Notes on TDD and Unit testing in general.
Most of the benefits of TDD are unquantifiable, Its tough to correlate better design, better ability to refactor the code to TDD practice. It requires a real practice to experience and appreciate the productivity benefits. When we are working with legacy code and is not easily testable, advocating TDD there is counter productive and refactoring such code to suit TDD is a hard sell & taking up such task is usually a thankless exercise.For green field projects TDD should be applied without a second thought, to my mind its not debatable anymore.

Here are some notes that I collected can be used to sell TDD during the discussions. For me TDD as effective tool of "COMMUNICATION" is the single most important point that should to enough to employ TDD. The lesser defects as result of clear executable communication should make all the stake holders namely Developers, Managers, Testers, Clients and finally the real users of software happy.
Tests prove that your code actually works resulting in fewer bugs, if we catch all the scenarios. Testing before the code is a design activity.These tests can’t replace system and acceptance testing,but they do supplement it & also fewer bugs that make it to QA.
We can improve the design without breaking it. Having unit tests in place, we can do powerful refactorings that can untangle the most challenging of system psychoses.Refactoring not only becomes cheaper, it changes the developer mindset to strive for better quality. When refactoring becomes cheaper, the quality of software continuously improves.Fixing PMD and FindBugs errors should not require management approval.
Unit tests are a way to make programmers have documentation as they hate to write a MSWord document & is a project management technique. So when we can’t remember how to use a class APIs, read the unit tests to find out.MS Word documents are not meant to be compiled and deployed, if we can avoid as much as possible, its good for everyone.Unit tests reduces the communication pain points. Its a shared language.You know what your code needs to do. Then you make it do it. Even if you don’t have a working system, you can see your code actually run and actually work. You get that great “I’ve done it!” feeling. Developers can avoid nagging questions "Have you written it", "Have you tested it" & finally "Have you really tested it". If tests are written and published, developer can always go & check from web tool.
Just try test-first if you want to be high on endorphins, proud about your work, and motivated to do more.They demonstrate concrete progress. You don’t have to wait a month for all the pieces of the system to come together. You can show progress even without a working system. Not only can you say you've written the code, you can actually demonstrate success. Of course, this is another distinction that traditional programming teaches us to ignore.
“Done” doesn’t mean you’ve written the code and checked it in.“Done” means the code actually runs in the system without bugs. Running unit tests is a step closer to the latter.Unit tests are a form of sample code. We all encounter library functions and classes we don’t know how to use and one of the first places we go is the sample code. Sample code is documentation. But we don’t usually have samples for internal code. So we’re left shifting through the source or through the rest of the system.

Test-first forces you to plan before you code. Writing the test first forces you to think through your design and what it must accomplish before you write the code which results in better code. This not only keeps you focused, it makes for better designs.
Test-first reduces the cost of bugs. Bugs detected earlier are easier to fix. Bugs detected later are usually the result of many changes, and we don’t know which one caused the bug. So first we have to hunt for and find the bug. Then we have to refresh our memories on how the code is supposed to work, because we haven’t seen it for months. Then finally we understand enough to propose a solution.
Anything that reduces the time between when we code the bug and when we detect it seems like a obvious win. We consider ourselves lucky to find out about bugs within a few days, before the code is shipped to QA or to customers. But how about catching them within a few minutes? That’s what test-first accomplishes with the bugs it catches.It’s even better than code inspections. Code inspections, they say, are better than testing, because using them to detect and fix bugs is cheaper than testing. After the code ships, it’s much more expensive to fix the bugs. The earlier we can detect and fix bugs, the easier and cheaper and better. That’s the advantage of having code reviews.Code inspections catch more bugs within a few days, rather than a few months, but It virtually eliminates coder’s block. Ever wonder what statement to write next? Like writer’s block, coder’s block can be a real problem. But test-first systematizes the structured part of coding, allowing you to concentrate on the creative part. You may get stuck on how to test the next bit or how to make the test pass, but you’ll never be left puzzling over where to go next. In fact, usually you’re left with the opposite problem: You know you need to take a break before you burn out, but you’re on a roll and don’t want to stop.Failed tests make better designs. Testing a piece of code forces you to define what that code is responsible for. If you can do this easily, that means the code’s responsibility is well-defined and therefore that it has high cohesion. And if you can unit-test your code, that means you can bind it as easily to the test as to the rest of the system. Therefore, it has loose coupling to the pieces around it.High cohesion and loose coupling is the definition of good, maintainable design. Code that is easy to unit-test is also easy to maintain.It’s faster than writing code without tests! Or to put it another way, skipping unit tests is faster, unless you actually need the code to work.
Most of the effort we spend on code, we spend fixing it after we’ve checked it in to the source-code repository. But test-first eliminates much of that waste by allowing us to get more of it right to start with and by making bugs easier to fix.
One of the real value proposition of unit tests is that we get a low-level regression-test suite,we can go back at any time and see not only what broke but where the bug is. Arguably with many frameworks around it’s a low-effort way to catch bugs before the build goes off to QA. Whenever a bug comes it will make life lot easier without the need of using a debugger many a times.Test-first catches some bugs within a few minutes instead of a few days. It is even cheaper than code inspections, code reviews.

2011-01-17T09:17:00.000-08:00

2011 - My Technical Goals

Make JavaIQ.in useful & interesting to Java developers

My child-hood ambition has been to become writer and editor to a magazine. My obsession with politics/history/technology, I guess hasn't helped me to make money, but it made me always feel complete and happy. Not sure by end of 2011, I will make JavaIQ.in a respectable online magazine, but I will definitely give my best shot at that. I also would like to refine my applications that I created as my weekend projects & publish them in this site.

Android - Develop mobile applications

My guess is that developing applications for smart phones, iPads will be the "big bet" in coming decade and Android is likely to lead in this area of innovation. My decade java programming experience should help. btw, I bought my android phone this year.

HTML5/Java Script - Learn it's effective usage

Some how I never liked Flash/Flex. Although I tried JavaFX (which apparently was waste of effort), never spent significant time with Flash. I hope create awesome applications with HTML5. JavaScript has been excellent lanaguage I hope to do plenty of coding using YUI, JQuery and GWT/JSNI.

Play/Grails - Developing web applications in simpler way.

The dumb struts & its similar web frameworks will have to be replaced with these awesome frameworks. I never had such a first time nice experience with any framework compared to Play. I intend to teach and develop few web applications with that this year.

JVM lanaguages - continue to invest in learning

I will continue to invest in learning Groovy and Scala & continue to look for places where I can use these to accomplish the tasks.

Happy & prosperous new year to all.

2010-06-03T07:04:00.000-07:00

Using groovy to search name for my newly born son.
I had the un-sorted list of names of the God Vishnu from where I wanted to select a name, so wanted to sort them out alphabetically & used Groovy for the same.
Groovy Code in Java style - Click to see the code

1    def top = """
2        <html> <head>
3        <script LANGUAGE="JavaScript">
4        <!--

5            function toggle(id) { var e = document.getElementById(id); e.style.display = e.style.display == "none" ? "block" : "none"; return false}
6        //-->
7        </script>

8        </head>
9        <body>
10       """
11   def htmlOut = top
12   def set = new TreeSet<String>()

13   new File('c:/vishnu.txt').eachLine {
14     set.add(it.toLowerCase().trim())
15   }
16   def ref = "a"
17   htmlOut += """

18           <a href="#" onclick="return toggle('$ref')">$ref</a>
19           <div id='$ref' style="display:none">

20       """
21   set.each {
22     if (ref != it.charAt(0)) {
23       htmlOut += "</div>"
24       ref = it.charAt(0)

25       htmlOut += """
26               <a href="#" onclick="return toggle('$ref')">$ref</a>
27               <div id='$ref' style="display:none">

28           """
29     }
30     htmlOut += it
31     htmlOut += "<br/>"
32   }
33   htmlOut += "<br/></body>"

34   writer = new FileWriter('vishnu.html')
35   writer.println(htmlOut)
36   writer.flush()
37   
38   println 'done :)'
39

Groovy with DSL MarkUpBuilder -Click to see the code

1    def writer = new FileWriter('c:/Names1000.html')
2    def src = new groovy.xml.MarkupBuilder(writer)
3    src.html {
4      head {

5        script """
6            function toggle(id) { var e = document.getElementById(id); e.style.display = e.style.display == "none" ? "block" : "none"; return false}
7          """

8      }
9      def set = new TreeSet<String>()
10     new File('c:/vishnu.txt').eachLine {

11       set.add(it.toLowerCase().trim())
12     }
13     body {
14       for (alphabet in 97..122) {

15         char current = ((char) alphabet)
16         Set<String> names = set.findAll { it.charAt(0) == current}
17         def size = names.size()

18         if (size != 0) {
19           a(href: "#", onclick: "return toggle(\"$current\")") {
20             span "$current ($size)"

21           }
22           div(id: current, style: 'display:none') {
23             names.each {
24               li it
25             }

26           }
27         } else {
28           span "$current ($size)"

29         }
30         br ''
31       }
32     }
33     println 'done :)'
34   }

Unsorted 1000 Names
Sorted 1000 Names with groovy
Groovy is indeed Java++, It adds nicer syntax, closures and fluent interface making coding a joy without much noise.
Here is the result from groovy program. Click on letters to hide/expand the names starting with those letters.

a (113)
b (54)c (20)d (57)e (6)f (0)
g (25)h (13)i (3)
j (22)k (58)l (10)m (64)n (33)
o (4)
p (81)q (0)
r (14)s (243)t (19)
u (8)v (119)w (0)
x (0)
y (22)z (0)

2009-09-20T22:53:00.000-07:00

Google App Engine & Cloud computing : Making world flat for Developers all over the world.
The playing field for developers all over the world has been levelled with the advent of cloud computing & web services. It's pretty much free for any developer to develop/deploy application over the Internet. There is no start-up cost.Scaling up can happen incrementally based business requirement with upfront investment & I think that's revolutionary in my opinion.
Here are my thoughts on Google App Engine, after deploying my first sample application.

Salient features of Google App Engine

Google started supporting java on their engine in April - 09 - Ref
Support for Google Id and Sign In
Automatic Persistence -JDO or JPA (standards based approach, implementing standard Java APIs on top of App Engine where possible. So instead of using the underlying App Engine datastore API, developers can program against Java Data Objects or Java Persistence API)
Local Development - Remote Depolyment
Scalability and free Use - Pretty cheap
Monitoring - Nice overview
Eclipse plug in - I did not face any problem with while deploying.
Limited support for JDK classes - When I tried converting my swing application to GWT one I found many of classes I used were not supported like java.util.Timer classes
Native threads can't be spawned

Google unleashed App Engine with support for Python in 2008(April). This was Google's first entry into on-demand application development and deployment. Developers were able to build, develop and easily deploy apps using Python. I think it has been successful in that.The economic impact of is that cloud computing disrupts the data center world by slashing the capital and skills required to deploy a web application. I am betting on success of this model.

About the current application: java typing tutor

I created this sample application within an hour with GWT & published it. Initially I tried to porting my one of Swing application "Java Typing Tutor" which I created after reading the steve yegge's blog long back Programmings dirtiest little secret - BTW I don't believe that fast typist should also be better programmer in general, but it was interesting write up. Whenever I get time I do intent to make this application on par with my feature rich swing application.

GWT - Although I feel very comfortable coding with this framework, (may be because I spent lot of time in coding Swing Apps) it's API limitations are quite annoying(java.util.Timer does't work etc...). Initially I had the impression that converting a Swing application to GWT application is straight forward & I had even the ambition to write even converter!(paint() for GWT widgets), But I guess now I realized that it's not possible. The programming style, approach are quite different. Once I used wingS framework to convert one of swing application to web - it was straight forward because of deep integration with swing, but now apparently that project has died. It definitely requires a different mindset to develop web application than developing swing application.

Finally some happy notes for swing programmers:
Wicket & GWT are great saviours for struggling swing developers : With Swing GUIS losing their relevance, these 2 framework provides great pathway to move into web development.

Over all I feel GWT & Wicket are great framework for Swing developers to develop web application in terms productivity. Rails/Grails (or any request/response framework) guys can never match component developers in terms of productivity :-)

Advantage of GWT clients (or any fat client) is that they can tap into the resources of the client PC they are running on (such as memory to store state) and thus scale much better for large numbers of users. GWT produces something that is more akin to applets, independent applications that happen to be hosted on web page.

But where as,

Wicket still assumes that you want to build at least part of your application the 'old fashioned'

way, so that it will work without JavaScript, turn up in search engines, can be bookmarked so on...

So component developers can satisfy both the worlds.

"It's stupidity. It's worse than stupidity: it's a marketing hype campaign," -Richard Stallman

Amazon looks to be better than Google for Cloud

2009-09-13T09:10:00.001-07:00

Elements of Java Coding Styles

Java Coding Styles: Here are some of the tricks which I felt useful for java developers & I have been using them a lot. Again these are personal preferences, consider what you like & disdain if you don't. By end of the day the software professional is all about getting "good working software, quickly and at low cost" that can sustain for longer time & solves the business problem at hand (with YAGNI caveat). I want to keep updating this whenever I see interesting ones.

Prefer smaller methods
They look beautiful, easy to understand/reuse/change & test.

'As I did 20 years ago, I still fervently believe that the only way to make software secure, reliable, and fast is to make it small.' - Andrew Tanenbaum

Dr. Venkat gives wonderful explanation on - how to convince your fellow developer to write short methods?

Use JUnit test cases; assert statements and method/variable names as documentation, writing comments in JavaDoc are obsolete and useless many a times.

The purpose of the assert statement is to give you a way to catch program errors early, using assertions to state things you know (or think you know) about your program can improve readability & are great in serving as comments, they are intended to be cheap to write, just drop them into your code any time you think of them. They are great tool of communication for a programmer & can get rid of dumb English stuff in the code with // & /** */.

"There is nothing as useless as doing efficiently, which should not be done at all." - Peter Drucker.

I hate to hear someone telling me to add javadoc comments especially for private methods. The method name variable name should be wise enough to reveal the intent. Fluent interface & design pattern can help a lot in naming variables, methods & class names. Over the years my variable & function names have become more verbose. It's so much easier to understand code from years ago when it reads like a sentence.

Big caveat with respect to assertions,

Assertions should be used to specify things that to be true at various points in your program for providing the documentation & definitely NOT for error checking and any active code. Assertions are disabled ("turned off") by default. Assertions were introduced in Java1.4 & can be turned on with -enableassertions (or -ea) flag on the java command line.

& for JavaDoc,

Code that isn't fully documented is unfinished and potentially useless. Javadoc comments on methods and classes should normally indicate what the method or class

It was necessary document the type of keys and values in a Map, as well as the Map's purpose, but now with Java5 Generics that is also not required. Prior to Generics I always used to code List/*String*/ = new ArrayList();

Some Java 5 features are smart; Look for smarter new libraries that exploit these features well

Use basic infrastructure libraries like Google Collections (I prefer this over apache collections), sl4j, Guice etc… as much as possible. We will learn to appreciate the value of Java-5 new features by looking into the source code of these libraries, Old coding styles have to be abandoned in favor new features.

Commenting the code with "if(false)"

In many a times we want to use different implementation for testing & commenting out code temporarily. If the method is very long it's painful to comment out the whole block. Let us say we have method like this.

public void sendMail(){

if(true) return;

// Lot of code for adding to database to event

...

// Transforming text with XSL etc...

....

}

if(true) return; -> This line has the same impact as the commenting out the whole stuff :)

Use double brace initialization (For non-performance intensive code), If possible use Google Collections in all the cases.

As swing programmer I have been using these quite long time. I have seen quite number of experienced java programmers aren't aware this. This really useful while testing. It's concise, requires less typing & more readable but comes with cost. Google Collections provide better same feature without any performance cost.

Map map = new HashMap() {{

    put(1, "one");

    put(2, "two");

    put(3, "three");

    put(4, "four");

    put(5, "five");

}};

Use protected & final where ever possible

Using final does not make much sense from performance point of view as modern JVM is intelligent enough to make use them efficiently, but still I feel they have lot of value in making the intent clear & in multi threading simple with im-mutable state.

Usage of "private" is over-rated & I think it's better to have it has protected making setting the code simpler.

Person p = new Person(){{

age=30; name="Suresh";

}}

where age & name are protected variables.

More & more you start using inner classes, you will get frustrate with java for lack of "closures" & will eventually move to Groovy or Scala (Like me :-))

Use Null Object Pattern & throw IllegalArguementException wherever applicable

The Null Object provides intelligent do nothing behaviour that helps to avoid problems with Null references (NPEs) & so called "A billion dollar irreversible mistake"

Prefer Unchecked Exception over Checked Exception

90% of time Unchecked Exception makes more sense than the Checked Exception. This is one of the most significant attribute of successful libraries like Spring & Hibernate. Use Checked Exception for recoverable errors (Business Exception) & Unchecked for remaining stuff.

Understand Soft, Weak and Phantom references in Java

WeakReference is a reference which doesn't have enough force to prevent Garbage Collector(GC) deleting object - Useful for caching
Soft Reference behaves like weak references, except when GC determines object is softly reachable & can be used to avoid OutOfMemoryError.
Phantom references are enqueued when objects are deleted from memory and get()method always returns null to prevent resurrecting object. So phantom references are good for determining exactly when object is deleted from memory.
__________________
Interesting quotes from Rock star programmers that I collected.

"Write lots of code. Have fun with it!" — Joshua Bloch

"Learn to use your tools. And I don't mean just enough to get by. I mean really learn how to use your tools." — Tor Norbye

"Don't use line numbers. Don't put your entire application in one method." — Chet Haase

"Don't be overwhelmed by the language or the platform." — Raghavan Srinivas (In the same line Neal Ford says "When you were hired by your current employer, you may think it's because of your winning personality, your dazzling smile, or your encyclopedic knowledge of Java. But it's not. You were hired for your ability to sit and concentrate for long periods of time to solve problems")

"Millions of people have been employed because someone at Sun Microsystems invented Java." - Masood Mortazavi. Master it & you will never regret for that

"There will always be opportunities for great engineers, but as I said earlier, I think the number of these opportunities will shrink as other, less technical personnel play larger roles in the software-development process, using more productive, higher-level tools and frameworks than we have used in the past." - Ben Galbraith

"Google makes finding information easier than ever, but nothing beats interacting with an expert."- Ben Galbraith. So always associate with people from whom you think you can learn.

From Pragmatic Thinking & Learning

There is no expertise without experience.It takes something on the order of ten years/ 10,000 hours
of practice to be expert in a field Deliberate, thoughtful practice is what makes the difference—not
just going through the motions.Practice doesn't make perfect, but it does make permanent:
neuroplasticity will cause your brain to re-wire itself according to what you do.You may not become
what you dream, or what you aspire to be, but you will become what you do.Unfortunately there is no substitute for hard work.

References:

http://java.sun.com/developer/technicalArticles/Interviews/studentdevs/index.html?intcmp=2225

http://www.agiledeveloper.com/blog/PermaLink.aspx?guid=8a745e85-2a34-4d9c-8c25-ca371530e281 - How to convince your fellow developer to write short methods?

http://weblog.raganwald.com/2007/04/rails-style-creators-in-java-or-how-i.html - Rails style initializers

http://www.refactory.org/s/double_brace_initialisation/view/latest

http://c2.com/cgi/wiki?DoubleBraceInitialization

http://bwinterberg.blogspot.com/2009/09/introduction-to-google-collections.html

http://thestrangeloop.com/sessions/ghost-virtual-machine-reference-references

http://juixe.com/techknow/index.php/2009/11/18/favorite-programming-quotes-2009/

http://www.readwriteweb.com/archives/top_10_software_engineer_traits.php

2009-06-26T00:03:00.001-07:00

ALL ABOUT UNIT TESTING

"First about Boring Theory"

"Unit Test is the smallest piece of testable part of an application" In computer programming, unit testing is a software verification and validation method where the programmer gains confidence those individual units of source code is fit for use. A unit is the smallest testable part of an application. The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect. Each unit is tested separately before integrating them into modules to test the interfaces between modules. Unit testing has proven its value in that a large percentage of defects are identified during its use.

"In Java Unit Test cases means JUnit test cases, the single most importance of Spring & Guice (or any dependency injection framework) is to make unit testing easier"

JUnit is the de-facto framework for unit testing in java world. JUnit is a simple library, although there are mock objects, test code generators, behavioral test design & many other tools based on dynamic languages JUnit remains viable option while testing libraries or API where developer is a end user. Bob Lee (Author of Guice) stresses on the point that single most importance of dependency injection framework or interface driven design for matter is easier testability. All Google great applications like Gmail, Google Adsense, Calendar are the great testimony of this fact.

"Developers don't like writing unit test cases; Management needs to understand the technical debt associated with un-availability of test cases"

Let's face it, developers don't like writing unit tests & write documentation. Kent says, Software, like golf, is both a long and short game. JUnit is an example of a long game project – lots of users, stable revenue, where the key goal is to just stay ahead of the needs of the users. So it's hard to sell writing JUnit cases for small projects that don't have longer life. It's clearly avoidable overhead in such cases (Most of the web applications). It may not economically make sense to write extensive test cases for short lived & small applications.

In some cases developers hate to be embarrassed & look stupid when someone finds a mistake or highly technical guys think they don't need to write test their solid code. The first case can be handled through management as it's a purely competence issue which can be sorted out through training & other means, in second case it's hard to convince as these guys very much correct in their assertions in their own way. It's an attitude problem, the best way would to be deploy someone to write unit test cases. A "high level of quality code" is great, yet most software lives on and on and people expect to add/modify features in that software or debug it, it's economic requirement that super stars need to prove that their code works with unit test cases. How much will the maintenance costs are without unit tests? How much more risk does that add?

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" -- Brian Kernighan

"Unit testing seems to a lot of managers and developers like pure overhead, but professionally responsible developers know that it is one of the keys to quality." - Neal Ford

"Solid test cases with 100% coverage provides the courage to refactor the code & the reduces the testing effort in future"

We need to educate ourselves it's economically makes sense to have solid test cases especially for pure library (API)providers, as cost associated testing & testers can be completely (almost) avoided as anyway applications will test the APIs & we can avoid duplicating of the testing effort of testers in testing the APIs. Unfortunately these long sighted approaches are difficult to sell to the management and it's become thankless job when the unit tested solid code can't be differentiated with the working code. Developing APIs is a marathon job & not a 100 meters race. Stamina & perseverance plays very important role. In most of the cases committed can't be taken back, I guess JDK deprecated APIs must be haunting, humiliating the initial designers. One of the benefits of JUnit test cases are that developers get first had experience of the developers using the same.

As time goes on, there will be cases where the code works a bit less, some minor bugs, and some dirty quick fix (or hack) happens. Since you don't want to touch that code, you'll put fixes/enhancements/workarounds in other parts of the code, slightly but constantly degrading the quality of your design. You won't even upgrade a depending library, since you can't easily run regression tests over it. In a shorter time than you expect, that good designed and implemented project will turn into a nightmare. So it's just not about changing code, it's about changing environment - RDBMS vendor, JDK version, Library versions, OS versions…

"JUnit can be used to write End to End functional Tests & Unit test cases needs to be reviewed"

From definition, a test is not a unit test if:

It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.

If we go by above principle 90% of our JUnit test cases don't pass above rules. Although POJO driven frameworks like Spring tries to solve it, I don't think we can use JUnit in it's pure form. It's ok to use the JUnit for functional and integration test cases as well.

"Test Before Code, or perhaps Test Before Design." – That means unit test cases need to be reviewed before the design or coding. These reviews probably should be more thorough than the code reviews itself.

"JUnit test cases can serve as great tool to document"

Probably writing documentation through Javadocs is a bad idea. Usage of verbose class names, method names & JUnit test cases is more scalable & efficient way of documenting classes. Communicating through the code is the best way communication.

"JUnit test cases have to be efficient & succinct"

Manual test hurts both economic wise & manageability wise. But by end of the day if test cases are not capturing the correct scenarios & worst part if we have repetitive test it really JUnit really doesn't help. Best of the people involved with software development needs to do this type of unit testing. Garbage in & Garbage out rule is perfectly applicable here.

I am done with all my legal points to sell unit testing? Do you guys buy this argument? J

My next topic on unit testing would be on patterns and anti-patterns while writing test cases.

Resource:

http://www.artima.com/weblogs/viewpost.jsp?thread=126923

http://www.theserverside.com/news/thread.tss?thread_id=51615

http://www.junit.org

http://c2.com/cgi/wiki?WhoIsUsingJunit

http://www.theserverside.com/news/thread.tss?thread_id=51615

http://c2.com/cgi/wiki?FunctionalTest

http://www.davenicolette.net/articles/functional_tdd.html

http://www.logigear.com/newsletter/api_vs_unit.asp

http://www.exubero.com/junit/antipatterns.html

http://www.infoq.com/news/2009/06/test-or-not

http://www.agitar.com/solutions/why_unit_testing.html

2009-04-24T12:27:00.000-07:00

Search Domain Basics : Search is probably the most pervasive technology domain of this century. Here I have tried to cover some basic concepts & some software implementation details with java as the focus. I thought this information can help new comers to this domain & provides enough starting pointers to dig into more details.

"Content is King" - Content is what that drives the web & "search" is the engine for that. All the services providing the content has to employ search techniques to fetch the required information with minimum possible user inputs in fastest possible way. Google/Yahoo which have become synonyms with search is implementing all the applications in relation with search one way or the other. Apparently unstructured content forms the major portion of the content available in the universe. Unstructured data (or unstructured information) refers to (usually) computerized information that either does not have a data model or has one that is not easily usable by a computer program or in simple words any data that is not represented in terms of column names in RDBMS table schema. Parallel computing, Data sharding, Schema definition, Scale of data & nature of the content acquisition related with un-structured content makes it unsuitable to be solved from 100% RDBMS solution, although full text indexing does exist in the RDBMS world.

Raw data with context is called 'Information' & Information search and retrieval is all about locating relevant material from collection of raw data in a fastest way taking minimum possible input from the users. The ability to aid and assist a user in finding relevant information is the primary goal of information engineers & information Retrieval (IR) libraries.

Major parts for search engine:

fetching/Loading the document – downloading content (lists of pages) that have been referenced.

analysis – analyzing the database to assign a priority scores to pages (PageRank) and to prioritize fetching.

indexing – combines content from the fetcher, incoming information from the built-in data source, and link analysis scores into a data structure that’s quickly reachable usually using cache services.

searching – returns set of content that ranks pages against a query using an index.

database – keeping track of what documents with various context information helping the ranking better.

To scale to billions of documents, all of these must be distributable, i.e., each must be able to run in parallel on multiple machines. This should happen by throwing more hardware into the pool, without massive reconfiguration whenever scale up is required.As we cannot offer to have failure of any single component cause a major hiccups; a search solution must be able to easily scale by throwing more hardware into the pool, without massive reconfiguration; and things should largely fix themselves without human intervention. This can only be possible with the stateless implementation of software services.

Search Engine with Java for unstructured content: Studying the Information library APIs gives better insights into what search engine is capable of providing the service & I am taking the lucence as the sample for that.

Lucene library has become defacto IR library in java world, now lucene has been ported to almost all the major languages showcasing the popularity & capability of this small library.

Lucene is a search and retrieval library providing key word and fielded search. It can use boolean AND, OR, and NOT to formulate complex queries & can use fuzzy logic that is useful when searching text created by optical character recognition. Un-structured content far exceeds the structured content in the web world. Lucene mainly deals unstructured content & can effectively search structured content also with the field tags.Lucene provides minimum required information retrieval functionality. We can call this a "SearchKernal" library that provides full-text search and indexing functionality. Instead of an out of the box application, Lucene offers a usable API for programmers and operates on a lower level.There are off the shelf libraries (Compass, Nutch, Solr...) providing monitoring, transaction utilities over lucene. Commercial enterprise search offerings include from vendors such as, Autonomy, Google, Oracle & FAST (MS).Lucene does not search file by file. The search space is analysed first and translated into a normalised representation - the index. Lucene uses a reverse index. All words in the index are unique, that means the index is a compressed representation of the search space. Lucene only supports plain-text files. However, a variety of free open source document parsers are available for document types such as, RTF, PDF, HTML, XML, Word etc. Depending on the nature of the text content various analysers are on offer. For example, text can be analysed with a white space analyzer which breaks down the text in tokens separated by white space. To keep the response time short the process of generating and optimising the index is separated. The index gets normalized by applying a stemming and lemmatisation algorithms. Lucene beats the RDBMS in full text search in terms of processing speed, manageable reduced the size of the index footprint (now about 25 percent of the source documents’ size), easy incremental updates, support for index partitions, price & flexibility (index methodology, deploy options & schema evolution). RDBMS way of searching with where clause & LIKE % is not only scalable but ineffecient, although RDBMS like Oracle includes full text indexing capabilities they have not been as popular to independent solutions such as Lucene & also it's not easy to implement the parallel processing (map/reduce) invloving multiple machines & terabytes of data. It's the dynamic nature & context understanding nature of search bringing huge changes search domain with navigating the hierarchy is becoming old fashioned. With technical problems pretty solved in this domain, now the key thing to remember is "search methodology is much more important than the underlying technology".

Search Terminologies:

Proximity search:A search where users to specify that documents returned should have the words near each other.

Concept Search: A search for documents related conceptually to a word, rather than specifically containing the word itself. Involves parallel computing.

Boolean search: A search allowing the inclusion or exclusion of documents containing certain words through the use of operators such as AND, NOT and OR.

Proximity search: A search where users to specify that documents returned should have the words near each other.

Stemming: The ability for a search to include the "stem" of words. For example, stemming allows a user to enter "running" and get back results also for the stem word "run."

Lemmatisation: is the process of grouping together the different inflected forms of a word so they can be analysed as a single item.Lemmatisation is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech.

Noise or Stop words:Conjunctions, prepositions and articles and other words such as AND, TO and A that appear often in documents yet alone may contain little meaning.

Thesaurus: A list of synonyms a search engine can use to find matches for particular words if the words themselves don't appear in documents.

Index: Normailzed presentation of words

Semantic Search: is a process used to improve online searching by using data from semantic networks to disambiguate queries and web text in order to generate more relevant results.

Web Search : Content is public & Generic.Uses keywords, Links (relevency) based some kind of historic traffic.

Enterprise Search : Also contains private documents that domian specific, Quality of content should be highest quality content & not necessarily popular. Information/metadata needs to be secure with role based access to the content.It has to support security (Realms, Roles), SLAs and many other requirements. Google & Yahoo do not provide enterprise search.

As of now I am interested in researching this topic in the search retrieval field, So will keep updating this blog with my research findings.

"Automatic annotation/Summary addition for content":Lengthy documents/text are boring to read. It will be great if someone or computer can automatically creates the gist of the content. Automatic creation of annotation is tough task especially for non-domain specific topics but can be predictable in domain specific cases. For example it might be easier to extract the information from judgments copy automatically (at least in routine cases that hardly requires special knowledge from legal experts to annotate) or through workflow for review with automatically annotating the content/documents. I see this as an interesting area.

Summary:

IR & Search domain is pretty complex subject requiring mastery over algorithms & data structures. Hope that I have been able assimilate the information related to search taking "lucene" as sample search engine library.

References:

Lucene Book : http://www.manning.com/hatcher3/

Google Search:http://en.wikipedia.org/wiki/Google_search

Useful wrapper libraries over Lucene.

http://lucene.apache.org/solr/features.html - Solr Features

http://www.compass-project.org/overview.html - Compass Features

2009-04-17T05:23:00.000-07:00

Some statistics about Java source code of popular open source libraries. Currently I am looking/learning a big system that has multi-million number of source code. I wrote a simple utility to extract information about the java source code just for fun. I ran this utility on many of "src" directory of open source libraries as well.This java utility takes a source code directory as inputs & traverses all the java code recursively inside that directory. It collects total number of active lines of code excluding comments, package count...

& here is the result.

********* JDK1.5 ********

Total # Lines = 850918

Total # of Files = 6556

Avg # Lines per file = 129

Total # of packages = 368

******** Hadoop 0.18.3 ******

Total # Lines = 129742

Total # of Files = 926

Avg # Lines per file = 140

Total # of packages = 66

******* Lucene 2.4.1 *********

Total # Lines = 69606

Total # of Files = 528

Avg # Lines per file = 131

Total # of packages = 16

******** Struts *********

Total # Lines = 63707

Total # of Files = 1040

Avg # Lines per file = 61

Total # of packages = 120

********* iText 2.1.5 ***********

Total # Lines = 96830

Total # of Files = 544

Avg # Lines per file = 177

Total # of packages = 55

******* Tapestry 5.1.0.3 ******

Total # Lines = 96395

Total # of Files = 1937

Avg # Lines per file = 49

Total # of packages = 103

It's not all surprising that one of the best prolific java coder comes best ("Howard") in the java world when it it comes to modularity.

********* iBatis *********

Total # Lines = 14132

Total # of Files = 202

Avg # Lines per file = 69

Total # of packages = 45

***** Hibernate 3.3.1.GA *******

Total # Lines = 173698

Total # of Files = 2102

Avg # Lines per file = 82

Total # of packages = 292

I guess these figures can be considered as standard while reviewing the modularity of any library.Do let me know if any one interested in code & having ANT target to analyze the

source code b/w releases.

2009-03-10T07:24:00.001-07:00

Interview with one of the best API designer in the world -Joshua Bloch.

Adding some other interesting wise quotes on API design that I have read...

When you design user interfaces, it's a good idea to keep two principles in mind:

Users don't have the manual, and if they did, they wouldn't read it.In fact, users can't read

anything, and if they could, they wouldn't want to.

Same rule applies for API designers as well:

Developers don't have the java docs, and if they did, they wouldn't read it.In fact, Developers can't read anything other than pressing "." against the object reference in IDE & wait for something to select, and if they could, they wouldn't want to.

Learnability,Effeciency,Memorability,Errors & Satisfaction remains the core of good interface design.

APIs should emerge from the needs of real applications, and that they should make common tasks super-easy as the demand for quality, validated designs far exceeds our capacity to create them.In 1996 it wasn't clear we could create a sufficiently fast language without primitive types and arrays, It wasn't clear how much boilerplate code would be required by anonymous callback classes or checked exception. So Java couldn't resist including primitive types,excluding closures in favour of anonymous classes & over-using checked exceptions.

Grady Booch->

"Great thing about objects is that they can be replaced". The great thing about Spring is it helps you replace them. Flexibility is much more important than the re-use.

Joshua Bloch says,

Public APIs are forever - one chance to get it right
Good code is modular–each module has an API
Thinking in terms of APIs improves code quality
Easy to use, even without documentation
Don’t let implementation details “leak” into API
Make classes and members as private as possible
Make variables "final" wherever possible

Now some economics:

Economic reality suggest that buying more memory can be easier and cheaper than to pay someone to debug code & it pays more in the long run to have understandable slow code than super fast cryptic code.

Once I heard from Architect in IBM conference (don;t remember the name) for choosing Java over C++ saying this,

"The most compelling reason for adopting Java over C++ is automatic memory management. It protects application from mediocre programmers. It eliminates many embarrassments of memory leaks & crash that randomly occur in production". He went on say that "So as a result we are trading with inexplicable crashes for slow performance (automatic memory management & database-centric storage).This makes sure that application at least works anyway"

"Rushing is at the root of all lack of quality" - Peter Calthorpe, architect

Reference:

Design Slides
Defensive Programming

Spring is Good

2009-03-09T13:48:00.001-07:00

javaiq.in - A playground for learning Java RIAs. I am planning to expose my experimental applications with GWT, JavaFx & Felx. I will also be creating applications mixing multiple open services from various vendors (Google,Yahoo, Amazon, eBay...) & create a combined value. I guess this is the area that has huge scope.

I created this site & made it public 2 months back. It was result of my experimenting with new techniques with sample applications/code snippets,I always felt that best way to sell any new technique is with real application. It's great to have useful apps while learning new technologies/framework,I hate to write throw away examples. I have been following web frameworks from past few years, I have spent large amount of time in validating/learning over the year. I am SWING programmer & was always trying out the samples from Internet/books. Learning (or Stealing) the good code snippets that I liked & thought of exposing them as useful apps. Having spent a considerable amount of time and energy with Swing, it was queasy feeling in my stomach to see all my Swing programming heroes (Like Chet, Romain Guy...) either have gone to Adobe/Google or have moved to JavaFx/Flex way. I have also lost faith in Swing. I don't expect to develop new pure Swing apps any more, but I guess I was able to grasp GWT, Echo, Wicket frameworks much better than any pure web MVC framework (Struts, Spring MVC...) developers who were not having exposure to swing. I guess that's the advantage still I can leverage.

Chasing entrepreneurship dream- "When you are not into inheriting money and because you do not belong to a rich family and when you aren’t an individual blessed with the talent of a sportsperson or an actor, the next best thing to do is to become an entrepreneur" - Anand Morzaria

A man is a success if he gets up in the morning and gets to bed at night, and in between he does what he wants to do - Bob Dylan

Now experts say it has become relatively cheaper to start a new web application or a startup. Moore's law has made hardware cheap; open source has made software free; the web has made marketing and distribution free; and more powerful programming languages & techniques are making development teams smaller & powerful. But actual life it is not that simpler, especially if you are fighting a lone battle.

It's really pretty time consuming process to create & manage applications. I went with cheap shared Java hosting which was unable to run my "stripes" web framework based application & no one was there to help me out by providing tomcat logs & I had to settle with few JSPs with trimmed version. It's pretty costly affair to run java apps in a proper manner. It's big road blocker if anybody wants to develop/expose web applications in java today. Sun has to fix this problem (shared hosting) if it wants to replace PHP/Ruby On Rails apps in small scale applications. (I will have separate notes with java hosting) Enthusiasm to code is very hard to maintain, actually I wrote most of the applications in a few days, but was difficult to keep the momentum. Perseverance, relentless resourcefulness (as paul graham calls it) is difficult to achieve without full time dedication. Anyway it was good exercise to know all these limitations.

If I look back, I seriously doubt my spending of countless hours in browsing blogs, learning 10s of java web frameworks has significant impact on carrying out my day today work or think better. In fact my guess is I have wasted >30% of total time spent in gleaning through useless/marketing literature links. Hopefully from now on I will be able channelize all my energy in building better context for my day today work rather than learning all java web frameworks under the Sun. BTW all my web framework heroes (Wicket, Appfuse, Stripes, Tapestry...) that I have been following from past one year are apparently looking for jobs themselves & are investing time in using Groovy, Scala & Clojure. :-( not really a happy situation. I guess innovation with pure java in this space has reached dead end. Now I want to to learn/invest more on my current work so that I can become better at what I am doing currently, that's my priority #1 & I will resist experimenting with what blog writers say on latest technologies. One of the trouble I had disclosing this site was public embarrassment! of people knowing that I created this half baked application that I hesitated to own. But now it's quite decent set of useful applications that I started as my week-end project & I have good ideas to make it still better & this will be always my low priority work & also make sure to extract the best out of I am currently working on.Oh! Now I can call myself CTO of javaiq. nice feeling. :-)

2009-03-06T00:02:00.000-08:00

Popular techie words: we hear below words a lot in discussion & blogs. I thought it's good to have their definitions (easier ones) & explain if anyone asks "what's that?" as I also use them to have buzzwords compliance.

Technical Debt:
Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.
The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.
Reference:
http://www.c2.com/cgi/wiki?TechnicalDebt
http://martinfowler.com/bliki/TechnicalDebt.html
http://www.youtube.com/watch?v=pqeJFYwnkjE

MapReduce:
MapReduce is hierarchical scatter/gather operation.
MapReduce is a library that lets you adopt a particular, stylized way of programming that's easy to split among a bunch of machines. The basic idea is that you divide the job into two parts: a Map, and a Reduce. Map basically takes the problem, splits it into sub-parts, and sends the sub-parts to different machines - so all the pieces run at the same time. Reduce takes the results from the sub-parts and combines them back together to get a single answer.
Reference:
http://scienceblogs.com/goodmath/2008/01/databases_are_hammers_mapreduc.php

Cloud Computing:
With cloud computing, everything is web-based instead of being desktop-based; access all programs and documents from any computer that’s connected to the Internet is possible. Cloud computing helps to do it more easily than ever before.

Wikipedia Cloud Computing is "a style of computing in which resources are provided as a service over the internet". Cloud computing user need not have worry about managing a machine or service at the physical level (Machine & location). Amazon Simple DB is an example for this as it handled operating system or database maintenance functions, SLA & operational issues.

Sharding:
Sharding or horizontal partitioning is about splitting up data sets. If data doesn't fit on one machine then split it up into pieces, each piece is called a shard.
Sharding is used when you have too much data to fit in one single relational database.

Reference:
http://highscalability.com/sharding-hibernate-way
http://highscalability.com/unorthodox-approach-database-design-coming-shard
http://lethargy.org/~jesus/archives/95-Partitioning-vs.-Federation-vs.-Sharding.html

Refactoring:
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations

Reference:
http://www.refactoring.com/
http://en.wikipedia.org/wiki/Code_refactoring

Code Smell:
In computer programming, code smell is any symptom in the source code of a program that possibly indicates a deeper problem.
Reference:
http://en.wikipedia.org/wiki/Code_smell

2009-02-28T10:51:00.001-08:00

DHH - The best talk I heard on startups

I was quite impressed by DHH (Creator of Ruby On Rails framework) speech on "The secret to making money online" when I viewed this video again today.

There are just 3 steps for this.
1. Create a gr8 product that people like & is useful to them
2. Have a price for the product & ask people to pay for it or it's usage.
3. Make profits!

This is so ridiculously simple grand old rule! he goes on to say that you don't need to be f***ing genius to achieve this.

He also came up with nice probability analysis that strike rate of success following above simple model is much higher rather than Yahoo, Google,facebook & youtube (setting up BillBoards/hoardings) way of attracting users providing free services & make use of web page real estate to show ads & make money. He also suggest to go slow on implementing the idea as "Finding a good cause is incredibly hard & time consuming" - Craign Newmark.

Simply superb.

I have been following many startup blogs including paul Graham's, but never impressed like this before, so simple words but telling you the hard truth :-)

2008-11-20T10:24:00.000-08:00

Nice way to tell what all you know with wordle

2008-11-11T04:52:00.000-08:00

Do we really need a web service api for serving static data when we already have a good query and mining language? (SQL, XML)

Google, Amazon & eBay provide the static data by publishing through their docs.
- Location information (City, Country...)
- Categories
- Currency data
- Hierarchy information

These are the advantages that I see from the above approach.
- Uses local CPU power, saves unnecessary traffic to the web service
- Supports online/offline scenarios
- Supports programming logic flow, since we will have the information at the compile time it's easy to code
- Solves the performance problem, what ever be the improvement network calls are always costly compared to local calls
- data model is essentially a flat data model, easy to import,save & navigate

But I guess they should be better exposed as XML rather than as HTML

The problems I see with this approach
- We need to have stringent way of updating the local cache of static code (usually stored in SQL or XML or as text file with some delimiters)
- static information that are easy to represent in code (say number is less than 10 & unlikely to change) are represented as Enums , so we will be 2 rules, some will be represented with static enums & some will be represented with static code
- There will be different rules for representing the data & we will be sacrificing the strong typing which I think is not a issue as benefits to the both parties (client & server) exceeds the pain with String based APIs.

As from the mature real SOAP based web services, it's clear that we don't need to expose API StaticCodeService() to get these static information. The demand for quality, correct designs far exceeds our capacity to create them. Web services should emerge from the needs of real applications, and that they should make common tasks ridiculously easy.

In Summary,
It's not worth wasting SOAP web service API calls for getting static information.

2008-11-06T20:58:00.000-08:00

Why Fluent Interface? Here are some thoughts on using fluent interface while designing APIs & their impact
Definition : In software engineering, a fluent interface (as first coined by Eric Evans and Martin Fowler) is an object oriented construct that defines a behavior capable of relaying the instruction 'context' of a subsequent call.

First philosophy,

The success of Conversation/Communication/Integration depends on the quality of signals between two entities involved & the signal is clean & is understood well. Communication is always based on context, shared context. Context also influences interpretation. For example , "use a fork" differently depending on context (Unix, dining table, Java or Ant build file). Well understood shared context improves the signal-to-noise ratio in communication it makes communication effective, expressive, easy to understand and easy to work with.
Domain Specific languages (DSLs) have implicit context, context is never mentioned, once the context is established repeating it again & again is a noise,. The end goal for involved parties in communication should be to reduce the noise to zero.
We have 2 types of DSL, Internal - based on exiisting language & external - A new language with parser & full fledged grammar, evidently external DSL is tough to implement & very effective (SQL,HTML) on the other hand Internal DSL is easy to implement especially with dynamic languages.

That's all about the boring philosophy...

The benefits,

No need to document the APIs.
- They are all self explanatory
- Examples should do all the explanation if any
- Testing/adaptability becomes easier

Less effort required to use, resulting in better economics
- There is very less chance to make mistakes
- All the extra noise is hidden which does not add domain concepts
- User has to write less & less number of lines of code, LOC most
of the time translate to number of bugs. Less the the code less the bugs

Correctly written fluent API gives the satisfaction of well written novel to
the author & the same reading experience to the consumer.

Market is always right, now all the new libraries are coming up using fluent APIs
Big software houses are putting more efforts to make code less noisy. The trend from Microsoft (C#3, IronRuby, F#...) Sun (JRuby & other dynamic language support) points towards that.

Experts think Fluent APIs are cool.
- Martin Fowler is writing a book on DSL
- Well proven frameworks like JUnit are coming up with fluent API alternatives
- Google collections, JMock, Fest, Guice... countless popular APIs are based
on fluent APIs
- Ruby on Rails Active Record is best example how a DSL can simplify the job & has successfully forced to think differently
- Market Signals shows that, best brains are talking more about DSL, functional
langauges & is definately are the way go about to develop software.
- Joshua Bloch (Google Java Architect) in his new Effective Java book talks about Builder pattern for building immutable fluent java objects.
buying more memory can be easier and cheaper than to pay someone to understand/debug code. Well written code (read fluent interface) results in good economics.

Now the real world code using Fluent APIs,

No marketing is better than showcasing real working code for technique or technology.

New Java Date APIs JSR 130:
Period thePeriod = Periods.periodBuilder().years(8).months(3).build();
This is what experts feel how the code should be written & this code is definitely looks better than java.util.Calendar,java.util.Date APIs.

Fluent way of handling XML marshalling un-marshalling:

Here we have sample example showcasing the XML usage with fluent APIs

<contacts>
<contact>
<name>praveenm</name>
<phone type="mobile">98862342333</phone>
<phone type="office">080-2344234233</phone>
<email>pm@aol.com</email>
</contact>
</contacts>

Groovy Sample:
def mkp = new MarkupBuilder()
mkp.contacts {
contact {
name("praveenm")
phone(type: "mobile", "98862342333")
phone(type:"office", "080-2344234233")
email("pm@aol.com")
}
}
Ruby Sample
require 'builder'
x = Builder::XmlMarkup.new(:target => $stdout, :indent => 2)
x.contacts {
x.contact {
x.name('praveenm')
x.phone '98862342333', :type => 'mobile'
x.phone '080-2344234233', :type => 'office'
x.email 'pm@aol.com'
}
}

Google Collections:
A excelellent example how JDK Collections can be simplified with fluent APIs
public static final ImmutableSet FAVORITE_NUMBERS
= ImmutableSet.of(2,9, 8, 15, 16, 50);

FEST Example: DSL-oriented API for functional Swing GUI testing

dialog.comboBox("domain").select("Users");
dialog.textBox("username").enterText("alex.ruiz");
dialog.button("ok").click();
dialog.optionPane().requireErrorMessage()
.requireMessage("Please enter your password");

JaxB common:

USAddress address = new USAddress()
.setName(name)
.setStreet(street)
.setCity(city)
.setState(state)
.setZip(new BigDecimal(zip));

I guess this is not a good example for fluent.

Guice:
@Override
protected void configure() {
binder().bind(IUserService.class).to(UserServiceMockImpl.class);
binder().bind(AuditInfo.class).to(DummyAuditInfo.class);
}

Hibernate:
List cats = session.createCriteria(Cat.class).setMaxResults(50).list()

DesignGridLayout:
A Fluent Layout manager

layout.row().label(label("Last Name")).add(lastNameField, 2).add(label("First Name")).add(firstNameField, 2);
layout.row().label(label("Phone"))
.add(phoneField, 2).add(label("Email")).add(emailField, 2);
layout.row().label(label("Address 1")).add(address1Field);
layout.row().label(label("Address 2")).add(address2Field);

Frameworks like Grails, JMock, JPA utilizes the fluent APIs. We also have some samples in standard JDK itself like StringBuffer, StringBuilder, ProcessBuilder etc...

These are the commonly seen techniques in Fluent APIs with Java.
- Method Chaining
- Nested Interfaces
- Builder Pattern
- static imports

And finally the problems with fluent APIs,

It's definitely not the 'wow' technique. People have been using this for a quite long time & now we have just fancy name 'Fluent API' that's it. So let's not think this is like OOP or OOAD or MDA. It's a simple programming technique for making code look better.

- DSL is about writing good essay. Programmers are not essayists
- Java is not suitable for DSL- It doesn't have closures, open classes , quite verbose & is filled badly written APIs.
- It's very difficult to get correct in developing DSLs.Fluent API can be useful for highly used API,otherwise the investment of developing may not be as effective as intended. Thus, not all API can be made fluent.
- Difficult to track down null return value issues that occur somewhere in the chain
- Difficult to handle the exceptions, especially while dealing with existing APIs having checked exceptions.
- One of my 'java friend' was not ready to believe some of the fluent samples that I showed him were actaully a java code, :-), so there is also -ve impact on readability
- Need to write more code to make code fluent.

Summary:
I strongly believe that DSL, both internal & external helps in developing better software that is easy to learn,extend, use & hard to misuse.

That's it, Hope that I was able sell fluent APIs to the new guys.

References:
Domain Specific Language Book by Martin Fowler
Domain Specific Language by Martin Fowler
DSL Boundary by Martin Fowler

2008-10-17T15:08:00.000-07:00

I am currently working on building a fluent interface over soap toolkits. I did discussed the possibilities of rich APIs value addition. One of the problem with fluent APIs is that we have to get rid of checked exception to make it easier, but there are cases where fault containment is necessary. I am just thinking the better appraoach would be to generate the unchecked Exception counterparts for each of the fault code & provide the easier error handling.

For Example: (I have modified sample shown by the Martin Fowler), Let us assume that newOrder process will talk to 2 other web services & there is possibilities of 2 soapfault errors.

import static test.Customer.newOrder;

private void orderNew(){
newOrder() .
with(6, "TAL") .with(5, "HPK").skippable()
.withDollar(35.d) .withAccountId(23423l).withLicense("AS900980") .
.with(3, "LGV")
.priorityRush();
}
@Test
public void test(){
try{
orderNew();
}catch(GenricException exp){
switch(exp.getErrorCode()){
case SOAPFaultsCodes.E1001:
// handle the exception possibly giving more useful message or some other recovery action
case SOAPFaultsCodes.7003:
}
throw new RuntimeException("Unhandled Error");
}
}

Here again GenericExcption is abstract Exception extedning RuntimeException that is implemented by all the soap fault exception correspding to the errorcode & will be thrown by the rich APIs.

I guess with JDK7 I guess we can have String based switch() & upgraded catch block.

Well, for unit testing exception we can laverege the annotations support to assert with JUnit4
@Test(expected = E7069Exception.class)

I wanted to generate all these Exceptions automatically from soap faults document, For that I wrote this groovy script. (show casing the usage of multiline string, closure & file I/O)

constantsFile=""
converFileLineIntoException = {
sarray = it.split(" ")
exception = sarray[0]+"Exception"
errorCode = exception.substring(1,sarray[0].length())
constantsFile+="\n public static final int E$errorCode = $errorCode;"
className =
"""package com.yahoo.sm.ws.builders.exception;
// Generated code
public class $exception extends GenericException {

private String description;
private String shortDescription;

public $exception(String description, String shortDescription){
this.description=description;
this.shortDescription=shortDescription;
}

public int errorCode(){
return $errorCode;
}

public String getDescription(){
return this.description;
}

public String shortDescription(){
return this.shortDescription;
}
}
"""
new File(sarray[0]+"Exception.java").write(className)
}

def lines = new File("soapFaultList.txt").eachLine(converFileLineIntoException)
soapFaults = """
package com.yahoo.sm.ws.builders.exception;
public interface SOAPFaultsCodes {
$constantsFile
}"""
new File("SOAPFaultsCodes.java").write(soapFaults)
println "--- Gr8 I am done "

Groovy, JUnit4 & static imports just rocks :-)

2008-10-12T10:39:00.000-07:00

Dependency Injection with Guice, JUnit 4 & Get-Set problem:
I was evaluating Guice,JUnit-4,while trying out with these same I wrote a sample application using the same. & also tried out a possible solution for the problem with data conversion.
In a typical web application we also use heavily with get/set to convert from UI representation of object to back end implementation object. Many a times (In my experience most of the time) they are all heavy parallel structures required by framework (Like classic struts forces UI object to extend & ActionFormBean, limited data type support) or data type representation in the different UI models. With the advent of domain driven design we have more rich domain objects which usually contains many other data/behaviour which should not be or need not be exposed to the UI layer.
The problem with this conversion is that they are not only look dumb consuming lot of source code lines but they are error-prone & since there is no contract with back end we are forced to unit test the code as we cannot rely on compile time checks. I also tried a possible solution for this, 3 years back I tried out implementing this in a classic struts based web application & was successful in reducing the number of bugs.

The sample application is user management system. (Please note that the code has been written in such a way to show the usage Guice, JUnit & type safe data conversion & shouldn't be mistaken as real time design,there are no exceptions, validation etc...). It follows a typical MVC pattern followed in web applications.

Model Layer

User.java Represents a User domain object. As we notice it also contains the information that is populated through context (Logged in user, context, date, etc... usually through HttpSession or EJBContext) in the form of AuditInfo object. Here the fullName field (which doesn't make sense to UI) representing the firstName & fullName as 'firstName,lastName' in a single field. This is done to show how easily back end object can be refactored to accomadate client requirements. This can be even applied to data types. Since extracting of interface from any class is supported by all the IDEs, there is no coding effort required here.

\src\model\User.java


 1 /*
 2  * To change this template, choose Tools  Templates
 3  * and open the template in the editor.
 4  */
 5
 6 package model;
 7
 8 import controller.IUser;
 9
10 /**
11  *
12  * @author praveenm
13  */
14 public class User implements IUser,java.io.Serializable {
15
16     private String userName;
17     private String eMail;
18     private int age;
19     private String fullName;
20     private static final String DELIMETER=",";
21  
22     private AuditInfo auditInfo = new AuditInfo();
23
24     public String getUserName() {
25         return userName;
26     }
27   
28     public User(){
29       
30     }
31  
32
33     public void setUserName(String userName) {
34         this.userName = userName;
35     }
36
37     public String getEMail() {
38         return eMail;
39     }
40
41     public void setEMail(String eMail) {
42         this.eMail = eMail;
43     }
44
45     public int getAge() {
46         return age;
47     }
48
49     public void setAge(int age) {
50         this.age = age;
51     }
52
53     public String getFirstName() {
54         return fullName.split(DELIMETER)[0];
55     }
56
57     public void setFirstName(String firstName) {
58         if(getLastName()!=null){
59             fullName = firstName+DELIMETER+getLastName();
60         }
61     }
62
63     public String getLastName() {
64         return fullName.split(DELIMETER)[1];
65     }
66
67     public void setLastName(String lastName) {
68         if(getFirstName()!=null){
69             fullName = getFirstName()+DELIMETER+lastName;
70         }
71           
72           
73     }
74     public void setFullName(String str){
75         fullName=str;
76       
77     }
78
79     public AuditInfo getAuditInfo() {
80         return auditInfo;
81     }
82
83     public void setAuditInfo(AuditInfo auditInfo) {
84         this.auditInfo = auditInfo;
85     }
86  
87  
88
89 }

IUserService - Exposing the functionality of user management service

\src\model\IUserService.java


 1 package model;
 2
 3 import ui.IUser;
 4 import java.util.List;
 5
 6 /**
 7  * A simmple service representing the general User management activities.
 8  * @author praveenm
 9  */
10 public interface IUserService {
11
12     User get(String userName);
13     boolean saveOrUpdate(User user);
14     boolean delete(String userName);
15
16     boolean saveOrUpdate(IUser user,AuditInfo info);
17     List&lt;IUser> getUsers();
18 }
19
20

UserServiceMockImpl.java - Implements the service by saving the content in a file using object serialization.

\src\model\AuditInfo.java


 1 /*
 2  * To change this template, choose Tools  Templates
 3  * and open the template in the editor.
 4  */
 5
 6 package model;
 7
 8 import java.io.Serializable;
 9 import java.util.Date;
10
11 /**
12  *
13  * @author praveenm
14  */
15 public class AuditInfo implements Serializable {
16     private String updatedBy;
17     private String createdBy;
18     private Date updatedDate;
19     private Date createdDate;
20     private boolean isAdmin;
21
22     public String getUpdatedBy() {
23         return updatedBy;
24     }
25
26     public void setUpdatedBy(String updatedBy) {
27         this.updatedBy = updatedBy;
28     }
29
30     public String getCreatedBy() {
31         return createdBy;
32     }
33
34     public void setCreatedBy(String createdBy) {
35         this.createdBy = createdBy;
36     }
37
38     public Date getUpdatedDate() {
39         return updatedDate;
40     }
41
42     public void setUpdatedDate(Date updatedDate) {
43         this.updatedDate = updatedDate;
44     }
45
46     public Date getCreatedDate() {
47         return createdDate;
48     }
49
50     public void setCreatedDate(Date createdDate) {
51         this.createdDate = createdDate;
52     }
53     @Override
54     public String toString(){
55         return "updatedBy=["+updatedBy+"] "+
56                 "createdBy=["+createdBy+"] ";
57     }
58
59     public boolean isIsAdmin() {
60         return isAdmin;
61     }
62
63     protected void setIsAdmin(boolean isAdmin) {
64         this.isAdmin = isAdmin;
65     }
66 }
67
68

UI Layer

IUser.java - Now this is the trick we have IUser that's required by UI layer (which is always a subset of back end object) that's being implemented by the UI model object.


 1 package ui;
 2 /**
 3  *
 4  * @author praveenm
 5  */
 6 public interface IUser extends java.io.Serializable {
 7
 8     int getExperience();
 9     String getEMail();
10     String getFirstName();
11     String getLastName();
12     String getUserName();
13     void setExperience(int Experience);
14     void setEMail(String eMail);
15     void setFirstName(String firstName);
16     void setLastName(String lastName);
17     void setUserName(String userName);
18
19
20 }
21
22

UserForm.java - UI bean honouring java bean spec & can extend the classes like ActionFormBean representing the HTML form in the screen.

\src\ui\UserForm.java


 1 package ui;
 2
 3 import ui.IUser;
 4 import java.io.Serializable;
 5
 6 /**
 7  *
 8  * @author praveenm
 9  */
10 public class UserForm implements IUser,Serializable {
11     private String userName;
12     private String eMail;
13     private int experience;
14     private String firstName;
15     private String lastName;
16
17     public String getUserName() {
18         return userName;
19     }
20
21     public void setUserName(String userName) {
22         this.userName = userName;
23     }
24
25     public String getEMail() {
26         return eMail;
27     }
28
29     public void setEMail(String eMail) {
30         this.eMail = eMail;
31     }
32
33     public int getExperience() {
34         return experience;
35     }
36
37     public void setExperience(int experience) {
38         this.experience = experience;
39     }
40
41     public String getFirstName() {
42         return firstName;
43     }
44
45     public void setFirstName(String firstName) {
46         this.firstName = firstName;
47     }
48
49     public String getLastName() {
50         return lastName;
51     }
52
53     public void setLastName(String lastName) {
54         this.lastName = lastName;
55     }
56
57 }
58
59

controller

UserController.java - Controls the logic flow b/w UI & the back end. Here is the place we will introduce the dependency injection. In a typical web application audit information is captured in HttpSession object, since in test envioronment we will not be having references to HttpRequest, HttpResponse etc... we will inject those data with our Guice & also we will inject service implementation which could be changed without disturbing the other layers.

\src\controller\UserController.java


 1 package controller;
 2
 3 import ui.IUser;
 4 import com.google.inject.Inject;
 5 import java.util.List;
 6 import model.AuditInfo;
 7 import model.IUserService;
 8
 9 /**
10  *
11  * @author praveenm
12  */
13 public class UserController {
14
15     private final IUserService service;
16     private AuditInfo info;
17
18     @Inject
19     public UserController(IUserService _service){
20         service=_service;
21         System.out.println("UserController Instaniated ");
22     }
23     @Inject
24     public void setAuditInfo(AuditInfo info){
25         this.info=info;
26     }
27
28     public boolean save(final IUser user){
29         return service.saveOrUpdate(user,info);
30    }
31
32     public IUser get(String userName){
33         return service.get(userName);
34     }
35
36     public boolean delete(String userName){
37         if(!info.isIsAdmin()){
38             throw new IllegalAccessError("You don;t have the permission to delete");
39         }
40         System.out.println("users"+service.getUsers());
41         return service.delete(userName);
42     }
43     public List<IUser> users(){
44         return (List<IUser>)service.getUsers();
45     }
46
47 }
48
49

Unit test layer

MockBinder.java - As I am obsessed with fluent interface, generics & type safety, it's really enjoyable to wire up dependencies using Guice APIs rather than through XML.

\test\MockBinder.java


 1 package test;
 2
 3 import com.google.inject.AbstractModule;
 4 import java.util.Date;
 5 import model.AuditInfo;
 6 import model.IUserService;
 7 import model.UserServiceMockImpl;
 8
 9 /**
10  *
11  * @author praveenm
12  */
13 public class MockBinder extends AbstractModule {
14
15     final boolean isAdmin;
16
17     public MockBinder() {
18         isAdmin = false;
19     }
20
21     public MockBinder(boolean isAdmin) {
22         this.isAdmin = isAdmin;
23     }
24
25     @Override
26     protected void configure() {
27         binder().bind(IUserService.class).to(UserServiceMockImpl.class).asEagerSingleton();
28         if (!isAdmin) {
29             binder().bind(AuditInfo.class).to(DummyAuditInfo2.class);
30         } else {
31             binder().bind(AuditInfo.class).to(DummyAuditInfo.class);
32
33         }
34     }
35 }
36
37 class DummyAuditInfo extends AuditInfo {
38
39     public DummyAuditInfo() {
40         super.setIsAdmin(true);
41         super.setCreatedBy("Praveen M");
42         super.setCreatedDate(new Date());
43         super.setUpdatedBy("Praveen");
44         super.setUpdatedDate(new Date());
45     }
46 }
47
48 class DummyAuditInfo2 extends AuditInfo {
49
50     public DummyAuditInfo2() {
51         super.setIsAdmin(false);
52         super.setCreatedBy("someone");
53         super.setCreatedDate(new Date());
54         super.setUpdatedBy("someone");
55         super.setUpdatedDate(new Date());
56     }
57 }
58
59
60

UserTest.java - & now finally we have unit testing code with JUnit4 test case show-casing the usage of the applications.

test\UserTest.java


 1 package test;
 2
 3 import com.google.inject.Guice;
 4 import ui.IUser;
 5 import controller.UserController;
 6 import junit.framework.Assert;
 7 import org.junit.BeforeClass;
 8 import org.junit.Test;
 9 import ui.UserForm;
10
11 /**
12  * The power & simplicity of JUnit4 test cases.
13  * @author praveenm
14  */
15 public class UserTest {
16
17     private static UserController controller;
18
19     @BeforeClass
20     public static void bootStrap(){
21         controller = Guice.createInjector(new MockBinder(true)).getInstance(UserController.class);
22     }
23
24     @Test
25     public void save(){
26         IUser form = new UserForm();
27         form.setExperience(9);
28         form.setEMail("praveen.manvi@yahoo.com");
29         form.setFirstName("praveen");
30         form.setUserName("pmanvi");
31         form.setLastName("Manvi");
32         Assert.assertTrue(controller.save(form));
33
34         form.setExperience(10);
35         form.setEMail("praveen.manvi@yahoo.com");
36         form.setFirstName("praveen");
37         form.setUserName("pmanvi123");
38         form.setLastName("m");
39         Assert.assertTrue(controller.save(form));
40     }
41
42     @Test
43     public void get(){
44         IUser user = controller.get("pmanvi");
45         if(user!=null)
46         Assert.assertEquals("praveen", user.getFirstName());
47     }
48
49     @Test
50     public void delete(){
51        // if(controller.get("pmanvi123")!=null) 
52             Assert.assertEquals(controller.delete("pmanvi123"), true);
53     }
54
55     @Test(expected= IllegalAccessError.class)
56     public void unauhorizedDelete(){
57         controller = Guice.createInjector(new MockBinder(false)).getInstance(UserController.class);
58         controller.delete("pmanvi123");
59     }
60
61 }
62
63

------- Final Summary-------

Guice simplifies the testing a lot doing everything in java, If you just want dependency Injection, Guice beats all other frameworks (Spring...) hands down in ease of use

JUnit4 is much easy to use with annotations & now we don;t have to extend any class & don't have to write methds with test...(). I really liked the way we can test the Exceptions

Adding interface to both backend & UI objects we can get rid of get/set noise.
Although it makes(or forces) back end to be aware of UI & other form of clients, in some cases (like we we use ORMs instead of JDBCTemplate) one time conversion logic shifts from UI to backend & it might put more development load on persistent layer team. My guess is these objections can be ignored because of the benefit of to cleaner & less error prone code. In case of distrbuted enviornment all the extra fields can be set to null to reduce the payload, but anyway number of rows are much more important than the number of columns in deciding the data size.

Hope that this sample application help new comers to understand/appreciate the value of Guice & JUnit4 libraries.

References;
Guice Dependency Injection Framework from Google

JUnit 4 - with Java5 features & is quite different from older versions.

2008-08-14T09:03:00.000-07:00

What value could custom client SDK can provide over SOAP toolkit(s)?

1. Single library to deal.
Developers need not have to spend time in knowing or generating stubs & which framework to choose for for the same (XML marshaling/un-marshaling).This single library client can hide this implementation methodology by exposing easy to use APIs & hiding these techniques behind the scene using some kind of dependency injection APIs. (Jax-ws, Axis, XFire, Xstream so on...). The single library can be collection of core + optional service libraries in the same fashion provided by the Spring Like spring-jdbc.jar, spring-jndi.jar so on... with spring-core.jar as kernal providing infrastructure for all other services.

2. Handling backward compatibility issues
Carefully designed client SDK can solve backward/upward compatibility issues without compromising on the design elegance, If we don't want to use new SOAP version, it should be OK to use existing version most of the time. Client SDK can be designed in such a way user is shielded from SOAP schema changes that does not require recompilation or rebuild what so ever.For using new APIs user just has to drop in new library(jar) that's it. Well designed Mock objects can even save the need to goto sandbox during development.

3. Easy to use fluent APIs.
Domain specific language (DSL) design can bring lot of changes & provide better user experience with fail safe APIs. Fluent APIs can bring lot of +ve changes. For me this is the single most important feature.
For example:

MailBuilder.mail().
.from(praveen.manvi@yahoo.com)
.to(pmanvi@aol.com)
.withSubject("Fluent Mail API")
.withBody("Fluent API & DSL can make developer life simpler")
.attach(new File("c:/test.txt")).
.send();

is definitely much easier to use. This is ok for 90% of users to start, if we have hooks to get into Transport & other Java Mail APIs,it can simplify the way we code & importantly understand without compromising the power for advanced users.

4. Meaningful Exception hierarchy stack for better recoverable actions
Users don't have to deal with SoapFault Error code. Just like Spring DAOException hierarchy translating the SQLException & SQL error codes giving meaningful RuntimeException (even exteding it to hibernate, ibatis,jpa) API providers can make developer's life easier. Like SQLException & SQL error code APIFaultException & fault code is difficult use & introduces lot of boiler plate code (because of checked exception) without providing the meaningful recovery options. Hibernate, Spring & C# have proved that checked exception is like communism, works best in theory only.

http://static.springframework.org/spring/docs/2.5.x/reference/dao.html

5. Automatic logging soap XML handshake (Making this switch on & off on need basis) between client & server. This comes as very handy & we can even make them asynchronous & log this information to to different data sources like file & database by providing proper hooks. This automation improves the performance as well as avoids the need to develop auditing APIs.

6. Most of error handling/validation can done by the SDK framework, there is very less chance for error requests coming to server improving server performance. 99% of the time server will deal with valid requests

7. Meaningful defaults for all the APIs & actions, can reduce initial learning curve & improve the perception. It can help user to get into the the details step by step instead of throwing him everything upfront.

8. Can enforce the best practices. & develop new interfaces that are very difficult to achieve with simple plane SOAP.

For example a utility can be developed to export the data in chunks from large files like Excel sheet, a csv file or a PDF., as the sending large files & batch processing with SOAP is not mature (mtom...)

9. Encourage more developer community participation. Easy to use APIs improves the possibility of
developer to develop new tools & application using the SDK especially desktop tools.

10. Get rid technical limitation of programming technique. SOAP inherintly not object orieneted & it's procedural style of programming is not what a modren developer would like to work.

I just evaluated different SDK provided by various well known API providers & most of them don't really provide above facilities & there is very good opportunity of 3'rd party developers to fill the gap over here.

Same concepts can be carried out for developing C# APIs. As for as dynamic langauges(Groovy,Ruby, PHP) concerned I guess they don't need any SDK, they are smart enough to deal with XML directly without any code generation.

Many a times developers need to design wrappers over SOAP apis -- that involves substantial effort. Client SDK can save the time & money here. Like Spring removed the necessity of having in-house framework for standard java enterprise project by not tying the implementation any specific technology but to the interface & also DSL developed with business in mind can make writing code like writing an essay.

References:
http://developer.ebay.com/DevZone/XML/docs/HowTo/BestPractices/JavaToJsdkToSoap.html - eBay SOAP SDK sample
https://www.paypal.com/IntegrationCenter/ic_sdk-resource.html - Paypal API
http://code.google.com/p/fluentmailapi/ - Fluent Mail APIs
http://www.javaworld.com/javaworld/jw-08-2008/jw-08-dsls-in-java-3.html - DSL in java
http://www.toolsforteams.com/code/facebook-api/ - Facebook APis
http://code.google.com/p/java-twitter/ - Twitter APIs
http://www.oracle.com/technology/tech/soa/mastering-soa-series/part4.html - Rich UI for web services
http://www.javabeat.net/articles/29-introduction-to-google-guice-5.html - Guice Samples

2008-07-24T10:08:00.000-07:00

Some wise words on design - I recently was listening to a presentation in InfoQ on design principles of successful large scale enterprise financial application based on Spring & Hibernate. I noted some points while listening to that. Although these are well known established principles I thought it's easier to refer & document when ever we need to come up new design document.

Minimize data movement
Task Parallel execution where ever possible
Physically partition of data
Optimized reads & writes of volatile data
Minimize contention -- Different connection pooling
Asynchronous Decoupling
Complete Business Logic performed by data
Caching frequently accessed data

Like all popular scalable architecture (eBay, Yahoo) the transaction (distrubuted) & state management were simple (stateless & transactionless for 90% of operations)
JDBCTemplate - fantastic example of Dependency injection
Spring promotes good design - modularity, loose coupling separation of concerns
Testability provides better confidence for refactoring
Hibernate promotes domain driven design - 2'nd level caching is gr8

Contents of design document can be divided into following categories:

Architectural review
Jumpstart programs
Deployment strategy
Platform change
Proof of concept
Performance tuning
Development

Capacity Planning questions:

How much memory footprint the application requires
How many threads the application components need
How many connections each application has to handle
How many system resources like file handles are required
How much parallelism is attainable in a single process
How many CPUs can be leveraged at the same time from within the same hardware
What are the synchronization primitives available in case of multiple processes and nodes
How we coordinate shared accesses and leases
The allowable number of nodes that a unit of application job can cross, taking into consideration the performance criterion

I also listened to Joshua Bloch's presentation on design guide lines, Joshua Bloch is currently Java Architect at Google & has great influence on Java's future. He also influenced (JDK5) & wrote many JDK classes as distinguished Engineer at Sun. His assertions carries lot of weight.
Importance of API design for internal and external use.

Public APIs are forever - one chance to get it right
Good code is modular–each module has an API
Thinking in terms of APIs improves code quality
Easy to use, even without documentation
Don’t let implementation details “leak” into API
Make classes and members as private as possible

I guess the most important one was "when in doubt, don't provide the API, leave it".

"There's a natural law in programming language and API design : as backwards compatibility increases, elegance decreases." - Bill Venners
So be minimalist, because of the compatibility requirement, it's much easier to put things in than to take them out. So don't add anything to the API that you're not sure you need

2008-07-13T01:39:00.000-07:00

Bio-Rhythmic Cycles - I got curious about bio-rhythmic cycles & wanted to try out the the results on my life.

For those who don't know what bio rhythmic cycles are, here is explanation:

The theory of biorhythms claims that one's life is affected by rhythmic biological cycles, and seeks to make predictions regarding these cycles and the personal ease of carrying out tasks related to the cycles. These inherent rhythms are said to control or initiate various biological processes and are classically composed of three cyclic rhythms that are said to govern human behavior and demonstrate innate periodicity in natural physiological change: the physical, the emotional, and the intellectual (or mental) cycles. Others claim there are additional rhythms, some of which may be combinations of the three primary cycles. Some proponents think that biorhythms may be potentially related to bioelectricity and its interactions in the body

I wrote a sample application in swing, which basically takes the date of birth as input & generate the charts.

Here is the sample screenshot, I used JFreecharts & some Date manipulating APIs to create above application. BTW is there any way I can show applet in this blog? Please let me know.

The results were interesting, Most of the events in my life had some kind of correlation with the above graph & explanation:

Physical cycle - 23 days; coordination strength well-being

Emotional cycle - 28 days; creativity sensitivity mood perception awareness

Intellectual cycle - 33 days; alertness analytical functioning logical analysis memory or recall communication

I also tried correlate our god Sachin's best innings with these cycles & surprisingly, cycle behaviour used to match with his best time :-)

Anyway I will try out with some other important figures before coming to conclusion these cycles have any impact or it's just a humbug. BTW, I am little skeptical about all astrology stuff.

So I am planning to develop a RESTful sample as well sample mobile application accepting date of birth & generating the charts possibly using Google charts instead of Jfreecharts

2008-07-09T10:24:00.000-07:00

GeoNames.org - provides geocoding services. GeoNames provide a huge databases of place names (with coordinates) organized in a hierarchy (eg. country/state/region/province/etc...). Also there is a query that allows to retrieve the geo tags within a range from a specific point. Geonames is integrating geographical data such as names of places in various languages,elevation, population, alternate names, Administrator division type,continent, Address,timezone & weather observation.

Geonames provides RESTful based webservices with both XML & JSON rendering. These seems to defined by seasoned developer & have Style enum (Short,Medium,Long, Full) for defining result verbosity & many such developer friendly criteria APIs.

For off-line applications, geographical database is available for download free of charge under a creative commons distrubution license & can leverage the service by downloading all the data. Geonames.org claims that it has already serving up to over 3 million web service requests per day & has the ability scale.

I guess applications (especially free & open source ones) have excellent opportunity to integrate with this service. Web applications can be relieved from maintaining the huge database of geo services. They can just store the geo-ids & can rely geonames to fetch the details. Hierarchical data is always to difficult store & fetch in standard RDBMS & now there is no need to maintain this data. GeoNames includes the reference implementation in all popular languages. I just checked the java implementation & it was not that good. It could be have been made more easier (or fluent).

The important point to note is that high-profile users of GeoNames include Slide.com & LinkedIn.

Reference
http://www.geonames.org/export/ws-overview.html
http://www.geonames.org/export/place-hierarchy.html
http://en.wikipedia.org/wiki/GeoNames

2008-06-22T08:13:00.000-07:00

Some notes on web services, REST & Fluent Interface. Theoretically we can make use any language to write web service client, & of course that's the exact reason web service was invented offering loose coupling for implementing distributed computing environment for businesses. But there were many requirements (schema definition/validaion, security,performance, service discovery, SLAs,BLAs, language etc..) to achieve high cohesion, hence SOAP, XML, UDDI & WSDL were necessary. I guess for most of integrations these technologies make developer's life un-necessarily tougher & are forced to learn/unlearn these vocabulary.They are all XML-RPC in disguise & may not be much better than the existing binary RPC solutions (RMI,CORBA, COM...). I am leaning towards REST proponents who are arguing that this complexity is not really worth of benefits it offers & is really waste of time, bandwidth & finally money. These are looking similar to EJBs as I am going through this book (considered as first printed book on REST)
If we see the existing web service landscape (I am referring WSDL, SOAP only), most of them are giving SDK for different languages (eBay SDK, Google Data API...) proving that SDK is more important than the WSDL. SDK always beats the SOAP toolkit hands down.

SOAP prevent independent evolution of client & server. For exampleSOAP toolkits provide (Jax-ws,Axis) code generation with strongly typed APIs, even adding new optional parameter, system will break until we generate thte stub again,Which basically diefies the logic of loose coupling.

With REST,HTML forms provide default & hidden variables,Service end point change is effortless, we can have re-directs based on situation & also partition, scale well depending on the requirements.Means don't have redeploy & test specifically for this.

separating reads from writes. SOAP has no inherent support for this, although we can design.
caching GET requests including hardware solutions, Caching @ HTTP always yields better results.
compression: Since REST uses HTTP, you can use compression such as gzip.
No need for additional jars or library
explorability -clicking around a RESTful API in their browser & use right away.
REST architecture serves as the basis for the most biggest and most successful information system the world has ever seen

Cons of REST
security: Though, security considered as weakest area of REST not sure why (HTTPS, authentic)
Tools will make working with SOAP just simple, IDE & tools cannot provide same kind features as SOAP to REST.
(Anyway we had same arguments for using EJB in the past as well)

Here is a sample Java APIs for web service APIs to help out developer community to directly working java instead of WSDL. I noted following points from going through the code.

Contains 28 methods & 768 lines

Expects user to update the Source code or use the properties file to inject startup values (Namely username, password, AccountID so on & so forth) & is not easy to pass/change these values @ runtime.

Uses Exception, RemoteException providing little recovery options from service faults & forcing client code to become ugly, clearing suggesting RuntimeExceptions are better suited here to use.

One single class for many APIs, making shpping client API size bigger if we want to deal with single API only

No way to see the SOAP XML files that get handshaked b/w client & server.

Uses Axis for marshalling/unmarshiling, We have Sun benchmarks suggesting latest jax-ws API/implementaion beat Axis/xfire hands down.

The API is not fluent (or user friendly) & does not make use latest JDK1.5 APIs

Uses lot of Array values as input/output params making client code uglier

Before providing my solution for this I do understand that I don't have much knowledge of the context (of client applications) using the APIs. So please be adviced I might have over-looked many issues.

I generated stub code using jax-ws APIs & netbeans. JAX-WS is the successor to JAX-RPC. It requires Java 5.0, and is not backwards-compatible to JAX-RPC.Netbeans has amazing plugins for about everything! I also wrote some wrappers around the campaign service API & here is how client code looke in the JUnit test case. (BTW JUnit4 has amazing features with POJO annotations & I guess this unit testing library is the best testimonial of effective usage of the java annotations.)

Now JDK6 users can turn their normal POJO in to web services and deploy it with ease without getting to deal with different web services stack. Before JDK6 one had to download some webservices toolkit and learn how to use

@Test
public void testGetCampaigns() {

List<Campaign>campaigns = new CampaignServiceTemplate() {
@Override
public YAccount getYAccount() {
return YAccount.builder.userName("uyiui")
.password("test123")
.license("uyiuu7687687gg")
.masterAccountID("988533")
.accountId("2086880350")
.build();
}
}.getCampaigns(13897897l,797897l, 89678l);
Assert.assertNotNull(campaigns);
}

Above code definately looks much cleaner easy to use. It makes use of generics, varargs, SOAPHandler java-ws interceptor,template pattern & fluent interface to make code look simpler & robust by validating all the inputs that takes care of all the issues I listed above sample code.

One task which I wanted to avoid was generating the source using tools Axis/jax-ws, if I want to write some adhoc testing I don't think doing in java make really sense. Ruby, Groovy where dynamic class reference options are amazing solutions here.

For stock quote client Groovy code looks something like this.

import groovyx.net.ws.WSClient
proxy = new WSClient("http://www.webservicex.net/stockquote.asmx?WSDL", this.class.classLoader)
quote = proxy.GetQuote('YHOO')
print quote

We, Java programmers are unable to make use of Groovy APIs like this, we will be the losers. Coding can't be simpler/easier than this :-)

But don't get me wrong, Java has a place & will continue to occupy the major portions of software development as Joshua Bloch says "One thing that makes Java such a pleasure to use is that it is a safe language."

So when we stick to Java,fluent interface or easy to use APIs have lot of importance.A fluent interface is a DSL embedded in a language,it consists of a set of classes that have been organized to allow you to write code that nearly reads as English essay. Beautiful Code is often associated with essay-writing.

Reference:

Immutable objects improves the code reliability.

Joshua bloch discusses about the design consideration

Jax-ws faster than Axis - Benchmarks

Java options for writing web service clients

SoapUI has wonderful set of applications & plugins to make web service testinf very easy task. I tried out their NetBeans plug in , It's simply amazing.SoapUI is a great tool for any developer who wants to test web services to verify that they are working correctly.

Good DZone link explaining functional testing using these APIs. I will update my experience after trying these out.

GoAPI -> Good for searching APIs.
Restas Architecture - "Simpler is better, and REST is generally simpler than SOAP"

2008-06-08T07:42:00.000-07:00

Java Logging Frameworks - why to use sl4j?
One of my friend was asking why should I with sl4j over log4j & java logging. My answer was "because it was written by the same author who wrote log4j & is now adopted by finest java framework authors (Tapestry & Wicket)" sl4j is the best available way to solve logging problems. It's latest & hence best suited for any new projects. This way some decisions are easy & safe to make. Logging is one of the most boring & non-debatable topic. At least for issues like this it's best to follow the best minds. During my initial days of coding I have seen mindless java wrappers over existing logging frameworks without any value addition. Log4j is definitely superior option to standard java logging both in terms of speed & availability of appenders. But when we write client apps or want to make our jar size compact & cannot afford to include log4j.jar java logging might be still better option which is rare case anyway. With log4j , in order to bypass the expensive string concatenation if statement was necessary. It would have been nice if APIs to provide some features that would alleviate things like this, and maybe make it easier to toggle the display of log statements for code readability.
if (log.isDebugEnabled()){
log.debug("Logging " + String.valueOf(a) + "some information :"+someBigEntity.toString());
}
I never liked double if conditions. Less if conditions directly proportional to good code quality.
sl4j utilizes parameterized messaging (like logger.debug("The new value {} is replacing {}.", newValue, oldValue);) & does'nt have class loader & memory leak problem. As JVM is becoming smarter & smarter with every new version, micro bench marking knowledge are becoming totally irrelevant. The experience is becoming baggage & anti pattern most of the time as for micro bench marking is concerned.

Here is one experiment I did with latest JVM with decompilation .

public class JavaLogTest {
// Get status from Logger.isDebugEnabled();
private static final boolean DEBUG=false;
public static void main(String[] args) {
if(DEBUG) {
System.out.println("Performance issue");
debug("Some Big"+"Concatenation here"+"& also creation of objects.... "+new java.util.Date()+" which is not good for performance");
}
debug("testing"+" ! testing again");
debug("testing"+new Integer(10).toString() +new java.util.Date());
}
public static void debug(String str){
System.out.println(str);
}
}
I decompiled the class with decompiler & here is the result what I got;

import java.io.PrintStream;
java.util.Date;
public class JavaLogTest{
private static final boolean DEBUG = false;
public JavaLogTest() { }
public static void main(String args[]) {
debug("testing ! testing again");
debug((new StringBuilder()).append("testing").append((new Integer(10)).toString()).append(new Date()).toString());
}
public static void debug(String s) {
System.out.println(s);
}
}
& the results were interesting;
1. It doesn't make sense to use StringBuffer for appending, JVM is intelligent to make use of StringBuilder which is not synchronized also
2. When we append strings only there is no need to use if statement even with log4j
3. We can take out the code from the byte code (pre-processor directive DEBUG) which can reduce the total jar size
I found scala solution for this was interesting. Dynamic languages indeed bring lot of paradigm shift in thinking!