Nirav's Contemplations

Avoid storing references of java.net.URL

2011-12-05T22:58:00.001-05:00

Normally, I avoid writing something so obvious but since I'm bitten multiple times now, it might help future me.

Never ever store references of java.net.URL in Java collections. The reasoning is pretty simple, 'equals' and 'hashCode' methods of this class does extremely expensive synchronous DNS lookup on every call.

It is not uncommon to see most of your thread's time being spent on monitors:

"pool-2-thread-2" prio=10 tid=0x92061400 nid=0x1744 waiting for monitor entry [0x91fad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.net.URLStreamHandler.getHostAddress(URLStreamHandler.java:429)
    - waiting to lock <0x9731b200> (a sun.net.www.protocol.http.Handler)
    at java.net.URLStreamHandler.hashCode(URLStreamHandler.java:354)
    at java.net.URL.hashCode(URL.java:875)
    - locked <0xaac87290> (a java.net.URL)
    at java.util.HashMap.getEntry(HashMap.java:361)
    at java.util.HashMap.containsKey(HashMap.java:352)
    at java.util.HashSet.contains(HashSet.java:201)

"pool-2-thread-1" prio=10 tid=0x9205e800 nid=0x1743 runnable [0x91ffe000]
   java.lang.Thread.State: RUNNABLE
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1211)
    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
    at java.net.InetAddress.getAllByName(InetAddress.java:1063)
    at java.net.InetAddress.getByName(InetAddress.java:1013)
    at java.net.URLStreamHandler.getHostAddress(URLStreamHandler.java:437)
    - locked <0x9731b200> (a sun.net.www.protocol.http.Handler)
    at java.net.URLStreamHandler.hashCode(URLStreamHandler.java:354)
    at java.net.URL.hashCode(URL.java:875)
    - locked <0xaac97228> (a java.net.URL)
    at java.util.HashMap.getEntry(HashMap.java:361)
    at java.util.HashMap.containsKey(HashMap.java:352)
    at java.util.HashSet.contains(HashSet.java:201)

This stack-trace just depicts hashCode but expect similar blocking code for 'equals' too. If you care a bit about performance, just stay away from this goddamned class.

Optimizing string memory footprint in Java - Part 1

2011-11-24T21:25:00.001-05:00

This is first in a series of blog posts where I will try to describe my miserable attempts at storing large amount of strings to build efficient spell correction and auto-suggest facility in Java. I learned some important lessons doing this and it will be a waste if I didn't share it. So here it goes..

In order to efficiently store large amount of strings, we first need to understand how JVM stores string objects in memory. In this post I will try to summarize memory layout of String object.

Each object in JVM has fixed, unavoidable structural overhead called object header that JVM uses for various tasks such as Garbage Collection, Identification, addressing and others that I don't understand. A 32-bit HotSpot VM uses 8 bytes header per object, 64-bit HotSpot VM with heap larger than 32g as well as IBM's J9 VM takes 16 bytes per object. As always, arrays are treated differently. Array is an object so it has a fixed header of its own as described above. However, since Java spec guarantees array bounds check on every array operation it has to store length of each array which is another 4 byte, making effective array header size 12 bytes instead of regular 8 for objects. And finally there's object padding in multiples of 8 or 16 bytes, depending on CPU word size and JVM to make memory to CPU communication efficient.

Assuming 32-bit JVM, the java.lang.String class has three bookkeeping integer fields called offset, count, hash occupying 12 bytes plus a 4 byte char array pointer and 8 byte header totaling 24 bytes fixed overhead per string. This is inefficient but slightly clever because padding is not required for string object itself making hash a free cache.

Here's the formula to calculate String's shallow and deep sizes:

Shallow size = HEADER + ( offset + count + hash + char array pointer) = 24 bytes
Retained size = Shallow size + 12 byte array header + (nchars * 2 + PAD)

Where PAD = ((nchars * 2) mod 8 or 16) is padding rounded off to 8/16 bytes if array elements' size is not multiple of 8/16.

So, on a 32-bit HotSpot VM, 6 byte word "Memory" takes :
24 + 12 + (6 * 2 + 4 PAD) = 24 + 12 + 16 = 52 bytes

That is 77% JVM imposed overhead v/s 23% actual data considering it a unicode string.

One way to make string representations efficient is to amortize structural overhead by storing larger strings. To illustrate, 100 char string will be 80% data v/s 20% overhead, 200 char string will be 94.33% data v/s overhead, 500 char string will be 97.6% data v/s overhead and so on. This will be one of the key technique in reducing string's memory footprint I will write about in later blog posts. Following is a rough graph that depicts this theory.

It is easy to see that storing a lot of small strings actually waste huge amount of memory which in no less part the reason why Java is considered a memory hog.

That's all with the basics of understanding overhead involved with storing strings. In next blog post, I will write about various implementations I tried and general rant on Java Collections' memory efficiency.

On using "PermGen" as application level cache

2011-10-24T22:25:00.001-04:00

I was reading an interesting article 'Assualt by GC' by stack exchange guy and it felt like a Déjà vu with my past couple of years of development on the JVM. It struck to me that we can definitely do better, so here it goes..

Automatic GC is really a great step forward in software development, except when it is not. If you have deployed application on a JVM with large heap (4g+) you probably know what a long GC pause really feels like [insert familiar knock-knock joke about java]. Jokes aside, JVM's GC advancement is unprecedented. The amount of tuning you can do with different garbage collectors can define a niche profession.

For most applications where GC latency isn't an issue; default garbage collector works just fine. For applications which need to scale GC can (and does) become a 'bottleneck'. If you disagree, try running a JVM under memory pressure and see app response times. It should be surprising because in most data driven applications bottleneck is usually IO or other IO bound resources (e.g. a DB). This situation generally happens when GC is completely thrashing the process because there are too many "tenured" objects which don't fit in the allocated heap or heap is fragmented and GC wastes a lot of time compacting the heap. Unlike .NET, Java folks are not very lucky with platform specific optimizations such as locking pages to prevent swapping. So it is not uncommon that a full GC causes excessive paging, making GC IO bound.

Turns out that GCing large "tenured" object space is expensive compared to short lived young objects (sweep). A large population of "tenured" objects is generally a genuine requirement for long running server processes relying on large amounts of data and this requirement shouldn't really punish application with long GC pauses. While not impossible, it is not really practical to set size of tenured generation very large because it may adversely affect young generation collections. JVM GC optimization is a skill not in abundance but the problem is all too common. So what can we do about it?

One way to eliminate GC on predictable "tenured" application data is to just not store it on JVM heap (i.e. use direct byte buffer etc. ). I've been watching solutions like Terracotta's BigMemory which uses similar approach to address GC issues. However all such solutions seem a mix of manual memory management with hacks to circumvent GC which end up being half-baked reinvention of JVM's copy-on-write "permgen".

Most of the java developers I know consider "permgen" to be some kind of evil which causes all kinds of problems including "eclipse crashing", crying JSP/[insert other template library] compiler, unpredictable class unloading and really large interned strings which stick around. "permgen" is going to go away from the hotspot vm, which is kind of sad because I think it could be a great way to achieve GC free heap storage for application level data (more specifically cache). This is not really possible unless "permgen" is used for one specific purpose, and if that specific purpose allows application to store its data, we can have standard supported GC free application data without the need of third party solutions which achieve the goal poorly. Even better would be java.cache using "permgen" for cache storage.

One of the commenter at HN talked about Smalltalk VM's way of using permgen (just send a message to object to move itself to "permgen"). I like this approach because applications can control which objects are long lived which is sensible because they have the best knowledge about long lived objects. The only similarity in JVM we have is String.intern, which unfortunately caches strings forever and it is not really as useful as having some kind of eviction control.

So, what do you think about this approach?

Thoughts on Event sourcing

2011-09-06T23:24:00.000-04:00

I read about event sourcing a while back and I couldn't stop thinking about it so it had to explode here as a blog post.

A typical data driven application involves CRUD operations on domain entities which are important to business. Such application typically capture data from user or other system in a centralized database for reporting or for further distribution. Architecture of such application is generally simple and there's a vast ecosystem of platforms, frameworks, tools and libraries to support it.

If we wanted to represent traditional design of such application in terms of a state machine, then we can say that such application capture, distribute and allow reporting of domain in a certain state. The primary objective of such application is to facilitate data manipulation (CRUD) which changes state of the domain. For most applications it is generally a sound design ignoring familiar caveats.

Thinking in terms of state machines, there's an interesting alternative: we can store all the state transitions that led initial domain model to its current state. This second perspective to application design has many interesting repercussions. In this approach, current state of domain is no longer as important as the earlier approach because it can be recreated just by repeating all the transitions. This second approach is named "Event sourcing" where event is just a little familiar name for state transition.

Not every application care about the easy recreatibility aspect of domain, most business applications care only about current state of domain. However many applications, especially the ones with mandated audit trails or domain with significant historical data, can benefit from event sourcing. A common example of such application is a version control system, version control systems capture state transitions (diffs) of domain entities (source files) so you can switch to any state (version) and rebuild it to desired state by successive applications of state transitions (diffs).

As far as business domains goes, Insurance domain by far seems to be a great area of application for event sourcing given that audit trail is a legal compliance requirement and insurance domain models tend to be really complex. Think of an insurance policy as a state and all the changes to it (endorsements) as transitions. By just tracking the transitions, one can rebuild a policy to its current state and reason about it for underwriting analysis and audit. Compare it, instead, with our initial approach of capturing multiple states (identical records in relational database) with complex logic to diff them and comparing them. This approach has profound positive implication on usability of application as well as testability, with this approach it is easy to visualize and rebuild data of interest to any point in time.

One more interesting application of event sourcing, I think, is in data mining. If data is stored as events it is fairly easy to sample, plot and build historical and predictive models. My limited experience in mining data has always involved custom (expensive) efforts to store historical data, such custom efforts usually involve complex development efforts just to extract marginally meaningful information.

It shouldn't be surprising that event sourcing can have significant influence on application architecture which may not be an easy sell especially in a larger setting. There are many related concepts to event sourcing, specifically CQRS which lead to wild architectures (which I'm not quite fond of yet).

I'm learning this is not new or revolutionary and has been done in past but never caught up for whatever reasons, nonetheless I find it interesting. As far as my technical curiosities go I'm very much inclined to try it out with a pet project to see how far these benefits are viable.

Why Concurrency is hard

2011-03-07T22:17:00.007-05:00

Concurrency is hard because we haven't figure out how to make it easy. For most developers, specifically web developers, concurrency doesn't really matter. I envy that assuasive confident feeling of a sequential execution of http requests. The number of cores on my machine quadrupled in last three years and I don't know a single reliable, comforting (easy) way of harnessing it as much as possible, I feel a little sad about current state of concurrency support.

Utilizing all the processing power consistently is a lot easier for well defined and not so concurrent tasks such as map-reduce. I have done it a lot, processing gigabytes of data by reducing the problem to independent subsets is programmatic triviality. On the other hand, I have always found developing a relatively concurrent application the "right way" to be a nightmare. Concurrency applications come in two mutually exclusive flavours: slow or complex.

At this point enthusiasts will point out java.util.concurrent and move on. While j.u.concurrent is nice and a significant improvement over explicit synchronization, it still mandates that API users be concurrency wizards and its complexity exposure is nearly at par with explicit synchronization. Here's one example blog post explaining common gotcha with ConcurrentHashMap. The only benefit j.u.concurrency provides is finer grained control over where to do CAS. I am a huge fan of j.u.concurrent and have been using it pre-1.5 but I still don't think it makes concurrency so easy. For one more example,

synchronized(this){ aRef = newVal;  return aRef;}

v/s

while (true) {
   V x = atomicRef.get();
   if (atomicRef.compareAndSet(x, newValue))
   return atomicRef.get();
}

Which one do you think is easier to grasp?

Many people think that Actors are the next big thing to tackle concurrency monster and complexities introduced by these shared memory model primitives. I too initially thought so, but then I found that Actor model isn't really the sweet spot in practice as it is touted. The very notion that Actors can fail and code must handle the tricky bits to recover from it makes it even more complex than using locks/mutexes etc. I am in constant a awe to see people talking so lightly about fault tolerant/fail safe systems without giving thought on the amount of complexity it adds. I am not necessarily protesting that philosophy but that behaviour is just not common in yer average regular applications (will your user be happy if one actor failed to process her payment and was asked to retry?). We still live in dark ages of transparent concurrency.

I remain as ignorant and unsatisfied about concurrency support as I was several years ago. For me, concurrency is hard so I am off to shopping!

Tackling nulls the functional way

2010-12-13T22:49:00.001-05:00

Most programmers have suffered null pointer one way or other - usually a core-dump followed by a segmentation fault on development machine or on a production box with application in smokes. NullPointerException results in a visible embarrassment of not thinking about "that something *could be* null".

Tracking Null Pointer ranges from loading core-dump in gdb and tracing dereferenced pointer to stack traces pointing to exact location in source. However, ease of tracking nulls opens up the doors to ignore them in practice and throwing null-checks just becomes as common as throwing one more div to fix IE's layout problems which is bad.

Problems with Null:
I hate having to ignore nulls as it is not always enough just to add one more null check. The reason why I am writing this blog is because I have several problems with Nulls:

All and Every reference can be a null in languages like Java. This covers everything: method parameters, return values, fields etc. There's no precise way for a programmer to know that some method might return null or accept null parameters. You absolutely have to resort to actual source code or documentation to see if it can possibly return null (and you are going to need good luck with that). All of it adds extra work when you really want to be focusing on fixing the real problem.
The problem with NullPointerException is that they point to causal eventuality and not usually the actual cause. So what you see in stack traces are usually the code paths where damage is not really initiated but done when we are normally interested in case where damage is initiated.
Null is actually very ambiguous. Is it the uninitialized value or absence of value or is it used to indicate an error? The paradigm of null fits well in database but not in programming model.
Having Nulls in your code has major implications in code quality and complexity. For example, it is not unusual to see code branches with null checks breeding like rabbits when an API "may" return null which in turn results in extremely defensive code. This significantly taxes readability.
Null makes Java's type system dumber when a method is overridden and you want to call it. Writing code like methodDoingStuff((ActualType)null, otherArgs) isn't exactly a pretty sight. This results in subtle errors when arguments are non-generic.

In many ways Nulls are necessary evil. For those of us who care about readability and safety we can't ignore them yet we shouldn't just let it overtake safety and readability.

I have come to know several techniques to tackle nulls. First, there is Null Object pattern which is not entirely as ridiculous as the name implies but it's not practical in real life software having hundreds of class hierarchies and thousands of classes, and so, I will not talk about it. Then there are languages like Haskell and Scala with library classes that try to treat nulls in, IMO, a better way. Haskell has MayBe and Scala has Options. After using options in Scala for a while in a side project, I found that I was no longer fighting with nulls. I knew exactly when I had to make a decision that a value is really optional and I must do alternate processing.

The central idea behind Haskell's MayBe and Scala's Option is to introduce a definitive agreement on a value's eligibility to be either null or not-null enforced with the help of type system. I will talk about Scala's Option since I have worked with it, but the concept remains same. I will also introduce how to implement and use Options in Java since this is much more of a functional way of thinking about handling nulls and it doesn't (almost) take Scala's neat language features to implement it.

Treating nulls the better way:

Usual course of action when you are not sure what value to return from a method is:

Most of the times it results in the last option because you don't have to fear about breaking everything (well, mostly) and everyone passes the buck like this.

We can do better with Scala's Option classes. We can wrap any reference in to Some or None and handle it with pattern matching or "for comprehension". For example:

Some(x) represents a wrapper with x as actual value; None represents absence of value. Some and None are subclasses of Option. Option has all the interesting methods you can use. When a variable in question is null we can: fall back to default value, evaluate and return a function's computed value, filter and so on.

Options in Java:
Implementing Options in Java is surprisingly a trivial task. However, it is not as pleasant as Scala's options. Implementation boils down to a wrapper class Option with two children: Some and None. None represents a null but with a type (None[T]) and Some represents non-null type.

To make Option interesting we make Option extend List, so we can iterate on it to mimic poor man's "for comprehension". We will also go as far as tagging both types with Enums so we can do poor man's pattern matching with a switch. You can find example implementation of Options in Java with a test case demonstrating use of Options. Here's the small snippet which covers the essense:

As you can see, Option opens up several doors to fix the null situation. You now have choice to compute a value, use default value or do arbitrary stuff when you encounter nulls.

How are my null problem solved with Options:

Using Options for optional/null-able references I have at least avoided "all things could be null" problem in my code. When an API is returning a Option, I don't have to wonder if it can return null. Intention is pretty clear.
When I am forced to handle null right at the time of using an API, I have to handle it right there: do alternate processing or use default. No surprises.
Option is a very clear way of saying a variable represents possibly an absent value.
Option doesn't really solve this problem completely. For example, method signatures with wrapper Option type can get really long (e.g. def method1(): Map[String, Option[List[Option[String]]] = {}). However, compared to null checks, I would prefer long method signature any day. Other benefits out-weight this limitation.
Clearly, Option[Integer] always means only Option[Integer] and not Option[Integer], Option[String], Option[Character], Option[Date] and so on. Compiler can infer exact method call from generic types.

As good as the concept behind optional values is, it doesn't and will not always save you from Null. You will still have to deal with existing libraries which return nulls and cause all these problems and more. However, most of the time null is problematic in your own code.

Where to use Options:

Here are the common places where I think using Options makes more sense:

APIs: Make your API as specific and as readable as possible; all optional parameters and return values should be Option.
Use in your domain model: You already have fair understanding on null-able columns, use Option for null-able fields in your table. It is not hard to integrate using Options if you are using an ORM with interceptable DB fetch; you can initialize fields to None if database contains null and so on.

In the interest of keeping this post relevant and on topic, I have completely avoided heavy theoretical baggage (monads et. al.) that's inevitable when theoretical functionalists (functional programmers) talk about Options. I really hope this post generates some interest in this topic. If you disagree or would like to share more on this topic, please leave a comment.

Future of a Java programmer

2010-05-01T12:34:00.006-04:00

As a long time Java only programmer professionally, I have been pondering about how things are changing around me as a Java programmer. Ever since I remember I had no choice but to use subset of C++ dialect (Java lang) with an extremely rich class library and ecosystem (Java platform).

In last few years there has been a drastic shift in number of languages targeting JVM. For example: dynamic (javascript, jruby, jython, groovy), functional + OO (scala) and a lisp dialect (clojure) and so many others. While I am excited about all the options I have today I don't think a single language will dominate on JVM anymore like Java did so far.

In a way this is a good thing, one tool rarely fits all needs (I couldn't curse Java enough for GUI programming). Like C, Java was never designed to be used for developing dynamic web apps, but we still tried and miserably failed with JSP/JSF and plethora of frameworks against PHP/Rails/Python in terms of productivity. One really good thing Java did was to raise a level of abstractions from platform specific details and memory management. These new languages on top of JVM raise the abstraction level even further for its area of strength.

It is not a remote future when we will see concurrent processes being programmed in clojure and presented with jruby/rails with intermediate code written in Java. Each layer of application is going to be implemented in different programming languages while interfaces being transparent for developers working in each layer. This is a big thing, it has never been envisioned before for Java Platform, the lowest coupling we have seen so far is through remoting (web services et. al.) where clients and servers are on different runtimes and languages.

What this means for a Java developer is if you are

A web developer: you are going to learn things which are extremely different from struts/jsf/jsps, no more artificial model1/model2 MVCs.

A non web-developer : you are going to write code which is far more readable and very specific to your business domain via DSL created in any of the languages mentioned above without worrying about accidental complexity Java and its frameworks imposed on you.

While I can keep classifying developers on Java platform all day long, these two are major ones whose life (and resumes) are going to change soon, they will be expected to know more than one programming languages rather than frameworks now. Contrary to the cool kids on interwebz, I don't think Java the language is going to die anytime soon not because many of the existing libraries are written in it, but because of the number of programmers on earth who know Java, tooling around it and the native JVM support for it. Java is like C in a way, you can do whatever is supported by underlying implementation.

Many of you who are like me are going to see change around them soon, I am thrilled to see how my career is going to transform as polyglot programmer are you?

Eclipse Refactoring for legacy code

2010-03-07T12:58:00.004-05:00

It has been quite sometime since I wrote anything on this blog, twitter probably spoiled me. If you are following me on twitter, you probably noticed announcement of a small eclipse plugin for automated refactoring for Legacy Code.

Before few months, I wrote an LTK refactoring mainly in Scala (yes, eclipse plugin in Scala language) to forward static method calls in a Java method to an instance method in same class. I am not really an expert at Scala (or functional programming for that matter) so the code is more of javaish Scala but I am improving and thats the best thing about Scala. This is also an attempt to prove how easy it is to write eclipse plugins in Scala and how seamlessly it integrates with Java source; Scala is not only a better language, it is much more suitable to deal with Eclipse APIs (you can define views etc. on extremely verbose interfaces like JDT AST).

Enough about Scala; the real purpose of the refactorings provided in this plugin would be to ease development with Legacy Java code, most of the code generators (think JavaCC) generate ugly Java code which is not only non unit testable but pain to comprehend and is often not usable with concurrent routines. Using this automated refactoring should make such code better. You can read more about the motivation behind the plugin in wiki.

For the performance heads who are worried about method chains introduced by this refactoring are encouraged to run their own "sane" micro-benchmarks to be sure JVM is really in-lining the method calls created by this refactoring.

The plugin is currently in beta and I am open for any new refactoring proposals. If you have any feedback, you are welcome to comment on this blog post or create a bug. The update site is here.

Real Refactoring

2009-08-13T21:00:00.012-04:00

A lot of really good material has been written about refactoring. However, just like a lot of other things in software development, refactoring is also victimized by becoming the buzzword implying 'rewrites' and every other thing that is not refactoring. A lot of people really don't understand what they mean when they refer to 'refactoring', this post aims to provide a systematic explanation of refactoring for such audience.

Refactoring is merely a process of restructuring existing code so that its end result and meaning remains intact. However, this restructuring has a great potential to become far better design than the original one. (It is implied that the refactored design is easier to refactor and hence better) In rest of this post I will try to summarize how to approach and refactor a code in real life application development.

1. Get familiar with code

I am assuming here that we are almost always refactoring code written fully or partially by others, refactoring your own code should be much more simple exercise. Primarily you would be dealing with Legacy code or testable code. If you are lucky enough to get the code which is almost covered with unit tests, your job becomes substantially easy and you can start refactoring very fast without a lot of trouble. However if you are working with existing code which just exists and works somehow, you will have to start slowly. The first step is to understand the incomprehensible. There are several ways you can get familiar with code:

Read the code

Reading code is one of the fundamental activity we developers do most of the time, if any employer is in the illusion that she is paying developers just to write code then they are paying 1/3 of the salary. Reading code is an art which develops over the time, reading good code can make you better developer (with a precondition that one possesses the ability to judge good and bad code for the language). The more you read bad code the more you familiarize yourself with the repeating mistakes in the code (duplication, absolute hostility towards testability, extremely fearful defensive checks and other hilarities). The goal is to grasp the mindset of the original developer(s) who wrote it. You will know it immediately how deeply thought out the code is in a few passes. Use any modern IDEs to navigate through files, ignore the comments which don't make sense.

Write test

Once you have some understanding of inputs/outputs and/or interactions in code's life cycle you can write some tests just so that you can identify if you made some mistake in next step (Step 2). These tests might not be unit-tests (unless the code itself is unit testable). Writing test is incredibly useful way to get familiar with the code base.

2. Isolate the refactoring hot-spot (or "The Inflection points" as Michael Feathers calls it)

There has to be a really good reason why you are refactoring and it certainly can't be a full 'rewrite'. Irony will kill itself when it finds out that the problems in a bad code manifests itself in popular hot-spots. By isolating these hot-spots you should be able to mock out the uninteresting and untestable (or very slow) dependencies. If you have multiple inflection points; deal with them one by one in isolation. Isolation may become tricky because you may have to introduce some inevitable changes before you actually have any unit tests. For example, you might want to eliminate inherently untestable 'statics', extract methods/interfaces which you can stub with NullObjects/mocks. The tests created in earlier step will be valuable to identify any problems you introduced while isolating the code. The goal here is to not concern yourself with trivialities other than the refactoring hot-spots.

3. Write unit tests

Now that you have code which can be isolated, start writing unit tests. If you are working on Java sources I highly recommend using Mockito to mock the dependencies. Don't waste time on accidental complexities of other mock libraries. A reasonable analogy would be: if Agile software development is a good thing than other mocking tools are worse than Waterfall - Mockito is Agile.

Your primary goal is to cover code under refactoring as much as possible, there are no hard numbers on code/branch coverages (but anything beyond 90% should be good enough :)), use common sense and quality metrics like CRAP. The more unit tests you will write, more you will learn about the code base. There may be totally unused code for anti-YAGNI stronghold, or there might be bugs, Your unit test will reveal these naturally. Depending on the code in question, A good test suit generally reaches the size of the code under test.

4. Refactor

Refactoring is particularly painful when it has to be done in an environment with highly volatile code base and for the parts of code which are critical to overall application functionality. Make small changes; make sure all tests pass.

If you are using Eclipse - turn off auto build and add a ANT or Maven builder so that it runs your tests after each compile - you will be running tests a lot.

Use a real source control system, it might be much harder for you to revert the changes otherwise. You might consider using something like git in between upstream and your local repository if your primary SCM is not good enough. In practice, this becomes a lot more important than what IDE you use, you will need the ability to identify and rollback a commit specifically in code base with many hands. Commit after each change, the smaller the commit the lesser chance of breaking a lot of things together by a long shot.

With each commit going in repository without external complains (like "You broke my build!"), you will gain confidence to make bigger changes. You should continue writing tests while refactoring because if at this point you are making changes without violating functional contracts you are evolving a new design and doing something good. If you have broken something and your justification involves the word 'refactoring', you are doing it wrong! you either don't have enough tests or you failed to understand the code entrails.

So this is refactoring in real life. For the primary audience, please consider following these simple steps before tossing the word 'refactoring' around. If you disagree, then invent the new word and let me know :).

Scala v/s Java arrays

2009-05-12T22:01:00.010-04:00

Here's a Java puzzler for the curious (and a good interview question too!). Given a array merge method below, what will be the output of following program?



public class Generics {

 static class A {
 }

 static class B extends A {
 }

 public static void main(String[] args) {

  A[] copy = merge(new B[] { new B() }, new A[] { new A() }, new B[1]);
  System.out.println(copy.length != 1);

 }

 static  Z[] merge(Z[] arr1, Z[] arr2, Z[] store) {
  List list = new ArrayList();
  list.addAll(Arrays.asList(arr1));
  list.addAll(Arrays.asList(arr2));
  return list.toArray(store);
 }

}

If you didn't guess it already, the program above results in a runtime exception (java.lang.ArrayStoreException).


Exception in thread "main" java.lang.ArrayStoreException
 at java.lang.System.arraycopy(Native Method)
 at java.util.Arrays.copyOf(Unknown Source)
 at java.util.ArrayList.toArray(Unknown Source)
 at name.nirav.Generics.merge(Generics.java:23)
 at name.nirav.Generics.main(Generics.java:16)

I am not a huge fan of generics in Java because we are left with whatever type safety we get from a half-hearted implementation (and I'm not even criticizing). It is too much to expect from a Java compiler to check that the program above has type safety compromised at call site, mostly because that's how arrays in Java are handled by VM. Arrays are special types of mutable objects with components as anonymous members which are accessed with indices. An array itself isn't a type, it assumes whatever type its components are. This is where the problem starts.

With current generics implementation, generic arrays are treated as covariant by default i.e. an array of component type T is also array of component type S where T is a subclass of S. This introduces type issues such as above where syntactically valid programs are victimized, making Java's "statically typed, type safe language" designation an irony. If arrays were regular objects, compiler will report an error in code without type variance information.

Arrays are regular objects in Scala, each array is an instance of Scala.Array class. The code below is equivalent to Java program above with some syntactic differences, unlike Java code the Scala code below is not syntactically valid. Scala arrays are non-variant, and Scala compiler uses what is called "conservative approximation" to ensure type safety at compile time.


object App extends Application{

 class A

 class B extends A
 
 def merge[T](arr1 : Array[T], arr2: Array[T], store: Array[T]) : Array[T] = {
    val list = new ArrayList[T]
    list.addAll(Arrays.asList(arr1:_*)) // :_* is for vararg conversion
    list.addAll(Arrays.asList(arr2:_*))
    list toArray store
 }

 merge(Array[B](new B), Array[A](new A), new Array[B](1)) //Error, type mismatch 

}

The Scala compiler will report an error on "merge" call, complaining about type mismatch.

Not everyone likes to know about such details until it bites back with million dollar bugs. Why are Java arrays co-variant? Who needs more run time checks?

Fork/Join Concurrency with Scala Actors

2009-05-05T22:23:00.000-04:00

Have you ever wondered why there are no special frameworks to address concurrency in a Java based application? Considering Java's rich (NIH) ecosystem, I do wonder why I have to write same old state management code while introducing even a small amount of concurrency in Java application.

The reason why I think it is almost impossible to consider concurrency as an aspect in arbitrary application is because of JVM's native support for shared memory concurrency. As a result every developer is forced to think in terms of threaded shared state with guarded blocks. If you have read or written non-trivial piece of code using shared memory concurrency primitives (Mutex, Semaphore etc.) you probably know that the resultant code is hard to visualize and test.

I have been reading about Scala's Actor library and its share-nothing message passing abstraction built over existing concurrency model of JVM. While it doesn't try to solve the fundamental problem, it provides an alternative to address concurrency in your application from a different perspective which is testable and easier to understand.

In actor model, an Actor is a forkable task which runs independently, something like a serializable+immutable object with its private data and behavior. Each actor can send and receive (or react to) messages asynchronously, very similar to object oriented programming with objects responding to messages, but in a concurrent way. This abstraction can seamlessly be applied to a given application of divide and conquer nature and can be made concurrent with minimal efforts as compared to adapting to Java's concurrency primitives.

To explain my point further take a look at classically trivial Producer/Consumer example in Java.


public class Consumer extends Thread {
private final Buffer buffer;
public Consumer(Buffer buffer) {
 super("Consumer");
 this.buffer = buffer;
}
@Override
public void run() {
 while (true){
  System.out.println(buffer.next());
 }
}
}
public class Producer extends Thread {
private final Buffer buffer;
public Producer(Buffer buffer) {
 this.buffer = buffer;
}
@Override
public void run() {
 Random random = new Random(System.nanoTime());
 while (true) {
  String num = Integer.toString(random.nextInt());
  System.out.println(getName() + "=putting: " + num);
  buffer.add(num + ": " + getName());
  try {
   sleep(400);
  } catch (InterruptedException e) {
  }
 }
}
}
public class Buffer {
private String string;
private boolean ready = false;
public synchronized String next() {
 if (ready != true)
  try {
   wait();
  } catch (InterruptedException e) {
  }
 ready = false;
 return string;
}

public synchronized void add(String string) {
 while(ready == true)
  try {
   wait();
  } catch (InterruptedException e) {
  }
 this.string = string;
 notifyAll();
 ready = true;
}

}
public class Test {
public static void main(String[] args) throws Throwable {
 Buffer buffer = new Buffer();
 new Consumer(buffer).start();
 Producer producer = new Producer(buffer);
 producer.start();
 producer.join();
}
}

Take a look at Buffer class, we have used some concurrency primitives there since that's the place where state is being manipulated. We didn't declare variable ready as volatile since primitive assignments are guaranteed to be atomic (except long and double), Even a simple problem like this involves fair bit of understanding of the underlying threading model. There's no doubt this complexity will extrapolate in non-trivial applications e.g. multi-phase concurrent incremental compiler, SEDA based server etc.

Now take a look at the equivalent Producer/Consumer example in Scala.


import actors._
import actors.Actor._
import util.Random

case class SimpleMessage(num: Long)

class Producer(c: Consumer) extends Actor{
val random = new Random(System nanoTime)
def act = {
 loop{
  val num = produce
  println("Sending: " + num )
  c ! SimpleMessage(num)  // asynchronous message passing
 }
}
def produce(): Long = {
 Thread sleep 400
 return random.nextLong
}
}
class Consumer() extends Actor{
def act = {
  loop{
   receive{ //blocks here
     case SimpleMessage(num) => println("Received: " + num);
   }
  }
}
}
object PCTest {
def main(args : Array[String]) : Unit = {
var c = new Consumer()
var p = new Producer(c)
c.start;p.start
}
}

Even if we don't compare the amount of code, the Scala code above is much more clear in terms of its functionality. In Scala, Actors can be mapped to a single native thread with 'receive' (similar to Thread#wait()) or we can replace 'receive' with 'react' which is event based invocation but doesn't cost a blocked thread. The code within 'react' is executed by any non-blocked thread from a pre-created thread-pool. Just a single change and your application is scalable!

The Java example code above can be equally trivialized with the util.concurrent BlockingQueue, but the important point to take away is, writing shared memory concurrency code is inherently difficult and error-prone. With JDK1.7 we will get similar fork/join abstraction in Java itself (JSR166y), which will add new alternative to how we design and write concurrent applications.

Scala borrowed Actors from Erlang and similar libraries exist for Java as well. If you are curious about interesting details on Actor based OO concurrency implementation in Java, take a look at some of the thoughts Sebastian is sharing with his ConcurrentObjects library.

How Scala's pattern matching can replace Visitors

2009-04-21T22:29:00.015-04:00

The primary motivation of Visitor design pattern is to separate model traversal from operational logic. A visitable model takes the responsibility of model navigation while the behavior is defined by arbitrary visitors. In this post I will try to explain problems associated with Visitors in general and how Scala's pattern matching feature can eliminate such problems cleanly.

Consider a simplified Insurance Policy model as follows (In Java):



public class PolicyElement {
 static class Quote extends PolicyElement {
  protected final Risk risk;
  public Quote(Risk risk) {
   this.risk = risk;
  }
  public void accept(PolicyVisitor visitor){
   visitor.visit(this);
   visitor.visit(this.risk);
  }
 }

 static class Risk extends PolicyElement {
  protected Coverage coverage;
  public Risk(Coverage coverage) {
   this.coverage = coverage;
  }
  public void accept(PolicyVisitor visitor){
   visitor.visit(coverage);
  }
 }

 static class Coverage extends PolicyElement {
  protected final Premium prem;
  public Coverage(Premium prem) {
   this.prem = prem;
  }
  public void accept(PolicyVisitor visitor){
   visitor.visit(prem);
  }
 }

 static class Premium extends PolicyElement {
  protected final double amt;
  public Premium(double amt) {
   this.amt = amt;
  }
  public void accept(PolicyVisitor visitor){
   visitor.visit(this);
  }
 }
}

public interface PolicyVisitor {
 public void visit(Quote quote);
 public void visit(Risk risk);
 public void visit(Coverage cvrg);
 public void visit(Premium prem);
}
public class PolicyTest {
 static class PremiumCalcVisitor implements PolicyVisitor {
  private double totalPremium;

  @Override
  public void visit(Premium prem) {
   totalPremium = getTotalPremium() + prem.amt;
  }

  @Override
  public void visit(Coverage cvrg) {
  }

  @Override
  public void visit(Risk risk) {
  }

  @Override
  public void visit(Quote quote) {
  }

  public double getTotalPremium() {
   return totalPremium;
  }
 };

 public static void main(String[] args) {
  Quote quote1 = new Quote(new Risk(new Coverage(new Premium(10))));
  Quote quote2 = new Quote(new Risk(new Coverage(new Premium(30))));
  PremiumCalcVisitor visitor1 = new PremiumCalcVisitor();
  PremiumCalcVisitor visitor2 = new PremiumCalcVisitor();
  quote1.accept(visitor1);
  quote2.accept(visitor2);
  assert visitor1.getTotalPremium() + visitor2.getTotalPremium() == 40;
 }
}

(Generally, we introduce one more abstract class to omit empty implementations in Visitors but I have left it for brevity.)

Now, not so apparent problem here is that if the object model changes (which is more frequently the case in real life), we have to add one more method to PolicyVisitor interface, all visitor implementations if change is substantial and have new Policy elements implement visitor methods. This invasive nature of Visitor couples it tightly with the model.

With pattern matching and views in Scala, you can have alternative implementation which is precise as well as non-invasive unlike visitors.


class PolicyElement 
case class Quote(risks: Risk) extends PolicyElement
case class Risk(cvrg: Coverage) extends PolicyElement 
case class Coverage(limit: Premium) extends PolicyElement 
case class Premium(amt: Double) extends PolicyElement
object PremCalcTest {
  class PremCalculator(pol: PolicyElement){
 def calcPrem : Double = calcPrem(pol)
 
    def calcPrem(policy: PolicyElement): Double = policy match{
   case Quote(risk)   => calcPrem(risk)
   case Risk(coverage)  => calcPrem(coverage)
   case Coverage(premium)=> calcPrem(premium)
   case Premium(amt)  => amt
 }
  }
 
  implicit def calPremV(pol: PolicyElement)= new PremCalculator(pol)
  
  def main(string: Array[String]){
   val risk1 = Risk(Coverage(Premium(10)))
   val risk2 = Risk(Coverage(Premium(30)))
   println(Quote(risk1).calcPrem + Quote(risk2).calcPrem)
  }
}

This code requires some explanation. What we have done here is we labeled domain classes with a 'case' keyword in Scala. If you tag a class with 'case' it can be used for pattern matching in a switch-case like structure as done in method 'calcPrem'. You don't need to create members or setter/getters for them, they are created by compiler for you. A case class can be instantiated without 'new' keyword; So Risk(Coverage(Premium(0)) is translated as new Risk(new Coverage(new Premium(0D))) in equivalent Java code.

The code in 'calcPrem' function can be assumed to be something similar to instanceOf checks for each possible case in Java, for example:


if(object instanceOf Premium)
return ((Premium)object).amt;

What we also have done silently is added a method 'calcPrem' to PolicyObject class. This is done through implicitly defined function 'calPremV', this will allow us to call 'calcPrem' method on any PolicyObject without actually modifying the domain model code. This type of lexically scoped class extension is known as a View in Scala and is similar to what is available in Ruby as open classes except without scoping.

In case if the model changes in this case, we just need to modify a single function and we are done. These programming language features of Scala frees us from coupling introduced by inheritance.

So it is easy to see that Scala's language features can be elegant and far more powerful than other languages (specifically Java) without sacrificing compiler checks and type safety.

Your Language is not SLOW!

2009-04-16T21:38:00.003-04:00

Do you really give a thought when you say "Ruby is slow or Python is slow"? Just because Twitter moved their back end messaging to JVM doesn't necessarily make Ruby any slower than it already was. If a language runtime can't handle concurrent load or long running processes it is not a limitation of the language!

This is dejavu for me as Java was considered "a slow language" back in 1.1 days when HotSpot JVM was in its poorer life and ISO/IEC C++ was hot girl in town. Now that you realized your pony implementations can't catch up with real life performance expectation you start blaming languages, are you serious?

Whether it's global interpreter lock or lack of native threads it's the runtime (re-read the runtime) and not the language that is slow. For the love of technology, Stop blaming languages for poor performance of your suboptimal reinventions!

Every language implementor should at least consider the host platform support before coming up with their "Not Invented Here" approach, sadly that's what is happening with most modern languages.

Scala: First impression

2009-04-09T22:21:00.004-04:00

If you are curious enough about programming languages then you probably have heard about Scala - the 'statically typed' functional and object oriented language. Scala is the new sun rising to balance 'the burn factor' between functional and object oriented schools of thoughts.

Unlike what this paper suggests[pdf], The reason why I think Scala exists is because functional v/s object oriented groups are moving in opposite directions, which is not only inefficient but they can't leverage achievements of one another. If you read about functional v/s object oriented programming comparison, every argument boils down to productivity, tooling and maintainability of code. While functional languages (Lisp, Haskell, Python etc.) offer excellent productivity compared to OO languages (Java, C++ etc.), Object oriented languages offer excellent tooling and are relatively maintainable. The reason why I think OO languages have been so popular is due to its easier to understand concept which is easy to map in real life, so for most people who have never took computer science course OO is still easier to grasp compared to Functional programming methods like list comprehension, closures or the higher order functions which are rooted from the formal systems of mathematics.

Scala tries to satisfy both of the groups by providing grammar and a type system which seamlessly integrates with mainstream platforms (Java, .NET) and offers powerful functional abstractions only available in dynamic languages. What this means is, Java developers can write their code in same fashion they write in Java using existing libraries and frameworks but with an added advantage of functional programming techniques wherever they feel it might be productive. Functional programming language enthusiast get access to rich class libraries and powerful tooling (eventually).

If you take a look at the Scala language grammar[pdf] you will notice that what you can create with Scala is limited by your creativity. Based on what I have learned so far, I find Scala much more refreshing than Java, Scala feels a lot more like a programming language of the 21st century! Scala compiler itself is pluggable so you can do heck of a stuff you can only dream with javac, ecj. What is missing is tooling, the existing tooling is scrap but that will improve hopefully with an active community.

Bill Venners of Artima has presented Scala wonderfully, take a look at the presentation[requires flash] on 'The feel of Scala'.

OPath ported for Java object model

2009-03-24T23:04:00.010-04:00

I have been thinking about alternate uses of OPath I created for EVars plug-in. And it occurred to me that I can very well use it for my unit testing (which involves pretty complex insurance object model) and for things like JSPs where I really hate to write ten lines of Java code just to display some value.

So I wrote a port of OPath for Java object model (less than 200 lines of real code). Following example should explain how it can add some value to your development efforts:

Consider an example of code for simple Accessible Swing Table :

If you are writing a UI test to see specific table's column heading, you just write following

Collection<Object> findAll = OPathReflectiveInterpreter.findAll(frame, "//dataModel//@val\\$column.*");
assertEquals("First Name",((Object[])findAll.toArray()[0])[0]);

This is very trivial test, of course. But it is sufficient to express the power of OPath micro-scripting.

If you like to try it out for unit-testing or templating check-out the download here (You will also need opath.jar).

Disclaimer: This is experimental work at best, it might be slow and it might have bugs at this time.

EVars Update

2009-03-22T22:58:00.005-04:00

Based on the awesome feedback, I have been doing some offline development on EVars plug-in lately. I have added some basic documentation on wiki. Installation is now easy, let P2 do the work to install/update the plug-in (it's a struggle to host update-site with google code).

One of the most exciting feature (Which I can't stop talking about) is value based filtering. Now you can have all the powers of predicate expression, how about searching a map like map//key['@user.*']/.. to select entries matching regular expression user.* ? This update also includes support for relational operators so you can search exactly you want like this //value[count > n] .

I have uploaded OPath grammar for the language enthusiasts (left factored, left recursion free). One feedback was 'experiencing some unresponsive behavior for huge graphs' I have addressed that to some extent by integrating progress monitoring and asynchronous interpretation job.

Thanks for the feedback! Also, for those at EclipseCon have fun with your sessions!

Improve your debugging speed with EVars Eclipse plugin

2009-03-10T19:37:00.012-04:00

Update: Added alternate link to screen-cast. Workaround to start plug-in

I have been debugging a lot lately which included open source libraries, closed source web-service engine and parts of closed source application server (WebLogic). During this painful debug-fest I felt a strong need for several features missing in Eclipse (NetBeans as well, for that matter) . Debugging closed applications is a pain beyond imagination, even with IDE integrated decompilers, it takes incredible amount of patience and time to debug.

To improve productivity (or prevent burnout), I jotted down a plugin to export/import live variables, filter variables view with a xpath like expressions etc. The plugin, called 'evars', features a small expression interpreter (similar to xpath) which can do a wonderful job of filtering variables on current stackframe. It also allows you to export variables to a file and reload them at a later point.

For the first time, I have attempted to create a screencast to explain its usefulness. You may watch it over here [10mb non-streaming, 1275x860], alternate link (open with your browser).

I have also created a beta release to see if it finds any interest which you can download it from here [JAR ~800kb, Jdk6] (Click here for Java 5 version) and drop it in dropins folder [Eclipse 3.4+]

If you find it interesting, please leave feedback here.

Two types of Architects

2009-03-09T20:05:00.000-04:00

I am amused by these creatures called 'Architects' in software development. In recent phenomena, They are available under numerous headings such as 'Software architect', 'Solution architect', 'Front end architect', 'Enterprise architect' and so on. To me, in what they really do, there are only two broad types of architects. My brain timed out while guessing fancy names for them so their description is in order with uninteresting machine generated type names, it is up to you to decide what to call them.

The Type1 Architect:
A Type1 architect lives near 'The well of Eternal dreams' in his tranquility. He speaks high of latest technologies, buzzwords from internet meme. He actively attends all expensive technology conferences, management meetings, and is excellent at presenting management with summaries of what he may possibly be capable of.
The ideal Type1 architect rarely talks to developer and/or QA teams, he doesn't really waste his time for trivial details and is religious about development-by-exception (don't stop coding/testing till you hit the roadblock); he would rather express his greatness with another Type1 architect who can really appreciate him. He is a huge fan of his super-mega automated, ultra-modern cloud-based, universally-ultimate end-of-the-world web 9000.0 ready architecture, which he thoughtfully insists in applying wherever it's not needed. His thinking is beyond languages, frameworks, platforms and business needs, no mortal can grasp what he thinks. He usually masters the skill of overkill and is full of without-the-box ideas. I can really go on for next few weeks talking about this type of architects but I need to talk about the other type so I will stop here.

The Type2 Architect:
This type of dude is a very simple earthliving guy, he lives by the belief that success of software development efforts is dependent only on the people who are involved in it rather than tools and technologies. He holds the shared vision of what the end product is likely to be and thrives at driving the team efforts in that direction without managing them. He doesn't fear coding (more like the existing code fears for its life when he starts coding), testing or training whenever he needs to do it. He believes in pleasing management with frequent stable releases than presentation of the obvious. He communicates with everyone, right from tester to business analysts and managers frequently, this guy is often found roaming around discussing with different team members.

So if you are an architect, what type are you?

Formal grammars, parsing etc. for dummies

2009-01-30T18:49:00.015-05:00

I thought it may be useful to post summaries of what I have been reading recently. In this post, I will try to explain some basic grammar and parsing stuff in as simple way as I can (omitting all the academic complacency) .

A grammar defines a language with a set of rules of type x -> y, where x is left hand side of the rule and y is right hand side of the rule. x and y can be either expandable or literals depending on grammar type (math zealot? read terminals and non-terminals).

Chomsky hierarchy defines four levels of grammar (although these are not the only grammar types in literature but Chomsky is cool dude in linguistic literature, hence the list):

0. Unrestricted grammar:

Languages based on such grammar can have any rule for derivation (which usually means any text can belong to a possible unrestricted grammar like 'yo dawg', where 'yo dawg -> u'). However, it is possible to define a language with unrestricted grammar which is also Turing complete. A language based on unrestricted grammar is extremely difficult to parse, but not completely impossible. Such language doesn't necessarily make sense, e.g. It is possible to create a math expression such as this 1++1=1--*1=1 in an unrestricted language, as well as an unrestricted English statement such as this: "all your bases are belong to us".

1. Context sensitive grammar:

A language defined with CSG may represent all the natural languages. Such grammar is defined with a set of rules where both sides may have expandable symbols (e.g. "a" B "c" -> "a" "x" "c"). It is very difficult to build a parser which can parse CSG.

2. Context free grammar:

All programming language grammars are context free grammars. It can have only one expandable symbol on left hand side of a grammar rule and is very easy to parse. E.g. X -> "if" (y) z | "if" (y) z "else" (X), where y and z can be represented as other statements of the same grammar. There are several well known parsing techniques to parse CFGs such as:

- LL(1)-LL(k) with a recursive descent parser (top down parsers with look-aheads 1 to k) and,
- Bottom up parsers such as LR(1) and LALR which can parse more CFGs than LL parsers (generally shift reduce parsers).

LL parser starts with Left to right and always expands Left expandable symbol first (think of queue) where as LR starts with Left to right but expands Right symbol first (think of stack).

3. Regular grammar:

A regular expression is a language defined by a regular grammar. It is represented as X -> "t" | "t" Y, where Y represents expandable symbol and always follows literal (if it exists). E.g. ab*, where * may be an expandable symbol defined to mean one or more occurrence of the literal it followed as a grammar rule.

Each grammar type above is a subset of previous grammar type, meaning every regular grammar is context free grammar, every context free grammar is also context sensitive grammar and so on.

Mutual recursion fun with Eclipse Debug model

2009-01-21T19:22:00.012-05:00

I rarely come across real-world data structures which are inherently mutually recursive (a.k.a. Daisy chain recursion in CS literature). Recently, while writing a plug-in related to eclipse.debug, I came across this interesting data structure.

Source Eclipse.org

Observe the IVariable ←→ IValue relationship (although most relationships in this object-model is mutually recursive), given an implementation of this model (say JDI) how do you clone an object-tree representing variables on a stack-frame?

The answer is simple: assuming implementation classes Variable and Value just write a simple function to clone variables one by one:

public class Variable {
…
private Value value;
…
public static Variable create(IVariable var) {
  Variable v = new Variable() ;
  try {
      v.setName(var.getName());
      v.setType(var.getReferenceTypeName());
      v.setValue(Value.create(var.getValue()));
      return v;
  } catch (DebugException e) {
      ..
  }
}
}

Similarly for Value class:

public class Value {
…
private  Variable[] variables;
…
public static Value create(IValue var) {
    Value v = new Value() ;
    try {
        v.setType(var.getReferenceTypeName());
        v.setValue(...);
        IVariable[] variables = value.getVariables();
        if (variables != null) {
            for (IVariable variable : variables)
                v.add(Variable.create(variable));
        }
        return v;
    } catch (DebugException e) {
        ..
    }
}
}

Statement 'Variable.create(yourJDIVariable);' clones entire JDI object. This is one of those intuitive recursion examples which you die to find out IRL. Interesting, isn't it?

This is how your regular stack-frame looks (Variables view):

And this is how it looks when you represent it using an example debug model :

Introducing PyConsole!

2008-12-05T23:37:00.004-05:00

I have been pretty inactive this year, unlike what I planned (yawn, haven't heard from DLTK since June either) . I started learning Python earlier this year and felt the need of an embedded python shell inside Eclipse. This was something PyDev is missing [Edit: Simone pointed out that PyDev already has this neat feature, I was late to discover it] as compared to other python tools in Linux such as Eric. So I decided to write one.

PyConsole is direct result of my irregular and intermittent Eclipse development (lately), it's very simple plug-in with a basic exploit of Eclipse Console View and manages two way synchronized I/O from the python interpreter process controlled by a background Eclipse job. It provides few basic features anyone might expect from a shell such as syntax highlighting, command history etc.

Here's the code repository if anyone would like to check it out. If you have suggestions or feedback please write to me or create a bug.

Productive keyboard shortcuts for Eclipse Editors

2008-08-16T14:35:00.008-04:00

I've seen very few developers using true power of Eclipse editors, so I thought of listing few very handy shortcuts that can significantly improve your source editing and navigation speed with Eclipse.

1. Ctrl + J / Ctrl + Shift + J : Inline search, meaning you can type on the fly to search text in current editor. You can use Up/Down arrow keys to navigate between matching text fragment in forward/reverse direction. This is one of the most powerful features you will find in advanced editors like Vim or Emacs.

2. Ctrl + K/ Ctrl + Shift + K : Search text in forward or reverse direction.

3. Ctrl + L : Go to line.

4. Ctrl + Alt + Up/Down : Duplicate current line above/below. Very useful in repeatative assignment statements (Java code, eh?).

5. Alt + Up/Down : Move current line up or down. Also, very handy to select current line.

6. Ctrl + . OR , : This is contextual and may be specific to Java editors: if you have error or warnings in current file in editor it will navigate to them.

All of these except for no. 6 are generic text editor features which are inherited by all eclipse text editors. You will realize that you will work much faster than average user if you start using these key bindings.

Synchronizing client requests using Ajax

2008-08-06T11:08:00.005-04:00

Before few months I was dealing with a problem where I needed to synchronize the server call sequence on browser. The problem was something like this: Client code will open new window navigating to a different page, but that should happen only after post on current page. The link that new window was opening needed to be server synchronized because of database updates etc. on server.

Being ignorant java-script developer I went ahead and put 'window.open' in onsubmit event of submit button, hoping it will be executed before form is submitted; Apparently, it failed. The event handling/dispatching/capturing in java-script is in sad state - it's specific to browser and I must make it work for IE so the pain is obvious. To reveal some more ignorance, I thought of putting 'window.open' in onmousedown event assuming form will not be submitted till the code in onmousedown is executed
fully, that too failed. Then, it was clear that 'window.open' was essentially a thread fork and will execute in parallel so this event capturing is not going to work here.

Since there was no obvious fix to my problem, I thought of using XMLHTTP for asynchronous wait to synchronize the request. I created new page which will open in a new window with javascript that handles form post in parent window, looks like following:

<html>

<body onload="postParentWindow()">

<%

 out.print("Loading Print Preview, Please wait...");

%>

</body>

</html>

getFormValues is a function which concatenates all the input's selected values and makes a request string. More interesting things are done in makePOSTRequest method:

var http_request = false;

function makePOSTRequest(url, parameters) {

http_request = false;

try {

     http_request = newActiveXObject("Msxml2.XMLHTTP");

} catch (e) {

      try {

          http_request = new ActiveXObject("Microsoft.XMLHTTP");

      } catch (e) {}

   }

}

http_request.onreadystatechange = navigateToOtherPage;

http_request.open('POST', url, true);

http_request.setRequestHeader("Content-type", "application/x-www-form-urlencoded");

http_request.setRequestHeader("Content-length", parameters.length);

http_request.setRequestHeader("Connection", "close");

http_request.send(parameters);

}



function navigateToOtherPage() {

if (http_request.readyState == 4) {

  if (http_request.status == 200) {

    window.navigate('newpage.jsf');

  } else {

    alert('There was a problem with the request.');

  

}

}

(Note: This code completely disregards browser compatibility and good error handling etc.) The important thing here is, we can assign a function to XMLHTTP object that will be executed after server response is received so that serves my purpose of synchronizing the request. I find this use of AJAX pretty useful in building stateful web-applications. The code might look very crude but powerful abstractions can be built over it to make a nice utility framework for such use-cases.

What's wrong with HP Quality Center?

2008-08-05T17:00:00.006-04:00

This entry is rant so you might just want to skip it.

HP Quality Center (formerly Mercury) is a, how so amusingly called, web-based test management tool. You will find a lot of marketing information if you ask Google, most of which is irrelevant and may only have 1-800 number for support; because they know when you are looking for it on web, you are in trouble and no one can help you. Unfortunately, I've to spend a good amount of time on this tool these days and it inspired me a great deal to write my inexpressible ovation for this tool here.

As a programmer, HP Quality center is one of the worst and most annoying tools you may have to work with (some may compare it with Lotus notes though).

This tool is implemented as a giant Active X; Apparently, they didn't want to make it web-based but when sales team threatened their developers, they some how adjusted their extra-thick client to web by (evil) means of Active-X(TM) technology. It only takes one or two more minutes to load than my Eclipse instance does, in Intra-net i.e.. It works only on lame Internet Explorer, in a sense that makes it 70% less web based. To add to the insult of using this tool, you have no option but use god-forsaken IE6 (or 7 if you are slightly lucky).

HP Quality center crashes at will, each crash is designed scientifically so that you waste a few hours to get back to normal work before it crashes again. For example, I add attachment to a defect; next, I go and try to add comment on it and it crashes - right on time. I open new IE window; browse it and wait for it to load, log-in again (this special web based tool doesn't have feature to remember your password in an enterprise-wide SSO environment). After recalling what I was doing last, I go to the defect where it crashed and try to add comment again; only to discover that it still thinks I'm working on it. 'The object is locked by user: [my userid]'. You will praise arcane brilliance imbued in this tool because these problems disappears randomly once it pisses you off - it knows when.

Some more praise on it's resource utilization, so one can't wonder why it is rightfully web-based. HP Quality Center uses ~120mb of your core if you have no defects in your view. This amazing tool can make your wish to see memory usage on crack come true, have 50-40 defects in your view and see the difference.

This is only the tip of the iceberg but gives you bigger picture and idea why companies make more money in support, there's always room for new version, patch, hotfix etc. Why in the world people pay for such crap? there are ton of free alternatives which are hundred times better and are totally free. And the other day someone was complaining about open-source tool usability...

How to: Always eagerly load objects from database with Hibernate 3 (without changing configuration xmls)

2008-04-24T15:40:00.007-04:00

So it was getting annoying to debug my app. with Hibernate 3 for it's 'lazy load everything by default' feature. Every time I wanted to see what's the value of some collection I will see these weird CGLIB proxy members lurking in debug views.

CGLIB$CALLBACK_0    org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer@9be3aa
class$net$sf$cglib$proxy$InvocationHandler    interface net.sf.cglib.proxy.InvocationHandler
class$net$sf$cglib$proxy$NoOp    interface net.sf.cglib.proxy.NoOp
class$org$hibernate$proxy$pojo$BasicLazyInitializer    null
componentIdType    null
constructed    true

For every association I would either evaluate the code or create expressions to see what's coming from DB. Lazy loading makes debugging more slower than it already is. What's the big deal?, you will say, Just go ahead and change configuration to lazy-load = "false". Well, I can do it but changing more than five hundred .hbm.xml files will help me more with retiring early than fixing the problem at hand.

So before it becomes nightmare for me I thought of hacking hibernate to make it eagerly load entire object from database. I'm saying hacking because there's no API in Session or SessionFactory interfaces that would allow you to unwrap these un-dead proxies automatically.

Here's how you can hack hibernate to prevent lazy loading:

The basic idea of making hibernate do what you want is to invert it's assumption, i.e. make everything load eager by default, we'll plug ourself to Mappings class and will force eager load implicitely. Follow these steps:

1. First create a class in org.hibernate.cfg package which extends Mappings:

package org.hibernate.cfg;

import java.util.List;
import java.util.Map;
import org.hibernate.MappingException;
import org.hibernate.mapping.PersistentClass;

public class MyMapping extends Mappings {


public MyMapping(Map classes, Map collections, Map tables, Map queries, Map sqlqueries, Map sqlResultSetMappings,
                Map imports, List secondPasses, List propertyReferences, NamingStrategy namingStrategy,
                Map typeDefs, Map filterDefinitions, Map extendsQueue, List auxiliaryDatabaseObjects,
                Map tableNamebinding, Map columnNameBindingPerTable) {

   super(classes, collections, tables, queries, sqlqueries, sqlResultSetMappings, imports, secondPasses,
         propertyReferences, namingStrategy, typeDefs, filterDefinitions, extendsQueue, auxiliaryDatabaseObjects,
         tableNamebinding, columnNameBindingPerTable);

}

@Override
public void addClass(PersistentClass persistentClass) throws MappingException {

   persistentClass.setLazy(false); // No Proxies, load everything in one shot
   super.addClass(persistentClass);

}

}

The 'addClass' method achieves all we need, just make persistenceClass non-lazy, it would be lazy by default.

2. Now hook this class with Configuration class.

    public class MyConfiguration extends Configuration {
    @Override
    public Mappings createMappings() {
        return new MyMapping(classes, collections, tables, namedQueries, namedSqlQueries, sqlResultSetMappings,
                             imports, secondPasses, propertyReferences, namingStrategy, typeDefs,
                             filterDefinitions, extendsQueue, auxiliaryDatabaseObjects, tableNameBinding,
                             columnNameBindingPerTable);
    } 
}

There you go, You can now create session factory out of it and you will get fully debug-compatible Java objects loaded from db.

This was way easier to achieve than I could think hooking hairy SessionImpl beast and making org.hibernate.event.LoadEventListener.LoadType INTERNAL_LOAD_EAGER from INTERNAL_LOAD_LAZY without changing SessionImpl itself.

Other than that, I found some talking about lazy loading being problematic with XStream serialization. Threre's a JIRA defect for XStream relating to this as well. I faced the same few weeks before when I thought of removing hibernate from test cases, XStream basically throws up when it finds a confusing CGLIB proxy like this:

com.thoughtworks.xstream.converters.ConversionException: Cannot handle CGLIB enhanced proxies with multiple callbacks
 at com.thoughtworks.xstream.converters.reflection.CGLIBEnhancedConverter.marshal(CGLIBEnhancedConverter.java:85)
 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:55)
 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:50)
 at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:73)
 at com.thoughtworks.xstream.core.ReferenceByXPathMarshallingStrategy.marshal(ReferenceByXPathMarshallingStrategy.java:34)
 at com.thoughtworks.xstream.XStream.marshal(XStream.java:765)
 at com.thoughtworks.xstream.XStream.marshal(XStream.java:754)
 at com.thoughtworks.xstream.XStream.toXML(XStream.java:735)
 at com.thoughtworks.xstream.XStream.toXML(XStream.java:725)

I faced this problem before as well and now I don't have to worry about it anymore. You guys can probably just go ahead and load your objects as I described above and your serialization would work fine.