Praveen Manvi's Technical Diary

Cloud Design Patterns

2016-04-26T22:33:00.000-07:00

Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications (Microsoft patterns & practices)

Design patterns serve as great communication tools during design conversations with various stake holders.Most of these things we already know but are packaged in succinct manner and I think their importance are limited if majority of stake holders are unaware of those terminologies.Examples like GoF patterns and enterprise patterns from Martin Fowler served the great purpose in capturing the imagination of developers and helped the design communication better.
I believe this documentation from microsoft has same level maturity and very well written with excellent diagrams. This is the best I have seen in this space.
This 236 page PDF document on Cloud Design Patterns is divided into 8 Focus Areas each area consist a list of design patterns. Also note that some of the design patterns fall into multiple categories but intention remains the same.

Most of the samples are in C# and I intend to re-create the sample using open source implementations in java eco system (best of breed of course). Will publish all the same samples github as I keep reading these pattern implementations and best practices. I believe this will serve as great learning tool. I will update this reference list as I complete coding for each pattern.

Availability

Data Management

Design and Implementation

Messaging

Management and Monitoring

Performance and Scalability

Resiliency

Security

And Big Thanks to Microsoft engineers for writing this.

Java-8 For Python Developers

2015-08-02T04:33:00.001-07:00

Abstract

This write up I wrote for providing session on java programming semantics for python programmers.The theme was to convey that Java (especially Java8) is not as verbose we (python programmers) think :)

Java 8 release is touted as biggest release since its inception in terms of paradigm shift with the support for functional programming. Given the backward compatibility burden, Java engineering team has come up brilliant effort with seamless integration with functional programming. Most importantly adding power to all the existing old single method classes and enabling java collections with stream classes. Boiler plate code is going to be reduced drastically. This write up on "Java 8 for Python developers" is for developers who are well versed python and familiar with basic java syntax. This content should provide enough orientation to navigate the Java ecosystem's libraries, tools and programming semantics.

As this is most the significant release adding support to functional programming, many of the previous assumptions (with verbosity) on Java language deserves a re-look. Java is a platform that supports multiple languages and surely ecosystem is bigger compared to python, so here the focus is only on language semantics. Its really hard to be popular for a long time for any language. I guess java has re-invented well and is giving tough fight to other languages despite its age and baggage.

In this I will go through over-view and in the next series we will see how new Java-8 features like Lambdas result in less boiler plate code (Method Reference,Concurrency), new annotations improves decorator coding and how many of the popular external libraries (Guava, apache commons, GS Collectioins, Joda time) can be removed (its obsolete as most of these features now available with JDK itself). The improved java script engine as part core JDK helps to utilize the existing java script code can make code as closer its python counter part in most of the cases.

Topics

------------

History
------------
Java:
Created by Canadian 'James Gosling' in 1996 influenced by Oak/C++. This is supported by corporate (Sun/Oracle) and is regarded on #1 language/platform for developing all kinds of application.

Python:
Created by Dutch 'Guido van Rossum' in 1994 influenced by ABC. Its pure community driven project although its creator is called "Benevolent Dictator For Life" and regarded as most developer friendly languages.

Both are object oriented (classes, encapsulation, inheritance, polymorphism) but python is more scripting/functional friendly language. Python is more object oriented in the sense that it does not have primitives and treats everything including functions as object. Java still deals with low level primitives and hence more perform-ant.

---------------------
General Mapping
---------------------

Python	Java
.py	.java
.pyc	.class
python.exe	java.exe + javac.exe
pypi	Maven central repo
PyCharm	IntelliJ IDEA
pip	maven
requirements.txt	pom.xml/gradle
PYTHONPATH	CLASSPATH
CPython	HotSpot/JIT
Jython/IronPython/PyPy	JRockit/oracle jdk/Dalvik

Java is a statement oriented language. You write a statement, and when it executes, it has an effect. REPLs by contrast are expression-oriented: You write an expression, and the REPL shows the result,like a calculator. Hence REPL was not super useful with java. (I remember hearing this from scala's creator Martin Odersky )

Definitely REPL(Read Eval Print Loop) will be greatly missed by python developers. But if you use the jython/groovysh, they serve the purpose. It should give the same feel of interactive mode development with REPL.
REPL for java coming with Java9 looks impressive.

For ex:

D:\jython2.7.0\bin>jython

Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)

[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_25

Type "help", "copyright", "credits" or "license" for more information.

>>> 1+1

>>> from java.util import Arrays as A

>>> numbers = A.asList(3,4,2,5,1)

>>> print numbers

[3, 4, 2, 5, 1]

>>> from java.util import Collections as C

>>> C.sort(numbers)

>>> print numbers

[1, 2, 3, 4, 5]

>>>

In general python has linux orientation and code quality is much better than corresponding java code because of the people involved. Its favorite with sys admins and scientist community. Lot many awesome libraries exist in python compared to any other language, Its a great glue language i.e. the performance intensive stuff can easily moved to native (C++) and python has the best interface mechanism to include. As I was listening to Jessica, a great Python evangelist on python future - the focus is to get python working equally well in "windows" and "mobile"

-------------------------
Language semantics
-------------------------
In Java, all variable names (along with their types) must be explicitly declared. Attempting to assign an object of the wrong type to a variable name triggers a type exception.That’s what it means to say that Java is a statically typed language. In java you have to declare variables with their type and use '{' '} for scope. Python does not explicitly type parameters (python 3 allows notation optionally) and apparently google moving their legacy to python 3 mainly because of types. If we see carefully, type checking is actually winning, people are dying to type. "You don't have to type" is not really a cool thing.

In Python if a name is assigned to an object of one type, it may later be assigned to an object of a different type. That’s what it means to say that Python is a dynamically typed language.

In Java there are access specifier (private, public, protected) for encapsulation, but there is nothing like in python but follows a convention with '_' for making developers to stay away from using it.

Because of private variables there are lot of get/set verbose code for all fields in a class. Python solves this problem pretty well making it develop on demand basis (get_attr). But sealing of fields/methods with 'final' serves a great tool to communicate design.

In java getters/setters are required because public field don't give opportunity to go back and change. To be fair to have there is builder patterns and libraries like lombak and immutables that makes it type safe and faster to implement as well.

Java uses method overloading heavily, Python does not have overloading of method names, with default values its not necessary for python.

In java there is support for 'switch' statement its a great replacement for huge if/else or the dictionary in python to declare and work with multiple alternate flows.

-------------------------------------

Basic Data structures and usage
--------------------------------------

Some of the basic data types in Python are numbers, strings and collections. The collections are natively supported and hence makes to breeze to work with them.
Java has huge eco system for designing java collections (Interface, Class and Abstract class) and separate set of classes to support primitives.(long,int,double ...).

List

numbers = [1, 2, 3, 4, 5 ]

ArrayList is usually replacement for mutable lists

List numbers = new ArrayList() {{ add(1);add(2);add(3);add(4);add(5); }}

Immutable List

tuple = (1, 2, 3, 4, 5 )

There is no equivalent of of tuple but there are immutable list implementation like
Arrays.asList() or Collections.unmodifiableList(), mutating is stopped at runtime.

List tuple = asList(1, 2, 3, 4, 5);

Map
called as dictionary

numberToText = { 1:"one",  2:"two", 3:"three" }

HashMap is the equivalent to the dict jack of all trades.

Map numberToText = new HashMap<> {{ put(1 :"one"); put(2:"two");put(3:"three"); }}

Similarly set() translates to HashSet<>()

"Java has huge of set of data structure classes in JDK and other external libraries dealing with concurrency and performance probably unmatched compared to any language including python"

But dealing with map (dictionary) could be single most important feature missed by python developers. Probably most important and used dictionary python makes it breeze but definitely java makes it very painful.

String
Like in Python, strings are immutable in java. In python String values can be wrapped in either single or double quotes. To differentiate between vanilla ASCII strings and Unicode strings, Python uses the u prefix to denote the latter.

Multiline strings will be greatly missed by python programmers in java. Its painful and ugly to define large strings in java.

Here are the list of features that are missing in java (or how they could be acheived in java)

------------------------
comprehension (List ...)
------------------------
>>> numbers = [1,2,3,4,5]
>>> print numbers
1, 2, 3, 4, 5]
>>> even_numbers = [n for n in numbers if n%2==0]
>>> print even_numbers
[2, 4]

[TODOs]

--------------------------------------------------
Returning multiple values from a function
--------------------------------------------------

------------------------------------------------
sequential substitution of actual/formal parameters
-------------------------------------------------

-------------------------------------------
Defining functions inside a function
-------------------------------------------

-------------------------------------------------
@staticmethod, @classmethod, decorator
--------------------------------------------------

-----------------------
Meta programming
------------------------

-------------
Generators
-------------

References:
https://dzone.com/storage/assets/4018-rc193-010d-python_2.pdf

oracle has comprehensive documentation available compared to any language.

http://docs.oracle.com/javase/8/

http://docs.oracle.com/javase/tutorial/getStarted/intro/definition.html

http://docs.oracle.com/javase/tutorial/tutorialLearningPaths.html

Java collections overview:

http://docs.oracle.com/javase/8/docs/technotes/guides/collections/overview.html

Elastic search experimentation with enron emails.

2014-10-12T11:52:00.000-07:00

ES (Elasticsearch) is distributed, real-time search and analytics engine. I have been using this product from more than 18 months. It is a de-facto distributed search solution built on lucene. Its fast and easy to develop.
Features could be broadly classified as

Full text search, search by words with boolean operators (Ex: (java OR techonlogy) AND (bangalore OR hyderbad)
Easy to build filters, facets, & aggregations
Fast response times with large data volume (usually measured in millis)
Ability to make geo-localized searches & configurable robust ranking system

Its a incredible tool for collecting analytic information & possibly with my experience easiest to build

Once we got convinced ourselves that ES is best option for given use case, usually product managers/analysts come up with questions like "tell me what is the node/RAM/Disk requirement for indexing (say 1TB) data?" & "what's the indexing strategy to use?" (like index schema, index sharding that could be time based OR data based etc...). Unfortunately there is no immediate correct answer, there are lots of data points & query characteristics needs to be collected before we even try to attempt to answer questions although people asking these questions are usually uncomfortable to wait and lengthy answers.
Here are few guide-lines for asking questions before trying to come up with answer.

required indexing rate (acceptable/desirable limits)
total # of docs with payload characterstics
latency requirements for searches (well, its for types of searches)
Full text search requirements (Highlight, analyzers, internationalization)
Indexing schema requirements (Is there any join requirement with nested elements)
Indexing schema metadata requirements (especially for date ones & the precision requirements)

Although internet literature claims ES scaling to insane numbers, Its all contextual (nature of data) and situational (nature of hardware), there is no option but to do experiment with real required data and assert the claim and quantifying hardware/software requirements as suggested by the creator ES Shay Banon.
There are too many indexing options and fine tune parameters for the ES. The exercises like below should help to find holy grail combination of ES fine tune parameters for the given use case.

Here is an effort OR sample guidelines that could be used to carry out such experiments. Usually the approach to create data using random values has the disadvantage that data it will have less/more unique content tailored that's not intended for the domain may not give desired results & worse it could be misleading. Lucene is mainly used mainly for full text search and stores the data in reverse fashion i.e. terms to docs. Here the nature of terms plays very important role in deciding the index size and query latency. In lucene as term will be stored once per field, searches work by locating terms and finding documents associated with those.

There is set of free large corpus of data available that could be used. Here I have picked enron email data as they are pretty decent in numbers given that mostly we restrict the index size to these numbers, making it easy to extrapolate the numbers as ES indexes are sharded to scale horizontally. Its also nice to do experiment with ES features with this publicly available mass collections of "real" emails and is not bound by privacy and legal restrictions.

Although I wrote all my framework code to deal with ES in java (making use spring data elasticsearch wrapper), I used python for the current exercise. Its awesome for running deploy tool-chain (pants), data processing pipelines and other offline systems. Now its my de-facto scripting language replacing groovy. Its easy to share as its concise and reader friendly language.

Before the approach & the code, here are some highlights & observations about this data.

1. If we enable the source (which is required highlighting feature) while indexing the storage cost increases by 76%
http://localhost:9200/_cat/indices
enron-email_without_source 746.5mb
enron-email 1.2gb
It was observed that there was no change search latency with storing source.

Although general assumption that lucene index size 20-30% of actual content may not be correct, as it depends on the indexing field. Even if I used better compression its unlikely to change the final figure drastically here. Its ~50% here. (considering source is not enabled).

2. Payload size distribution, Its important to match the use-case on how the real domain are getting pumped into ES.

Here are again ES aggregation feature comes handy for finding out the payload size distribution.

Payload range	Count
0KB to 1KB	124181
1KB to 5KB	237870
5KB to 10 KB	42331
10K to 1MB	303

You can find the awesome ES aggregation used here to extract this information. I introduced a size field to capture the size of each document & above is the result. I guess this is most killer feature in ES for building data analytics applications.

3. There were totally 66043 'to' unique addresses & 21021 unique 'from' addresses. Enron mails were extracted by email in-boxes (with complete folder structure) of 148 employees.

Transform the raw enron data into JSON and push into ES.

ES works with JSON & raw data needs to be transformed into ES. Download the data & extract the content into a directory.

Here is the python script used to create in windows machine. The script expects a firs parent directory where enron mails have been extracted & recursively parses the emails in EML format and index into ES. It also creates a fresh index (deleting previous one if exists) making it easy to test with experiments.

Search and test the generated data.
Here is python script shows how the we can do search and other powerful analytics feature like aggregations

Although I quashed the idea of using ES (storing the content along with index) as primary storage rather than secondary storage in the beginning itself, now I do think its worth re-vising again with the availability of snapshot feature.

Hope this helps some one trying out newly with ES.

About my current server side software stack

2014-07-13T07:32:00.000-07:00

Here are my thoughts on working with NoSQL technologies from past more than a year while building a platform in archival/e-discovery space where the cost of data storage matters a lot. Clearly its very difficult claim to be expert on every possible technology trend especially in NoSQL where technology/usecase over-lap to solve a problem domain & its not easy to pick one against other. There is also a cost of maintaining multiple data sources (code base & relationship). Here I am documenting overview of technologies that we picked and plan to document all the major issues and best parts separately for each of the stack in the next set of write ups that can help developers to make informed technical decisions especially with my favorites "Elasticsearch","Hazelcast" with spring data wrapper libraries.

I guess the paradigm shift is "Think about the queries(business), horizontal scaling & work backwards." rather than data first approach (storage, integrity) and make queries to work on the model. Data storage is driven by query usage patterns (or functional requirements) and cost of building and maintaining it. The latency and storage costs have moved from non-functional to basic functional requirements, distributed computing has become basic necessity and not the luxury. In NoSQL context also data is still the king but he is at the mercy of query access patterns and the storage cost.By end of the day most of the time software is all about storing & retrieving the useful data in cost effective way. Performance/transaction should not be the reason to abandon RDBMS, but distribution of data (because of scale) and ease of development are the reasons. Shortening development cycle, lowering costs and enabling new ideas should be goal for any technology professional & non-relational data store mechanisms increasingly fit the bill.

There is no single silver bullet for data storage requirements (for anything for that matter) that servers any serious project, it has to be polygot persistence & the trend looks to be ir-reversible.The important here is that each of them provide their scalability & performance by offering a limited data model. They compromise one of the feature of the CAP (Consistency, Availability, Partition). Although everyone claims to do almost anything. You name feature, the evangelists from NoSQLs comes with post saying "Look ma! I can also do this", but its important for us to make informed decisions after doing POCs making sure that these are not stretching themselves too much to achieve the goal (latency, storage efficiency). It was key to pick the storage nodes & technology behind them considering their sweet spots & after evaluating with their counterparts.

Based on use cases we made following decisions.
Search from key - Document Search (MongoDB) Serves most of the RDMS requirements. Ex: Blog
Search the term or value - Full Text Search (ElasticSearch) serves the ability look for text and then the document. Ex: conversations, documents.
Search and store by order - Column Family (Cassandra) serves the ability to constant write for data associated usually with time series Ex: twitter
Search for large binary objects one at a time- Object store provided by cloud vendors, ceph
Search/process for results in millis - In memory data grid solution (Hazelcast)
Although all are scalable solutions cassandra stands out the best in linear scalability offering both in read (for range based queries) and writes efficiency ir-respective data size as column family keeps the columns that fit together & work with append only mode making writes faster than reads & doesn't depend on the current size of the data.
Future plan is to use graphs.
Search by relationship - Graph DB (Neo4J) Users and their relationship Ex: social graph

Although there numerous links suggesting selection criteria, I found following links interesting.
http://blog.nahurst.com/visual-guide-to-nosql-systems - explains following CAP theory & what you will be giving up among three (Consistency, Availability & Partition aware) with most prominent solutions in this space.
http://highscalability.com/blog/2011/6/20/35-use-cases-for-choosing-your-next-nosql-database.html - Documenting use cases
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis - Easy to follow well documented introductions.

KAFKA
A distributed pub-sub messaging system designed for throughput. It is maintained/used in LinkedIn & more importantly written in scala which is my favorite language these days. Kafka is log centric, ordered, im-mutable and sequence based (offset). It is replication/persistence by default. It concentrates durable, scalable message storage system & does pretty well.
As author defines it as "kafka is a system that supports long retention of ordered data" - mainly used for pipeline pushing data to the system that's re-playable.

Buzzwords:
topic - category of data.
partition - log of records (data maintained for duration, week, month ...)
log sequence number (offset) - acts as state of system.
consumer/producer - for pulling and pushing the messages.

Biggest success story:
LinkedIn claims of processing 175 TB of in-flight data with ~1.5 ms latency of 7 million writes and 35 million reads/second. This is what I heard from kafka author himself when I recently listened to a webinar hosted from orielly publishers.

Uses:
Zookeeper

Issues:
No easy way to trap re-balancing events. Scala code is difficult with work especially with existing java code.

STORM

Storm is like a pipeline where you push individual events and then get processed in a distributed fashion, it makes it easy to process massive streams of data in a scalable way, and provides mechanisms for doing things like guaranteeing that the data will be processed.

there is always some kind of confusion its relationship with hadoop, storm creator clearly says they are complimentory & comes up with following thumb rule while adopting,
"Any time you need to look at data historically(already saved) , use batch processing (i.e. Hadoop) as its cheap & scalable, whenever you need to look at all the data once, then anything you do as it comes in in real time, use Storm for that."

The ability of storm to run multiple copies of a bolt’s/spout’s code can be run in parallel (depending on the parallelism setting) is the crux of the solution and helps to scale to insane numbers by taking care of cluster communication, fault-tolerant fail-over, scalable & durable messaging with order and distributing topologies across cluster nodes. Each worker could be assigned process, memory & threads(tasks) depending on their requirement making ideal for any growing company for horizontal scale requirement.

Buzzwords:
Spout - is a source of streams in a topology making it a starting point.
Bolts - any piece of code, which does arbitrary processing on the incoming tuples
tuple - basic unit of data that gets emited in each step
topology - layout of the communication which is static and defined upfront, deployable workflow manager that ties spout and bolts together. It can also be described as a graph consisting of spouts (which produce tuples) and bolts (which transform tuples).
Nimbus - similar to hadoop job tracker, service that tracks all the workers executing the bolt.
supervisor - worker (collection of executors)

Spouts are designed and intended to poll, we can't push to them. So we have kafka to keep the messages and spout is consumer triggering the indexing process.

Built on:
LMAX Disruptor : intra-worker communication in Storm (inter-thread on the same Storm node)
ZeroMQ or Netty: Inter-worker communication (node-to-node across the network)
Zookeeper: cluster management of Storm (automatic fail-over, automatic balancing of workers across the cluster)

Biggest success story:
Twitter claims to index four hundred million tweets within 50 milliseconds with the help of storm.

The issues:
Inter-topology communication: nothing built into Storm. (we used kafka). The UI isn;t mature. Storm is not well suited to handle OOM kind of errors. Its tough to manage the state across the bolts. (errors, progress)

Samza & Spark are also tries to solve similar kind of problems that storm is trying to solve. Here are some notes in comparison with spark which apparently is doing great & looks to be has the largest momentum.

Storm Vs Spark
Both frameworks are used to parallelize computations of massive amount of data.With Storm, you move data to code. With Spark , you move code to data (Data parallelism). Spark is based on the idea that, when the existing data volume is huge, it is cheaper to move the process to the data (similar to Hadoop map/reduce, except memory storage is aggressively used to avoid I/Os which makes it efficient for iterative algorithms)
However, Storm is good at dynamically processing numerous generated/collected small data items (such as calculating some aggregation function or analytics in real time on a Twitter stream) also called task parallelism. Spark applies on a corpus of existing data (like Hadoop) which has been imported into the Spark cluster, provides fast scanning capabilities due to in-memory management, and minimizes the global number of I/Os for iterative algorithms.
Storm is like a pipeline where you push individual events in which then get processed in a distributed fashion.
Instead, Spark follows a model where events are collected and then processed at short time intervals (few seconds) in a batch manner.

Spark’s approach to fault tolerance is that instead of persisting or checkpointing intermediate results, Spark remembers the sequence of operations which led to a certain data set. So when a node fails, Spark reconstructs the data set based on the stored information

LOGSTASH
provides centralized logging server that listens for connections from Java application servers, accepting streams of logs when they connect & it filters, modifies, and routes those streams to the appropriate outputs.Its one of the corner stone product offered from elasticsearch family.

ELASTICSEARCH

Distributed (through shards) and Highly Available (through replicas) Search Engine. Its Document oriented, dynamic and provides reliable, asynchronous write Behind for long term persistency with (Near) Real Time Search support.

Buzzwords:
Shard : a single Lucene instance
Replica : is a copy of the primary shard for fail over and performance
Index Type : is like table

Index can be sharded with a configurable number of shards & each shard can have one or more replicas.

Used native java ES APIs instead of HTTP - spring data wrapper
Increased refresh time to higher number (index.engine.robin.refresh_interval). (default is 1 second)
Increased the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10%
Increased the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (default is 5000).
Increase the memory allocated to elasticsearch node default is 1G
Decreased replica count (or nothing with 0), and increase it later for HA and search performance. Replicas can be changed @ runtime but not the number of shards.
Increased the number of machines you have so you get less shards allocated per machine.
Use scan mode for search while search all results.
Planning to use SSDs for translog & check the performance.

Lucene (search engine behind elasticsearch) requires these to be in RAM, so we should be paying attention to these "deleted docs", " norms", "terms index", " field cache", "doc values"
In lucene as term will be stored once per field, searches work by locating terms and finding documents. So initially size will decrease and will get stabilized after most of the terms are indexed. So don;t worry about initial numbers usually it translates into 20-30% of the content.

CASSANDRA

column-oriented way of modeling system that is eventually Consistent relies.

Buzzwords:
Column. The basic element which is a tuple composed of a timestamp, a column name and a column value. The timestamp is set by the client and this has an architectural impact on clients’ clocks synchronization.
SuperColumn. This structure is a little more complex. You can imagine it as a column in which can store a dynamic list of Columns.
ColumnFamily: A set of columns,
KeySpaces: A set of ColumnFamily.
Row in cassandra sense is a list of Columns or SuperColumns identified by a row key.

Cassandra allows only N (under 3) number of executions to be checked and immediately returns to value. Therefore, write can be conducted successfully even with failures in the replication node, which raises usability. Histories of failed write operations are recorded on a different node, and the operation can be retried at a later date (this is called “hinted handoff”). Since the success of replicated writing is not guaranteed, the data suitability is checked in the reading stage. Generally, if there are multiple replications, it is collected into one when reading. However, Cassandra keeps in mind that not all replications match, and reads the data from all three replications, checks whether it is identical, and restores the latest data if it is not suitable (this is called “read repair”).

HAZELCAST
As compute nodes doing processing, In memory data grid solution is great for building applications that requires real time latency.

MONGODB
is mmap’d linked lists of BSON documents with B-tree indexing.

OSGI - JBoss Fuse
OSGi can help in managing "ClassNotFoundExceptions" by ensuring that code dependencies are satisfied before allowing the code to execute. It also help with running with multi version jars and hot deployments with life cycle management.

OSGi verifies that the set of dependencies are consistent with respect to required versions and other constraints.
Package application as logically independent JAR files and be able to deploy. (gracefully without affecting or managing others)
Manage new level of code visibility in a bundle(JAR), can make public classes unavailable which is not possible with java access specifier symantics (private, protected,public, package) making deployment/upgradation easy.
Extensibility mechanism with like plugin (eclipse,netbeans)
Helps to dynamic service model in your application, where services can be registered and discovered both declaratively and programmatically during execution.
Shell scripting for JVM to investigate
JVM footprint for special-purpose installs could be greatly reduced (with positive implications for disk, memory, and security).
Benefits although looks great, but having other systems (containers) sharing the same jars (storm, hazelcast, etc..), I don't think its worth spending efforts with OSGI.

Its always better to streamline on library versions and manage them in a consistent manner. Its a nightmare with OSGI containers & we end up solving the jar issues most of the time. I think OSGI is not worth the effort when we have different container with shared jars.

References:
http://wiki.apache.org/cassandra/ArticlesAndPresentations
http://blogs.atlassian.com/2013/09/do-you-know-cassandra/

Faceted Navigation for mobile applications and NoSQL

2013-01-17T02:58:00.002-08:00

Recently I developed a android based application as part of proof of concept initiative. It was search application based on pre-defined static data that will get shipped along with application. Using lucene with android was pretty simple and straight forward. Having working with information retrieval space from past 4 years and having worked with one of best known swing based UI application (sitebuilder.yahoo.com), it did not take time to come up with decent working application. One of the challenge was to move data stored in database to lucene so as to make it full text searchable and build facets.
There were 2 points that were revelation for me "the power of facet based navigation in building user interface" and "using lucene as data store and viable NoSQL solution" after this exercise.

The interfaces for mobile(or web for that matter) that requires keyword text entry aren't suited for browsing. Facet based navigation is pretty effective when exact keyword is not known. Amazon is a pioneer in this space & their web site is great example of faceted navigation. Their cross sell and up sell features directly rely on faceting.Keyword entry is tedious,using hierarchical faceted metadata for content selection results in awesome experience especially for mobile application. With ever increasing power and storage power of mobile devices (or desktop client for that matter), we should effectively use them to give better user experience. Let's say if some-one looking for job with key words such "java architect technologist", giving auto completion with total results (like java(200), architect(10) location(20)...) improves the over-all searching experience better. We should actively look into our existing web/mobile applications where faceting can be effectively utilized.

Lucene makes creating a facets a breeze. TaxonomyWriter & TaxonomyReader classes are well designed that helps to build categories/dictionaries easily. All the existing applications can be re factored to make use this feature rather than using groupby SQL queries. Its is also pretty straight forward to extract categories from full text and not column oriented.
Developing a android application is a tedious job. Since all the java APIs aren't supported its painful exercise to migrate & also its really greatly time-consuming to debug the android application with simulator although we can appreciate the effort from eclipse plugin team. I used groovy swing builders to develop quick mock up before doing it with Android. I love JVM scripting languages for that.One of the main reason to learn these although we may not use it in our daily jobs is the ability to come up with quick working prototypes. Here is the script that I created in Groovy, it prints the violations in the code that from each package level. It would be easier to build such things in GWT as well. So we can refactor all these upfront.
The problem that got solved with the application was storing the data & query any information using the lucene (Both with API and lucene syntax) in sub-second. It can deal with huge data & schema-less, the main characteristics of NoSQL. Although lucene is not generally counted as one of the NoSQL solution, it actually is.
When some one asks to me about NoSQL, I can safely tell I have used one :) and buzzword compliant. But this is a much bigger issue, recently I was hearing session from Martin Fowler where he argued very well for the need to both forms of data storage, SQL and NoSQL types & used the term "Polygot Persistence". The challenge for technologists is to identify the best contextually applicable use cases.
I started looking more into subject more & here are my notes. Although I am not in position to strongly recommend one over the other, definitely readers will appreciate the (NoSQL against SQL)value they bring in to the table and can make technically informed decision.

Schemalessness is one of the main reasons for interest in NoSQL databases - But we can generate schema-less design in relational database as well. There is a masterpiece discussion from asktom where "Generic Data Models" with dynamic capabilities have been regarded as anti-pattern defeating strong-typeness and query performance of relational database. I completely agree with that sentiment. These are types of cases, we should generally looking for Non SQL solution.

Generally, the data is significantly more important than the applications which use it - content is the king. The main reason is data outlives the applications that create it, the applications that use the data usually change over time. The schema and strong typeness is very important and in the long run will help the organization. We should be doubly cautious before deciding to put the data in non-SQL based storage & SQL is a safe option from manageability and staffing perspective for utility projects.
After all - "Misuse is the greatest cause of FAILURE & we need to understand the merits/demerits before dismissing it.

High performance from NoSQL is not free, it comes with cost (high-performance look-ups at the cost of no or no easy support of relational queries through "joins" and reliability through "transactions", "consistency", referential integrity) that are inherently addressed/provided by SQL products. NoSQL solutions needs to bend too much to match with RDBMS.

When data no longer fits on a single RDBMS server or when a single machine can no longer handle the query load, some strategy for storing/sharding and replication is required. Doing with RDBMS (rows and columns with index) is not easy and viable from cost and scalability requirements. The primary reason to integrate in-memory-based solutions with a NoSQL DB is to reduce the cost per GB of data. The need of systems that can run across multiple machines is going to be an absolute requirement & NoSQL shines over there.

Here is quick overview of options that we have.

Key-Value Stores
USP - Caches/Simple domain with fast read access
Use case - Massively concurrent systems
Examples - Redis/Memcached/Tokyo Cabinet/Coherence

Column-Family Stores
USP - Write on a big scale for reading and writing
Use case - Massively concurrent systems
Examples - Cassandra/Google BigTable/Apache HBase

Document-Oriented Databases
USP - Contents are documents with changing schema
Use case - Dynamic Data Structure
Examples - MongoDB/CouchDB

Graph Databases
USP - interconnected data with nodes and relationship
Use case - Social Graphs with many to many relationships/Recommendation engines/Access Control Lists
Examples - Neo4J/AllegroGraph/OrientDB

Ir-respective data source chosen, CAP theory states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees Consistency, Availability & Partition tolerance(CAP), Rules change with data and the context, but below rule remains the same to improve the performance.

"Look for opportunity to turn scan into a seek (cache) and use event based concurrency (asynchronous and parallel computation)"

Read Cache with In memory data grid solutions - Coherence, Hazelcast, CDN
Make computations asynchronus & parallel wherever possible. - JMS, Threads (event based)
Move computation and not data. - Map Reduce
Events in efficient machine parseable forms for effecient serialization - protobuff
Route new data through new partitions
Avoiding shared mutable state is the secret weapon to winning concurrency battles

I got interested in graph database, especially Neo4J and the problem that it tried to solve. Spring Data library samples were interesting. I highly recommend to try out spring data, spring source guys are awesome in creating the abstractions making developers life easier as usual.

here is the extract from "Neo4J in Action" book that explains the use case and technology behind the same.Social graph is being used as sample. Friends with many to many relationships. We can relate easily the differentiation it brings.

To find all friends on depth 5, MySQL will perform Cartesian product on 't_user_friend' table 5 times, resulting in 50,0005 records, out of which all but 1,000 are discarded. Neo4j,on the other hand, will simply visit nodes in the database, and when there is no more nodes to visit, it will stop the traversal. That is why Neo4j can keep constant performance as long as the number of nodes returned remains the same, opposed to significant degradation in performance when using MySQL queries
Each join creates a Cartesian product of all potential combinations of rows, and then filters out those that are not matching the where clause. With one million users, the Cartesian product of 5 joins (equivalent of query on depth 5) contains huge number of rows – billion billion billions – way to many zeros to be readable. Filtering out all the records that don’t match the query is too expensive – such that the SQL query on depth 5 never finishes in a reasonable time.
The secret is in the data structure – the localized nature of graphs makes it very fast for this type of traversals. Imagine yourself cheering your team on the a football stadium. If someone asks you how many people is sitting five meters around you, you will get up and count them. If the stadium is half empty, you will count people around you as fast as you can count. If the stadium is packed, you will still do it in a similar time! Yes, it may be slightly slower, but only because you have to count more people because of the higher density. We can say that, irrespective of how many people are on the stadium, you will be able to count the people around you at predictable speed – as you’re only interested in people near you, you won’t be worried about packed seats on the other end of the stadium for example.
This is exactly how Neo4j engine works in our example – it counts nodes connected to the starting node, at the predictable speed. Even when the number of nodes in the whole graph increases (given similar node density), the performance can remain predictably fast. If you apply same football analogy to the relational database queries, we would count all people in the stadium and then remove those that not around us – not the most efficient strategy given the inter connectivity of the data."

References:
Here is a pretty neat effort in to explain to support Unit testing NoSQL. A great way to learn as well. https://github.com/lordofthejars/nosql-unit#readme
Identifying the use case
http://highscalability.com/blog/2011/6/20/35-use-cases-for-choosing-your-next-nosql-database.html

Java.next() -> Scala or Groovy?

2012-12-29T09:55:00.000-08:00

The future is already here—it’s just not evenly distributed. —William Gibson

The Java.next() is going to be Java followed by Groovy and Scala. The dominance of java language is likely to continue but there are more and more use cases that compels us to use other JVM languages (with their functional programming capabilities) for developing newer platforms. Java7's new features like "Improved Type Inference","Multi-catch","try-with-resources statement","Strings in switch","New Garbage First Collector" can still fight with newer languages. Java8 is likely to close the gap further with the support of closures. Hence Java.next() would be multiple languages led by java, a technique also popularly known as polygot programming.

Project team will have heterogeneous level of competence (best/good/average bad developers), technical superiority is not always the best choice, cost of maintenance always exceeds the cost of construction. Java has proved to be the best choice although it may not be technically superior, there is no reason to believe that it will change dramatically in coming days.
I am personally sold on the productivity benefits of JVM languages (Scala/Groovy/JRuby) and their functional programming abilities. Having developed several internal projects during day work and few personal projects during free time given a chance, I always prefer to code with JVM languages rather than in java.

The assumption that Java is getting old its time look for new ideas is getting stronger day by day. Irony of the history that .NET started as truly multi-language with single platform as compared to java claiming one language running in multi-platforms, the current trend looks like there are lots of JVM languages gaining lots of mind-share but .NET still heavily tilted towards C#.
The question of what is java.next() itself seems to be strange because all who came to replace java as language compile to byte code and runs on rock solid JVM. I have been following JVM languages for a quite some time. I picked these Groovy,Scala languages (Only because I can code in these languages and following seems to be higher for these 2 compared to any other JVM languages) as contenders.
I also picked up the parameters that suits java developers thinking.
1. Can I get my things faster and hence spend less time in office. (Here "my things" are general programming requirements as I perceive)
2. How easily I can pick them and can convince both technically literate and il-literate stake holders.
3. How easily I can get jobs and is it going to offer better salary compared to java

Thought Leaders opinion about the the languages
Generally the understanding is that scala has the ability to disrupt the status quo rather than incrementally improving it like Groovy. Scala has done and doing good job of reducing the cost of abstraction transliterate Java into Scala and end up with bytecode that is almost exactly the same. Scala has some original good ideas and well thought out type system.
It does look like java programming Intellectuals move towards scala.
Here are the examples.
- Spring creator (Rod Johnson) endorses Typesafe and his dislike for groovy is quite evident as I heard from a podcost
- Java creator (James Gosling) prefers scala over ther JVM languages
- Groovy original creator (James Strachan) says scala as better option in the long run to replace java and so the JRuby creator (Charles Oliver Nutter )
But I think there will be some vested interest by some of these folks taking extreme stand and hence we need to take them with pinch of salt.

Winner : Scala

Job Trends
"Where the rubber meets the road" - The moment of truth.
Making more money with enjoyable effort (I don't say least effort) is a general end goal with technologies. Job opportunuties are the best borometer. They are the real reason for anyone to invest in a technology & probably single most important parameter to believe the hype in technology.
As web technology development is dominant one Grails popularity and active SpringSource support is driving the results.

scala,groovy Job Trends

Scala jobs - Groovy jobs

Winner: Groovy

Learning curve
Compared to Groovy, Scala has been harder to learn for me (still learning of course). The ability to write scary programs in scala is big NO for enterprise development. We are always going to have mixture of good/averege/bad and great developers.
Although Martin (author of scala) tried hard to prove other way round brilliantly From my experience I say it just more hard for java developers.In groovy I can just close my eyes and type java code if I get struck with some groovy concepts while learning that can't happen with scala.

Winner : Groovy

Multi-core programming
For the past 30 years, computer performance has been driven by Moore's Law;(the number of transistors on integrated circuits doubles approximately every two years). We don’t get the same increase in clock frequency as we used to, instead, we get more cores. CPUs aren't get faster anymore, it’s physically difficult to do so. Programming languages that can handle this better is likely to succeed more in coming days.
Scala supports multi-core friendly language constructs and libraries than others. (Im-mutability and functional programming is baked into the language). Although there is nothing inherently great about scala (which runs on JVM anyway) compared java. GPars from groovy is also fair. Scala folks says if we stick our head in the sand we will be left behind. This is most preferred argument from scala evangelists to promote. I do think there is huge merit in this argument by seeing what libraries like akka are able to achieve although famous TestNG creator thinks otherwise.

Winner: Scala

Web Applications
Grails and play framework are the show pieces of Groovy and Scala respectively. They beat other java web frameworks in the page oriented MVC architecture hands down. The productive gains are pretty huge to overlook. The approach is not only new and intelligent.
More and more work is getting done at the web browser, mainly through java script. This trend likely to continue the future as well. With lots of great library pushing the limits of ability of java script (JQuery, AngularJS, Backbone ...).
Both the frameworks are way ahead of other frameworks in supporting new java scrpit libraries, web soeckets etc... JSON/XML rendering from controllers of play and grails looks so innate behaviour of the framework unlike others. Both supports MVC and stateless architecture much better than their counterparts.
I think there is scope to write component oriented single web applications is pretty huge. GWT is still the best option for java programmers. Relying on some code generators is always not good. Coffee script(better way to write java script) also looks nice.

Coming to performance, performance of most web applications is I/O bound, so the practical performance impact of Groovy may be largely offset by the (developer) productivity gains it offers. Play framework folks thought otherwise and moved from groovy based templates to scala one. But I still feel for most of the applications the first assertion stands, and groovy offers awesome productivity benefits.
The plugin framework eco system of grails is miles ahead compared to play, definitely a big plus.

Winner : Groovy

Mobile platform
Its difficult to imagine any new application that runs only on web. Although web apps will work with mobile as well with new frameworks fluently with mobile centric CSS/Script supoort, I believe HTML5's inherent advantages with distribution, monetization, platform power and network effects aren't enough to match with rich user experience with native in mobile apps.
I firmly believe that both of them will have their own space in the days to come and we have to live with building 2 kinds of applications. Java android SDK is the viable option for java programmers. (although there are some options to run other way)

Winner : Java

Writing DSLs
"Say What You Mean, Mean What You Say" - bridging the gap between what the user and a programming language. Ability to defining Domain Specific Language is very important feature. There are lots of use cases where it can be used to simplify most of the tasks. (new rules, Data entry, API usage, Testing, build systems, deployment etc...). Writing fluent API and XML are definitely verbose of way of defining new DSL. Resorting into using ANTLR, JavaCC is also overkill in most of the cases with huge learning curve/maintenance overhead that they bring in.
Ability to modify the existing class at runtime (meta programming) and language ability for operator overloading flexible syntax plays very important role.All functions can be used infix (obj.method(arg) or obj method arg) makes the writing of DSL a joy. I created fluent interface library myself and a great fan of functional libraries like Guava, and functional java. But we are bending too much to use functional programming with java and isn't natural.

Winner : Groovy

Both Scala/Groovy wins hands down over java.

Database operations
Most of the development involve large chunk of code dealing with bringing data from various data sources (mainly relational) and updating the same. JPA (Its originator Hibernate) tried solve impedance mismatch between the object oriented code and relational data. After evaluating some of the scala framework I have come conclusion, JPA fixed the issue and not solved it. Functional programming is the best way to deal with row oriented data and success is not hiding the SQL details from developer but making as easy and transparent to developer without losing type safety.
Its thumbs up for slick and other several scala frameworks making JDBC operations simpler from scala world.

Winner : Scala

Big Data
In general appetite for insight into the details is increasing.Data inflow is increasing with larger velocity, its becoming difficult to assimilate the insight information. When we deal with larger data we need software that monitors and heals the software. Pretty big challenges.
We need software that exploits the hardware and new multi-core cpus better. The functional programming paradigm and immutable friendly programming language plays very important role.

Winner : Scala

Here are some interesting threads on this topic I found interesting against Scala. You can also see spirited fight from scala enthusiasts as response.
Switching back to Java:
Scala feels like EJB-2
Yammer moves back to java from scala

Introduction to Hadoop

2012-03-03T00:49:00.009-08:00

My notes on Hadoop - I never got an opportunity to directly work with Hadoop but currently looking into in house POC application that analyzes logs, Hadoop remains attractive for solving huge data that was majorly required by big companies (& mainly social software companies that deal with big data) but likely to expand into other companies as the data emitted by the world is growing exponentially. As I see from job description and salary perspective hadoop engineer is at the top & rightly so.
Hadoop has passed the hype cycle & is stable now. Now Hadoop is with Plateau of Productivity.

My notes here on hadoop should make readers hadoop buzz words compliant & give better idea of over-all picture.

What is Hadoop?

A cost-effective, scalable, distributed and flexible big data processing framework written in java based on Map-Reduce algorithm. MapReduce is a algorithm proposed by Google that breaks complex tasks down into smaller elements which can be executed in parallel.
Hadoop at core consists of two parts, one a special file system called Hadoop Distributed File System (HDFS) and Map-Reduce Framework that defines two phases to process the data a map and reduce.

Why Hadoop?

Traditional storage with RDBMS is costly and unsuitable in follwing scenarios

Too much of data that too in unstructured format (100s of Terabytes, Peta bytes)
Large # of applications to use a single filesystem namespace in distributed execution space
In-expensive storage of large data, but can be still easily queried.
Big data with high volume & velocity

Relational databases are designed with good principles of data durability, isolation and independence but the design is centralized and tends to get disk IO bound for high write workloads. The use of locking at different levels (row, page, table level) for consistency/isolation and the need to flush transaction state to disk introduces scalability challenges requiring users to scale by deploying heavy machines (vertical scaling).

Many tech vendors like EMC,Y!, Twitter, IBM,Amazon,Oracle & even Microsoft has got Hadoop-oriented "big data" strategy. Hadoop has proven to work with many of the big companies & huge investment in hadoop and its related technology made by them makes it viable option for handling big data in years to come.

Hadoop Use Cases:

This wiki-page lists out 100s of companies and their usages alphabetically,

Here is the list that I picked from there.

Log files processing,
Machine learning
User experience optimization/Prediction of behaviours based on the patterns and build the recommender system for behavioral targeting with pattern discovery/analysis.
Billions of lines of GPS data to create TrafficSpeeds for accurate traffic speed forecast

(proximity operators, hub and spoke search terms, customized relevance ranking, stemming, white and black lists, Data mining, Analytics and machine learning) Data from various wide verity of sources Ex: sensors, cameras, feeds,streaming, logs, user interactions

Where Hadoop doesn't make sense

You cannot directly use Hadoop as a substitute for a database that takes query & returns result in milliseconds, so if there is requirement to have sub-second interactive reporting from your data, or using the data in multi-step, updates/insertions, complex transactions, an RDBMS solution may still be your best bet.

By design Hadoop is suited for batch index building,and is not proper for incremental index building and search.

Hadoop eco system

There are software add on components developed on hadoop to make life simpler, VMWare recently announced that Spring also will have support for Hadoop under their "Spring Data" umbrella.

HBase - Column-store database (of the order of terabytes) based on Google's big table
Pig - Yahoo owned DSL for Data-flow or routing data
Hive - Facebook owned DSL for routing data but based on SQL
ZooKeeper - Distributed consensus engine with concurrent access

All the big companies (Twitter,Amazon, Yahoo, Facebook) have something to offer over the Hadoop which is a good thing.

Hadoop Core components:

At core hadoop can be grouped into following

HDFS - Hadoop Distributed File System, is responsible for storing huge data on the cluster.
Hadoop Daemons - A set of services offering to work with the data.
Hadoop HDFS API - APIs to communicate with the various nodes (services) from applications.

Here is brief write up on each of the components:

HDFS:

This is a distributed file system designed to run on commodity(low cost) hardware & highly fault tolerent.

It provides the high throughput with large data sets (files). HDFS supports write-once-read-many semantics on files

In HDFS data is split into blocks and distributed across multiple nodes in the cluster. Each block is typically 64Mb or 128Mb in size.

HDFS Vs NAS

In HDFS Data Blocks are distributed across local drives of all machines in a cluster. Whereas in NAS data is stored on dedicated hardware.

HDFS is designed to work with MapReduce System, since computation are moved to data. NAS is not suitable for MapReduce since data is stored seperately from the computations.

HDFS runs on a cluster of machines and provides redundancy using a replication protocal. Whereas NAS is provided by a single machine therefore does not provide data redundancy.
Hadoop Daemon services or modules:

Hadoop is comprised of five separate daemons. Each of these daemon run in its own JVM.

Master nodes:

NameNode - This daemon stores and maintains the metadata for HDFS.
Secondary NameNode - Performs housekeeping functions for the NameNode.
JobTracker - Manages MapReduce jobs, distributes individual tasks to machines running the Task Tracker.

Slave nodes

DataNode – Stores actual HDFS data blocks.
TaskTracker - Responsible for instantiating and monitoring individual Map and Reduce tasks.

NameNode :

NameNode is heart of HDFS file system. It keeps the directory hierachy information of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself but just metadata. "NameNode" is a Single Point of Failure for the HDFS Cluster & makes all decisions regarding replication of blocks.Any hadoop user applications will have to talk to NameNode through Hadoop HDFS API to locate a file or to add/copy/move/delete a file.

Data Node :

A DataNode stores data in the Hadoop File System HDFS. DataNode instances can talk to each other, this is mostly during replicating data.

JobTracker :

A daemon service for submitting and tracking jobs(a processing unit) in Hadoop & is single point of failure for the Hadoop MapReduce service.

As per wiki:

Client applications submit jobs to the Job tracker.
The JobTracker talks to the NameNode to determine the location of the data

The JobTracker locates TaskTracker nodes with available slots at or near the data

The JobTracker submits the work to the chosen TaskTracker nodes.

The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.

A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.

When the work is completed, the JobTracker updates its status.
Client applications can poll the JobTracker for information.

Task Tracker:

Task Tracker is a slave node daemon in the cluster that accepts tasks (Map, Reduce and Shuffle operations) & monitors these task instances(Task instances are the actual MapReduce jobs), from a JobTracker.

Miscellaneous:

MapReduce programming model does not allow reducers to communicate with each other. Reducers run in isolation & hence can be zero as well.

Combiners are used to increase the efficiency of a MapReduce program. They are used to aggregate intermediate map output locally on individual mapper outputs. Combiners can help you reduce the amount of data that needs to be transferred across to the reducers.

Speculative execution is a way of coping with individual Machine performance.

Some Criticisms:

"Hadoop brings a tons of data, but until you know what to ask about it, it’s pretty much garbage in, garbage out." - There are limited use cases for this especially for generic programmer to fully invest on this.

"Does querying huge data sets win over the advanced algorithms applied over limited data" - I am skeptical about querying huge data

"While most of Hadoop is built using Java, a larger and growing portion is being rewritten in C and C++" - I thought Google Map-Reduce must be better & converting some components to C++ is not a good sign for Java

"Configuration parameters are pretty huge" - that's a design smell I guess it shouldn't that complex.

Implementing equals and hash

2012-02-08T21:46:00.000-08:00

All about equals() and hashCode()

Java does not provide direct support for associative arrays -- arrays that can take any object as an index. In Java we have Object class has two methods for making inferences about an object's identity: equals() and hashCode()
HashMap helps to lookup based on object as index, other hash-based data structures such as HashSet, LinkedHashSet, HashMap, Hashtable, WeakHashMap does the same thing.

There are two approaches to defining equality and hash value: identity-based, which is the default provided by Object, and state-based, which requires overriding both equals() and hashCode(). If an object's hash value can change when its state changes, be sure you don't allow its state to change while it is being used as a hash key.

There are some restrictions placed on the behavior of equals() and hashCode(), as explained in JavaDoc of Object class.
In particular, the equals() method must exhibit the following properties:
Symmetry: For two references, a and b, a.equals(b) if and only if b.equals(a)
Reflexivity: For all non-null references, a.equals(a)
Transitivity: If a.equals(b) and b.equals(c), then a.equals(c)
Consistency with hashCode(): Two equal objects must have the same hashCode() value

Here are some rules to follow:
    - if a class overrides equals, it must override hashCode
    - equals and hashCode must use the same set of fields
    - if two objects are equal, then their hashCode values must be equal as well
- Consistency with the equals() contract is a fundamental requirement to   every implementation of equals() . Not only the hash-based collections rely on reflexivity, symmetry, and transitivity, but everybody who calls equals() will expect exactly this behavior. Failure to comply to the equals() contract leads to subtle bugs that are difficult to track down, because they are conceptual problems.
    - The value that's likely to change or unique should be first to compare (like Id etc...)
    - Make sure hashCode() of the key objects that you put into the collection never changes while the object is in the collection or make it immutable
    - if the object is immutable, then hashCode is a candidate for caching and lazy initialization
Caching of hashCode value is useless with modren JVMs.The last one in the above proved to be wrong after my experiments, JVM seems to be intelligent enough to make it fast.

Morale of the story : If you think you can optimize some obvious stuff,don't do that & JVM might be already doing that.

Generally its assumed that hashCode provides a unique identifier for an object. But actually it does not.
According to The java.lang.Object documentation it should be perfectly ok to always return 0 for the hashCode(). The positive effect of implementing hashCode() to return unique numbers with the use prime number for unique objects, is that it might increase performance. The downside is that the behavior of hashCode() must be consistent with equals(). For object a and b, if a.equals(b) is true, than a.hashCode() == b.hashCode() must be true. But if a.equals(b) returns false, a.hashCode() == b.hashCode() may still be true. Implementing hashCode() as 'return 0' meets these criteria, but it will be extremely inefficient in Hash based collection such as a HashSet or HashMap.

Recently we had one problem with the old code, where in the equals was utilizing the hashCode() method in the equals(), but since some of the string values were getting the same hashCode(), the hashmap was giving unexpected results.
assert 2627 == "RU".hashCode()
assert 2627 == "S6".hashCode()
So its always better to be careful about this as it can manifest itself after a long time

I was looking into code generated by 3 dominant IDEs in java, I thought its interesting to share some of the information. Its safest way to use IDEs to generate code from code review & consistency purpose. I don't see anything wrong with generated code here.

For me IntelliJ is doing better here (anyway nothing wrong with others).
- provides a option to mark certain fields non-null and hence null check can be avoided which I think is a good thing to do.
- makes use of instanceof others uses getClass(), All classes in the hierarchy either allow slice comparison and use instanceof or they disallow it and use getClass() . The use of 'instanceof' if need be, it can match any supertype, and not just one class & it renders an explict check for "that == null" redundant, since it does the check for null already - "null instanceof [type]" always returns false. (Effective Java)
- code looks compact compared to others

There are excellent helper classes EqualsBuilder and HashCodeBuilder from the Apache Commons Langlibrary. can make life simpler, but I think code generated from IDEs are good enough

Reference:
Effective Java - Joshua Bloch chapter about equals() and hashCode()
Java theory and practice: Hashing it out

2011-09-30T07:29:00.000-07:00

Backward and Forward compatibility with Java serialization protocol:-

Java offers a nice protocol for storing and communicating the persisted information. In a distributed deployment environment the server code has to support both the new and old information when its anticipated the control over the deployment of different versions of the servers are not available. The code has to deal with both backward as well as forward compatibility. i.e. VERSION N should be able to read both VERSION N-1 & VERSION N+1 object.

Maintaining the backward compatibility is easy. i.e. Version N code can read Version N-1 code by maintaining the version attribute. The version attribute in the read() API will make sure that the reading of new information is skipped. This is quite straight forward to achieve through “java.io.Externalizable” interface provided java where we have the full control of the context information on what we need to save. But it becomes tricky when we try to read the information created by future versions. The problem with forward compatibility is that the Java de-serialization it has to know what the new stuff that has been added & also should know where the information has been added.

The solution explained below tries to solve the problem in a pure java with less overhead & suitable for systems that have already established using Java serialization with ”java.io.Externalizable” interface.

Here the top down approach is followed for the explanation, it starts with how the usage of API should look like VERSION 0,1 & 2 then show how that can be achieved through code.
The challenge here is to read VERSION-1 (N) data from VERSION-0 (N-1) class. The current solution targets supporting only N+1 version & not the future versions which is the only important part of 24 X 7 support systems in the event of release back out.
Let's take an example of class "Test" having 2 attribute in VERSION-0, VERSION-1 introduces a new attribute called "versionData1" & VERSION-2 introduces the one more new attribute "versionData2". To simulate the case the data is inserted in between the data. In real-time scenario we will have lots of classes with readExternal()/writeExternal() methods spread across all over the code, but usually controlled with single mother object.
SKIP_START & SKIP_END are the marker interfaces used to identify the new information and ObjectInputWrapper is the new class used to move the cursor to skip the details when we read new information from the old class.
"Test" is Externalizable class having 3 attributes. We will include a new attribute in the second version and see how a "Test" object with VERSION -0 (N) reads VERSION -1 (N-1) object.

This plugin architecture is quite scalable. ObjectInputWrapper can be injected with different serialization protocols like Google proto buffer, Oracle Coherence's POF, etc... using a using factory pattern like ObjectInputOutputProviderFactory that provides different implementation including custom logic to efficiently store & retrieve the information

VERSION = 0;

    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       out.writeObject(name);
    }
    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        name = (String) in.readObject();
    }
"Test{VERSION=0, number=10, name=praveen}"

VERSION = 1

This class has new String attribute called versionData1 with the value "versionData1"
    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       // Added for version1 in between
       out.writeObject(new SKIP_START());
       out.writeObject(version1Data);
       out.writeObject(new SKIP_END());
       out.writeObject(name);
    }
A marker interface class has been included to identify the new information that the older version can safely ignore.

    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        if(VERSION>=1) {
             in.readObject(); // we can safely ignore skip signals
            version1Data = (String)in.readObject();
            in.readObject();
        }
        name = (String) in.readObject();
    }

"Test{VERSION=1, number=10, versionData1=versionData1, name=praveen}"

VERSION = 2

    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
       out.writeInt(VERSION);
       out.writeInt(number);
       out.writeObject(version1Data);
       // Added for version2 data in between
       out.writeObject(new SKIP_START());
       out.writeObject(version2Data);
       out.writeObject(new SKIP_END());
       out.writeObject(name);
    }
    @Override
    public void readExternal(ObjectInput _in) throws IOException, ClassNotFoundException {
        ObjectInputWrapper in = new ObjectInputWrapper(_in);
        VERSION = in.readInt();
        number = in.readInt();
        if(VERSION>=1) {
           if(VERSION==1){
                  in.readObject();
           }
           version1Data = (String)in.readObject();
           if(VERSION==1)
       in.readObject();
           }
        }
        if(VERSION>=2) {
            in.readObject();
            version2Data = (String)in.readObject();
            in.readObject();
        }
          name = (String) in.readObject();
    }
"Test{VERSION=2, number=10, versionData1=versionData1, versionData2=versionData2, name=praveen}"
The skip data version can safely be ignored in future versions. It will be removed in write first and also from read on subsequent versions.

The 2 key classes used to implement are explained here.

The ObjectInputWrapper class will have read method implementations for each basic data type. The sample below shows just two methods reading and readObject for illustration. For real implementations, we need to create final ObjectInputReaderTemplate classes for each basic data type and reuse them in the read methods of ObjectInputWrapper & they can be cached to avoid lots of object creations. The profiling of this code showed that these light weight are pretty cheap to handle, JVM will make sure that there is overhead in terms of memory or the cpu by in lining them automatically.

This class makes use of generics and anonymous inner classes to move the cursor to skip the new information that version N was un-aware while reading.

One sample code having all the classes published in github

2011-08-20T02:02:00.000-07:00

Caching Exceptions in server applications -

In web applications it’s common that response page of certain types can be cached for given http request, similar approach could be used to cache the exceptions in the server side applications as well. If the reason for any exception known upfront that it will be applicable for given amount of time, there is a opportunity to cache these to reduce the load on resources. This could be huge gain in cases where a particular piece of code is doing heavy operations like cpu intensive tasks, making remote calls, Database/File manipulation etc..

Here is a solution making use of Java generics.

For Ex:

public static int mockHeavyOperation(String someData) throws FileNotFoundException,Exception{

Thread.sleep(3000);//simulating some heavy operation

if(true) throw new FileNotFoundException();

return 10;

}

Can be converted into

public static int mockHeavyOperationWithCacheExceptionHandler(String someData) throws FileNotFoundException{

return new ExceptionCacheTemplate<Integer,FileNotFoundException>(){

@Override

public Integer handle() throws Exception {

Thread.sleep(3000); //simulating some heavy operation

if(true) throw new FileNotFoundException("Checked Exception");

return 10;

}

}.runIn(new ExceptionCacheTemplate.ExceptionKey(someData,40000));

}

As we can see from above code extra 4 lines are doing the trick.
If we run the first operation 10 times it takes 30000 millis where as 2nd example takes only 3000 millis, that's a huge gain I suppose.
1. The input information that resulted in exception can be stored as part of Exception Key class. It can be any class including String as shown above as long as it implements equals/hashCode(). runIn method works directly with String as well. If any Exception re-appears within the interval 40000 as shown above the Exception will be thrown from cache.
2. Only the exceptions that are included as part of the generic exceptions,
3. A custom map can be provided to manage the expiring of cached exceptions, Default implementation will be managing through the value provided while creating cache key.
"public Map getCache()" can be overridden to provide different implementation
For example Google's Guava library provides awesome fluent interface to manage expiry of keys.

Map<Key,Graph> graphs = new MapMaker()
       .concurrencyLevel(4)
       .weakKeys()
       .maximumSize(10000)
       .expireAfterWrite(10, TimeUnit.MINUTES)
       .makeComputingMap(
           new Function() {
             public Graph apply(Key key) {
               return createExpensiveGraph(key);
             }
           });

It can be handled at the framework level making it completely transparent to the applications.

ExceptionTamplate is available in github

2011-07-17T10:13:00.000-07:00

Notes on TDD and Unit testing in general.
Most of the benefits of TDD are unquantifiable, Its tough to correlate better design, better ability to refactor the code to TDD practice. It requires a real practice to experience and appreciate the productivity benefits. When we are working with legacy code and is not easily testable, advocating TDD there is counter productive and refactoring such code to suit TDD is a hard sell & taking up such task is usually a thankless exercise.For green field projects TDD should be applied without a second thought, to my mind its not debatable anymore.

Here are some notes that I collected can be used to sell TDD during the discussions. For me TDD as effective tool of "COMMUNICATION" is the single most important point that should to enough to employ TDD. The lesser defects as result of clear executable communication should make all the stake holders namely Developers, Managers, Testers, Clients and finally the real users of software happy.
Tests prove that your code actually works resulting in fewer bugs, if we catch all the scenarios. Testing before the code is a design activity.These tests can’t replace system and acceptance testing,but they do supplement it & also fewer bugs that make it to QA.
We can improve the design without breaking it. Having unit tests in place, we can do powerful refactorings that can untangle the most challenging of system psychoses.Refactoring not only becomes cheaper, it changes the developer mindset to strive for better quality. When refactoring becomes cheaper, the quality of software continuously improves.Fixing PMD and FindBugs errors should not require management approval.
Unit tests are a way to make programmers have documentation as they hate to write a MSWord document & is a project management technique. So when we can’t remember how to use a class APIs, read the unit tests to find out.MS Word documents are not meant to be compiled and deployed, if we can avoid as much as possible, its good for everyone.Unit tests reduces the communication pain points. Its a shared language.You know what your code needs to do. Then you make it do it. Even if you don’t have a working system, you can see your code actually run and actually work. You get that great “I’ve done it!” feeling. Developers can avoid nagging questions "Have you written it", "Have you tested it" & finally "Have you really tested it". If tests are written and published, developer can always go & check from web tool.
Just try test-first if you want to be high on endorphins, proud about your work, and motivated to do more.They demonstrate concrete progress. You don’t have to wait a month for all the pieces of the system to come together. You can show progress even without a working system. Not only can you say you've written the code, you can actually demonstrate success. Of course, this is another distinction that traditional programming teaches us to ignore.
“Done” doesn’t mean you’ve written the code and checked it in.“Done” means the code actually runs in the system without bugs. Running unit tests is a step closer to the latter.Unit tests are a form of sample code. We all encounter library functions and classes we don’t know how to use and one of the first places we go is the sample code. Sample code is documentation. But we don’t usually have samples for internal code. So we’re left shifting through the source or through the rest of the system.

Test-first forces you to plan before you code. Writing the test first forces you to think through your design and what it must accomplish before you write the code which results in better code. This not only keeps you focused, it makes for better designs.
Test-first reduces the cost of bugs. Bugs detected earlier are easier to fix. Bugs detected later are usually the result of many changes, and we don’t know which one caused the bug. So first we have to hunt for and find the bug. Then we have to refresh our memories on how the code is supposed to work, because we haven’t seen it for months. Then finally we understand enough to propose a solution.
Anything that reduces the time between when we code the bug and when we detect it seems like a obvious win. We consider ourselves lucky to find out about bugs within a few days, before the code is shipped to QA or to customers. But how about catching them within a few minutes? That’s what test-first accomplishes with the bugs it catches.It’s even better than code inspections. Code inspections, they say, are better than testing, because using them to detect and fix bugs is cheaper than testing. After the code ships, it’s much more expensive to fix the bugs. The earlier we can detect and fix bugs, the easier and cheaper and better. That’s the advantage of having code reviews.Code inspections catch more bugs within a few days, rather than a few months, but It virtually eliminates coder’s block. Ever wonder what statement to write next? Like writer’s block, coder’s block can be a real problem. But test-first systematizes the structured part of coding, allowing you to concentrate on the creative part. You may get stuck on how to test the next bit or how to make the test pass, but you’ll never be left puzzling over where to go next. In fact, usually you’re left with the opposite problem: You know you need to take a break before you burn out, but you’re on a roll and don’t want to stop.Failed tests make better designs. Testing a piece of code forces you to define what that code is responsible for. If you can do this easily, that means the code’s responsibility is well-defined and therefore that it has high cohesion. And if you can unit-test your code, that means you can bind it as easily to the test as to the rest of the system. Therefore, it has loose coupling to the pieces around it.High cohesion and loose coupling is the definition of good, maintainable design. Code that is easy to unit-test is also easy to maintain.It’s faster than writing code without tests! Or to put it another way, skipping unit tests is faster, unless you actually need the code to work.
Most of the effort we spend on code, we spend fixing it after we’ve checked it in to the source-code repository. But test-first eliminates much of that waste by allowing us to get more of it right to start with and by making bugs easier to fix.
One of the real value proposition of unit tests is that we get a low-level regression-test suite,we can go back at any time and see not only what broke but where the bug is. Arguably with many frameworks around it’s a low-effort way to catch bugs before the build goes off to QA. Whenever a bug comes it will make life lot easier without the need of using a debugger many a times.Test-first catches some bugs within a few minutes instead of a few days. It is even cheaper than code inspections, code reviews.

2011-01-17T09:17:00.000-08:00

2011 - My Technical Goals

Make JavaIQ.in useful & interesting to Java developers

My child-hood ambition has been to become writer and editor to a magazine. My obsession with politics/history/technology, I guess hasn't helped me to make money, but it made me always feel complete and happy. Not sure by end of 2011, I will make JavaIQ.in a respectable online magazine, but I will definitely give my best shot at that. I also would like to refine my applications that I created as my weekend projects & publish them in this site.

Android - Develop mobile applications

My guess is that developing applications for smart phones, iPads will be the "big bet" in coming decade and Android is likely to lead in this area of innovation. My decade java programming experience should help. btw, I bought my android phone this year.

HTML5/Java Script - Learn it's effective usage

Some how I never liked Flash/Flex. Although I tried JavaFX (which apparently was waste of effort), never spent significant time with Flash. I hope create awesome applications with HTML5. JavaScript has been excellent lanaguage I hope to do plenty of coding using YUI, JQuery and GWT/JSNI.

Play/Grails - Developing web applications in simpler way.

The dumb struts & its similar web frameworks will have to be replaced with these awesome frameworks. I never had such a first time nice experience with any framework compared to Play. I intend to teach and develop few web applications with that this year.

JVM lanaguages - continue to invest in learning

I will continue to invest in learning Groovy and Scala & continue to look for places where I can use these to accomplish the tasks.

Happy & prosperous new year to all.

2010-06-03T07:04:00.000-07:00

Using groovy to search name for my newly born son.
I had the un-sorted list of names of the God Vishnu from where I wanted to select a name, so wanted to sort them out alphabetically & used Groovy for the same.
Groovy Code in Java style - Click to see the code

1    def top = """
2        <html> <head>
3        <script LANGUAGE="JavaScript">
4        <!--

5            function toggle(id) { var e = document.getElementById(id); e.style.display = e.style.display == "none" ? "block" : "none"; return false}
6        //-->
7        </script>

8        </head>
9        <body>
10       """
11   def htmlOut = top
12   def set = new TreeSet<String>()

13   new File('c:/vishnu.txt').eachLine {
14     set.add(it.toLowerCase().trim())
15   }
16   def ref = "a"
17   htmlOut += """

18           <a href="#" onclick="return toggle('$ref')">$ref</a>
19           <div id='$ref' style="display:none">

20       """
21   set.each {
22     if (ref != it.charAt(0)) {
23       htmlOut += "</div>"
24       ref = it.charAt(0)

25       htmlOut += """
26               <a href="#" onclick="return toggle('$ref')">$ref</a>
27               <div id='$ref' style="display:none">

28           """
29     }
30     htmlOut += it
31     htmlOut += "<br/>"
32   }
33   htmlOut += "<br/></body>"

34   writer = new FileWriter('vishnu.html')
35   writer.println(htmlOut)
36   writer.flush()
37   
38   println 'done :)'
39

Groovy with DSL MarkUpBuilder -Click to see the code

1    def writer = new FileWriter('c:/Names1000.html')
2    def src = new groovy.xml.MarkupBuilder(writer)
3    src.html {
4      head {

5        script """
6            function toggle(id) { var e = document.getElementById(id); e.style.display = e.style.display == "none" ? "block" : "none"; return false}
7          """

8      }
9      def set = new TreeSet<String>()
10     new File('c:/vishnu.txt').eachLine {

11       set.add(it.toLowerCase().trim())
12     }
13     body {
14       for (alphabet in 97..122) {

15         char current = ((char) alphabet)
16         Set<String> names = set.findAll { it.charAt(0) == current}
17         def size = names.size()

18         if (size != 0) {
19           a(href: "#", onclick: "return toggle(\"$current\")") {
20             span "$current ($size)"

21           }
22           div(id: current, style: 'display:none') {
23             names.each {
24               li it
25             }

26           }
27         } else {
28           span "$current ($size)"

29         }
30         br ''
31       }
32     }
33     println 'done :)'
34   }

Unsorted 1000 Names
Sorted 1000 Names with groovy
Groovy is indeed Java++, It adds nicer syntax, closures and fluent interface making coding a joy without much noise.
Here is the result from groovy program. Click on letters to hide/expand the names starting with those letters.

a (113)
b (54)c (20)d (57)e (6)f (0)
g (25)h (13)i (3)
j (22)k (58)l (10)m (64)n (33)
o (4)
p (81)q (0)
r (14)s (243)t (19)
u (8)v (119)w (0)
x (0)
y (22)z (0)

2009-09-20T22:53:00.000-07:00

Google App Engine & Cloud computing : Making world flat for Developers all over the world.
The playing field for developers all over the world has been levelled with the advent of cloud computing & web services. It's pretty much free for any developer to develop/deploy application over the Internet. There is no start-up cost.Scaling up can happen incrementally based business requirement with upfront investment & I think that's revolutionary in my opinion.
Here are my thoughts on Google App Engine, after deploying my first sample application.

Salient features of Google App Engine

Google started supporting java on their engine in April - 09 - Ref
Support for Google Id and Sign In
Automatic Persistence -JDO or JPA (standards based approach, implementing standard Java APIs on top of App Engine where possible. So instead of using the underlying App Engine datastore API, developers can program against Java Data Objects or Java Persistence API)
Local Development - Remote Depolyment
Scalability and free Use - Pretty cheap
Monitoring - Nice overview
Eclipse plug in - I did not face any problem with while deploying.
Limited support for JDK classes - When I tried converting my swing application to GWT one I found many of classes I used were not supported like java.util.Timer classes
Native threads can't be spawned

Google unleashed App Engine with support for Python in 2008(April). This was Google's first entry into on-demand application development and deployment. Developers were able to build, develop and easily deploy apps using Python. I think it has been successful in that.The economic impact of is that cloud computing disrupts the data center world by slashing the capital and skills required to deploy a web application. I am betting on success of this model.

About the current application: java typing tutor

I created this sample application within an hour with GWT & published it. Initially I tried to porting my one of Swing application "Java Typing Tutor" which I created after reading the steve yegge's blog long back Programmings dirtiest little secret - BTW I don't believe that fast typist should also be better programmer in general, but it was interesting write up. Whenever I get time I do intent to make this application on par with my feature rich swing application.

GWT - Although I feel very comfortable coding with this framework, (may be because I spent lot of time in coding Swing Apps) it's API limitations are quite annoying(java.util.Timer does't work etc...). Initially I had the impression that converting a Swing application to GWT application is straight forward & I had even the ambition to write even converter!(paint() for GWT widgets), But I guess now I realized that it's not possible. The programming style, approach are quite different. Once I used wingS framework to convert one of swing application to web - it was straight forward because of deep integration with swing, but now apparently that project has died. It definitely requires a different mindset to develop web application than developing swing application.

Finally some happy notes for swing programmers:
Wicket & GWT are great saviours for struggling swing developers : With Swing GUIS losing their relevance, these 2 framework provides great pathway to move into web development.

Over all I feel GWT & Wicket are great framework for Swing developers to develop web application in terms productivity. Rails/Grails (or any request/response framework) guys can never match component developers in terms of productivity :-)

Advantage of GWT clients (or any fat client) is that they can tap into the resources of the client PC they are running on (such as memory to store state) and thus scale much better for large numbers of users. GWT produces something that is more akin to applets, independent applications that happen to be hosted on web page.

But where as,

Wicket still assumes that you want to build at least part of your application the 'old fashioned'

way, so that it will work without JavaScript, turn up in search engines, can be bookmarked so on...

So component developers can satisfy both the worlds.

"It's stupidity. It's worse than stupidity: it's a marketing hype campaign," -Richard Stallman

Amazon looks to be better than Google for Cloud

2009-09-13T09:10:00.001-07:00

Elements of Java Coding Styles

Java Coding Styles: Here are some of the tricks which I felt useful for java developers & I have been using them a lot. Again these are personal preferences, consider what you like & disdain if you don't. By end of the day the software professional is all about getting "good working software, quickly and at low cost" that can sustain for longer time & solves the business problem at hand (with YAGNI caveat). I want to keep updating this whenever I see interesting ones.

Prefer smaller methods
They look beautiful, easy to understand/reuse/change & test.

'As I did 20 years ago, I still fervently believe that the only way to make software secure, reliable, and fast is to make it small.' - Andrew Tanenbaum

Dr. Venkat gives wonderful explanation on - how to convince your fellow developer to write short methods?

Use JUnit test cases; assert statements and method/variable names as documentation, writing comments in JavaDoc are obsolete and useless many a times.

The purpose of the assert statement is to give you a way to catch program errors early, using assertions to state things you know (or think you know) about your program can improve readability & are great in serving as comments, they are intended to be cheap to write, just drop them into your code any time you think of them. They are great tool of communication for a programmer & can get rid of dumb English stuff in the code with // & /** */.

"There is nothing as useless as doing efficiently, which should not be done at all." - Peter Drucker.

I hate to hear someone telling me to add javadoc comments especially for private methods. The method name variable name should be wise enough to reveal the intent. Fluent interface & design pattern can help a lot in naming variables, methods & class names. Over the years my variable & function names have become more verbose. It's so much easier to understand code from years ago when it reads like a sentence.

Big caveat with respect to assertions,

Assertions should be used to specify things that to be true at various points in your program for providing the documentation & definitely NOT for error checking and any active code. Assertions are disabled ("turned off") by default. Assertions were introduced in Java1.4 & can be turned on with -enableassertions (or -ea) flag on the java command line.

& for JavaDoc,

Code that isn't fully documented is unfinished and potentially useless. Javadoc comments on methods and classes should normally indicate what the method or class

It was necessary document the type of keys and values in a Map, as well as the Map's purpose, but now with Java5 Generics that is also not required. Prior to Generics I always used to code List/*String*/ = new ArrayList();

Some Java 5 features are smart; Look for smarter new libraries that exploit these features well

Use basic infrastructure libraries like Google Collections (I prefer this over apache collections), sl4j, Guice etc… as much as possible. We will learn to appreciate the value of Java-5 new features by looking into the source code of these libraries, Old coding styles have to be abandoned in favor new features.

Commenting the code with "if(false)"

In many a times we want to use different implementation for testing & commenting out code temporarily. If the method is very long it's painful to comment out the whole block. Let us say we have method like this.

public void sendMail(){

if(true) return;

// Lot of code for adding to database to event

...

// Transforming text with XSL etc...

....

}

if(true) return; -> This line has the same impact as the commenting out the whole stuff :)

Use double brace initialization (For non-performance intensive code), If possible use Google Collections in all the cases.

As swing programmer I have been using these quite long time. I have seen quite number of experienced java programmers aren't aware this. This really useful while testing. It's concise, requires less typing & more readable but comes with cost. Google Collections provide better same feature without any performance cost.

Map map = new HashMap() {{

    put(1, "one");

    put(2, "two");

    put(3, "three");

    put(4, "four");

    put(5, "five");

}};

Use protected & final where ever possible

Using final does not make much sense from performance point of view as modern JVM is intelligent enough to make use them efficiently, but still I feel they have lot of value in making the intent clear & in multi threading simple with im-mutable state.

Usage of "private" is over-rated & I think it's better to have it has protected making setting the code simpler.

Person p = new Person(){{

age=30; name="Suresh";

}}

where age & name are protected variables.

More & more you start using inner classes, you will get frustrate with java for lack of "closures" & will eventually move to Groovy or Scala (Like me :-))

Use Null Object Pattern & throw IllegalArguementException wherever applicable

The Null Object provides intelligent do nothing behaviour that helps to avoid problems with Null references (NPEs) & so called "A billion dollar irreversible mistake"

Prefer Unchecked Exception over Checked Exception

90% of time Unchecked Exception makes more sense than the Checked Exception. This is one of the most significant attribute of successful libraries like Spring & Hibernate. Use Checked Exception for recoverable errors (Business Exception) & Unchecked for remaining stuff.

Understand Soft, Weak and Phantom references in Java

WeakReference is a reference which doesn't have enough force to prevent Garbage Collector(GC) deleting object - Useful for caching
Soft Reference behaves like weak references, except when GC determines object is softly reachable & can be used to avoid OutOfMemoryError.
Phantom references are enqueued when objects are deleted from memory and get()method always returns null to prevent resurrecting object. So phantom references are good for determining exactly when object is deleted from memory.
__________________
Interesting quotes from Rock star programmers that I collected.

"Write lots of code. Have fun with it!" — Joshua Bloch

"Learn to use your tools. And I don't mean just enough to get by. I mean really learn how to use your tools." — Tor Norbye

"Don't use line numbers. Don't put your entire application in one method." — Chet Haase

"Don't be overwhelmed by the language or the platform." — Raghavan Srinivas (In the same line Neal Ford says "When you were hired by your current employer, you may think it's because of your winning personality, your dazzling smile, or your encyclopedic knowledge of Java. But it's not. You were hired for your ability to sit and concentrate for long periods of time to solve problems")

"Millions of people have been employed because someone at Sun Microsystems invented Java." - Masood Mortazavi. Master it & you will never regret for that

"There will always be opportunities for great engineers, but as I said earlier, I think the number of these opportunities will shrink as other, less technical personnel play larger roles in the software-development process, using more productive, higher-level tools and frameworks than we have used in the past." - Ben Galbraith

"Google makes finding information easier than ever, but nothing beats interacting with an expert."- Ben Galbraith. So always associate with people from whom you think you can learn.

From Pragmatic Thinking & Learning

There is no expertise without experience.It takes something on the order of ten years/ 10,000 hours
of practice to be expert in a field Deliberate, thoughtful practice is what makes the difference—not
just going through the motions.Practice doesn't make perfect, but it does make permanent:
neuroplasticity will cause your brain to re-wire itself according to what you do.You may not become
what you dream, or what you aspire to be, but you will become what you do.Unfortunately there is no substitute for hard work.

References:

http://java.sun.com/developer/technicalArticles/Interviews/studentdevs/index.html?intcmp=2225

http://www.agiledeveloper.com/blog/PermaLink.aspx?guid=8a745e85-2a34-4d9c-8c25-ca371530e281 - How to convince your fellow developer to write short methods?

http://weblog.raganwald.com/2007/04/rails-style-creators-in-java-or-how-i.html - Rails style initializers

http://www.refactory.org/s/double_brace_initialisation/view/latest

http://c2.com/cgi/wiki?DoubleBraceInitialization

http://bwinterberg.blogspot.com/2009/09/introduction-to-google-collections.html

http://thestrangeloop.com/sessions/ghost-virtual-machine-reference-references

http://juixe.com/techknow/index.php/2009/11/18/favorite-programming-quotes-2009/

http://www.readwriteweb.com/archives/top_10_software_engineer_traits.php

2009-06-26T00:03:00.001-07:00

ALL ABOUT UNIT TESTING

"First about Boring Theory"

"Unit Test is the smallest piece of testable part of an application" In computer programming, unit testing is a software verification and validation method where the programmer gains confidence those individual units of source code is fit for use. A unit is the smallest testable part of an application. The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect. Each unit is tested separately before integrating them into modules to test the interfaces between modules. Unit testing has proven its value in that a large percentage of defects are identified during its use.

"In Java Unit Test cases means JUnit test cases, the single most importance of Spring & Guice (or any dependency injection framework) is to make unit testing easier"

JUnit is the de-facto framework for unit testing in java world. JUnit is a simple library, although there are mock objects, test code generators, behavioral test design & many other tools based on dynamic languages JUnit remains viable option while testing libraries or API where developer is a end user. Bob Lee (Author of Guice) stresses on the point that single most importance of dependency injection framework or interface driven design for matter is easier testability. All Google great applications like Gmail, Google Adsense, Calendar are the great testimony of this fact.

"Developers don't like writing unit test cases; Management needs to understand the technical debt associated with un-availability of test cases"

Let's face it, developers don't like writing unit tests & write documentation. Kent says, Software, like golf, is both a long and short game. JUnit is an example of a long game project – lots of users, stable revenue, where the key goal is to just stay ahead of the needs of the users. So it's hard to sell writing JUnit cases for small projects that don't have longer life. It's clearly avoidable overhead in such cases (Most of the web applications). It may not economically make sense to write extensive test cases for short lived & small applications.

In some cases developers hate to be embarrassed & look stupid when someone finds a mistake or highly technical guys think they don't need to write test their solid code. The first case can be handled through management as it's a purely competence issue which can be sorted out through training & other means, in second case it's hard to convince as these guys very much correct in their assertions in their own way. It's an attitude problem, the best way would to be deploy someone to write unit test cases. A "high level of quality code" is great, yet most software lives on and on and people expect to add/modify features in that software or debug it, it's economic requirement that super stars need to prove that their code works with unit test cases. How much will the maintenance costs are without unit tests? How much more risk does that add?

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" -- Brian Kernighan

"Unit testing seems to a lot of managers and developers like pure overhead, but professionally responsible developers know that it is one of the keys to quality." - Neal Ford

"Solid test cases with 100% coverage provides the courage to refactor the code & the reduces the testing effort in future"

We need to educate ourselves it's economically makes sense to have solid test cases especially for pure library (API)providers, as cost associated testing & testers can be completely (almost) avoided as anyway applications will test the APIs & we can avoid duplicating of the testing effort of testers in testing the APIs. Unfortunately these long sighted approaches are difficult to sell to the management and it's become thankless job when the unit tested solid code can't be differentiated with the working code. Developing APIs is a marathon job & not a 100 meters race. Stamina & perseverance plays very important role. In most of the cases committed can't be taken back, I guess JDK deprecated APIs must be haunting, humiliating the initial designers. One of the benefits of JUnit test cases are that developers get first had experience of the developers using the same.

As time goes on, there will be cases where the code works a bit less, some minor bugs, and some dirty quick fix (or hack) happens. Since you don't want to touch that code, you'll put fixes/enhancements/workarounds in other parts of the code, slightly but constantly degrading the quality of your design. You won't even upgrade a depending library, since you can't easily run regression tests over it. In a shorter time than you expect, that good designed and implemented project will turn into a nightmare. So it's just not about changing code, it's about changing environment - RDBMS vendor, JDK version, Library versions, OS versions…

"JUnit can be used to write End to End functional Tests & Unit test cases needs to be reviewed"

From definition, a test is not a unit test if:

It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.

If we go by above principle 90% of our JUnit test cases don't pass above rules. Although POJO driven frameworks like Spring tries to solve it, I don't think we can use JUnit in it's pure form. It's ok to use the JUnit for functional and integration test cases as well.

"Test Before Code, or perhaps Test Before Design." – That means unit test cases need to be reviewed before the design or coding. These reviews probably should be more thorough than the code reviews itself.

"JUnit test cases can serve as great tool to document"

Probably writing documentation through Javadocs is a bad idea. Usage of verbose class names, method names & JUnit test cases is more scalable & efficient way of documenting classes. Communicating through the code is the best way communication.

"JUnit test cases have to be efficient & succinct"

Manual test hurts both economic wise & manageability wise. But by end of the day if test cases are not capturing the correct scenarios & worst part if we have repetitive test it really JUnit really doesn't help. Best of the people involved with software development needs to do this type of unit testing. Garbage in & Garbage out rule is perfectly applicable here.

I am done with all my legal points to sell unit testing? Do you guys buy this argument? J

My next topic on unit testing would be on patterns and anti-patterns while writing test cases.

Resource:

http://www.artima.com/weblogs/viewpost.jsp?thread=126923

http://www.theserverside.com/news/thread.tss?thread_id=51615

http://www.junit.org

http://c2.com/cgi/wiki?WhoIsUsingJunit

http://www.theserverside.com/news/thread.tss?thread_id=51615

http://c2.com/cgi/wiki?FunctionalTest

http://www.davenicolette.net/articles/functional_tdd.html

http://www.logigear.com/newsletter/api_vs_unit.asp

http://www.exubero.com/junit/antipatterns.html

http://www.infoq.com/news/2009/06/test-or-not

http://www.agitar.com/solutions/why_unit_testing.html

2009-04-24T12:27:00.000-07:00

Search Domain Basics : Search is probably the most pervasive technology domain of this century. Here I have tried to cover some basic concepts & some software implementation details with java as the focus. I thought this information can help new comers to this domain & provides enough starting pointers to dig into more details.

"Content is King" - Content is what that drives the web & "search" is the engine for that. All the services providing the content has to employ search techniques to fetch the required information with minimum possible user inputs in fastest possible way. Google/Yahoo which have become synonyms with search is implementing all the applications in relation with search one way or the other. Apparently unstructured content forms the major portion of the content available in the universe. Unstructured data (or unstructured information) refers to (usually) computerized information that either does not have a data model or has one that is not easily usable by a computer program or in simple words any data that is not represented in terms of column names in RDBMS table schema. Parallel computing, Data sharding, Schema definition, Scale of data & nature of the content acquisition related with un-structured content makes it unsuitable to be solved from 100% RDBMS solution, although full text indexing does exist in the RDBMS world.

Raw data with context is called 'Information' & Information search and retrieval is all about locating relevant material from collection of raw data in a fastest way taking minimum possible input from the users. The ability to aid and assist a user in finding relevant information is the primary goal of information engineers & information Retrieval (IR) libraries.

Major parts for search engine:

fetching/Loading the document – downloading content (lists of pages) that have been referenced.

analysis – analyzing the database to assign a priority scores to pages (PageRank) and to prioritize fetching.

indexing – combines content from the fetcher, incoming information from the built-in data source, and link analysis scores into a data structure that’s quickly reachable usually using cache services.

searching – returns set of content that ranks pages against a query using an index.

database – keeping track of what documents with various context information helping the ranking better.

To scale to billions of documents, all of these must be distributable, i.e., each must be able to run in parallel on multiple machines. This should happen by throwing more hardware into the pool, without massive reconfiguration whenever scale up is required.As we cannot offer to have failure of any single component cause a major hiccups; a search solution must be able to easily scale by throwing more hardware into the pool, without massive reconfiguration; and things should largely fix themselves without human intervention. This can only be possible with the stateless implementation of software services.

Search Engine with Java for unstructured content: Studying the Information library APIs gives better insights into what search engine is capable of providing the service & I am taking the lucence as the sample for that.

Lucene library has become defacto IR library in java world, now lucene has been ported to almost all the major languages showcasing the popularity & capability of this small library.

Lucene is a search and retrieval library providing key word and fielded search. It can use boolean AND, OR, and NOT to formulate complex queries & can use fuzzy logic that is useful when searching text created by optical character recognition. Un-structured content far exceeds the structured content in the web world. Lucene mainly deals unstructured content & can effectively search structured content also with the field tags.Lucene provides minimum required information retrieval functionality. We can call this a "SearchKernal" library that provides full-text search and indexing functionality. Instead of an out of the box application, Lucene offers a usable API for programmers and operates on a lower level.There are off the shelf libraries (Compass, Nutch, Solr...) providing monitoring, transaction utilities over lucene. Commercial enterprise search offerings include from vendors such as, Autonomy, Google, Oracle & FAST (MS).Lucene does not search file by file. The search space is analysed first and translated into a normalised representation - the index. Lucene uses a reverse index. All words in the index are unique, that means the index is a compressed representation of the search space. Lucene only supports plain-text files. However, a variety of free open source document parsers are available for document types such as, RTF, PDF, HTML, XML, Word etc. Depending on the nature of the text content various analysers are on offer. For example, text can be analysed with a white space analyzer which breaks down the text in tokens separated by white space. To keep the response time short the process of generating and optimising the index is separated. The index gets normalized by applying a stemming and lemmatisation algorithms. Lucene beats the RDBMS in full text search in terms of processing speed, manageable reduced the size of the index footprint (now about 25 percent of the source documents’ size), easy incremental updates, support for index partitions, price & flexibility (index methodology, deploy options & schema evolution). RDBMS way of searching with where clause & LIKE % is not only scalable but ineffecient, although RDBMS like Oracle includes full text indexing capabilities they have not been as popular to independent solutions such as Lucene & also it's not easy to implement the parallel processing (map/reduce) invloving multiple machines & terabytes of data. It's the dynamic nature & context understanding nature of search bringing huge changes search domain with navigating the hierarchy is becoming old fashioned. With technical problems pretty solved in this domain, now the key thing to remember is "search methodology is much more important than the underlying technology".

Search Terminologies:

Proximity search:A search where users to specify that documents returned should have the words near each other.

Concept Search: A search for documents related conceptually to a word, rather than specifically containing the word itself. Involves parallel computing.

Boolean search: A search allowing the inclusion or exclusion of documents containing certain words through the use of operators such as AND, NOT and OR.

Proximity search: A search where users to specify that documents returned should have the words near each other.

Stemming: The ability for a search to include the "stem" of words. For example, stemming allows a user to enter "running" and get back results also for the stem word "run."

Lemmatisation: is the process of grouping together the different inflected forms of a word so they can be analysed as a single item.Lemmatisation is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech.

Noise or Stop words:Conjunctions, prepositions and articles and other words such as AND, TO and A that appear often in documents yet alone may contain little meaning.

Thesaurus: A list of synonyms a search engine can use to find matches for particular words if the words themselves don't appear in documents.

Index: Normailzed presentation of words

Semantic Search: is a process used to improve online searching by using data from semantic networks to disambiguate queries and web text in order to generate more relevant results.

Web Search : Content is public & Generic.Uses keywords, Links (relevency) based some kind of historic traffic.

Enterprise Search : Also contains private documents that domian specific, Quality of content should be highest quality content & not necessarily popular. Information/metadata needs to be secure with role based access to the content.It has to support security (Realms, Roles), SLAs and many other requirements. Google & Yahoo do not provide enterprise search.

As of now I am interested in researching this topic in the search retrieval field, So will keep updating this blog with my research findings.

"Automatic annotation/Summary addition for content":Lengthy documents/text are boring to read. It will be great if someone or computer can automatically creates the gist of the content. Automatic creation of annotation is tough task especially for non-domain specific topics but can be predictable in domain specific cases. For example it might be easier to extract the information from judgments copy automatically (at least in routine cases that hardly requires special knowledge from legal experts to annotate) or through workflow for review with automatically annotating the content/documents. I see this as an interesting area.

Summary:

IR & Search domain is pretty complex subject requiring mastery over algorithms & data structures. Hope that I have been able assimilate the information related to search taking "lucene" as sample search engine library.

References:

Lucene Book : http://www.manning.com/hatcher3/

Google Search:http://en.wikipedia.org/wiki/Google_search

Useful wrapper libraries over Lucene.

http://lucene.apache.org/solr/features.html - Solr Features

http://www.compass-project.org/overview.html - Compass Features

2009-04-17T05:23:00.000-07:00

Some statistics about Java source code of popular open source libraries. Currently I am looking/learning a big system that has multi-million number of source code. I wrote a simple utility to extract information about the java source code just for fun. I ran this utility on many of "src" directory of open source libraries as well.This java utility takes a source code directory as inputs & traverses all the java code recursively inside that directory. It collects total number of active lines of code excluding comments, package count...

& here is the result.

********* JDK1.5 ********

Total # Lines = 850918

Total # of Files = 6556

Avg # Lines per file = 129

Total # of packages = 368

******** Hadoop 0.18.3 ******

Total # Lines = 129742

Total # of Files = 926

Avg # Lines per file = 140

Total # of packages = 66

******* Lucene 2.4.1 *********

Total # Lines = 69606

Total # of Files = 528

Avg # Lines per file = 131

Total # of packages = 16

******** Struts *********

Total # Lines = 63707

Total # of Files = 1040

Avg # Lines per file = 61

Total # of packages = 120

********* iText 2.1.5 ***********

Total # Lines = 96830

Total # of Files = 544

Avg # Lines per file = 177

Total # of packages = 55

******* Tapestry 5.1.0.3 ******

Total # Lines = 96395

Total # of Files = 1937

Avg # Lines per file = 49

Total # of packages = 103

It's not all surprising that one of the best prolific java coder comes best ("Howard") in the java world when it it comes to modularity.

********* iBatis *********

Total # Lines = 14132

Total # of Files = 202

Avg # Lines per file = 69

Total # of packages = 45

***** Hibernate 3.3.1.GA *******

Total # Lines = 173698

Total # of Files = 2102

Avg # Lines per file = 82

Total # of packages = 292

I guess these figures can be considered as standard while reviewing the modularity of any library.Do let me know if any one interested in code & having ANT target to analyze the

source code b/w releases.

2009-03-10T07:24:00.001-07:00

Interview with one of the best API designer in the world -Joshua Bloch.

Adding some other interesting wise quotes on API design that I have read...

When you design user interfaces, it's a good idea to keep two principles in mind:

Users don't have the manual, and if they did, they wouldn't read it.In fact, users can't read

anything, and if they could, they wouldn't want to.

Same rule applies for API designers as well:

Developers don't have the java docs, and if they did, they wouldn't read it.In fact, Developers can't read anything other than pressing "." against the object reference in IDE & wait for something to select, and if they could, they wouldn't want to.

Learnability,Effeciency,Memorability,Errors & Satisfaction remains the core of good interface design.

APIs should emerge from the needs of real applications, and that they should make common tasks super-easy as the demand for quality, validated designs far exceeds our capacity to create them.In 1996 it wasn't clear we could create a sufficiently fast language without primitive types and arrays, It wasn't clear how much boilerplate code would be required by anonymous callback classes or checked exception. So Java couldn't resist including primitive types,excluding closures in favour of anonymous classes & over-using checked exceptions.

Grady Booch->

"Great thing about objects is that they can be replaced". The great thing about Spring is it helps you replace them. Flexibility is much more important than the re-use.

Joshua Bloch says,

Public APIs are forever - one chance to get it right
Good code is modular–each module has an API
Thinking in terms of APIs improves code quality
Easy to use, even without documentation
Don’t let implementation details “leak” into API
Make classes and members as private as possible
Make variables "final" wherever possible

Now some economics:

Economic reality suggest that buying more memory can be easier and cheaper than to pay someone to debug code & it pays more in the long run to have understandable slow code than super fast cryptic code.

Once I heard from Architect in IBM conference (don;t remember the name) for choosing Java over C++ saying this,

"The most compelling reason for adopting Java over C++ is automatic memory management. It protects application from mediocre programmers. It eliminates many embarrassments of memory leaks & crash that randomly occur in production". He went on say that "So as a result we are trading with inexplicable crashes for slow performance (automatic memory management & database-centric storage).This makes sure that application at least works anyway"

"Rushing is at the root of all lack of quality" - Peter Calthorpe, architect

Reference:

Design Slides
Defensive Programming

Spring is Good

2009-03-09T13:48:00.001-07:00

javaiq.in - A playground for learning Java RIAs. I am planning to expose my experimental applications with GWT, JavaFx & Felx. I will also be creating applications mixing multiple open services from various vendors (Google,Yahoo, Amazon, eBay...) & create a combined value. I guess this is the area that has huge scope.

I created this site & made it public 2 months back. It was result of my experimenting with new techniques with sample applications/code snippets,I always felt that best way to sell any new technique is with real application. It's great to have useful apps while learning new technologies/framework,I hate to write throw away examples. I have been following web frameworks from past few years, I have spent large amount of time in validating/learning over the year. I am SWING programmer & was always trying out the samples from Internet/books. Learning (or Stealing) the good code snippets that I liked & thought of exposing them as useful apps. Having spent a considerable amount of time and energy with Swing, it was queasy feeling in my stomach to see all my Swing programming heroes (Like Chet, Romain Guy...) either have gone to Adobe/Google or have moved to JavaFx/Flex way. I have also lost faith in Swing. I don't expect to develop new pure Swing apps any more, but I guess I was able to grasp GWT, Echo, Wicket frameworks much better than any pure web MVC framework (Struts, Spring MVC...) developers who were not having exposure to swing. I guess that's the advantage still I can leverage.

Chasing entrepreneurship dream- "When you are not into inheriting money and because you do not belong to a rich family and when you aren’t an individual blessed with the talent of a sportsperson or an actor, the next best thing to do is to become an entrepreneur" - Anand Morzaria

A man is a success if he gets up in the morning and gets to bed at night, and in between he does what he wants to do - Bob Dylan

Now experts say it has become relatively cheaper to start a new web application or a startup. Moore's law has made hardware cheap; open source has made software free; the web has made marketing and distribution free; and more powerful programming languages & techniques are making development teams smaller & powerful. But actual life it is not that simpler, especially if you are fighting a lone battle.

It's really pretty time consuming process to create & manage applications. I went with cheap shared Java hosting which was unable to run my "stripes" web framework based application & no one was there to help me out by providing tomcat logs & I had to settle with few JSPs with trimmed version. It's pretty costly affair to run java apps in a proper manner. It's big road blocker if anybody wants to develop/expose web applications in java today. Sun has to fix this problem (shared hosting) if it wants to replace PHP/Ruby On Rails apps in small scale applications. (I will have separate notes with java hosting) Enthusiasm to code is very hard to maintain, actually I wrote most of the applications in a few days, but was difficult to keep the momentum. Perseverance, relentless resourcefulness (as paul graham calls it) is difficult to achieve without full time dedication. Anyway it was good exercise to know all these limitations.

If I look back, I seriously doubt my spending of countless hours in browsing blogs, learning 10s of java web frameworks has significant impact on carrying out my day today work or think better. In fact my guess is I have wasted >30% of total time spent in gleaning through useless/marketing literature links. Hopefully from now on I will be able channelize all my energy in building better context for my day today work rather than learning all java web frameworks under the Sun. BTW all my web framework heroes (Wicket, Appfuse, Stripes, Tapestry...) that I have been following from past one year are apparently looking for jobs themselves & are investing time in using Groovy, Scala & Clojure. :-( not really a happy situation. I guess innovation with pure java in this space has reached dead end. Now I want to to learn/invest more on my current work so that I can become better at what I am doing currently, that's my priority #1 & I will resist experimenting with what blog writers say on latest technologies. One of the trouble I had disclosing this site was public embarrassment! of people knowing that I created this half baked application that I hesitated to own. But now it's quite decent set of useful applications that I started as my week-end project & I have good ideas to make it still better & this will be always my low priority work & also make sure to extract the best out of I am currently working on.Oh! Now I can call myself CTO of javaiq. nice feeling. :-)

2009-03-06T00:02:00.000-08:00

Popular techie words: we hear below words a lot in discussion & blogs. I thought it's good to have their definitions (easier ones) & explain if anyone asks "what's that?" as I also use them to have buzzwords compliance.

Technical Debt:
Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.
The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.
Reference:
http://www.c2.com/cgi/wiki?TechnicalDebt
http://martinfowler.com/bliki/TechnicalDebt.html
http://www.youtube.com/watch?v=pqeJFYwnkjE

MapReduce:
MapReduce is hierarchical scatter/gather operation.
MapReduce is a library that lets you adopt a particular, stylized way of programming that's easy to split among a bunch of machines. The basic idea is that you divide the job into two parts: a Map, and a Reduce. Map basically takes the problem, splits it into sub-parts, and sends the sub-parts to different machines - so all the pieces run at the same time. Reduce takes the results from the sub-parts and combines them back together to get a single answer.
Reference:
http://scienceblogs.com/goodmath/2008/01/databases_are_hammers_mapreduc.php

Cloud Computing:
With cloud computing, everything is web-based instead of being desktop-based; access all programs and documents from any computer that’s connected to the Internet is possible. Cloud computing helps to do it more easily than ever before.

Wikipedia Cloud Computing is "a style of computing in which resources are provided as a service over the internet". Cloud computing user need not have worry about managing a machine or service at the physical level (Machine & location). Amazon Simple DB is an example for this as it handled operating system or database maintenance functions, SLA & operational issues.

Sharding:
Sharding or horizontal partitioning is about splitting up data sets. If data doesn't fit on one machine then split it up into pieces, each piece is called a shard.
Sharding is used when you have too much data to fit in one single relational database.

Reference:
http://highscalability.com/sharding-hibernate-way
http://highscalability.com/unorthodox-approach-database-design-coming-shard
http://lethargy.org/~jesus/archives/95-Partitioning-vs.-Federation-vs.-Sharding.html

Refactoring:
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations

Reference:
http://www.refactoring.com/
http://en.wikipedia.org/wiki/Code_refactoring

Code Smell:
In computer programming, code smell is any symptom in the source code of a program that possibly indicates a deeper problem.
Reference:
http://en.wikipedia.org/wiki/Code_smell

2009-02-28T10:51:00.001-08:00

DHH - The best talk I heard on startups

I was quite impressed by DHH (Creator of Ruby On Rails framework) speech on "The secret to making money online" when I viewed this video again today.

There are just 3 steps for this.
1. Create a gr8 product that people like & is useful to them
2. Have a price for the product & ask people to pay for it or it's usage.
3. Make profits!

This is so ridiculously simple grand old rule! he goes on to say that you don't need to be f***ing genius to achieve this.

He also came up with nice probability analysis that strike rate of success following above simple model is much higher rather than Yahoo, Google,facebook & youtube (setting up BillBoards/hoardings) way of attracting users providing free services & make use of web page real estate to show ads & make money. He also suggest to go slow on implementing the idea as "Finding a good cause is incredibly hard & time consuming" - Craign Newmark.

Simply superb.

I have been following many startup blogs including paul Graham's, but never impressed like this before, so simple words but telling you the hard truth :-)

2008-11-20T10:24:00.000-08:00

Nice way to tell what all you know with wordle

2008-11-11T04:52:00.000-08:00

Do we really need a web service api for serving static data when we already have a good query and mining language? (SQL, XML)

Google, Amazon & eBay provide the static data by publishing through their docs.
- Location information (City, Country...)
- Categories
- Currency data
- Hierarchy information

These are the advantages that I see from the above approach.
- Uses local CPU power, saves unnecessary traffic to the web service
- Supports online/offline scenarios
- Supports programming logic flow, since we will have the information at the compile time it's easy to code
- Solves the performance problem, what ever be the improvement network calls are always costly compared to local calls
- data model is essentially a flat data model, easy to import,save & navigate

But I guess they should be better exposed as XML rather than as HTML

The problems I see with this approach
- We need to have stringent way of updating the local cache of static code (usually stored in SQL or XML or as text file with some delimiters)
- static information that are easy to represent in code (say number is less than 10 & unlikely to change) are represented as Enums , so we will be 2 rules, some will be represented with static enums & some will be represented with static code
- There will be different rules for representing the data & we will be sacrificing the strong typing which I think is not a issue as benefits to the both parties (client & server) exceeds the pain with String based APIs.

As from the mature real SOAP based web services, it's clear that we don't need to expose API StaticCodeService() to get these static information. The demand for quality, correct designs far exceeds our capacity to create them. Web services should emerge from the needs of real applications, and that they should make common tasks ridiculously easy.

In Summary,
It's not worth wasting SOAP web service API calls for getting static information.

2008-11-06T20:58:00.000-08:00

Why Fluent Interface? Here are some thoughts on using fluent interface while designing APIs & their impact
Definition : In software engineering, a fluent interface (as first coined by Eric Evans and Martin Fowler) is an object oriented construct that defines a behavior capable of relaying the instruction 'context' of a subsequent call.

First philosophy,

The success of Conversation/Communication/Integration depends on the quality of signals between two entities involved & the signal is clean & is understood well. Communication is always based on context, shared context. Context also influences interpretation. For example , "use a fork" differently depending on context (Unix, dining table, Java or Ant build file). Well understood shared context improves the signal-to-noise ratio in communication it makes communication effective, expressive, easy to understand and easy to work with.
Domain Specific languages (DSLs) have implicit context, context is never mentioned, once the context is established repeating it again & again is a noise,. The end goal for involved parties in communication should be to reduce the noise to zero.
We have 2 types of DSL, Internal - based on exiisting language & external - A new language with parser & full fledged grammar, evidently external DSL is tough to implement & very effective (SQL,HTML) on the other hand Internal DSL is easy to implement especially with dynamic languages.

That's all about the boring philosophy...

The benefits,

No need to document the APIs.
- They are all self explanatory
- Examples should do all the explanation if any
- Testing/adaptability becomes easier

Less effort required to use, resulting in better economics
- There is very less chance to make mistakes
- All the extra noise is hidden which does not add domain concepts
- User has to write less & less number of lines of code, LOC most
of the time translate to number of bugs. Less the the code less the bugs

Correctly written fluent API gives the satisfaction of well written novel to
the author & the same reading experience to the consumer.

Market is always right, now all the new libraries are coming up using fluent APIs
Big software houses are putting more efforts to make code less noisy. The trend from Microsoft (C#3, IronRuby, F#...) Sun (JRuby & other dynamic language support) points towards that.

Experts think Fluent APIs are cool.
- Martin Fowler is writing a book on DSL
- Well proven frameworks like JUnit are coming up with fluent API alternatives
- Google collections, JMock, Fest, Guice... countless popular APIs are based
on fluent APIs
- Ruby on Rails Active Record is best example how a DSL can simplify the job & has successfully forced to think differently
- Market Signals shows that, best brains are talking more about DSL, functional
langauges & is definately are the way go about to develop software.
- Joshua Bloch (Google Java Architect) in his new Effective Java book talks about Builder pattern for building immutable fluent java objects.
buying more memory can be easier and cheaper than to pay someone to understand/debug code. Well written code (read fluent interface) results in good economics.

Now the real world code using Fluent APIs,

No marketing is better than showcasing real working code for technique or technology.

New Java Date APIs JSR 130:
Period thePeriod = Periods.periodBuilder().years(8).months(3).build();
This is what experts feel how the code should be written & this code is definitely looks better than java.util.Calendar,java.util.Date APIs.

Fluent way of handling XML marshalling un-marshalling:

Here we have sample example showcasing the XML usage with fluent APIs

<contacts>
<contact>
<name>praveenm</name>
<phone type="mobile">98862342333</phone>
<phone type="office">080-2344234233</phone>
<email>pm@aol.com</email>
</contact>
</contacts>

Groovy Sample:
def mkp = new MarkupBuilder()
mkp.contacts {
contact {
name("praveenm")
phone(type: "mobile", "98862342333")
phone(type:"office", "080-2344234233")
email("pm@aol.com")
}
}
Ruby Sample
require 'builder'
x = Builder::XmlMarkup.new(:target => $stdout, :indent => 2)
x.contacts {
x.contact {
x.name('praveenm')
x.phone '98862342333', :type => 'mobile'
x.phone '080-2344234233', :type => 'office'
x.email 'pm@aol.com'
}
}

Google Collections:
A excelellent example how JDK Collections can be simplified with fluent APIs
public static final ImmutableSet FAVORITE_NUMBERS
= ImmutableSet.of(2,9, 8, 15, 16, 50);

FEST Example: DSL-oriented API for functional Swing GUI testing

dialog.comboBox("domain").select("Users");
dialog.textBox("username").enterText("alex.ruiz");
dialog.button("ok").click();
dialog.optionPane().requireErrorMessage()
.requireMessage("Please enter your password");

JaxB common:

USAddress address = new USAddress()
.setName(name)
.setStreet(street)
.setCity(city)
.setState(state)
.setZip(new BigDecimal(zip));

I guess this is not a good example for fluent.

Guice:
@Override
protected void configure() {
binder().bind(IUserService.class).to(UserServiceMockImpl.class);
binder().bind(AuditInfo.class).to(DummyAuditInfo.class);
}

Hibernate:
List cats = session.createCriteria(Cat.class).setMaxResults(50).list()

DesignGridLayout:
A Fluent Layout manager

layout.row().label(label("Last Name")).add(lastNameField, 2).add(label("First Name")).add(firstNameField, 2);
layout.row().label(label("Phone"))
.add(phoneField, 2).add(label("Email")).add(emailField, 2);
layout.row().label(label("Address 1")).add(address1Field);
layout.row().label(label("Address 2")).add(address2Field);

Frameworks like Grails, JMock, JPA utilizes the fluent APIs. We also have some samples in standard JDK itself like StringBuffer, StringBuilder, ProcessBuilder etc...

These are the commonly seen techniques in Fluent APIs with Java.
- Method Chaining
- Nested Interfaces
- Builder Pattern
- static imports

And finally the problems with fluent APIs,

It's definitely not the 'wow' technique. People have been using this for a quite long time & now we have just fancy name 'Fluent API' that's it. So let's not think this is like OOP or OOAD or MDA. It's a simple programming technique for making code look better.

- DSL is about writing good essay. Programmers are not essayists
- Java is not suitable for DSL- It doesn't have closures, open classes , quite verbose & is filled badly written APIs.
- It's very difficult to get correct in developing DSLs.Fluent API can be useful for highly used API,otherwise the investment of developing may not be as effective as intended. Thus, not all API can be made fluent.
- Difficult to track down null return value issues that occur somewhere in the chain
- Difficult to handle the exceptions, especially while dealing with existing APIs having checked exceptions.
- One of my 'java friend' was not ready to believe some of the fluent samples that I showed him were actaully a java code, :-), so there is also -ve impact on readability
- Need to write more code to make code fluent.

Summary:
I strongly believe that DSL, both internal & external helps in developing better software that is easy to learn,extend, use & hard to misuse.

That's it, Hope that I was able sell fluent APIs to the new guys.

References:
Domain Specific Language Book by Martin Fowler
Domain Specific Language by Martin Fowler
DSL Boundary by Martin Fowler