#BYTESCROLLS

Boyer and Moore's Linear Time Voting Algorithm

noreply@blogger.com (Haris) — Fri, 15 Jul 2011 17:42:00 +0000

This is a simple linear time voting algorithm designed by Robert S Boyer and J Stother Moore in 1980 which is discussed in their paper MJRTY - A Fast Majority Vote Algorithm.

This algorithm decides which element of a sequence is in the majority, provided there is such an element.

Suppose there are n characters (objects/candidates). When the ith element is visited, the set can be divided into two groups,ca group of k elements in favor of current selected candidate and a group of elements that disagree.After processing all, we can conclude that candidate selected can be considered majority if there's any !

When the pointer forward over an element e:

If the counter is 0, we set the current candidate to e and we set the counter to 1.
If the counter is not 0, we increment or decrement the counter according to whether e is the current candidate.
When we are done, the current candidate is the majority element, if there is a majority.

I have written a simple java implementation.

Sometime ties may occur. But this algorithm doesn't fit as the solution.For an assurance, if the vote is greater than n/2, the candidate which is returned as majority it is announced to be the selected one. This counting phase can be done when the increment for the candidate happens.This algorithm is really effective when the data is read from a tape.The algorithm only works when at least half of the elements constitute the majority.

Reference
http://www.cs.utexas.edu/users/moore/best-ideas/mjrty/example.html

Descending iterator

noreply@blogger.com (Haris) — Thu, 14 Jul 2011 15:38:00 +0000

There is a descending iterator in linked list implementation in Java SDK. A humble private class in LinkedList. A good example of adapter.

calls up

public Iterator<E> descendingIterator() {
return new DescendingIterator();
}

Using Avro to serialize logs in log4j

noreply@blogger.com (Haris) — Mon, 30 May 2011 12:42:00 +0000

I have written about serialization mechanism of Protocol Buffers previously. Similarly, Apache Avro provides a better serialization framework.

It provide features like:

- Independent Schema - use different schemas for serialization and de-serialization
- Binary serialization - compact data encoding, and faster data processing
- Dynamic typing - serialization and deserialization without code generation

We can encode data when serializing with Avro: binary or JSON. In the binary file schema is included at the beginning of file. In JSON, the type is defined along with the data. Switching JSON protocol to a binary format in order to achieve better performance is pretty straightforward with Avro. This means less type information needs to be sent with the data and it stores data with its schema means any program can de-serialize the encoded data, which makes a good candidate for RPC.

In Avro 1.5 we have to use (this is different from previous versions which had no factory for encoders)
- org.apache.avro.io.EncoderFactory.binaryEncoder(OutputStream out, BinaryEncoder reuse) for binary
- org.apache.avro.io.EncoderFactory.jsonEncoder(Schema schema, OutputStream out) for JSON

The values (Avro supported value types) are put for the schema field name as the key
in a set of name-value pairs called GenericData.Record

Avro supported value types are
Primitive Types - null, boolean, int, long, float, double, bytes, string
Complex Types - Records, Enums, Arrays, Maps, Unions, Fixed

you can read more about them here

An encoded schema definition to be provided for the record instance. To read/write data, just use put/get methods

I have used this serialization mechanism to provide a layout for log4j. The logs will be serialized to avro mechanism.

github project is here - https://github.com/harisgx/avro-log4j

Add the libraries to your project and add new properties to log4j.properties

log4j.appender.logger_name.layout=com.avrolog.log4j.layout.AvroLogLayout
log4j.appender.logger_name.layout.Type=json
log4j.appender.logger_name.layout.MDCKeys=mdcKey

Provide the MDC keys as comma seperated values

This is the schema

Bloom Filters

noreply@blogger.com (Haris) — Sun, 22 May 2011 19:28:00 +0000

A Bloom filter is a probabilistic data-structure. This can be used to store a set of data in a space-efficient manner. For eg; a distributed cache called Cache Digests shared as summaries between the nodes to have a global image.

The data-structure can be used to provide membership queries ie. checkIfDataPresentInStore() If it is to check an element is already inserted in the filter then it will return true, there are no false negatives. But there can be chance if the element not inserted may return true. But the check for that element can be done in the original store ie. the overhead is associated with the rate of false positives. This is different from dictionary in which the hit/miss is deterministic.

For a set of n elements, a bloom filter can be a vector of size m.Initially, all bits are set to 0. For each element e, k hash functions will set k bits in the bit vector to 1. When a query for membership executed, it will check for the bit positions for the set value. If matches all, the queried element is possibly present in the store else, it is sure not present.Each hash function returns the index to set. This means we have to store these m bits per key. So a total of m * N bits of space required. The use of different hash functions results less collision.

Uses

Design a spell checker.
Database join implementation (Oracle)
Peer to peer (P2P) communication and routing
In HBase, the Bloom filter is stored as meta block in the HFile. When a HFile is opened, the bloom filter is loaded into memory and used to determine if a given key is in that store file. This can avoid the scanning region for the key.
and more

I found a java implementation here
Cassandra's java implementation here

Reference

http://en.wikipedia.org/wiki/Bloom_filter
https://issues.apache.org/jira/browse/HBASE-1200
http://wiki.squid-cache.org/SquidFaq/CacheDigests
http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf

Labs

noreply@blogger.com (Haris) — Mon, 09 May 2011 18:23:00 +0000

avro-log4j - serialization mechanism to provide a layout for log4j

firetester - A simple RESTful services testing tool written in Groovy Griffon framework

gitter - Publishes github activities to Twitter

jfilemagic (jfm) is an utility for identifying files using magic numbers or signatures

cometd-chat - a comet based chatter for fun

Interesting uses of sun.misc.Unsafe

noreply@blogger.com (Haris) — Wed, 13 Apr 2011 17:54:00 +0000

Inspired from the question that found in stackoverflow, I started looking up for the uses. I found some pretty interesting ones...

VM "intrinsification." ie CAS (Compare-And-Swap) used in Lock-Free Hash Tables eg:sun.misc.Unsafe.compareAndSwapInt it can make real JNI calls into native code that contains special instructions for CAS

What is intrinsification?

They are optimization done like compiler generating code directly for called method or JVM native optimizations. We know that there are VM downcalls from JDK like wait method etc. Its about low level programming. For eg:- the Atomic classes for numbers, they are pure numbers represented by objects but atomically modified in which the operations are managed natively.

read more about CAS here http://en.wikipedia.org/wiki/Compare-and-swap

The sun.misc.Unsafe functionality of the host VM can be used to allocate uninitialized objects and then interpret the constructor invocation as any other method call.

One can track the data from the native address.It is possible to retrieve an object’s memory address using the sun.misc.Unsafe class, and operate on its fields directly via unsafe get/put methods!

Compile time optimizations for JVM. HIgh performance VM using "magic", requiring low-level operations. eg: http://en.wikipedia.org/wiki/Jikes_RVM

Allocating memory, sun.misc.Unsafe.allocateMemory eg:- DirectByteBuffer constructor internally calls it when ByteBuffer.allocateDirect is invoked

Tracing the call stack and replaying with values instantiated by sun.misc.Unsafe, useful for instrumentation

sun.misc.Unsafe.arrayBaseOffset and arrayIndexScale can be used to develop arraylets, a technique for efficiently breaking up large arrays into smaller objects to limit the real-time cost of scan, update or move operations on large objects

References

Design and implementation of a comprehensive real time java virtual machine

Demystifying Magic: High-level Low-level Programming

Implementing Fast JVM Interpreters Using Java Itself

TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones

Efficient Checkpointing of Java Software Using Context-Sensitive Capture and Replay

How To Write Directly to a Memory Locations In Java

Accessing data from storage system using XAM API

noreply@blogger.com (Haris) — Tue, 12 Apr 2011 17:00:00 +0000

If you are familiar with POSIX which is a collection of standards which enables portability of applications across OS platforms. It provide a standardized API for an OS, eg:- functions like 'open' to open a file, 'fdopen' to open stream for a file descriptor, fork to create a process etc. It defines file descriptor as a per-process, unique, nonnegative integer used to identify an open file for the purpose of file access. The Unix systems programming API is standardized by a large number of POSIX standards. Most operating system APIs share common functionalities like listing directories, renaming file etc.

XAM stands for "eXtensible Access Method". The XAM API will allow application developers to store content on a class of storage systems known as “fixed-content” storage systems.

The data centers have evolved into a complex ecosystem of SANs, fiber channels, iSCSI, caching technologies etc. Certain storage systems are efficiently designed to store the documents that donot change, which is useful for preservation or archival. Digital imaging storage systems which us used to archive XRAY DICOM images, ie. documents that are not edited and to be kept for a long time need can use fixed storage. The applications accessing these solutions need to be written using only those vendor-specific interfaces. What if one changes their infrastructure? The application has to be rewritten. This scenario is an analogy to the need for JDBC/ODBC driver specifications as a standard API to interact with different vendors. XAM API specified by SNIA is the storage industry's proposed solution to the multi-vendor interoperability problem.

Consider EMC Centera, the data written to it is fixed in nature. Most common systems uses a file location (address) based approach to store and retrieve content. Centera has a flat address scheme called content address. When the BLOB object is stored, it is given "claim" check derived from its content which is a hash value (128-bit) known as Content Address (CA). The application needn't know the physical location of the data stored. The associated metadata like filename etc is added into an XML called C-Clip Descriptor File (CDF) with the file's content address. This makes the system capable of WORM. When one tries to modify/update a file, the new file will be kept separately, enabling versioning and unique access through its fingerprint checksum. Also, there is an essential attribute for retention that tracks the expiry of the file. The file can't be deleted until time surpasses the defined value.

This is a basic vendor specific architecture. They provides the SDK/API to acces the data. So the industry came up with standard in which XAM API provides the capability to access these systems independently.

XAM has concepts like SYSTEMS, XSETS, properties and streams ("blobs"), and XUIDs.

SYSTEM/XSYSTEM - is the repository

XSET - the content

XUID - identifier for the XSET

All these elements include metadata as key value pairs. There are participating and non-participating fields. Participating fields contribute to the naming of the XSET known as XUID. If XSET is retrieved from a SYSTEM and a participating field is modified, it will receive a different XUID. XUID created must be formed by using the content of the property or stream (e.g. run an MD5 hash over the value).

So a vendor specific library implements the XAM Application Programming Interface (API) which is known as XAMLibrary connecting to the Vendor Interface Module (VIM), which internally communicates to the storage system. Usually, VIMs are the native interfaces. Once the application load XAMLibrary and connect to the source, it needs to open the XSET by providing the XUID which will open up the content as a stream, called XStream. The XStream object is used to read the transported data which can be a query result or a file.

A sample code to read the data,

// connect
String xri = “snia-xam://"
XSystem sys = s_xam.connect(xri);
//open the XSet
XSet xset = sys.openXSet( xuid, XSet.MODE_READ_ONLY);
//getting metdata value for the key
String value = xset.getString( “com.example.key” );
XStream stream = xset.openStream( “com.example.data”,
XStream.MODE_READ_ONLY);
stream.close();
xset.close();
sys.close()

XAM fields have type information described using MIME types.
eg:-“application/vnd.snia.xam.xuid" for the an 80-element byte array, xam_xuid.

The XAM URI (XRI) is the XAM resource identifier specification associated to a vendor identifying the storage.

As XAM generates a globally unique name (address), the objects may move around freely in time, changing their physical or technological location so as to enable a transparent information lifecycle management (ILM). So the metadata information and retention can be externalized ie, even if the application is gone ie. retired, the information management can be handled through an API. This makes the data center more adaptive to organizational information management.

And, unlike the file systems, XAM tags the information with metadata and provides a technology independent namespace, allowing software to interpret the content independent of the application.

Architecture specification

http://www.snia.org/tech_activities/standards/curr_standards/xam/XAM_Arch_v1.01.pdf

Enterprise Information Integration

noreply@blogger.com (Haris) — Sun, 10 Apr 2011 18:41:00 +0000

The relational model given by Codd enable a simple logical model, in mathematical foundation based on set-oriented queries. The relational database evolved in decades. The use cases for accessing information changed. Data marts, reporting, dashboards, security, governance and so on… RDBMS provided an efficient transaction semantics and support. But, world has changed, global economics became complex and data semantics became unmanageable. The corporations grown bigger, swallowing the tiny prawns around the globe. Many database vendors like Oracle, DB2, SQL Server. Many packaged apps SAP, Siebel, homegrown. Structured, unstructured, semi-structured and so many exotic data sources… Federated databases provided a centralized schema mapping logically the schemas from different databases. It faced challenges of schema integration, query translations etc. Virtual Databases, some called them.

Use of XML as standard for data exchange became a boon for all. It provided standards for schema, XSD, querying, XQuery etc. It is not a holy-grail though. The paradigm SOA, integrate systems, applications, processes and businesses. As the enterprise information ecosystem has grown like a behemoth, multiple producers/consumers forced to share data across its domain. Proliferation of data models results data chaos.

EII decouples data sources from data consumers without moving data. It provides universal data access facilitating dynamic dashboards or reports. By decouple application logic from integration; one can easily add/change sources with customization of the content delivery. It acts like a mediation engine for the source and client. Standardized data access APIs like ODBC, JDBC offer a specific set of commands to retrieve and modify data from a generic data source or one can use Web Services.

Then there are SDOs, SDO is an object-based programming specification for uniform access to various forms of data. It is designed to complement existing J2EE data access mechanisms, including EJBs, JDBC, and a number of popular frameworks for persistence. It is also intended to support service-oriented architectures (SOA), including Web services. SDO programmers work with data graphs (collections of tree-structured data) that consist of one or more data objects. The data graph structure is well suited to certain types of Web applications, such as those that display master-detail information and is easily understood by most object-oriented application programmers. It is based on Model Driven Architecture.

An EII system provides a single interface and extensible metadata that can encompass all data sources, speeding the development of applications such as enterprise portals. Development is faster as programmer’s code for access to a single virtual data source regardless of data storage. It is like a metadata-based view of sources. Virtual views optimize execution of distributed queries. So modeled with metadata, makes many data sources look like one single entity. Information modelers like XBRL could be used to publish data for reporting purposes. The EII server creates the query for each data source. It is intelligent to understand multiple data sources. The data sources are mapped to create the view i.e. data to xml binding to metadata. The metadata is stored in a repository. These reusable integrated logical entities or data objects can be exposed as a data service or as jdbc/odbc.

One of the key issues faced in data integration projects is locating and understanding the data to be integrated. Often, one would find that the data needed for a particular integration, application is not even captured in any source in the enterprise . ie the developer has to once again model the domain from different sources. Mapping from multiple systems and schema is really a cumbersome process and a human intervention becomes essential with more headaches. As an EII system do provide writing data, the transaction management across the system to be managed and information retrieval become complex when mapping data from structured and unstructured data. EII is not a perfect solution as it needs to have the performance and scalability similar to the ones expected from RDBMS. Even when it matures and succeeds, EII will not replace the data warehouse. Depending on the problem to be solved, integrated data will be persisted into a warehouse, or virtualized through EII.

Refrences

http://www.ibm.com/developerworks/data/library/techarticle/dm-0407saracco/

http://www.cs.washington.edu/homes/alon/files/eiisigmod05.pdf

http://www.jboss.com/products/platforms/dataservices/

Dissecting Protocol Buffers

noreply@blogger.com (Haris) — Tue, 31 Aug 2010 16:31:00 +0000

There are many ways to serialize data to formats like XML, JSON etc. Google's ProtoBuf is a way of encoding structured data. It is the de facto standard at Google for any server-to-server calls.

The choice of using the method of object serialization depends on speed,size,compatibility, metadata integrity, platform independent etc which make Protocol Buffers a good candidate especially if it is used for Java (the code can also be generated for languages like C,Python etc).XML has been a standard as data interchange and serialization format for most of the applications. Java do have a native serialization mechanism for objects.PB is like IDL describing the entity or data structure. PB can be considered as a high-level language describing the input and output types. Then the compiler-generated code is used to hide the details of encoding/decoding from application code.Isn't this similar to CORBA or EJB? Hoi, its from Google. They have been using this binary encoded structured data as input,output,writable formats for map reduce.

Protocol buffers are advantageous because they support multiple languages (i.e., Python, Java, C++ and others), are cross-platform, flexible, and extensible. They are forward and backward compatible. It uses descriptive message (ie entity or object) definition files (.proto files). The proto files are parsed by the compiler provided (eg:protoc.exe) to generate the java files based on the message definition.This java code will have the "builders" for creating the object.Good thing is that the message definitions can be modified without affecting the parsers derived from an older legacy version of the .proto definition.If we consider XML for persistence, it will need a metadata associated with entity.But, the protocol buffers are self descriptive and deprived of such unecessary details making it smaller in size.

The entity will be defined as a message.The message can refer other messages, but need to be defined or imported.Several data types and repeated data are supported.
The available wire types are as follows:

WireType	Meaning	Used For
0	Varint	int32, int64, uint32, uint64, sint32, sint64, bool, enum
1	64-bit	fixed64, sfixed64, double
2	Length-delimited	string, bytes, embedded messages, packed repeated fields
3	32-bit	fixed32, sfixed32, float

It include the concept of optional elements: fields that aren’t currently needed are not included in the binary representation. This is similar to an XML shema for an element with minOccurs=0. XSD/DTDs provide data integrity. But I think portocol buffer can also provide data type definitions and values within itself without compromising integrity.

A key is associated with each data value.For the first 15 values/members in the structure this key is stored in 1 byte; 2 bytes is required for
each key representing the 16th through the 2047th member.

Take an example.Define a proto file,

message student{
optional int32 id = 1;
optional string name = 2;
}

Generate java file. Then save the data.

Student.student.Builder builder = Student.student.newBuilder();
builder.setId(100);
builder.setName("Hary");
builder.build().writeTo(output);

Then view the binary file.If you have an hex editor one can see the hex dump.Use vim, it has xxd.exe. Use it to generate the binary dump.

Byte 1: The key 00001000 give info like :
bits 2-5 ie. 0001 says the field number ie 1
bits 6-8 ie. 000 says the wiretype is 0 ie int32 (see the table)
Byte 2:
The value 01100100 is 100

Byte 3: The key 00010010
bits 2-5 ie. 0010 says the field number ie 2
bits 6-8 ie. 010 says the wiretype is 2 ie string (see the table)

Byte 4: The 00000100 says next 4 bytes for UT-8 character string

....

This is how the binary is file is structured. As protocol buffers include data binding library, it makes easy for encoding/decoding. It is faster (7x) than JSON serialization.Even unknown fields can be set to the object.

References

ProtoBuf HomePage
http://code.google.com/apis/protocolbuffers/docs/overview.html

NetBeans IDE Plugin for code generation
http://code.google.com/p/protobuf-netbeans-plugin/

Performance Using Internet data in Android applications
http://www.ibm.com/developerworks/opensource/library/x-dataAndroid/index.html?ca=dgr-twtrAndroidPArcedth-OS

Google Protocol Buffers - the Good, the Bad and the Ugly
http://www.freshblurbs.com/google-protocol-buffers-good-bad-and-ugly
MapReduce: A Flexible Data Processing Tool
http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext

Encoding
http://code.google.com/apis/protocolbuffers/docs/encoding.html

Parallel Databases and Map Reduce

noreply@blogger.com (Haris) — Tue, 09 Mar 2010 03:25:00 +0000

n 2007 Google handled 400 petabytes of data per month.

It was Map Reduce. And the competitors were Dyrad and Hadoop

Architects were reinventing parallel databases.

Data is partitioned across multiple disks. The parallel I/O enables relational operations like join executed in parallel. Teradata, is one of the highly parallel database machines uses shared-nothing architecture Google called it sharding, infinitely scalable almost simply by adding nodes.

In parallel databases the horizontal partitioning is done so that the tuples of a relation are divided among many disks such that each tuple resides on one disk.

The strategies can be :

a. round robin, the ith tuple inserted in the relation to disk i mod n.
b. hashing - send to the result (ie i [1.. n-1]th disk) of hash function h applied to the partitioning attribute value of a tuple.
c. Range partitioning - for a partitioning vector [range] say [3,9], a tuple with partitioning attribute value of 1 will go to disk 0, a tuple with value 6 will go to disk 1, while a tuple with value 12 will go to disk2 etc.

In these databases, the queries/transactions execute in parallel with one another.Relational queries by SQL is good for parallel execution.

In a normal relational db, the execution plan for a query like this,
SELECT YEAR(date) AS year, AVG(fund)
FROM student
GROUP BY year

The execution plan is:
projection(year,fund) -> hash aggregation(year,fund).

In parallel dbs it will be,

projection(year,fund) -> partial hash aggregation(year,fund)
-> partitioning(year) -> final aggregation(year,fund).

Each operator produces a new relation, so the operators can be composed into highly parallel dataflow graphs. By streaming the output of one operator into the input of another operator, the two operators can work in series giving pipelined parallelism. By partitioning the input data among multiple processors and memories, an operator can often be split into many independent operators each working on a part of the data. This partitioned data and execution gives partitioned parallelism

image source and reference: link

Parallel databases uses the parallel data flow model makes programming easy as data is not shared.

MapReduce can be programmed in languages like C, Java, Python and Perl and process flat files in a filesystem, ie no need to have database schema definitions.This is an advantage when documents are processed.It uses a set of input key/value pairs, and produces a set of output key/value pairs. Programmer two functions: map and reduce. Map an input pair and produces a set of intermediate key/value pairs. Then intermediate values are grouped together based on the same intermediate key and passes them to the reduce function. The reduce function accepts the intermediate key and a set of values for that key and merges these values together to result smaller set of values, zero or one output value is produced in the process, thus helps in reducing memory and handles a lot of values making the job scalable. link

HadoopDB is a hybrid of DBMS and MapReduce technologies that targets analytical workload designed to run on a shared-nothing cluster.It uses Hadoop to push the data to existing databases engine to process the SQL queries.In order to push more query logic into databases (e.g. joins), hash-partitioning of data needs to be performed. The data is loaded into HDFS. for processing. HadoopDB's API provide the implementation of Map and Reduce functions.This makes hadoopdb to process huge analytical database with scalability using map reduce framework and performance with exisiting db engines.

refer:link

image source link

How Stuff Works: Spring Component Scanning

noreply@blogger.com (Haris) — Thu, 10 Dec 2009 21:20:00 +0000

Spring has an interesting feature of scanning its components defined and load it.So the configuration is tied to application ie the code, using annotations.Spring javaconfig also provides the capability to do convention over configuration.There are a lot of documents,references etc explaining how to do the spring configuration. I was looking into the under the hood flow of how the stuff works...

A minimal config to application context xml

<context:component-scan base-package="packageName"/>

will scan all the component classes in the package.The component classes in the classpath are detected and bean definitions are auto-registered for them.

As per the Schema URI and Schema XSD, the context namespace will be like this - Reference

There are stereotype annotations which are markers for any class that fulfills a role within an application.This is well showcased in SpringMVC.More about the annotations

For efficient configuration, we can have multiple context xmls for maintaining resources. The application can have one for DAOs, one for services and so on.The layers can be effectively scanned by the context loader with this usage. So MVC applications will have seperate xmls for @Repository (data access tier),@Service (service),@Controller (web tier) components.
So for the example in a simple java app, I used them in a single xml. But this is mot an mvc app.

A UserDAO Interface

package com.sample.data;

import java.util.List;

public interface UserDAO {

List<String> getUsers();
}

Its Implementation

package com.sample.data;

import java.util.ArrayList;
import java.util.List;

import org.springframework.stereotype.Repository;

@Repository("userDAO")
public class UserDAOImpl implements UserDAO {

@Override
public List<String> getUsers() {

List<String> l = new ArrayList<String>();
l.add("Roger Moore");
l.add("Pierce Brosnan");

return l;
}

}

Service Layer

package com.sample.service;

import java.util.List;

public interface UserService {
List<String> getUsers();
}

And its implementation

package com.sample.service;

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import com.sample.data.UserDAO;

@Service("userService")
public class UserServiceImpl implements UserService{

@Autowired
private UserDAO userDAO;

@Override
public List<String> getUsers() {

return userDAO.getUsers();
}

}

the client

package com.sample.client;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import com.sample.service.UserService;

public class SpringClient {

private UserService service;

public SpringClient(){
ApplicationContext appContext = new ClassPathXmlApplicationContext("resource/applicationContext.xml");
service= (UserService) appContext.getBean("userService");
((ClassPathXmlApplicationContext)appContext).close();
}

public void showUsers() {
for (String s : service.getUsers()) {
System.out.println(s);
}
}

public static void main(String[] args) {
SpringClient spc = new SpringClient();
spc.showUsers();
}

}

applicationContext

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-2.5.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.5.xsd"
default-autowire="byName">


<context:annotation-config/>

<context:component-scan base-package="com.sample.data">
<context:include-filter type="annotation"
expression="org.springframework.stereotype.Repository"/>
</context:component-scan>

<context:component-scan base-package="com.sample.service">
<context:include-filter type="annotation"
expression="org.springframework.stereotype.Service"/>
</context:component-scan>

</beans>

After bootstrapping, the set of application components and service that need to be created are identified. AbstractbeanDefinitionReader will read resource definitions. DefaultListableBeanFactory will be used as default bean factory based on bean definition objects. XmlBeanDefinitionReader.loadBeanDefinitions() load bean definitions from the specified XML file in which the BeanDefinitionParser will identify the context namespaces and parses the applicationContext xml. The resources are identified by the implementation of ResourcePatternResolver:, ie PathMatchingResourcePatternResolver in which the location patterns are found like an ant-style. Internally it uses ClassLoader.getResources(String name) method which returns an Enumeration containing URLs representing classpath resources. Then the ComponentScanBeanDefinitionParser will parse through the context defintion nodes. If annotation configuration is enabled, autowiring of components takes place as these "candidate patterns" can be set as autowired. A default AutowiredAnnotationBeanPostProcessor will be registered by the "context:annotation-config" and "context:component-scan" XML tags.If filters are added, then it will parse the type filters. In the example I have provided annotation is the type filter. So it will use AnnotationTypeFilter to load the annotation Repository which is provided as the DAO's annotation resolver.

And we know that spring classes are designed to be extended.I was going through the API docs and found that we can add and exclude filters programmatically too.

So I added showComponents to client code using a selected base package

public void showComponents(){

ClassPathScanningCandidateComponentProvider provider =
new ClassPathScanningCandidateComponentProvider(true);
String basePackage = "com/sample/data";
provider.addExcludeFilter(new AnnotationTypeFilter(Repository.class, true));
Set<BeanDefinition> filteredComponents = provider.findCandidateComponents(basePackage);
System.out.println("No of components :"+filteredComponents.size());

for (BeanDefinition component : filteredComponents) {
System.out.println("Component:"+ component.getBeanClassName());
}

provider.resetFilters(true);
provider.addIncludeFilter(new AnnotationTypeFilter(Repository.class, true));
filteredComponents = provider.findCandidateComponents(basePackage);
System.out.println("No of components :"+filteredComponents.size());

for (BeanDefinition component : filteredComponents) {
System.out.println("Component:"+ component.getBeanClassName());
}

}

So the output will be

No of components :0
No of components :1
Component: com.sample.data.UserDAOImpl

If code and configuration are static like beans, the scanner annotations are useful. For accessing resources, jndi or jdbc or anything dynamic like that better go for xml as it is easy to modify it without code change.Its widely used for request mappings and controllers in Spring MVC.The xml overrides the config.When classes are more scanning will be difficult, so we have to filter them based on the type required.

More Reading

Classpath scanning and managed components

Model Driven Development in Adobe RIA and Eclipse RCP new thoughts

noreply@blogger.com (Haris) — Thu, 03 Dec 2009 18:34:00 +0000

I had a chance to attend Adobe Dev Summit 09 Hyderabad. Among the plethora of Adobe "hoo-hah" products for the new age developers, a very good demo caught my attention. Model driven development in Adobe. Adobe is investing a lot in rooting themselves in the enterprise arena by their Adobe Livecycle stack.

The product has a LiveCycle Workbench ES to integrate with Flex studio which is an Eclipse based development environment. RIA has find its way to enterprise as Web 2.0 has moved from a hype to a standard development use case.But for me, the attention grabber is the powerful Eclipse development environment.More than an IDE, it is a wonderful platform for developing powerful applications with its handy plug-in architecture and widgets.From simple RSS readers to Rational platform, eclipse has its magic hand.

RIA for desktop is AIR.If one develops app in AIR we wont be able to use native widgets, as the runtime provides them.But we can access native windows . Also from the AIR2 next api release it is possible to run native applications from AIR.

What about RCP ? It has SWT runtime component uses native widgets..We may think most of the RCP apps will look like the dry eclipse IDE look and feel... Nope !

See the look of Lotus Domino with the power of custom skinning. Lotus Software has been purchased by IBM and has put a lot of resources to continue the development of different Lotus components (such as Domino and Notes). The UX feel of applications based on Eclipse 4 would be amazing.The upcoming e4 architecture of Eclipse is a promising space for next generation eclipse based applications.User experience guidelines for IBM Lotus rich client applications and plug-ins is here

There has always been a rivalry between desktop and web applications. Both of these genre of applications fight to conquer the business space farmed across the computing world.Web apps want to be in desktop and desktop apps want be in we.Its a fame race. RCP has got a wide enterprise users.The choice of RIA or RCP depends on use case.GWT and Flex in web are good in data visualizations. But 2D and 3D will require the power of native processors. The RCP could make use of Java 2D.There are a lot of solutions in market. More choice more pain ! So I may write about them later.

In the demo (Adobe Max)

rich data model can be modelled and exposed as a crud application with a rich UI
Similar presentation was done by Sujith Reddy where the presentation slides he shared

In Eclipse the model driven development is accomplished using EMF - Eclipse Modelling Framework.It provides an API to access the models. Access models by reflection and dynamic creation are possible.As it goes with RCP,it is like an UI for domain objects.The MVC architecture is done through EMF.It can make use of hibernate, JPA or eclipse link.

As seen in the demo, Flex became the UI of the model.

But there are developments going on in eclipse RCP framework. As Flex SDK is open source these eclipse guys wrote programs to create java for flash !

E4 ?

e4 is the incubator for Eclipse 4.0, to be released 2010 built on current Eclipse and OSGi technology as a solid foundation.

Use of web styling technology (CSS), allows the presentation of user interface elements to be infinitely tweaked and reconfigured without any modification of application code.

This is bringing Eclipse runtime technology into the JavaScript world, and enabling
software written in JavaScript to be executed in the Eclipse runtime.

E4 comes with a framework for defining the design and structure of Standard Widget Toolkit (SWT) applications declarative.This eliminates writing of repetitive boilerplate SWT code, thus reducing development cost, improving UI consistency, and enabling customized application rendering. Plug-ins are coded in Java. A typical plug-in consists of Java code in a JAR library, some read-only files, and other resources such as images, web templates, message catalogs, native code libraries, etc

E4/Eclipse Application Services -Eclipse APIs are refactored to services which should be offered as separate, independent APIs, so that clients can make use of them without having to buy into all of them. Structuring them as individual services also makes it easier to use them from other languages / environments such as e.g. JavaScript. In service programming models the consumers receive dependencies via dependency injection. This theoretically allows application code to completely eliminate its dependency on a particular container technology, thus enabling greater reuse.

Modeled UI - The E4 user interface is model based; everything that appears in the presentation is backed by a representation in the UI Model.

To that end, e4 is investigating bringing both the benefits of Eclipse to the JavaScript world (modularity, extensibility, and tooling), and JavaScript components into the Eclipse desktop environment.OSGi modularity makes integration of JavaScript Bundles easier.

Use of CSS and declarative styling a pluggable styling engine is used to customize the fonts, colors, and other aspects of widget presentation.

SWT Browser Edition- zero install widgets like RAP in which a JavaScript library (qooxdoo) is running on the client, rendering widgets that are manipulated from Java running on the server or Java is cross-compiled to Flex (ActionScript), Dojo (JavaScript) and Silverlight (.NET) technologies which is a "GWT-like" (Google Web Toolkit) approach.

This shows E4 SWT Java for Flash

More demos

Workbench - perspectives, toolbars, menus, parts. The e4 workbench greatly increases flexibility for application designers by providing a formal model of the elements that comprise an instance of the workbench.

XML UI for SWT (XWT), is a framework for writing SWT widgets declaratively in XML. In XWT, the complete structure of an application or widget hierarchy is expressed declaratively, along with bindings of the widgets to some underlying application model or to Java-based call-backs implementing the widget behavior.

More RCP apps

Some IdentityHashMap gotchas

noreply@blogger.com (Haris) — Wed, 11 Nov 2009 20:30:00 +0000

Most of java guys are familiar with IdentityHashMap, which is mainly used when there is a need of maintaining keys based on reference equality. Lot of people, have written about a lot of collections hashing and so on.. But I thought whats so interesting about this map. Even the algorithm behind it is about 45 years old !

Every object is having an "Identity", i.e., the internal "address" is unique for the lifetime of that object. So map will contain unique objects as the keys. Consider HashMap, HashSet, TreeMap, TreeSet, PriorityQueue etc, they are "equality-dependent". One can override the equals() method in the object stored to keep the object as the keys in these collections.But IdentityHashMap is "equality-independent". Even if the objects residing it changes its comparability or equality, the map will stride fine. But it violates the symmetry property of the equals contract. Symmetry means that for any two references a and b, a.equals(b) must return true if and only if b.equals(a) returns true. That sets the contract for IdentityHashMap at odds with the contract for Map, the interface it implements, which specifies that equality should be used for key comparison.

According to the spec in bold says so... "This class is not a general-purpose Map implementation! While this class implements the Map interface, it intentionally violates Map’s general contract, which mandates the use of the equals() method when comparing objects. This class is designed for use only in the rare cases wherein reference-equality semantics are required."

IdenityHashMap data structure is based on open addressing, which is a method in which when a collision occurs, alternative locations are tried until an empty cell is found. All the items are stored in an array. This map uses linear probing, wherein the next position is found out sequentially.

ie newlocation = (startingValue + stepSize) % arraySize (here the stepsize be 1)

This is basically storing all keys in a single array. The occupied addresses will form clusters (primary clusters). These clusters are distributed and the keys in them are localized to some are. So the collision will occur when the hash function returns the index of insertion wherein the keys are occupied, consuming time for insertion. Basically this is a form file/memory management.We can say that map behaves like of addressing of arrays. It puts the keys and values alternating in a single array (supposedly good for large data sets) and doesn't need to create or reuse entry objects. Even when the map size be large as it grows the insertions are not affected, while other map implementations(which uses chaining) gets slower. However, lookup is much cheaper than insertion, as we be looking items up much more often than you insert them. Using probing is somewhat faster than following a linked list, when a reference to the value can be placed directly in the array,For other hash-based collections, this is not the case because they store the hash code. This will be efficient when a get operation must check whether it has found the right key by reference (equality is an expensive operation, as it checks for the right hash code). So this takes less memory and its faster.

I read that linear probing was first analyzed by Knuth in a 1963 (at the age of 24 !) memorandum now considered to be the birth of the area of analysis of algorithms. Knuth’s analysis, as well as most of the work that has since gone into understanding the properties of linear probing, is based on the assumption that hash function is a truly random function, mapping all keys independently.

The keys are stored in an array Tab. The hash function h will be mapping keys.When a key x is inserted, index = h(x). If Tab[index] is occupied, then we scan sequentially until x is found. If an empty slot found, x is inserted. The next insertion location will be nearest next empty slot from h(x).In IdentityHasMap h(x) is basically System.identityHashCode (as per source some more calculations are done). As this algorithm uses modulo arithmetic it goes around until the size is filled up. So in the implementation MAXIMUM_SIZE is doubled up untill allowed space for object is filled in the permspace.

According to Knuth, it is possible to estimate the average number of probes for a successful search, where l is the load factor as 1/2(1+ 1/(1-l))

The put(k,v) method looks for the position for insertion. As the he index is found out based on calculations involve System.identityHashCode(returning 32-bit integer value), identity hash codes are well distributed, almost like random numbers. Using System.identityHashCode means that for objects that have overridden hashCode and equals, even if two objects are equal() they will be put in a different hash buckets ie it treats them as two distinct objects.

Some popular anomalies:


IdentityHashMap<long,> orderMap1 = new IdentityHashMap<Long,Order>();
orderMap1.put(1L,null);
orderMap1.put(1L,null);

System.out.println(orderMap1.keySet().size());

Prints 1. As the 1L will be autoboxed in Java 1.5. But as the values less than 127 could be pointing to same object there will be a single reference. When use 1000L for the keys, the size will be 2 as it creates distinct objects. Long has a private class for caching these values in consideration to performance as those values are commonly used. This is applicable to Integer,Character etc.



private static class LongCache {
private LongCache(){}

static final Long cache[] = new Long[-(-128) + 127 + 1];

static {
   for(int i = 0; i < cache.length; i++)
cache[i] = new Long(i - 128);
}
}

Take another example


IdentityHashMap<String,OrderI> orderMap6 = new IdentityHashMap<String,OrderI>();
OrderI oI2 = new OrderI(1); 
OrderI oI3 = new OrderI(2);

orderMap6.put("order1",oI3);
orderMap6.put("order1",oI2);

Here the key size will be one. As the string literals are referred from the String heap itself they point to same location.So the second insertion overrides the previous value.

Where can this map be used.Some says , for caches, as they perform very well. Only need to store references though. For cyclic graphs, we must use identity rather than equality to check whether nodes are the same. Calculating equality between two graph node objects requires calculating the equality of their fields, which in turn means computing all their successors and we are back to the original problem. A IdentityHashMap, by contrast, will report a node as being present only if that same node has previously been put into the map. Thus by saving the size and node traversals.

I just wanted to know about this map thats it...

References

http://www.ece.uwaterloo.ca/~ece250/Lectures/Slides/6.07.LinearProbing.ppt
http://www.owasp.org/index.php/Java_gotchas#Immutable_Objects_.2F_Wrapper_Class_Caching
http://www.siam.org/proceedings/soda/2009/SODA09_072_thorupm.pdf
http://www.cs.unm.edu/~moret/graz01.pdf
http://www.it-c.dk/people/pagh/papers/linear-jour.pdf
http://reference.kfupm.edu.sa/content/l/i/linear_probing_and_graphs_122432.pdf

ETags - Roles in Web Application to Cloud Computing

noreply@blogger.com (Haris) — Tue, 03 Nov 2009 21:05:00 +0000

A web server returns a value in the response header known as ETag (entity tag) helps the client to know if there is any change in content at a given URL which requested.When a page is loaded in the browser, it is cached.It knows the ETag of that page.The browser uses the value of ETag as the value of the header key "If-None-Match".The server reads this http header value and compares with the ETag of the page.If the value are same ie the content is not changed, a status
code 304 is returned ie. 304:Not Modified. These HTTP meta data can be very well used for predicting the page downloads thereby optimizing the bandwidth used.But a combination of a checksum (MD5) of the data as the ETag value and a correct time-stamp of modification could possible give quality result in predicting the re-download. An analysis of the effectiveness of chosing the value of ETag is described in this paper.

According to http://www.mnot.net/cache_docs/

A resource is eligible for caching if:

There is caching info in HTTP response headers
Non secure response (HTTPS wont be cached)
ETag or LastModified header is present
Fresh cache representation

Entity tags can be strong or weak validators.The strong validator provide the uniqueness of representation.If we use MD5 or SHA1, entity value changes when one bit of data is changed, while a weak value changes whenever the meaning of an entity(which can be a set of semantically related) changes.

More info on conditional requests explaining strong and weak ETags in here

In Spring MVC, Support for ETags is provided by the servlet filter ShallowEtagHeaderFilter. If you see the source here

String responseETag = generateETagHeaderValue(body);
.... ......

protected String generateETagHeaderValue(byte[] bytes) {
StringBuilder builder = new StringBuilder("\"0");
Md5HashUtils.appendHashString(bytes, builder);
builder.append('"');
return builder.toString();
}

The default implementation generates an MD5 hash for the JSP body it generated.So whenever the same page is requested, this checks for If-None-Match, a 304 is send back.

String requestETag = request.getHeader(HEADER_IF_NONE_MATCH);
if (responseETag.equals(requestETag)) {
if (logger.isTraceEnabled()) {
logger.trace("ETag [" + responseETag + "] equal to If-None-Match, sending 304");
}
response.setStatus(HttpServletResponse.SC_NOT_MODIFIED);
}

This reduces the processing and bandwidth usage.Since it is a plain Servlet Filter, and thus can be used in combination any web framework.A MD5 hash assures that the actual etag is only 32 characters long, while ensuring that they are highly unlikely to collide.A deeper level of ETag implementation penetrating to the model layer for the uniqueness is also possible.It could be realted to the revisions of row data. Matching them for higher predicatability of lesser downloads of data will be an effective solution.

As per JSR 286 portlet specification Portlet should set Etag property (validationtoken) and expiration-time when rendering. New render/resource requests will only be called after expiration-time is reached.New request will be sent the Etag. Portlet should examine it and determine if cache is still good if so, set a new expiration-time and do not render.This specification is implemented in Spring MVC.(see JIRA )

A hypothetical model for REST responses using deeper Etags could be effective while an API is exposed or two applications are integrated.I have seen such an implementation using Python here

When cloud computing is considered, for Amazon S3 receives a PUT request with the Content-MD5 header, Amazon S3 computes the MD5 of the object received and returns a 400 error if it doesn't match the MD5 sent in the header.Here Amazon or Azure uses Content-MD5 which is of 7 bytes.

According to the article here in S3 for some reason the entity was updated with the exact same bits that it previously had, the ETag will not have changed, but then, that's probably ok anyway.

According to S3 REST API,

Amazon S3 returns the first ten megabytes of the file, the Etag of the file, and the total size of the file (20232760 bytes) in the Content-Length field.

To ensure the file did not change since the previous portion was downloaded, specify the if-match request header. Although the if-match request header is not required, it is recommended for content that is likely to change.

The ETag directive in the HTTP specification makes available to developers to implement caching, which could be very effective at the transport level for REST services as well as web applications.The trade-off would be, there may be security implications to having data reside on the transport level.

But in the case of static files which is having a large "Expires" value and clustered files, Etag will not be effective because of the unique checksum for files that are distributed will be transported to client for each GET requests.By removing the ETag header, you disable caches and browsers from being able to validate files, so they are forced to rely on your Cache-Control and Expires header.Thus by reducing the header size which was having the checksum value.

A peek into metaprogramming

noreply@blogger.com (Haris) — Fri, 14 Aug 2009 04:31:00 +0000

Metaprogramming is about programming acting on other programs. Modifying the programs on the fly. Consider a compiler which is a program written to act,parse,execute the code written, analyzed on grammatical domain of its own lexical structure. It interacts and compose programs of small code components like importing required classes, #define macros, message passing etc. when we are to discuss about compilers, they are translating the source to it machine representation. In meta programming, its about source to source and machine independent. In C++, there are templates which is a data object for a set of programs,generates the code that is executed. The compilers use the program analysis method of type system, a check for the type correctness.an oracle, that is,some method for checking whether the system under test has behaved correctly on a particular execution. It is the assertions embedded in the code.Many programs read an input sequence and produce an output sequence, maintaining a logical correspondence between the input and output structures. When the type come into picture in a dynamic way,the system needs to be intact. This is being handled by the compiler itself as it asserts the type. Usually a program will have a structure ; a syntax tree,like an AST.The meta programs can manipulate these representations. The abstract syntax can then be used as an intermediate language, such that multiple languages can be expressed in it, and meta-programs can be reused for several source languages. Functional languages can have polymorphic higher-order functions take other functions as arguments and returns functions, treating functions as values. Java uses AOP, which introduces pointcuts to a program that can then be modified at runtime (using bytecode manipulation etc).

Take, Javascript, we can attach additional properties to an object ("expando").


var testObject = {};
testObject.variable1 = 'string';
testObject.variable2 = 3;

The type system is dynamic.

It has functions as first class objects.


testObject.functionA = function(){};

Function can return other functions as well as passed in as parameters.


testObject.functionC = function(functionA){ return functionB(){};};

Closure occurs when a function is defined within another function,and the inner function refers to local variables of the outer function. Currying a function is modified with each call so that the arguments passed into it become part of the function which will be helpful in partially evaluating functions. It is implemented using closures.

for the simple stupid code ,


function show(){

alert(testAB('A','B'));//AB
var testB = testAB('A');
alert(testB);
alert(testB('B'));//AB

}

function testAB(a,b){
   if (arguments.length < 1) {
         return add;
     } else if (arguments.length < 2) {
         return function(c) { return a + c }
     } else {
         return a + b;
     }
}

In Groovy,


def testAB = { a, b -< a + b }
assert testAB('A', 'B') == 'AB'// or testAB.call('A','B')

def testB = testAB.curry('A')
assert testB('B', 'World') == 'AB'

In Groovy,each object has it is own meta class defines the behavior of any given Groovy or Java class.Alter the metaclass (getMetaClass()) and change the object's behavior at runtime. A special MetaClass called an ExpandoMetaClass that allows dynamically adding methods or properties.


String.metaClass.echo = {->
return "This is an echo"
}

println "Test".echo() //will print This is an echo.

String is final, even methods can be added !

The behavior of program changes with a concise code.Instead of test driven development,the methodology of behavior-driven development do follow test-first code-writing. In this case tests are considered specs, or "expectations" about how code will behave. Its is checking what code should do, than what it had done. To understand the concepts and practical uses, need to explore more

Groovy and some Griffon experiments

noreply@blogger.com (Haris) — Thu, 06 Aug 2009 21:46:00 +0000

Groovy is dynamic with its superset java syntax to have fun with and be productive.It can be compiled to bytecode in .class and interpreted at runtime as a script.Adds closures,dynamic typing (duck), auto imports,no semicolon,no return... like a ruby-fied java.There can came railing Grails.Griffon for desktop, is very similar to developing web applications with Grails, basically the MVC + "convention over configuration".In Griffon, the Model is a POGO,the view is SwingBuilder and the controller (injected to) manage the Model and View together ().We know the applet coding and managing is cumbursome, for eg an applet form populated by an OrderDO etc.The coupling of model logic along with the applet controllers is mostly seen in web applications.So I decided to have some fun on this cute creature,Griffon.

A quick work around to display a bindable xml to the text area.I tried with tables, but some issues with events, so I will try later.Developing is easy and with less code.Scripting makes a flexible (agile) application layers like views and controller.Groovy can act as a super glue for different layers.It could combine XML parsing,widgets,networking...Each widget node written in the view classes is syntactically a method call.All of these method calls are dynamically dispatched.Most don’t actually exist in bytecode.Child closures create hierarchical relations.Closures are first class objects.This hierarchy of components would normally be created through a series of repetitive instantiations, setters, and finally attaching child to its respective parent.Child widgets are added to parent containers.
Like a ScrollPane having a text area etc.An event listener method is equivalent to adding a listener with that method; actionPerformed : { ... } ;in Groovy.

For this same Java does as

button.addActionListener(new ActionListener() {
void actionPerformed(ActionEvent evt) { ...}
})

Samples

source
Download source
Follow the quickstart guide and run the app from griffon-test directory

Groovy rocks!! says http://onestepback.org/articles/groovy/index.html

More here
http://groovy.codehaus.org/Griffon+Quick+Start
http://groovy.codehaus.org/Swing+Builder
http://griffon.codehaus.org/

JiBX - Part 1

noreply@blogger.com (Haris) — Mon, 03 Aug 2009 20:10:00 +0000

When we consider XML as a data model while programming, focus is mostly given on strings/tags/elements in parsing data.The ETL operations are done on this ubiquitous document interchange model when communication is made between applications.To make things simple in the internet; which is fond of documents ; XML became the foundation of services through web.The object, which is accessed from object graphs in the memory have features such as inheritance / polymorphism and attributes / object-based relationships while XML does not have any of these features as objects have.It is merely a grammatical (hierarchical) representation of data with all its branches mangled to itself .But they have similarity in the sense of representation of real world data.Therefore they can exists as the representation of each other which eases in programming.They are effective in defining business uses cases.But there is an impedance mismatch between objects and xmls.An application written in Java, will have it data types defined within the scope while XML Schema which defines the data is richer than Java.Complex objects find difficulty in serializing.Have a look on this paper which explains the X/O impedance mismatch.

We can generate Java classes from XML or vice versa.In this case, it is a "Schema-Centric" approach.We define an XML schema.Then one or more XML documents.Then generate java classes based on them.In this case you need a stable schema which can be used to validate data.Essential for a "reliable" web service.But application code is forced to use an interface that reflects the XML structure which makes them tightly coupled to contracts defined.Any change in schema make a need to regenerate the object model and change application code to match.

If we map the classes using bindings, then its "Java technology-centric " approach.This can be adopted when you don't want object model tied to schema.When there is a need to support different versions of the schema with the same object model or a common data exchange format for existing objects is needed.Binding definitions are themselves XML documents, with a schema defined for it.

JiBX is fundamentally a Java technology-centric (mapped binding) approach uses binding definitions.You define how XML relates to Java objects.Start from schema, code, or both.Binding code is compiled (or wired ?) into files (which uses BCEL byte code enhancements ). It can be done at build time, or on-the-fly at runtime.This makes JiBX compact and fast.JiBX achieves its performance by using post-compilation bytecode manipulation rather than the reflection.The advantage is that there is no need for getters, setters and no-arg constructors.One can write the class without considering mapping issues and then map it without modifications. XML schema describe domain objects.JiBX really maps well to XSD. JiBX is based on a XML pull parser architecture (XPP3) . Rather than generating code from a DTD or Schema, JiBX works with a binding definition that associates user-supplied classes with XML structure. The binding compiler is executed after code is compiled and the marshalling/unmarshalling code is added on to the class files.There are tools along with it, like Bindgen that can generate schema from existing classes.

Tutorials

http://jibx.sourceforge.net/binding/tutorial/binding-start.html
Pdfs - JiBX -Part1 JiBX-Part2 Intro

Previous articles
Creating a Java WebService using Axis 2 (a lazy approach)
A simple RSS parser

A slothy ride through "Work Flows"... a primer

noreply@blogger.com (Haris) — Sun, 28 Jun 2009 18:30:00 +0000

IT has become integral part of every organization.Business people need to be empowered with the tools that would keep an up hand on the competitive market.The execution of business processes in an organization can be done through work flow management systems.It is basically the flow of data that is associated with a work.Distribution of work is needed for efficient management.So the non technical people find it very easy if the applications they work on to control the process, is graphically manageable.In the deep level, the process models are based on graph theories and algorithms.As a programmer you are familiar with activity diagrams/ data flow diagrams,so we can observe these work flow patterns described by them.The distribution of work and its monitoring from the inception to the end can be effectively managed by work flow systems.Work flow systems helps to model the services for Business process management (BPM) (which is a management discipline that focuses on the designing them).Anyway the software used to model them did become role players of the industry.

I recently, read some articles on evolution of work flow tools.In 70's the work flow domain focused on office information systems.They were not successful in earlier days due to the limitations of computing, graphical user interfaces etc.But in these latest times, we could see the massive change in the concept of computing... semantic web,web 2.0,advanced networking etc that helped work flow systems to spread from managing a single organization to social networks(in the sense of social marketing tools, collaborative work etc ).There are a mindful of software like IBM MQseries WorkFlow, Tibco Staffware Oracle BPEL PM , jBPM ,OpenWFE (Ruby) used for work flow development.There is a consortium, formed to define standards for the interoperability of workflow management systems known as Workflow Management Coalition (WfMC) . According to their reference model

- workflow engine the instantiation and execution of process instances based on a predefined process model
- this model is specified using a process definition tool and is subsequently passed to the engine for execution
- workflow client interaction and managing work items on the worklist
- toools for adminstration and monitoring of processes and their instances
- external workflow engines can be communicated while processing done

take the case of jBPM

JBoss JBPM core component is the workflow engine (also referred to as core process engine) which takes care of the execution of process instances.
JBoss jBPM Graphical Process Designer (GPD), an plugin to Eclipse is the process definition tool which provides support for defining processes in jPDL both in a graphical format and in XML format. (processes can also be defined as Java objects or as records in the jBPM database)
jPDL (jBPM Process Definition Language) is the process language
JBoss jBPM console web application is the web based workflow client as well as an administration and monitoring tool

more here

At the programming side it uses Graph Oriented Programming.Graph Oriented Programming is a very simple technique that describes how graphs can be defined and executed on a plain OO programming language. Domain specific languages has a pivotal role in the development of work flows.Graph Oriented Programming is the foundation for all domain specific languages that are based on executing a graph. If you compare the system, it can be similar to any process model involving state changes,transitions etc.

Some cool stuffs i read

Making Web 2.0 Meaningful and Achievable by Jackbe
Take advantage of Web 2.0 for next-generation BPM 2.0
Evolution of the workflow management systems

More to know about the meta languages used in these systems.....

Java modularity and metadata

noreply@blogger.com (Haris) — Sun, 21 Jun 2009 16:41:00 +0000

As a java developer one will encounter the spaghetti world of class path hell.Loads of libraries,jars,dependencies etc are used in large projects.Manageability is the major part of any application that is evolving through continued engineering process.When we access a class in a library we have to add the library class (which can be in jar file) to class path.This helps the compiler to use the class when its get compiled and the JVM to load the class when used at runtime. Just the basic stuff.java.lang.ClassLoader plays the main role in the life of class objects.we have jars/plugin mechanisms to do all the required functionality of so called modular.A jar can have dependencies to other jar files.We use lot of jar files usually the Apache commons library and all.Some time we wont be needing all the classes for the required functionality.Intelligent packaging is one of the main concern of any project while deployment.Less dependencies could overcome performance bottlenecks.Better versioning could help a better way of continued engineering and maintenance.The default class loader in java runtime is immutable.Any class that is loaded by the class loader is added to a name space which cannot be changed i.e you can add any class to this namespace but unable to unload them.That's what happening when the object is instantiated with new operator.So when we create a new ClassLoader instance it creates a new namespace.One of the solutions provided by the tech commumity is the modularization of JDK itself.This would allow applications to be installed with just those components of the JDK that they actually require.A specification JSR 277 was targeted to be delivered as a component of Java SE 7.0.The specification defines an architecture with first-class modularity, packaging and deployment support in the Java platform, including a distribution format, a versioning scheme, a repository infrastructure, and runtime support.They introduced the Java module format called JAM.But later on they decided to halt its development because of difficulty in integrating with JVM.They started a project Jigsaw under open jdk.The modular system can be independent or be implemented based on the language or compiler changes.Anyway, the concept of modules is interesting.According to JSR-277 spec

A Java module is a unit of encapsulation. It is generally written as a development module in a programming language in the Java platform, and is ultimately compiled into metadata and packaged together with classes and other resources as a deployment module.

These modules provides the metadata related to itself, like name,version,imported classes,dependencies etc.

But there is already an evolving system known as OSGI that intends to do the modular concept.In this bundles will be the modular jar files.You can install, uninstall, start and stop these bundles(without restarting the VM ).Also, it does offer services that can be dynamically discovered in JAR files at runtime.You can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms( design point of OSGi has always been to support a broad set of Java ME). .The concept of bundles is powered by custom class loaders which will provide encapsulation as well as runtime dynamics.

A bundle is a group of Java classes and additional resources equipped with a detailed manifest on all its contents, as well as additional services needed to give the included group of Java classes more sophisticated behaviors, to the extent of deeming the entire aggregate a component.

We know that a jar will have MANIFEST.MF file attached to its structure inside the META-INF directorythat contains information about its contents.By adding OSGi headers to the MANIFEST.MF we can make the jar a bundle.These headers can be

Bundle-Name: a name for this bundle
Bundle-SymbolicName: a unique identifier for a bundle (naming like java packages).
Bundle-Description: description of functionality.
Bundle-ManifestVersion: which OSGi specification to use for reading this bundle.
Bundle-Version: version number of the bundle.
Bundle-Activator: the class name to be invoked once a bundle is activated.
Export-Package: packages contained in a bundle available to the outside world.
Import-Package: packages required from the outside world

These standardized metadata is used by the framework to provide robust modularity while integrating to most of the existing applications.This metadata can act as a semantic definition to integrate pluggable modules to be integrated in large applications.I think modern enterprise application need this type of modularity to provide an easy integration with low cost of maintanence and development.The ISVs can provide an efficient solution by the light weight applications instead of heavy application suites having complex integration structure.It can be an asset to the low budget IT solutions.Also the standardization helps to coexist (and interoperable) with different solutions provided by open jdk community or spring community or any other vendors.

More header information in this OSGi core specification
Some osgi implementations equinox,apache felix
About jigsaw
OSGi elearning

A snippet on Apache Sling

noreply@blogger.com (Haris) — Mon, 18 May 2009 20:08:00 +0000

We know everything is content.Content(Knowledge) management systems are widely used in various fields and is really an old concept.Usually data is stored in databases or file system as it is very easy to manage unless data is huge..When we consider filesystem, it is unstructured and hierarchical.Database is structured with a well defined schema but restricted by constraints and transactional behavior.Web 2.0 caused such an insurge of highly dynamic data into web.It became inevitable for collaboration and integration.If we have to access information of different
formats, then we can use content repositories; for its inherent nature of file system and database (with some extra features).Its a best of both worlds.Here is the definition of a content repository from from JSR-170 spec

A content repository is a hierarchical(tree like) content store with support for structured and unstructured content, full text search, versioning, transactions, observation, and more(they can be considered as content services).

It is like a high level information management system.It can have both text files as well as binary files like pdfs,docs,images etc

Apache Sling (with a new release of version 5) is an open source Web framework for the Java platform designed to create content-centric applications on top of a JSR-170-compliant (aka JCR) content repository like Apache Jackrabbit.If you use Sling, it will be easy to develop large websites with thousands of pages.It works on top of JCR.If we are familiar with configuration management systems like changeman,VSS or subversion we know how it is easy to manage content.It avoids spreading of files in webserver Sling process the HTTP request in a RESTful way.Sling uses scripts or servlets to render and process content.Modularization by OSGi (using Apache Felix ) concept which loads modules to memory dynamically.Sling uses JSP,ESP (server side ECMA script),Groovy etc.We can directly map URL to a content.I think I have to know more about OSGi to dig whats happening inside Sling.I believe Sling could be used in large websites like social networks.DevDay says how Open Social and Apache Sling fit together.

more on Sling later...

Apache Lucene - Indexing Part2

noreply@blogger.com (Haris) — Thu, 02 Apr 2009 20:10:00 +0000

I was going through some interesting sections of Apache Lucene these days.I found it really interesting because, the project is a very popular one and it made most of the web applications to integrate complex search modules in them.Some might be knowing JIRA tracker by Atlassian uses lucene which traverses huge buglists,comments,codes,documents etc.

For a vanilla search tool, comparing the search key with strings in the file is very slow.So the indexing like inverted index comes in handy.When indexing is done by lucene, it will create document ids for each document.It will collect all the words and associate them with each docId in which the word belongs. Therefore, each docId will be having alist of positions of words in the document.The index datastructure, the store of documents, with its associated fields is constructed to provide a random access data retrieval.The Lucene inverted index can be either opened to add more documents or delete existing documents at a time.
To update a document you must delete it first, close the index and add it again.
The Analyzer , specfied in the Indexwriter, will extracts the tokens to be indexed.There is a default analyzer for english texts(for multilingual one custom analyzers are needed).Before analyzing is done, the documents like pdf,doc etc are to be parsed.A Term is the basic unit for searching. Similar to the Field object, it consists of a pair of string elements: the name of the field and the value of that field.A term is defined as a pair of <fieldname,text>A term vector is a collection of terms.The inverted index map terms to documents.For each term T , it should store the set of all documents containing that term.So the duty of analyzer is to look for the terms in documents and create a token stream so that they can be mapped.Terms are stored in segments and they are sorted.The term frequency will tell how well that term describes the document contents.But term which appear in many documents are not very useful for filtering.The Kth most frequent term has frequency approx 1/K ie for 100 tokens, the index will contain 50% text.For the indexing strategies : - they can be chosen from

Batch based - like a simple file parsing and sorting-
BTree - indexing - similar to indexing by file systems and databases - as it is a tree the update can be done in place
Segment based which is common, created by lots of small indexes

The algorithm used for lucene indexing can be

indexing a single document and merging a set of indexes
incremental algorithm in which there will be a stack of segments and new indexes are pushed to stack (segment based)

References

Doug Cutting (the creator) Lecture

http://docs.huihoo.com/apache/apachecon/us2007/AdvancedIndexingLucene.ppt
http://www.im.ntu.edu.tw/~b90003/Lucene.ppt.
Lucene In Action

Apache Lucene - Indexing - Part 1

noreply@blogger.com (Haris) — Tue, 31 Mar 2009 19:42:00 +0000

"Information retrieval (IR) is the science of searching for documents, for information within documents and for metadata about documents, as well as that of searching relational databases and the World Wide Web."

Most of the application uses search features.If you are looking to add a powerful text search engine feature to your application then use Lucene, which can add advanced Search Engine capabilities to an application.This is a really powerful Java API which gave birth to powerful tools such as Nutch,Hadoop,Hibernate search and so on.Lucene was started in 1997 and adopted by Apache in 2001.The main functionality Lucene does is the powerful full text indexing of data.
Indexing with Lucene breaks down into three main operations: converting data to text, analyzing it, and saving it to the index.Lucene looks for strings only , so the documents has to be parsed and indexed.
To search large amounts of text quickly, you must first index that text and convert it into a format that will let you search it rapidly, eliminating the slow sequential scanning process. This conversion process is called indexing, and its output is called an index. So the searching is done on this index to find the data related with a cost of space 'storing indexes'.
These index files can be stored in a directory .A lucene index is divided into segments madeup of several index files(Lucene Documents).An index can be related to mutiple documents.So if new documents are indexed , it is added to segments than modifying the existing index file.Lucene uses a feature called incremental indexing ie there will be a global indexing and index those incremental documents so that it is searchable.Regarding the structure of a lucene index, it is an inverted index .While searching, lucene loads the index to memory .It uses a high performance indexing which has an index size roughly 20-30% of the size of text indexed which uses less memory. The documents in an index is a collection of fields which is a named collection of terms like <field,term>.These fields are independent search space defined at run-time.The segments or sub-indexes are independently searchable and the results of these segments are merged.Suppose a wiki article is indexed , we can set the field properties, so that the field objects contain actual indexed article data or stored one.

More about lucene index file formats - here

Few videos i liked from Mix 09

noreply@blogger.com (Haris) — Mon, 30 Mar 2009 17:50:00 +0000

I have worked on some .NET RIA projects before.I saw some videos from Mix09, and sharing some I liked.

Building a Rich Social Network Application

Learn how to build a social networking site using Microsoft Silverlight.Also explains how to mash up existing services, how to tag and store data in back-end services,and how to build a rich and engaging experience.

Advance Your Design with UX Design Patterns

Visit this website for different ux patterns

http://quince.infragistics.com

Building Data-Driven Scalable AJAX Web Pages

There are different patterns for designing an application .See this video for how to enable rich communication patterns between your AJAX Web pages and Web server using existing and new features in WCF, Windows Communication Foundation.

Modeling RESTful Data Services: Present and Future

I have written about RESTful services before.In this video, look for ADO.NET Data Services matching REST.

For Three Tier RIA Pattern

markItUp! a jQuery plugin

noreply@blogger.com (Haris) — Sun, 29 Mar 2009 18:21:00 +0000

I saw a very cool html editor plugin called markItUp!, which is a JavaScript plugin built on the jQuery library. It allows us to turn any textarea into a markup editor.Even our own Markup system can be easily implemented.It enables us to quickly modify any standard TEXTAREA within our page into a powerful markup editor, It’s so simple that instantiating the editor is as easy as:

$('#html').markItUp(Settings);

settings is a json defining the html settings for parsing text area - like shortcut key mappings,markup sets etc

For more features
http://markitup.jaysalvat.com

Develop an Open social application in 60 seconds

noreply@blogger.com (Haris) — Sun, 29 Mar 2009 13:11:00 +0000

Open social application development made a giant leap.An eclipse plugin that eases the development of opensocial apps.I have written about open social applications before and I had a chance to work on it.Apache shindig is the initiative to develop a SNS container for application development and testing.I would say OSDE plugin developed by Yoichiro Tanaka rocks!! It uses apache shindig and hibernate for a dynamic development, so the developer can create a single application for different data models.As apache shindig provides java REST support, the application development will become more extensible.The database packaged with the plugin is H2(a Java SQL database).Before this plugin came, we had to develop and run the applications iniside a sandbox which was really tiring.This plugin has features of wizard like development for both javascript widgets and Java REST client applications.We can have our own custom social data which can be easily persisted due to the excellent plugin architecture.So I tried to develop a simple application ...

After the plugin is installed create a new OSDE project

Specify the gadget.xml and the API specs etc

For the development we need to run the apache shindig in the background.

To have a custom social data create people and add relationships between them.

Write a simple gadget ... (templates can be generated by the plugin if needed)
----src-------

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><module><moduleprefs author_email="harisa@pramati.com" description="A friendly os app" title="Friends"><require feature="opensocial-0.8"><require feature="dynamic-height"></moduleprefs><content view="canvas" type="html">


<div>
<button onclick='fetchPeople();'>Fetch</button>
<div style="margin-left:20px;">
I am ... <span id='viewer' style="background-"></span><br/>My friends are ...
<ul id='friends' style="margin-top:5px;list-style:none;margin-left:75px;"></ul>
</div>
</div>
<script type='text/javascript'>
function fetchPeople() {
var req = opensocial.newDataRequest();
req.add(req.newFetchPersonRequest(opensocial.IdSpec.PersonId.VIEWER), 'viewer');
var params = {};
params[opensocial.IdSpec.Field.USER_ID] = opensocial.IdSpec.PersonId.VIEWER;
params[opensocial.IdSpec.Field.GROUP_ID] = 'FRIENDS';
var idSpec = opensocial.newIdSpec(params);
req.add(req.newFetchPeopleRequest(idSpec), 'friends');
req.send(function(data) {
var viewer = data.get('viewer').getData();
document.getElementById('viewer').innerHTML = viewer.getId();
var friends = data.get('friends').getData();
document.getElementById('friends').innerHTML = '';
friends.each(function(friend) {
document.getElementById('friends').innerHTML += '<li>»' + friend.getId() + '</li>';
});
gadgets.window.adjustHeight();
});
}
</script>

----src ends---------

Run the application

Gadget --->

Cool... Its simple. But this plugin will be really useful when we develop the complex applications and aiming for multiple containers supporting open social api.

More ....
opensocial-development-environment
screencasts
youtube