Sumit Bisht

Experiences as a software engineer

2012-01-10T10:09:00.000+05:30

I will soon be completing my first year working as a software engineer. An year ago, I was an eager college student and was dabbling in open source stuff heavily to make up of the extra time that I had spent in college. I had the prerequisite knowledge to be a developer even before college, but in India, you are worthless if you do not have anything on paper, but are deemed to be knowledgeable if having degrees and certifications.

To start off with, I consider myself lucky to be in my present job, last year during my fifth semester, a trip to various local companies to 'test the waters' proved out to be an opportunity in disguise. I was interviewed and then selected without much fanfare. During the first day as an intern(my last semester involved industrial training) I jumped headfirst into connection pooling using spring, which was required here and there was no looking back (In direct contrast to people working in multinationals, who with much fanfare start their training - but later migrate to management as growth as a developer is restricted).
Among other things, I was having a java and ruby background and was started as a java developer. One of the perks of being in a product company is you have time and opportunity to try out, learn and explore new stuff all while doing your job.
After working considerably in the RnD over product development, I was put to test in scaling web scraping applications, porting of our product in .net platform and creating a keyboard/mouse capture application for assembling a test management mashup during the course of the previous year. Although I miss web applications a wee bit, the middleware and logic is what keeps me busy. On a parallel basis, I am into lot of books involving me to be a better developer. The Clean Coder, The passionate programmer,
With regards to updating my blog with the new stuff once every month, I learnt and made some interesting excercises during my free time, but paucity of time as well as my frequent forays in dzone and other knowledge portals led to cancellation of various blog posts. I am thinking of reverting to small posts providing a high level overview of technology practices that I consider important in current scenario instead of writing tutorial like posts.
Am having quite a few of these items in my mind, and will post them as the time allows.

Book Review : User Interface

2012-01-08T23:13:00.003+05:30

Book Review : User Interface

2012-01-08T23:13:00.001+05:30

Aplying Domain Driven Design

2011-11-29T17:36:00.000+05:30

One of the common mistakes in constructing software that is sub-par with requirements is the lack of understanding of the domain in which it is to be operated. This fallout or mis-communication is perpetrated evidently by usage of different types of languages between the developers and domain experts (abstracted for the sake of simplicity) Evaluation of failures have enabled us to realize that underestimation of domain can lead into perils in project understanding. One of popular example occurs when you try to apply simple CRUD generators into, say something like medical instrumentation. Your productivity falls and you end up hitting a wall. This is evident as the technology is just the enabler, and not the end to means. I have learnt this the hard way in one of my recent project where over engineering and legacy code forced us to create entity framework configuration files via a self-created generator instead of the inbuilt edmx designer - an example of relying on reverse engineering over traditional process.

However, many software solutions which are designed with their domain in mind are essentially an object-oriented model expressed in terms of their domain, at its core. The domain, rather than data-oriented entities constitute this and it compasses the domain wisdom (logic, data, rules, terminology) expressed in simple and utilizable manner. One of the tangible benefits of utilizing this approach is that this increases the longevity of code as domain outlives its implementation. Moreover, this can be modularized into reusable components and mashed-up as the needs arise.

But to start off with, in existing applications, we can use metadata such as annotations or attributes to 'beautify' our code through using the domain logic without undergoing any unnecessary plumbing in the existing application architecture/stack. Another benefit from using them is that in java or c# multiple inheritance is not possible, but this can exist in the domain.
One thing that most domain-oriented tools tend to do is interface their data/stack through xml, which might be fine for domain experts, but is pain to maintain (think of all the xml documents for business rules and instances).

Thus domain annotations provide a starting point of integrating domain knowledge into our code, making it more reasonable. Other approaches include use of full fledged frameworks like naked objects that are geared for these tasks and like rails, play or asp.net mvc frameworks, can quickly generate abstract applications quickly.

Play Framework Cookbook :book review

2011-09-20T11:56:00.000+05:30

Play Framework Cookbook

Author : Alexander Reelsen

Play framework is an upcoming framework that comes in the growing breed of rapid application development, using convention over configuration based web application development.

The thing that sets apart this framework apart from others is the customizability of the framework in the jvm, while taking advantage of either java or scala as desired. However, owing to the late arrival of play, there has been a lack of tutorials and documentation in order to solve real world problems with this framework.

This book, being a cookbook covers the topics, head-on and gives practical insight into building web application while covering loads of practical topics, from installation to deployment.

The book has been divided into 7 chapters and a small appendix. The first chapter covers the basics and is good for those coming without rails experience. However, the chapter also presents recepies for customizing all parts of the framework- pretty much everything that other technology cookbooks pack into themselves from cover to cover. The middle chapters dig deeper into the framework and integration with external services. One thing that is special in this cookbook is it devotes 3 chapters to modules as extensibility via modules is one of the main differentiating factor that play framework offers. In line with practicality, the book devotes its last chapter totally on deployment and release functions. This does omits the cloud vendors that provide paas for play framework directly, but as a lot of cloud offerings for play are unfinalized, they do not find any mention. The appendix is a disappointment as there are no pointers for carrying out further. As with any upcoming technology, it is imperative to look out for more resources in order to update and correct ourselves with its ecosystem. This is especially important as play 2.0 would be out on beta by the end of this year and a reader always expects pragmatism from a hands-on cookbook. Also, due to the flexibility of this framework, various PAAS vendors are providing, or planning to provide play support, which deserves at least a listing of these services. The author clearly states his intention behind leaving scala based applications in play, and this hopefully might make its way in future editions.

Overall, this is a book that lives well up to its expectations on providing us with something apart from the documented portions and the stock examples supplied with the play distribution while providing insights into the activities covered and making them flexible enough to be used practically.

For anyone learning play framework and looking to go beyond the examples for making interesting and practical applications, I’d heartily recommend this book.

Book page : http://www.packtpub.com/play-framework-cookbook/book

Amazon : http://www.amazon.com/Play-framework-Cookbook-ebook/dp/B005HIK8BG

Up Next : Book review on Play framework

2011-08-24T16:00:00.001+05:30

Today I received a copy of the book, 'Play Framework Cookbook' by Alexander Reelsen, courtesy its publisher, Packt Publications.

As the name suggests, it covers various recipes regarding the play framework, which is a promising rapid application development framework in java/scala platform.

This is the first instance of any good documentation of this upcoming framework for professionals and looks promising. I will be going over this book for the next few days and would be posting my views about it, stay tuned!

Need Concurrency ? LMAX to rescue!

2011-08-05T15:02:00.000+05:30

In praise of LMAX.

Concurrency is a necessary evil beast in any application. Multicore architecture has ncessiated the use of programming for concurrency if there is a need for enhancing application performance.
Recently, on the martin fowler's blog, I came across an interesting architecture, LMAX which was used by in a financial application to run over 6 Million transaction simultaneously, over a single thread of execution only!
Even if this architecture is limited to finance domain, it makes up for an excellent solution for dealing with concurrency. With its single point of handling of the business logic processing there might seem a potential bottleneck, but is negated via alternative mechanisms to recreate this controller - Event Sourcing.
The concept of in-memory solution for performance is not new, but having concurrency without the taxing side effects of transactions is indeed interesting.
I'll be doing some more study through it during this weekend, and update here if necessary.

Links :

Found reference from Martin Fowler's blog : http://martinfowler.com/articles/lmax.html
One implementation of LMAX (Disruptor): http://code.google.com/p/disruptor/
Disruptor's technical paper that gives an excellent overview : http://disruptor.googlecode.com/files/Disruptor-1.0.pdf

The curious case of multithreading Sqlite

2011-06-25T17:48:00.000+05:30

In a recent project, I've embarked upon the process of saving information in a quick and lightweight manner for which the serverless database, sqlite was ideal. This is not only a flexible database supporting dynamic data types, but also an extendible one used from embedded web browsers and mobile platforms to rails based websites(good riddiance, mysql) .The database is so lightweight that you'll find yourself exclaiming it as db. It is less than 100 kb in some cases and can be used as a memory resident database for the performance conscious folks and applications.
However, the support of transactions and scalability leave a lot to desire as the documentation cleary states the goals of this database is not these.
In my case, getting started was just the matter of finding the jdbc driver and it s documentation from its website. However, upon customization, there was a need for normalization which in turn required a sequential read/write operations. As soon as this was done, the multi-threaded application that was accessing started experiencing concurrency issues around the database.
This was resolved via using a little used jni wrapper that I discovered, sqlite4java. This not only solved the concurrency issues, but also enabled transactions. The beauty of this approach is also the fact that it uses a single connection as opposed to a pooled/fresh connection per every request, which tends to be a real performance bottleneck as we scale up our application.

For such a nice framework, it is a surprise that these changes have not permeated to the java/native jdbc drivers so far and the users have to cope up with its api at the moment. This could be probably because of its tight integration with the underlying C-based api that is quite performant as compared to its
alternatives in python and java, effectively making it non compliant with the jdbc specification.

As the single connection served entire applications, exposing it as a singleton was the easiest approach :

static SQLiteQueue queue = new SQLiteQueue(new File("BiddingsDB"));

static {
queue.start();
}

For providing a transaction-like support, there was a need to wrap the existing operations in this queue :

public static void databaseInsert(final List list) { queue.execute(new SQLiteJob() { @Override protected Integer job(SQLiteConnection arg0) throws Throwable { insert(bidders, arg0); return null; }}); } The insert method performs the actual reads and writes in a row in a non-blocking manner which earlier used to cause locking problems with sqlite in the past. Integration tests verified this performance gains (recollections): DB insert (Read + Multiple writes in the operation) 1 Record : ~200 ms 2 s 100 Records : 5s 10s 10000 Records : Fail ~100s As an afterthought, you can have a plain vanilla solution via the scraperwiki.com that not only gives you free cloud access for your scripts, but also its data via sqlite. This aptly highlights the difference in the mindsets of java as well as scripting developers. Here, I was able to run a rudimentary scraper in a matter of minutes but was of no use for my client due to the gathering restrictions of the running jobs there. Hopefully, we get to see some mixing of both the worlds in the near future as scripting for jvm has been making a lot of noise for the past 3 years, but not much efforts have been made by the two sides so far. Oh, and I am really enjoying the rainy weather as I write this post!

PC Gaming on Linux

2011-04-23T18:46:00.001+05:30

Many people consider Linux as a serious OS for computing, but there is a fun aspect to it that is not really known as there is no marketing done for these games. However, for a moderate gamer, linux platform is satisfying especially if you are looking for playing games for your amusement without going too heavy on your wallet. Linux has also improved on the UI front considerably as now we can have some seriously GPU intensive graphics through out of box programs like compiz.

Cool 3D Desktop with Compiz

Those looking for a quick solution can use http://live.linux-gamers.net that allows you to create and start gaming from the live CD/USB directly.

In addition to the point and click games(like card based games), you can have some serious fun with 3D games that PC gamers have become familiar with. Be it a fast paced action game like AssaultCube or a racing game like TuxKart, there are options for everyone.

TuxKart : Funny racing game

AssaultCube : Fast paced action, a la Quake3 style

Thanks to ID software's initiative of providing a linux binary(and their stand on DirectX framework), we can install and play a game like DOOM3 on Linux http://sites.google.com/site/lgmscripts/scripts/games/doom-3 (which is definitively on my TODO list:)

There are certain caveats to this excercise too. Since my laptop has a Nvidia GeForce 8200MG card, I had to use a non open source binary driver as the cannonical one was not working well on Ubuntu. If you consider the freedom and the non presence of any cost/license, it forms a good overall personal entertainment package.

Domain-Driven Design Using Naked Objects

2011-03-27T13:55:00.000+05:30

I just had a chance to read a newly released book, 'Domain-Driven Design Using Naked Objects' by Dan Haywood [http://www.pragprog.com/titles/dhnako] that provides an insight into the world of DDD. If nothing else, this book is for techies & management people alike. Although Naked Objects is covered as the implementation framework (to explain practically, all aspects of DDD), this book serves as an excellent introduction text for all of us who are new into this concept of crafting an enterprise application according to domain. This book also deserves to be read specially because there has been considerable buzz around this 'Domain' stuff in the recent past.

According to the book ,
'Domain-driven design is an approach to building application software that focuses on the bit that matters in enterprise applications: the core business domain. Rather than putting all your effort into technical concerns, you work to identify the key concepts that you want the application to handle, you try to figure out how they relate, and you experiment with allocating responsibilities (functionality). Those concepts might be easy to see (Customers, Products, Orders, and so on), but often there are more subtle ones (Payable, ShippingRecipient, and RepeatingOrder) that won’t get spotted the first time around. So, you can use a team that consists of business domain experts and developers, and you work to make sure that each understands the other by using the common ground of the domain itself.'

Since being a domain expert is the next step for any developer naturally, as his experience increases, there is need to have more insight into the business stuff, instead of just the software. UML although does a great job for explaining the Object Oriented stuff, but ponder for a second, how is it really going to help a businessman that is interested in getting more profits. Naked Objects framework (http://www.nakedobjects.org) is based on the design pattern with same name(http://en.wikipedia.org/wiki/naked_objects) is an open source framework (only for java, .NET is a commercial one) that converts simple bean/components automatically into an interface(read as multiple applications).
Don't yet confuse this with prototyping because DDD incorporates both the developer and the domain expert teams and we are not just creating the UI.
DDD's 2 central premises explained :

A ubiquitous language for integrating & easing communication between domain experts instead of 2, which is the existing norm(like code & UML)
Using model-driven design that aims to capture the model of the business process. This is done in code, rather than just visually, as was the case earlier.

Naked Objects
The java based Naked Objects(NO) framework is an evolutionary step forward from Rails (and its other avatars : Grails, Spring Roo, asp.net MVC, etc) that focuses more on M & V rather than MVC and provides much more domain-specific applications in turn resulting in flexibility for all.
A typical NO application consists of multiple sub-projects like the core domain, fixture, service, command line and webapp project through a maven archetype. The coolest thing is that NO automatically displays the domain objects in an O-O based UI that offers display in more flexible manner than any other IDE.
NO also challenges the common frontend-middleware-backend convention and instead applies the Hexagonal architecture (http://alistair.cockburn.us/Hexagonal+architecture) that deals with the bigger picture in mind. The development in this framework is pojo centric and is heavily based on annotations, which should be pretty much regular stuff for any JEE developer. Also, during my initial evaluation of the framework, the code that's being generated during the development is of maintainable quality, which is virtually essential for maintenance and scaling in any enterprise application.

Hence this book, and its field of study is highly recommended for any enterprise developer/ team/ manager/ domain expert and as is repeatedly mentioned, becomes highly important when one has more years of experience under his belt. I am continuing my exploration in this and if it is really useful for me, would post some exercises here.

10 Years of Ruby!

2011-03-02T08:01:00.001+05:30

Let me recollect not very recent past. It was 2008 and recession was at its peak. Back then, I was largely a self-taught developer and had a working knowledge in struts & ejb (the java heavyweights at that time). After unsuccessfully trying to yield a job, I finally decided to pursue my masters in computer applications (out of necessity, or so it seemed back then).
During this frustrating period, I heard about Ruby On Rails, which was touted as the next-gen platform/language. As with other people, I was quite scared of it as the name sounded very unfamiliar. So, I didn't pursue it. However in the summers of 2009, as I was idling across the netbeans.org looking for some interesting activity to do, I came across this '5-minute Weblog' exercise on rails and decided to have a go.
Believe me, it was one of the turning point in my career and I spent more than an hour figuring out exactly how I was able to generate a web application just like that using practically no big tools.

As the things turned out, I was not the only one who leaded this entirely new approach. During a recent magazine interview(http://www.pragprog.com/magazines/2010-12/chad-fowler-on-ruby), Chad Fowler, a prominent figure on Ruby Conference and a ruby authority echoed similar views.
During this interview, it was interesting to observe the maturing of ruby as a language and a platform (performance wise and lightweight for mobile devices).

My current job deals with java, however Ruby & Scala have kept my other side of programming (hacking & open-sourcing) buzzing with activity and with a healthy mix of curiosity and satisfaction.

Increasing usage of Functional Programming in driving scalable architecture

2011-02-20T22:52:00.000+05:30

One of the most exciting technologies that I've been pursuing lately is nothing new, but as an old wine in a new bottle, provides an alternative method to solve emerging issues for addressing performance. I am referring to Functional Programming, that has been growing in importance during some period in past few years due to various ongoing development in today's huge data processing applications used in enterprise environments. I've been seeing the rise of this otherwise academic language, and is increasingly used/considered to be usable for the past few years due to various ongoing developments.

Features
Functional programming provides various features that separates it as a different paradigm for programming( http://en.wikipedia.org/wiki/Functional_programming ). I'll be explaining various features as per this post's context.
Higher Order Functions : FP revolves around functions! Think of this as a small version of class in OOP. If you've worked with, say anonymous or inner classes in the past. You've unknowingly been doing FP in OOP, and needless to say, it was difficult to learn and maintain. Generics solve the same problem as First Class Functions, but the resultant implementation leaves a lot to be desired.
Recursion : Again a handy feature that provides a lot of bang for small amount of buck, or code! If you know how to use your functions, you can do a lot with your code without having to write boilerplate or plumbing code in your algorithmic implementations.
Immutability : Now this is what I am talking about! We all are familiar with multi-threaded programming, yet the tools for using it are hardly ever used as multi threaded programming involves sharing of resources, which is not a good thing. In all other programming paradigms, we are familiar with the foll construct :
int a = 0; Initialization at this line in the memory

   a = 1;       Memory address pointed by a gets 'Updated'

The problem lies with the second statement as we don't know when and more importantly which thread is going to invoke it. But if you recall from your Mathematics, we used to have following notations there:

   let a=0      Initialize a

   a=5          Assign a something new, and ignore previous state

As the value of a is immutable, we can safely have as many threads to it as we need. This means, as our applications are thread safe, we can have concurrent / parallel executions of the same code by 2 or 200000 invocations without adverse effects.

Applicability in meeting the challenges in scaling and performance

Today's applications need to fully utilize the underlying hardware. Moore's law doesn't holds when we consider raw cycle performance, but is still applicable if we consider the overall increase in processing capability through hyper threaded and multi core processor hardware. The information generation and its processing needs has increased too, leading to use of parallel computations. This is where the FP comes in handy. Its true that infrastructure can be abstracted from the application (as with cloud platforms) but if we do that ourselves, we can have greater flexibility and scalability for our solutions.
One feature of many 'newer' breed of FP languages like F# or scala that are hybrid in nature that I haven't discussed before is the ease of use of these languages is the ability to construct Domain Specific Languages; DSLs. From the business' perspectives, DSLs are fast becoming key to map the technology-business divide and keep application agility and re-usability simultaneously.

My exposure these days is with scala, which is a hybrid FP language that runs on JVM, and is interoperable with java. It is attracting the same kind of curiosity for java developers right now which ruby (largely because of Rails) did a few years back. However, this language aims at scalability, which is also its full name (scala stands for scalable architecture) within the JVM itself, making it an ideal candidate for solving middleware performance and scalability related issues. The fact that it replaced Rails in handling message processing in Twitter speaks for itself. I am also working on a research paper that explains the use of scala (and FP in general) to process massive amounts of data in distributed/cloud environment. Will post some more information here as it gets finalized.

My initial foray into professional software development – Experiences as a Developer

2011-01-29T19:56:00.000+05:30

Its been a fortnight since I've started working full time in a professional software development firm and through this post, I am sharing my experiences. During this period, there have been a lot of revelations for me and many of my fears about work in an IT company have, needless to say, been allayed.
I started working with a small sized local software company that performs all the development activities for a UK based firm that sells products for converting mainframe legacy systems(IBM's i-Series) into modern ones(JavaEE based). Since this is solely focused business on software development, I was able to obtain various advantages that could've not been possible otherwise. Some of which are :-
No managerial bullshit about communication, teamwork & motivation (and what else the HR people can put in).
Simple product oriented workflow with flexible time lines.
A quiet workplace in the outskirts of the city (which is indeed a quaint place).

My only grouse so far (If I really need to proclaim one) has been long work hours (almost 10 hrs) which leave me with no time for outings and socialization on weekdays, but is fine by me as I am growing as a developer in leaps and bounds, and also because I am not much of a social fellow and all the other developers too are busy with themselves during work.

The technologies that I am working on are also my favorite and involve Enterprise Java (JSF, Hibernate, Spring and JPA). However, due to non proficiency in JSF, finally I am learning this framework. In the past, I've blogged against this framework and ever since 2007, have refrained until now in using this technology. The workplace requirements, however enforced this change and this is not as bad as I was imagining it to be.

The first day in the job was a huge eye-opener for me as I was handled a finished application to explore and document my findings. Upon seeing this massive piece of JEE mastery, despair set in quickly an I was beginning to feel panicked. Fortunately, I had done some open source development during my college years that proved to be quite handy for me (Just build the whole thing and unit tested it). By the end of the first day, I had some rough idea about that enterprise application that housed more than 100 JSF beans (that were integrated into similar number of service classes, DAOs, and other layers coupled with a complete in-house API that was stretching back to more than 10 years). To add my misery, the database was a DB2 instance running on AS400 platform remotely and was sluggish at the very best of the performance with no user manipulative tool at my disposal.

The subsequent days, surprisingly, eased this initial pain as I was informed by my boss that the application in fact was auto-generated by a in-house tool based on freemaker/velocity. Also, what was seemingly impossible one day, became reasonable the other and was accomplished on the third. For this success, I'll attribute to the long hours that I am putting at my workstation there instead of shying away from the problem, as the case was earlier.

However, before ending this, I'll sum up an excellent blog entry that I received quite earlier as a tweet:
Ingredients for a perfect technology workplace (read as IT)
Excellent people (Your boss & co-workers)
Excellent projects (That really stimulate you)
Excellent workplace (Or something similar, I can't recollect)

A paradigm mismatch ?

2010-12-31T21:33:00.000+05:30

Here are my comment on an often repeating topic of conversation about 6th semester projects in Masters Of Computer Applications course. A lot of people are considering programming for web based applications aka websites for their last semester projects in MCA. Since this involves a whole stack of technologies, one can observe that it in turn, helps a person or more importantly, his mentor to hide a poor design behind this complexity.
Coming back to the topic, a lot of my friends and people from various coaching centers have certainly got preconceived notions regarding different technologies to code with as their server side programming language. However, one can observe the faint rumblings of faint but still existent religious fervor of 'my language is better than yours'.
To start off with, students try and then ignore Java as an alternative because it is too complex (It is a big pain, but then has its benefits) and huge for their web applications. However, this fear can be allayed by the fact that one can choose from a wide variety of frameworks(MVC like Struts & SpringMVC or component based like Tapestry or JSF) and technologies (a plethora of middleware tech.) within the java ecosystem itself. PHP is touted as a compelling alternative, but for me, the scripting approach only works for demonstrative purposes. Anything larger either needs a huge patience, or frameworks, which are immature so far in this platform. I really am not impressed by frameworks like Zend here to name a few. Microsoft .NET deserves a special mention because it is already baked in (with IDE & app server ready, for instance) as well as has a meaningful architecture(through code behind). However the very strength of this framework becomes its nemesis as students are not encouraged to question/hack into the innards of any tool used. The upcoming framework like asp.net MVC too represents a copied and chaotic exercise to perform something that is coming too late.
My personal outlook about the whole situation is that no single technology rules the roost. If you ask me, it is java for building middleware (business intensive) & expandable (one where you can fit in a lot of components/features) websites. Php is really cute as you get the time to play around with css & javascript based functionality around your site (After all, the data is the king and your site functions as its glorified frontend). If I require a lot of versatility as well as ease of development, I'll go for .NET as it really hits that sweet spot when you are willing to work inside the confines of the framework.
I had this discussion with myself a few days back in a moment of clarity as I was contemplating mine and others' 6th semester project for my MCA course. I specifically ignored Rails (and frameworks based on it) as one generally threads a safe line in our course. Researching and hacking into a shiny new technology and writing a perfect but undocumented & incomplete project would only lead towards its cancellation.
BTW, I had time to write all this stuff on new year's eve as tomorrow is my 5th semester ERP examination. So good luck for me and a very happy new year 2011 AD for my readers!

The stuff behind the fluff – Is Green IT an exercise in vain?

2010-12-06T20:32:00.002+05:30

Cloud computing is generally thought to be efficient, but this study changes it all. According to the recent UK study as published by DW World, cloud computing study reveals higher environmental impact than was previously thought. The quantity of the data centers as well as the consumers would collectively create this problem, which would be further intensified by the use of richer digital media. What this study surprised me was the analysis that it would be casual home users who would lead the excessive usage through rich media sharing/duplication.
In the past, Information Storage and Management used to focus solely on the enterprise specific needs. For handling the binary media format, technologies like CAS(Content Addressing Systems) are already in place. But according to this study, there would be a demand of 3200 Mega Bytes per person per day in just few years time. Given the pace of Internet proliferation and advancement, this is not a vague guess but a worrisome one.
The article pointed out the following courses of actions :
Creation of better/faster computers – This is really not possible as stated due to the lag between demands and Moore's Law
Green data centers – Again, here too the politics play an important role, BRIC nations in particular do have a cavalier attitude towards implementing clean sources.

What was missing here in my option was the point that the research was addressing the solution of the problem rather than its cause. So, instead of addressing the data storage issues, we need to rigorously safeguard against the redundancy in such data. Semantic web is one such approach. Other approaches such as tagging the digital media and having a single copy for identical media (different locations with same signature) could solve this problem. Also, instead of consolidization, the work on distributed computing should also be promoted.
The distributed computing can easily be done using map/reduce algorithms or through application development platforms like Apache Hadoop. The issue of data can be addressed using peer-to-peer data exchange as in bit-torrent. These wouldn't create problems themselves because of the ever decreasing costs of online bandwidth and its performance.

Hurdles Towards Virtulization

2010-11-24T12:28:00.000+05:30

Hurdles Towards Virtulization : What we face on its way to implementation

This post is dedicated to the current challenges that we face in the creation and use of virtualization. I am writing this even at a time when virtualization is something which is sorely missing in everyday computing and beyond for me. For instance, during the past few weeks, I had my both hard disks developing bad sectors and had a miserable experience upgrading Ubuntu on my friends' disk. We need a mechanism to seamlessly move the operating systems from one machine to another. For this, we need virtualization.
Given the current hype and hoopla over this term, many people believe that this is a recent phenomenon which is not the case. Virtualization has its roots at least 40 years before, with the concept of time sharing systems. Similarly, the IBM OS/360 also heralded this new era of 'virtual machines'(VM). In order to manage these Vms, we need a special program which is known as Hypervisor. Now the challenge with hypervisor is that it needs to provide a secure, equivalent, efficient and controlled resource environment for its guests(which are the virtual machines).

So what are the challenges ?
The first challenge that comes in my mind is the performance of the VM. If the guest is not getting enough processing, then there is no benefit out of such an arrangement. This can often happen when the guest OS performs a privileged operation such as a hardware interrupt or a system call resulting in acquisition of hardware locks. Now, as the guest has to run safely, this has to be carefully emulated by the hypervisor.
There is also a concept of time loss or clock skew. As there may be more than one OS per CPU running on a hypervisor at a given instance of time, the instructions by them would be run collectively. However, from the VM's perspective, there were only its instructions, so the instructions take longer time to execute (as other VM processes also hog the CPU cycles) than what the VM actually expects it to. Naturally, this time delay propagates to larger proportions leading to problems such as time-slicing of multiple processes at the guest OS/VM level.
The processing sharing problem also has a profound effect on the performance of the overall system. For instance, on a high priority guest, a low priority process would get precedence over a higher priority process on a low priority guest.This problem has not been solved to this date and only fringe works have been done in this regard. Perhaps this has still pertained due to the fact that a lot of OSs still are created with a single operator at a time without any restrictions.
The memory management as well as addressing are also some of the prevailing problems. Memory cache management is also a problem that rises exponentially as the number of intermediaries between the processes and hardware increase. This can be understood using a simple example. Let us consider that we have a machine which as LRU method of page catching and its guest has MRU method of caching. So as soon as the process becomes old in guest, it is discarded from the memory. However, on the host it is not deleted. So new we have wasteful data on the host memory. The other extreme of this situation is when the data is frequently required in this setup. Now, the host would start to thrash. We need to control this thrashing and ensure page faults remain constant and distributed over a period of time through 'handshaking' communication between the guest and hypervisor with regard to occurrence of faults.The final problem that plagues virtualization systems is the addressing of memory which is there due to different memory addressing schemes being followed in both the hypervisor and guest. The memory addresses are divided for processes into segments, pages and further into offsets. So every process has 2 addresses and 3 memory references, leading to extra overhead. This is avoided through a translation look-aside buffer (TLB), which generally stores the recently used segment and page addresses. But for virtualized environments, this isn't enough as in normal TLB translations, the virtual addresses are transformed into a real ones but due to 3 address levels (Actual hardware address, the 'emulated' hardware address for guest and the virtual address at the guest), there exists an overhead of extra translation. Thankfully, this is being addressed by hardware vendors through hardware support such as Intel's VT-i or AMD's NPT on one end and through utilisation of these enhancements by hypervisors such as virtualbox and vmware.
Hopefully in the near future, we can expect to see more prevalence of virtualized environments, not only in heavy duty systems, but on hardware ranging from the least expensive personal computing to mission-critical systems, given the increasing maturity of virtualization .

Are we a dying breed ?

2010-10-28T19:06:00.000+05:30

Is it really beneficial to build and maintain an online persona when it is fraught with problems all around the corner? This thought was echoed on a thread on javaranch.com, which happens to be a very popular social forum for java technology enthusiasts.
One of my acquaintance had also, quite recently expressed reservations about the same due to the lack of appreciation. Whether his comment is just his perception or is it the truth, remains to be seen. The concept of professional blogging too remains an elusive concept for me and as far as I have observed, this remains largely and exercise in isolation or through and 'tool'.

In the google trends on blogging, it appeared that the search volume index dropped to an all time low in the last quarter of 2010 ever since the usage curve became consistent since 2007. The reference of blogs in the news also became erratic and dropped during this period.

This leads to an interesting conclusion; is twitter really ushering in an era of microblogging since the people don't really want to go that extra mile for sharing their thoughts ? Applying social networking in the above comparison definitely causes no raised eyebrows as it is quite clear that social networking has taken the lead for the online citizens.

As the aftermath of this change, you can expect greater emphasis on the management of information over these platforms - both for commercial and research purposes as well as a general change in the dissemination of information. This is because instead if the traditional Author-reader/follower role, we're fundamentally shifting towards an peer to peer approach with spontaneous interactions. So instead of following your favorite blog and waiting for the posts to come in, real time and probably, geographically aware interactions rule the roost. Social networking today has spread much beyond socializing with friends, family and that 'hot chick' profile(pun intended) to an information sharing portal. It is customizable as a mini internet with all the users being able to be the generators of the content.

So if you are a techie or business savvy, you can utilize this change for your benefit. What do you feel about this ? are there any new developments in this arena that you are aware of ? please do comment about it.

From theory to practice- Transitioning isn't as easy as it seems

2010-10-12T00:24:00.002+05:30

I've been involved recently with the task of creating different projects. In order to do this, I followed the simplest and most important rule in the book – just put the most dedicated individuals on your team. However, with the induction of each separate member in the team, the allocation of responsibilities became more complex and the communication too started becoming a thorn in the flesh for the project vision that I had in mind. However, through agile approaches, currently I am able to lead and integrate my team effectively.
I didn't had enough time to bask in the glory of my newly-founded team equilibrium when my junior students asked me to mentor for their project too (we're having a common project by IBM, namely TGMC 2010). Okay, I said, and went into their class for a briefing. Upon entering the class, I was surrounded by a group of around 70 students, all willing to enter (and most of them already have) in a contest requiring people to submit Ajax driven websites that leverage SOA as their middleware, but very few having experience in even basic html . To get the people up and running, I created a blog wrote a basic how-to (http://bbdnitm-tgmc.blogspot.com/2010/10/getting-up-to-speed-towards-building.html).However, after 4 days, I am yet to get a comment from these people and have resigned to the fate that they need to work really hard if they ever want to see anything at the end of the tunnel. A few of them might deliver a workable solution after 6 months, but for the rest of them, the pain would be inevitable. From my side, I am willing to mentor any person who is a quick learner of jsp and middleware stuff and although I don't know much about the IBM's product usage, I am willing to learn and help others ease into deployment into IBM specific technologies like DB2 and Websphere/RAD. This incident has led me to think about project management in a new light, and now what I am feeling personally is that it is as important to have an able team as it is to find a mentor. Apart from this, it is the approach that matters for an individual, rather than his/her skills.

A case for Security on cloud

2010-09-13T21:13:00.000+05:30

A case for Security on cloud – What vendors tell developers and what they actually do

There has been commensurate commentary over the issue of security over clouds by people. My observation so far has made me come to this consensus that on the Internet, there have been two fractions of crowds – one that favors CSPs(Cloud Service Providers) and place trust over them and the other, quite skeptical and $% one wary of the CSPs over privacy and security issues. This issue has permeated from individual to enterprises alike. In keeping with the main issue of 'trust', CSPs have been quick to point out (for their good more than anything else) the black-boxed nature of their offerings.
This discussion has become more important due to ever increasing share of cloud over the IT infrastructure in almost all organizations, which is not surprising considering the ever growing importance to economize and increase flexibility in all firms across every domain.
As apparent from the above chart, cloud computing has moved from network and storage level (SAN, NAS) to complete machine level virtualization (vmware, zen) to platform as a services, and is moving towards completely managed applications (of which Salesforce serves as an excellent example). This implies that interdependence over CSPs is increasing at almost an exponential pace. Apart from risks in reliability, interoperability and vendor lock-in, there has been jitters in the IT industry, and rightly so because of the black box approach towards solutions as I'll mention below that CSPs sell while explaining the benefits of abstracting security.
However, things aren't as rosy as they appear and life is not as simple as we imagine it to be (sigh!) and security for entire solution is different than that of an application. It is not an aspect or component that we can invert to container or add in addition to the application. In this case, the approach towards security differs as it has to be applied at all the levels of the application stack.
Infrastructure Security
This is important in context of public clouds as there tends to be misconceptions between the terms infrastructure security and infrastructure-as-a-service security. There is a slight difference between the two as there is a difference in the above mentioned terms. Both have a similar ramifications over the customer threat, risk and compliance management. Some of the leading CSPs like Amazon and Google have had incidents where their Data Confidentiality and Integrity have been breached. Proper access control over data is as important as having the information secure. Here again, the emphasis is not over the information security but also on the infrastructure security(There have been cases where discarded AMIs have been used maliciously) that is being offered.
In order to mitigate the increased risk factors, we need to rethink about whether we need a public, private or a hybrid cloud. Similarly for the data in transit, we need to implement rigorous encryption algorithms which again have a trade off between security and usage.
The security approach follows an onion pattern where host security forms the basis followed by virtualization software and virtual server security, which then propagates into network and finally application security.

Data Security(Information Storage and Management)
This is an important part of information storage and management because in a cloud setup, there tends to be a change about how the data moves about. Since the data gets remotely stored, security of the data takes a new dimension. Data security now requires the following aspects:
Data-in-transit
Data-at-rest
Processing of data, including multitenancy
Data lineage
Data provenance
Data remanence

Authenticity and Accessibility on cloud
The services pertaining to the trio of Authentication, Authorization and Auditing(AAA) comprise the Identity and Access Management. Unfortunately, it remains to be seen whether the cloud providers actually are committed towards providing them in a readily usable manner or not because it is inconvenient to provide it properly, but also due to frequent change of roles and rules of the stakeholders involved. However, there has been a lot of work going about here and largely due to federation of IAM entities and emergence of web service standards such as SAML and WS-Federation. What is needed here is something that is IAM-as-a-service. This will not only address weak ISM models, but also makeup for lack of federated structures.

Is Security as a Service (Gosh, another SAAS acronym ) sufficient ?
The Security-as-a-service model, like the original SAAS also is subscription based. This however is nothing new as many vendors have been providing email filtering and anti-virus scanning traditionally.
This service today needs greater scanning and importance that ever before largely due to the rise of crime over the cloud and before cloud platform assumes even greater dimensions than ever before. These are just some of the difficulties that I've mentioned. Similarly, vulnerability management and IAM has been around for a while, but has not truly been incorporated as a service up till now.
Hopefully in the near future, the shift of enterprises towards cloud computing and the maturity of the services oriented model would promote this service paradigm and the ever increasing security conscious customers would fuel the growth of security over cloud.

Commenting about Comments

2010-09-02T16:50:00.000+05:30

//TODO : Finish this rant
#There has been a lot of discussions about the use of comments that appear in program code. This was something that I overlooked so far and only used when it appealed to me, which is itself a rare occasion for me.

#In the book, '97 things every programmer should know', there have been a lot of insight over the same issue. In it, Carl Evans recollects a lesson that he learned from his teacher(getting poor grades without commenting). The same thing, in fact happened to me too.

#Recently, a lecturer teaching Software Engineering (who is quite vociferous in his teachings)in our class proclaimed, “Comments are just waste of time and people(coders perhaps ?) should be kicked in their ass for writing comments. Comments indicate that the programmer was wasting time in writing them instead of writing LOCs”.
However, I learned a lesson similar to Carl in my class exams held in the previous week. In the exam, there was a question that required us to print a pascaline triangle using C#. I wrote its program, that I conjured up at the moment and hoping that it'll yield returns in terms of marks. But to my utter surprise, the program in question was awarded a big 0.
While explaining the paper, this teacher(who teaches .NET framework and C#) declared the opposite of the commenting practices listed above.
In my opinion, the amount of comment that you really need to insert is when the code requires some external prodding to be able to be understood.

So, I guess that it is probably useful to create and maintain comments in a codebase if it is self-describing. According to me, for long term basis a comment should only written specifically for technical documentation. The use of javadocs or msdn styled comments not only serve the purpose for the ordinary comments, but also aid future developers into maintaining and extending the software as they can provide insight in various ways( separate documentation and IDE support).

In The Elements of Programming Style (Computing McGraw-Hill), Kernighan and Plauger note that “a comment is of zero (or negative) value if it is wrong.”. These, alongwith my thoughts over the usability and proportionality of comments are echoed in this javaranch discussion thread. Hopefully, in the near future such a situation may not arise when it became an embarrassing situation for me, just because of lack of comments in my otherwise correctly functioning piece of code.

Why performance matters #1

2010-08-28T10:00:00.006+05:30

One of the many issues that i've faced as an enterprise java developer has been performance. It is important to understand the issues surrounding performance and address them before the problems escalate into unresolvable issues. It is probably the fixation of architectural design patterns and conventions with the enterprise java community that lead us to overlook various performance related issues.

The community has today moved a long way from MVC frameworks and monolithic j2ee 1.4 servers to increasing adoption of various open source frameworks and protals. The enterprise applications too have evolved from websites having component based solutions to portals and coarse granular SOA applications, built on top of web services. However, one thing remains constant, that is the layered execution model of any application. This can be demonstrated as follows :

Our Java EE Application
Application Framework
Application Server
Java Runtime Environment
Operating System
Hardware

It is important to note that the performance of the solution does'nt depends on the application alone, rather, it depends on correct performance tuning and optimization of all the layers associated with the application. This form of issue is known as Vertical complexity.

In actual applications, however, there are a lot of discrete components that are present, each with its own 'stack' of complexity. Together, this forms into another complexity known as horizontal complexity. These issues are apparant only when the application faces heavy load in its lifecycle or when it begins to operate beyond a single JVM. This can result in the following problems :-

Slow execution of application- beyond the aggreed upon threshold in SLAs
Application performance degradation over a period of time- memory leaks, resource allocation bugs
Erratic CPU utilization and application freezing
Performance anomalies which occur in production- hard to reproduce, which escape load testing

So, it is not hard to imagine why organizations and teams look forward to solve the problems. Solution to these problems has also a fancy name, “Application Performance Management” which is a set of recepies and approaches meant to address the issues listed above. In development, this is applied to memory analysis, code profiling and test coverage. Memory analysis can be done using any standard debugger and most IDEs usually have a nice interface to do so. Profiling, however involves some serious investment into the profiler that you are planning to use. Fortunately, some IDEs like netbeans have in built profilers, but external profilers like yourkit are more relied upon. A code coverage tool does the same thing for performance testing as it normally does for unit testing- identifying whether a given code is covered under performace testing or not.

Returning to our 'stack' for a moment, we can easily visualize the tasks needed for performance optimization at different levels.

While the performance at application level can be done via APM, the underlying application framework also plays an important role. For instance, if the framework is in beta and the application is using some new feature of it, chances are that it might have a bug that can wreak havoc upon the application and create confusing problems. The application server also needs to be properly tuned and customized for clustering and performance scaling if needed. In other cases too, there is a need for system administration of these servers as they not only act as the host for the application, but also provide monitoring activities (which can further be used as performance measurement) and manage external resources needed by the application. JVM is an often ovelooked slice of the stack, which if used correctly, can cause precise Garbage Collection trips, memory management for the application as well as its dependencies. It never hurts to have knowledge of the JVM memory areas like stack, heap and perm spaces, which are the places where the application and its server reside and operate. The operating system and hardware are the obvious choices for performance optimization because these provide the underlying resource allocation, scalability, computation and other aspects that are needed for the software. From a developer's point of view, the operating system and hardware is not generally modifiable, but the overlying layers are. So it is better to have an understanding of these concepts. In a future post, i'll start with analyzing specific performance issues relating to a tier and post recepies and solutions that I came up with so far.

IBM Offers free System Z Mastery exam to students

2010-08-02T01:22:00.000+05:30

The IBM System Z is a part of IBM's zEnterprise system, which combines mainframe, power and system x in a single system and as claimed by IBM, can save more than 50% in costs.
This includes z/OS, which is a widely used mainframe operating system.Today the mainframe plays a central role in the daily operations of most of the world's largest corporations. Even the advent of private and hybrid cloud has yet to impact the mainframe market significantly and as mainframes continue to evolve, the demand for professionals working on them rise.

Enough of the briefest of the introductions about this technology.
I wanted to share this information that IBM are conducting the System Z mastery exam for free (they are giving vouchers) to students by the end of this year (before 31st December 2010)
You can visit http://www.ibm.com/developerworks/university/systemz/masterytest/st... for more details.
The course materials are also included at the website.
These exams are important as they can serve as your career decider; mainframe administration is a field in itself.
Unlike configuring server systems targeted for smb s, this involves programming, monitoring, fault tolerance and scalability challenges.
Hope you have fun!

An Open Source Summer Event

2010-07-15T19:27:00.000+05:30

We, the members of Open Source University Meetup recently conducted an on campus event this summer on 5th of July. As always, it was a successful event with a large turnout and participation of students from different years and branches.
This was an altogether a different event as compared to the few previous ones. In this event, we held discussions on how to conduct our group in future and also decided on what next things that we as an open souce lobby in our college ought to be doing.
As the markets for open source softwares continue to mature, so is the increasing number of tools, frameworks and libraries which in short, propell this phenomenon even further.

We are going to have new students coming into our college very soon in the upcoming academic year and this presents an excellent opportunity for us to reiterate and rekindle our spirit of openness and sharing. Special thanks for this time go to Gaurav, Sushant, Raghuvendra, Lalit and others for assembling everyone and making this event a sucess.

Mainly, we held brainstorming sessions on the following areas :

Cool Technologies that we can discuss in depth in future.
Promoting various Oracle based software.
Spreading the need of sharing and contributing in the IT industry.
Trying out open source technologies and helping spread its awareness and adoption.
Conducting events of varied themes and involving various activities- everyone in community can chip in.

As always, by conducting these events, we continue to equipt ourselves better in the event management as well as improving upon the contents of our presentation and doing more than ever for the community.

See the event pics at my OSUM profile albums
http://osum.sun.com/photo/albums/on-campus-event-5th-july

Enterprise Ruby : First Impressions

2010-06-27T18:08:00.000+05:30

I cannot stop marvel at the 'gem' of a useful software that I installed today (www.rubyenterpriseedition.com). As its name was enterprise ruby, earlier my impression was that it was some proprietary software (I had heard websites like twitter using it) but to a pleasant surprise, turned out to be an open source software distributed by Phusion (the same guys behind Passenger deployment tool). It is provided for all *NIX systems and ready to run .deb installer for Ubuntu.
Well, to the point itself, Enterprise Ruby features :
# An enhanced garbage collector. This allows one to reduce memory usage of Ruby on Rails applications (or any Ruby application that takes advantage of the feature) by 33% on average.
# An improved memory allocator. This increases Ruby's performance drastically.
# Various developer tools for debugging memory usage and garbage collector behaviour

A lot of this optimization is POSIX dependent, which is why there is no such version for windows yet. The FAQ page (http://www.rubyenterpriseedition.com/faq.html) demonstrated this and a lot more about the improved platform, which is why a lot of production servers are adopting it.

However, the nicest thing in the whole installation was that you got a lot of commonly used gems installed, which is really appreciative because we otherwise have to manually install or repeatedly try to install via the gem repository (a thing that sadly fails most of the time for me).

Hopefully you find this information helpful. If you feel anything contradictory to that posted above, please do comment as I might be wrong in some areas that I haven't checked out yet.

Towards RESTful Web Services

2010-06-17T10:34:00.000+05:30

When we talk about distributed applications, internet is the enabling technology that comes to the mind. Sure, one can think of many other ways and protocols of performing a software feat on more than one computer at the same time, but the fact that internet or the World WideWeb is the biggest network out there remains true and has the appeal of its ubiquity in the implementation of any such distributed application.
You can even think of a website as a distributed application, but here, the client (or you) is a passive person/computer that can only use what is fed to him. Even dynamic websites ultimately generate static content (or for that matter, even Web 2.0, which is built on user experience rather than anything else), which even if it appears as a distributed application, there is no such distribution of computation going on. However, in a truly distributed application, the portions of application reside in different computers and communicate with each other via internet. This interchange of the necessary data and process requests is what is needed in these kinds of applications.
Before I further befuddle you from the idiosyncrasy of complex distributed application technologies that are widely used like WS-* stack, we need to understand the simple phenomenon that on the web, the content generated on the web pages is in form of HTML, that is read by the web browsers on your computers and displayed to you as a web page. However, if we change this format to a different one (still textual, not changing into binary ) like XML or JSON, the end users of our application become different.
So ultimately, there is not much difference between a web site and a web service as the former can be used interchangeably with the latter and what make a web site easier to be used by a human being can be applied to a web service, which can be applied over internet on the same manner, to a program. The common web service protocol has been SOAP, which is essentially an application layer on top of whatever internet it is operating on. This created a lighter way of creating distributed and interoperable applications, but at an increasing cost of complexity. Instead of reinventing the wheel, wouldn't it be simply great if we simply set forth the best of the internet in a mix and obtain something refreshingly easy, elegant and say useful for our purposes.
This is what REST or REpresentational State Transfer is all about. It is not just an alternative method of creating web services, but is arguably the easier way of creating them as it addresses the addressing and discovery concerns based on the established design patterns of internet. The simplicity of this approach is not its weakness as you probably might be thinking, but its strength. Other web service provider technologies can claim maturity and tools to hide complexity, but the changing face of REST is negating this limitation of itself in this regard. In my future posts, I'll explain how the changing face of REST is going to make it a force to reckon with in future.