| Thomas Joseph's technical blog

Don't be a dick technologist

noreply@blogger.com (Thomas Joseph) — Sat, 02 Jul 2016 03:54:00 +0000

Being a technologist doesn't mean that everything we create is created from scratch. We Google around, find lots of write ups, blogs, articles, and code and we take them all for writing our own solution. However the effort we put to create our solution would vary a lot. Our solution may depend on a lot of libraries, or at times we have to copy some code and get started. All this is so usual to our normal routine work, but one thing many of us forget about is the license that needs to be respected - especially when dealing with Open Source licenses.

A decade back, people used to question the quality of the Open Source, but today with professional open sourcing in place this mindset has changed and people now have thinking that using Open source is good. However many people still believe that "Open source is free". People also wrongly believe that "Every visible source code is open source". Another wrong belief about Open Source is that for source code - "can-see is can-copy". This mindset needs to change. People must know the license that applies to any piece work (including a write-up), and must faithfully abide by the license.

But most of the time, reading the license text is very verbose and complex with legal terms that many of us would fail to understand it. But irrespective, we all must give due respect to the hard work of the people(s) who originally created the work, and which you are now using. This is rightly summarized by the DBAD license (Don't be a dick license). This license rightly brings out in simple and humorous way the courtesies that every user of the piece of work must follow. This license focuses on the moral responsibility of a person copying the work, rather than legally binding the person to do something or prevent it. It highlights the fact that many people copy other's work, and remain ungrateful, and do not attribute their work to the original work, or even worse - complain about it.

I personally may not apply a DBAD license due to its loose legal terms. However, I would love to put its text along with my work, to let other know how important it is to show some courtesy to the creator of work. I would love to reproduce some of the context text of the license here.

As developers we all want to protect our code from dicks that try to steal, sell, infringe or just generally rip us off. For this we have licenses like GPL, MIT, etc.
Sometimes however, you release a project where you just don't care what happens to the code. For this Sam Hocevar created the WTFPL which is a brilliant license.
For those of you who want something in between, try the DBAD license.

One thing which every person and company using other's work in their own work (especially when using Open sourced work) must understand is that although the work is free to copy, the license accompanying it is still a legal license and should be abided. The copyright owner (or someone on their behalf) of the work can always sue the infringing dick. When you make loads of money by selling someone else's work, do care about buying them a pint or a coffee please, or at-least acknowledge them. Don't be a dick!

Universal Applications with Universal JavaScript

noreply@blogger.com (Thomas Joseph) — Mon, 20 Jun 2016 01:22:00 +0000

Universal Applications with Universal JavaScript from ethomasjoseph

JavaScript is going Universal - Why should this matter you?

noreply@blogger.com (Thomas Joseph) — Mon, 26 Oct 2015 03:37:00 +0000

If you program for the web, and think that you are a "backend developer", and JavaScript isn't my cup of tea, that is not true any more. You can close your eyes and wish away that JavaScript on Server side is a hype, but the fact is that JavaScript is invading the Server side space each passing day, and you "must" know JavaScript and JavaScript based rendering technologies, if you intend to create next generation web and mobile applications, or may be - just to be employed!

In the recently shared roadmap of Angular 2 in Angular U, the Angular team presented the idea of splitting Angular into two pieces: core functionality and the renderer. The concept of renderer in Angular could be used for server side rendering, or native rendering on iOS or Android.

This roadmap clearly brings Angular into the family of Universal JavaScript (also known as Isomorphic JavaScript) rendering technologies. The family of Universal JavaScript rendering technologies already include Rendr, Flux, React, Dust and LazoJS - to name a few popular ones. Coursera, Airbnb, LinkedIn, Facebook, NetFlix, Walmart are some of the high volume web applications that have already moved to Universal JavaScript. This move has already helped these websites to write applications that run universally - on server, browser and mobile apps - significantly reducing the development and maintenance effort across various channels, and more importantly - you just maintain one code and be flexible about intelligently choosing a logic execution and rendering to be on server or client side while pushing up the web applications to be much faster!

JavaScript going Universal

Despite the creation of JavaScript as a lightweight look-alike brother of Java, JavaScript eventually became a key component in enabling the Web 2.0 evolution - especially through AJAX. JavaScript has moved much beyond its initial design goals, with more standardization with ECMA, optimizations done to every release of the JavaScript engines, JavaScript is all ready to rule the web (and more platforms).

Universal JavaScript
Image Credit - Netflix

Today JavaScript is a ubiquitous runtime. JavaScript is everywhere - from Browser, server side, embedded device, mobile and gaming.. The language's influence continues to grow on the server landscape via Node.js, and runtimes such as Rhino and Nashorn for Java; Jurassic and V8.NET of .NET and projects such a J2V8 and React aiming to go even higher by targeting at Server side and mobile at the same time.

Digital Marketing, CMS, Ecommerce and Omnichannel

If you are into Digital Marketing and Ecommerce, you cannot discount the fact that these worlds are increasingly growing omni channel. Engaging consumers across various channels require consistent experience. I am yet to gain some more insights into usage of JavaScript for native apps and work it out, but thinking your application architecture with universal JavaScript rendering itself lends the architecture to support mobile apps as well. Which means building omni-channel experiences would be much easier, with cleaner separation of concerns. More importantly many of the frameworks advocate - "learn once and write anywhere".

My Experiences with Universal JavaScript

Working at SapientNitro, last year I had initiated a humble beginning to use Universal JavaScript rendering with Handlebars being introduced for Adobe AEM on the server side, which later paved its way into the IEA - Integrated Experience Architecture. With almost one year down the line one extremely positive outlook common with people who have worked on this concept on different projects was the "separation of concerns" and the inherent advantages of it.

However such a model of development could be a cause of concern at times when there is no mediator who would understand the "backend" and the "frontend" frameworks patterns well, and allow the two teams to gel together. Even without such a person, projects would still move on, but what is produced at the end may not be elegant or easy to understand.

When I started out with this effort for a project for a leading financial consulting client, I learnt Handlebars, Grunt, and NodeJS to an extent that I wrote some Grunt tasks and Handlebar helpers on JavaScript. While this may look trivial to a seasoned JavaScript developer, for me it really helped a lot to understand both sides of the world and help the Interactive team and the AEM team with teething problems and integration glitches. This also helped us to create a better application "together". Looking back at the application, it is so seamless that ramping up a new member on the technology stack was really fast.

JavaScript - much beyond just rendering

JavaScript from a pure programming perspective itself is interesting - even when we leave out the goodies gained with universal rendering. JavaScript does not follow a purist programming paradigm, but it lends itself to almost any programming paradigm - Object Oriented (with prototypal inheritance), Functional or or just even no paradigm! Lot of libraries exist to help you get your flavour of JavaScript. The evolution of JavaScript specification itself is going through interesting phase - especially with ES6 advances on making the language better. The TypeScript from Microsoft makes JavaScript "strongly typed" and object oriented, while libraries like Underscore, Lo-Dash, Ramda make it a functional language. This further enables you to take the best of the different programming paradigms and write better programs. Add to that, the programs you write can run on Server or on the client. Isn't that interesting?

JavaScript based server side technologies - especially with NodeJS have advanced so well that you can use them to create full blown high-volume (I/O) web applications. Rendering page, connecting to database, crunching number or throwing out graphical charts, you name it - NodeJS applications do almost everything or even more than what you would expect from a traditional language or framework.

I would recommend that more and more people on the "backend" come forward and learn the front-end frameworks, more importantly the JavaScript. This would help us be better programmers and architects and create better applications. For when you plan to give training to the developers, do also consider that the training can also include the interested backed developer. A "frontend" aware backend architect and developer can make such a difference that you will end up more time productively creating application than arguing that "it works for me", or "please do whatever to make this work on your end". And remember the Atwoods Law - "Any application that can be written in JavaScript, will eventually be written in JavaScript." as a corollary to "The Rule of Least Power".

The improvements on the framework is ongoing. You can't wait for the perfect framework to come to learn it. Improvements in JavaScript frameworks seems to be there each passing day. If you are not already into the groove of learning JavaScript, do so. Understand the ecosystem of JavaScript - especially with things like NodeJS, Grunt, Gulp and JavaScrpt templating. Try joining your local JavaScript meetup group, or give yourself some hands on training.Try out the functional and object oriented features of JavaScript. It will make you a better and efficient programmers and architects.

Sling - Who is closing my JCR Session?

noreply@blogger.com (Thomas Joseph) — Sat, 26 Sep 2015 13:49:00 +0000

Has this happened to you? - You got the JCR session, but at times, you find that "someone" closed the JCR session! And that too - its very inconsistent. Sometimes it works and other times it does not! Well then its that you have hit the Sling / CQ development anti-pattern. And possibly this is what you see in your log:

javax.jcr.RepositoryException: This session has been closed.

As suggested by Jorg Hoh on his blog on CQ development patterns, you should rely on Sling objects as much as possible, rather than relying directly on the JCR objects. But at those cases (e.g. writing to a repository) when you will have to directly rely on the JCR repository you will have to still use the JCR Session, its recommended that you directly get the JCR session from the Sling repository, and not adapt it from the ResourceResolver, IF you do not intend to use the ResourceResolver directly.

@Reference
private ResourceResolverFactory resolverFactory;

// Wrong way of obtaining JCR Session
private Session getSessionAntipattern() throws LoginException {
 ResourceResolver resolver = resolverFactory.getServiceResourceResolver(null);
 return resolver.adaptTo(Session.class);
}

Code Listing 1: JCR Session Creation Antipattern

@Reference
private SlingRepository slingRepository;

// Correct way of obtaining JCR session
private Session getJCRSession() throws RepositoryException {
 return slingRepository.loginService(null, null);
}

Code Listing 2: JCR Session Creation Pattern

You may wonder, what's wrong. Logically thinking, why would you retrieve the ResourceResolver (and adapt it to JCR Session), if you do not intend to use the ResourceResolver? This logic has made the Sling developer's to close the underlying JCR Session of the ResourceResolver when the ResourceResolver object is Garbage Collected. Although, how this is done is being changed across the releases, the basic idea of releasing the resources held up (JCR Session) is still valid.

If you observe in the code snippet 1 (antipattern), the scope of the ResourceResolver object ends as soon as the method returns, which makes it eligible for garbage collection. This means that whenever the garbage collector runs, the JCR Session got through the antipattern would be closed - oops! This explains your JCR Session getting closed inconsistently.

I would like to emphasize that adapting to JCR Session from ResourceResolver itself is not wrong, if you intend to use the ResourceResolver along, other than just adapting it to JCR Session.

Hope this post helps someone save some time debugging on the JCR Session getting closed inconsistently.

Insane sanity (testing)!

noreply@blogger.com (Thomas Joseph) — Tue, 22 Sep 2015 17:30:00 +0000

You are busy writing your code on a complex piece and suddenly a team wide announcement comes - "The new code has gone in for the build, please do 'sanity' for your components".

"My unit tests would save me!" is not something which will save you here if this happens to be something which has a lot of visual and "content driven" elements. So, you break out of writing code half-way and start doing the "sanity" because it is crucial to test your components before lunch, as this code will be going to production soon after. You do this half-heartedly because this is so common for you whenever a major release happens or a content restructuring happens - and anyway "I am not a tester!". So you lose at-least one hour of your otherwise productive time to doing this insanely mundane task. On top of this, you also have your "professional testers" doing the same testing -just few extra eyeballs! This is the typical scenario if you are developing content driven (CMS) applications. Bless you if you are not so!

Insane testing
Image Credit - Wikimedia Commons

At this age in Digital Marketing, where content is the king, faster development and delivery overpowers the more structured delivery methods often followed in typical enterprise applications, where you invest for a lot of integrated and automated testing. Reason - "If we invest time to build integration testing and automation, we lose time to market". This is a broken argument often upheld by people who most often do not even care about integration testing in its real sense, and automated testing which can test the UI elements and the content of the page.

If we can invest some time in bringing about continuous integration and UI automation, a lot - really lot of time could be saved on developers doing "sanity testing". This may be difficult to achieve for shorter duration projects of the span less than 6 months - all out of their own budget, but may be easily feasible for bigger projects. Better - if this could be planned by "practice leads" who could invest on this once, and reap the benefits multiple times across projects. I do not have metrics to prove this by maths, but just a little retrospection into this would be useful.

Personally, this topic is lower on my radar. First, there are higher priority tasks for learning and experimentation on my TODO list. Second, I work more in a consulting mode than delivery, so budgeting it on my project is not possible. Additionally, I do not have metrics to convince my client to budget this either! However, I will try to squeeze in some time in the coming days and weeks to do some research into this topics and publish my findings.

Three great advice

noreply@blogger.com (Thomas Joseph) — Mon, 21 Sep 2015 18:38:00 +0000

Act on it. Don't think too hard. Be Agile. These are the three great advice that have been my favorite picks from the book - "The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009)".

"Act on it" means that a piece of good advice needs to have an action plan on it - however small it may be. Most of the chapters of the book had the "Act on it" section which forced us to review ourselves and try to find an action plan. Without an act, the advice just rusts in our minds.

Thinking hard - "to jump or not?"
Image Credits - stefaanroelofs

"Don't think too hard" means that it is not necessary to perfect the things in the first go. We should think on it, but not "too hard" to make it the best at the first attempt. This has been especially important to me because, often trying to do the "best" often resulted to the act not being done at the first place. The best example would be writing something for the blog. I would often have such a lot of topics but often either I choose a topic which requires itself to be expressed as an essay, or just make the writeup to be the best and cover all the aspects of the topic, leaving no open loops. Both of this essentially means a lot of time, which eventually just rusts in the backlog and never gets a chance to get published.

"Be Agile" when applied to career and life means that career and life are never done. Re-iterate over, and try to make them better every time, and correct the course of action if it needs so.

Putting this together would also mean that I should be blogging more often, even with not so well rounded topics still appearing in there. And of-course, this would also apply to other aspects of life as well.

Nice is just ordinary! You need to be Extra-ordinary - Amazing - Remarkable!

noreply@blogger.com (Thomas Joseph) — Mon, 14 Sep 2015 16:45:00 +0000

Today I happened to see a TV advertisement which had a beautiful story - "Nice is just ordinary!". This happened to rephrase what Chad Fowler had to say in his amazing book - The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009), which I had finished reading one round just today.

Winning is not about not-losing. Winning is about being awesome. Winning is about being Amazing - Remarkable!

My career, as I look back, seemed to have stagnated. I never had a roller-coaster ride in my career. I had pigeon-holed myself into the mediocrity - especially for the last 4-5 years. No one else to be blamed - just me. This was why I forced myself to read this excellent book to be inspired, and bring out the best in me. After-all I got into this profession for pure passion.

This book was in my reading list ever since 2010, but it was two weeks back that I finally ordered a paperback edition from Amazon, which I would religiously read during my travel to and from office.

The world around us always demands extra-ordinary and amazing things. Everything else is just OK, and CAN just live without. But I would like my life to be something which would create a dent in the world! Something awesome; something remarkable!

Yes, I do need a lot of overhauling. I have to invest in myself - to be a remarkable product - to create remarkable products. And this should mark the beginning of the journey to a remarkable career.

Over the next course of time, I would post critique to the book, apart from my brain dumps of some interesting ideas brewing.

The secret sauce of wrapping a jar as OSGi bundle

noreply@blogger.com (Thomas Joseph) — Sun, 13 Sep 2015 15:06:00 +0000

OSGi has been on the mainstream technologies for some time now. Many of the Java application servers now are either natively OSGi based, or support OSGi. This has also led many to package their libraries as valid OSGi bundles. But still there are a lot of libraries, that have absolutely no knowledge of OSGi - and when we have to use one of such libraries - we have to depend on one of those many open source projects which gives an OSGi bundle for the library, or prepare our own recipe to wrap them into OSGi bundles.

Before you begin

Before you pack your toolbox to wrap vanilla jars into OSGi bundles, do search public OSGi repositories and projects which can save a lot of your effort. Here are some of the popular ones:

SpringSource Enterprise Bundle Repository
ServiceMix Bundle Project Github Repository - https://github.com/apache/servicemix-bundles
Pax Tipi Github Repository - https://github.com/ops4j/org.ops4j.pax.tipi

If you still can't find what you are looking for, you may have to consider building your own! You can even consider contributing your bundles to the ServiceMix Bundle project or the Pax Tipi project.

Wrapping of vanilla jars can sometimes become complicated, especially when it is related to dynamic class generation or that of XML parsing. And that is why I am writing this - to share my experience of wrapping OSGi bundles when the library is complicated.

Choosing your tools

No surprise, I am also going to suggest the BND tool here. However, the BND tool itself is available in various flavours depending on your development workflow and tool as described in the Download page of the BND tool:

Bndtools - Bundling tool for the Eclipse Plugin
Apache Maven Bundle Plugin
Bnd4sbt - Bundling tool for Scala application
Gradle OSGi Plugin - A Gradle based OSGi plugin. Gradle is a Groovy based builder.
Ant - Available as an ant task
Osmorc - Plugin for IntelliJ IDEA
Command Line Utility

All of them however use the bnd library at its heart. However here, the BND tool command line utility can be handy to quickly wrap vanilla jars into OSGi bundles and do some basic analysis of the vanilla jars.

The choice of tool is mostly a personal preference, and also driven on your development environment and workflow. I personally prefer the Maven Bundle Plugin. However, irrespective of the tool used, there are a few fundamental principles which will help us wrap the bundles successfully. This need not be the best way, but has always helped me to build out OSGi bundles.

For the Maven users

To begin with, you can clone one of the projects - ServiceMix Bundle Project or Pax Tipi project, and add your own module to start out wrapping your bundle. If you need to manipulate the vanilla jar other than just adding the Manifest, you could consider using the Maven Shade Plugin - as it is being used in the ServiceMix bundle project.

Setting up the lab

First, I would recommend setting up an environment to test your OSGified library. You should have the following in your testing environment:

An OSGi container - Felix or Equinox or whatever you are using. Ensure that the version matches your target environment.
The target OSGified jar - yeah we are not done yet, but this is the test target. We will perfect it over some iterations.
The client bundle - the bundle where your code uses the API which exists in the target library (#2 above). You can use an OSGi activator (or the SCR activate) to run your code which will test if you have your library OSGified correctly.
The supporting bundles in your target environment. This may range from logging bundles to that of XML parsers and so on. Just be as close as possible.

All this environment should be something that is easy to provision and setup. I personally use a vanilla Felix instance with necessary environment bundles provisioned using Pax Runner. However you are free to use the tool of your choice to do this provisioning. The objective is to be able to start, test and stop the OSGi container, make changes to your target OSGified bundle and client bundle and repeat. And you do need an environment which should be easy to provision. You can also consider using Pax Exam to setup this environment.

Bundle it up!

If the above setup gave you a success right away at your first try, then you need not read the rest of this writeup. But most often, we do not get it right at the first attempt. So then you will have to do some experimentation with the packaging of your bundle which will ultimately give you the success.

Realize what you need - single or family pack?

It is important to understand and realize that what you ultimately need, and how the final target OSGified bundle will look depends on your use case. If operating in a pure and strict OSGi world, every library should be in its own bundle. In other cases, where there is no strict requirements around your dependencies being strictly separate bundles, you can combine a few libraries together to get a single bundle (or may be still a few more). However, both these ways of bundling has their own pros and cons.

Single bundles are the citizens of perfect OSGi world. This bundle would only represent one vanilla jar. The advantage of this approach is that you fully leverage the OSGi modularity. If there are multiple bundles which wants to use this library, one instance of this library would do (as long as the version requirements also match). Your artifacts would be light. However, you may have to deal with and manage a lot of these bundles if the transitive dependencies of your target library is just too big.

Family pack - as I love to call it - is usually made by combining a related set of libraries together to form a single bundle. In Maven world, this often can be related to as the transitive dependencies. This kind of packaging may sometimes be necessitated by the library itself, which often has code which has hard, non-OSGi compliant bad OSGi practices code for loading classes and resources, which requires some other dependent libraries to be on the same classpath (OK, you need to understand that in OSGi, every bundle has its own classpath). At other times this kind of packaging avoids you to manage a lot of bundles on your target environment. However, this would mean that tomorrow if another library needs to be added, which has one of the transitive dependencies that already exist somewhere in your existing family pack bundles, either you would duplicate the library over in your second family pack, or just expand your existing family pack. Family packs at times can also lead to cyclic dependencies being created between bundles, which really is just evil.

Things to Remember

Once you have decided the kind of packaging you require, you can move ahead and start creating and perfecting your bundle. The following are some of the things to be taken care of when you start your OSGification process:

1. Know what you are doing

Unless the vanilla jar you are trying to wrap is really simple, you would have to understand OSGi a little bit deeper - especially what each of the manifest headers mean, and how it affects the classloading and resource resolution. What each of the BND instructions mean and how it affects the bundle being generated. Read the BND documentation, and also the tool specific documentation - in my case the Maven Bundle Plugin's documentation.

2. Verify the vanilla jars versus your OSGified bundle

Verify that the contents of the vanilla jar and your OSGified bundle to ensure that you preserve the packages and the resources contained therein, unless you intend it otherwise. The resources iteself may contain some configuratoin files that are required by the library to function, or it may have some licenses. You should be particularly be aware that almost all the licenses and notices require that you keep them in the derived products. If you are merging more than one library, you may have to consider renaming the resources - Maven Shade Plugin can be useful here.

3. Perfect on the contract - Import and Export packages

The Export-Package and the Import-Package are the primary contract of your OSGi bundle. What goes into these should be absolutely clear. Try to understand the library clearly, and export those packages which are API, and hide the internal implementations (some libraries hardly distinguish this fact, so that forces you to have not-so-perfect OSGi practices). Also ensure that your exported packages has a logically valid version. For instance, if you are OSGifiying itext-2.1.7, you could start with an Export-Package of com.lowagie*;version="2.1.7".

What to keep for Import-Package is trickier sometimes. This is where you would have to experiment a bit. Start with all the import packages being optional (*;resolution:=optional), and then once your client bundle is all working perfect, come back here and understand what is absolutely necessary for the library to function, and which are optional. As an example, some libraries may give out dependencies to JUnit classes. Now, why would someone depend on JUnit classes, so quietly remove them from the import packages. On the other hand, dependencies to JAXB libraries could possibly be made as optional one. All this depends on the library under consideration.

4. Ensure no BND warnings

When you are perfecting the contract of Import and Export packages, ensure that you have no BND warnings. Especially around split packages. When you get the split packages, carefully study your vanilla jars (often split package warnings will come when you are making a family pack), and understand of how would you like to prioritize the packages across the libraries to apply the split-package instructions for BND.

5. Dealing with cyclic dependencies

As mentioned before, multiple family packs can lead to cyclic dependencies. When this happens, carefully study the bundles involved in cyclic dependencies and the packages on which these links are formed, and carefully remove the cyclic dependency by either separating out one of the transitive jar as a separate OSGi bundle, or simply keep all the packages of such a library as private. Take the decision based on your situation.

6. Adjusting your client bundle

At times, your bundle may be importing certain packages from the wrong bundle (instead of what you are intending). Example of such packages may be what is exported by the system bundle which is outdated, and you instead want your client bundle to import from a later version exported by a separate OSGi bundle. In these cases, use the version as your savior and explicitly restrict the package versions so that it would import from the bundle of your choice. Often the client bundle may have a relaxed import versioning, which makes it to import from the system bundle. When the version is stricter, you would often achieve what you intend to.

7. Classloader Hell

At times you may get into situations that even after following all the instructions above, your library is unable to load some classes or resources. Carefully try to debug through the library's source code and see where actually the code fails. Also compare this execution to one in a non-OSGi environment, and you would be able to identify that in OSGi, the library is unable to instantiate a class or load a library, because of the way that the code is written to do that. In these cases, the last option may be to patch the library. Download the source code of the original library and make modifications to the offending classes and put it back to the OSGi bundle. Follow the OSGi best practices when you do dynamic class loading or resource loading.

8. Test thoroughly

Test you client bundle thoroughly with all the possible use cases of using the library. If possible use the Pax Exam to write a test suite to be sure that you are covering all the use cases. Everytime you run the above environment, you will realize - most often with OSGi resolution error messages or error stack traces that something went wrong. This information is key to identifying what has to be tweaked to get it working. Repeat the all the steps above through this step, until you are done.

9. Be license compliant

Nothing technical, but respect the licenses of the libraries you are trying to OSGify.

For AEM / CQ developers

For sometime I have been working on the Adobe's marketing platform AEM /CQ which utilizes OSGi as the application container and so requires developers to package their code as OSGi bundles and many of them get panic when they have to OSGify an external dependency - especially when it is a complicated one. I have personally found that most of these AEM developers do not understand the OSGi beyond what is there in the product documentation. And this was my primary motivation for writing this article.

I have made some shortcuts for kick-starting your OSGification process for AEM environment, by creating Pax Runner profiles for few AEM versions at my Github repository https://github.com/ethomasjoseph/aem-pax-runner-profile. Feel free to tweak and use this and contribute to this if you find this useful.

Before you say - I am done (or not)!

Bundling a vanilla jar - especially the complicated ones require a lot of patience and experience. A lot of the above instructions are dependent on the context of which library are you trying to work with and what exactly is the need of the hour. Weigh the choices to choose your track. Feel free to reach out to me (blog comments are OK), in case you want some help in OSGification of your vanilla jar.

Happy Bundling!!!

References

Tuning your JCR Queries for the AEM & Jackrabbit OAK

noreply@blogger.com (Thomas Joseph) — Wed, 04 Mar 2015 13:11:00 +0000

Application tuning is an activity which which every developer and architect will encounter sooner or later for any application. AEM based applications are no different. One of the activities that is particularly important with AEM 6 is query tuning. AEM 6 uses Jackrabbit OAK as the repository and hence we would have to know more about OAK to tune the queries. Apart from tuning the queries itself a majority of the time you would be tuning your indexes based on the query. AEM itself uses a lot of queries within for the application to function. It is important to understand how the queries and the indexes work in AEM.

Update: This post was not written as a generic performance tuning guideline for AEM, but just on tuning the queries on OAK repository. Although a lot of the performance depends on how well tuned is your repository. Adobe's recent service packs on AEM 6, and AEM 6.1 in general are much stable and performant. Please refer to the Adobe's documentation for general guidelines on performance tuning.

While the instructions herein are primarily written for AEM applications, non-AEM applications which uses Jackrabbit OAK can also benefit.

There are no hard and fast rules that a particular index will work. This is very similar to database indexes. You would have to look at the current queries being fired and the current repository state and arrive at the best indexes. Often you arrive at the best indexes over time. You would have to apply an index and the re-evaluate your queries, then further tune your queries or try out a different index. Repeat these steps over and over again to find the best and optimized index.

However, there are some basic guidelines which should be followed, which can help you define the best indexes to you application specific needs:

Ground Rules (AEM Specific)

Apply the latest Service Pack available and Oak specific hotfixes (if applicable). As of date SP-2 is publicly available.
AEM has some built in indexes as well. Do not arbitrarily re-index these OOTB indexes. Indexing is a costly and resource-intensive operation. OOTB indexes should be re-indexed only when specifically advised by Adobe (this is a catch - sometimes you know that re-indexing a particular OOTB index will solve your problem at hand, but get a buy-in from Adobe for this). At other times you would have to create a new index (type) since the OOTB / existing index is in-efficient.
Be aware that there has been some important changes since OAK 1.0.9 - especially related to Lucene indexes, hence be mindful about reference resources that advice about using OAK indexes. Check the OAK's latest documenting on query and indexing.

Tools

If you are using AEM, the best tool to help you debug and explain your queries, find out slow and popular queries is to install Adobe AEM ACS Tools package (0.0.20 and above). Once installed, you can have access it using the URL: http://yourinstance/etc/acs-tools/explain-query.html
If the above is not possible, you may have to depend on JMX to find slow and popular queries, you can use the Felix console to identify slow and popular queries on any instance

http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D9%2Cname%3D"Oak+Query+Statistics"%2Ctype%3D"QueryStat".

You can also use the OSGi Service of org.apache.jackrabbit.api.jmx.QueryStatManagerMBean to programmatically find the slow and popular queries.

queryStatManagerMBean.getSlowQueries().values()
queryStatManagerMBean.getPopularQueries().values()

Use the following debug logs for details on various aspects of query and indexing:

Enable DEBUG logging on org.apache.jackrabbit.oak.query for logs to find the details on query cost calculation (explain)
Enable DEBUG logging on org.apache.jackrabbit.oak.plugins.index for logs to find the details on indexing

Use JMX to have high level overview about lucene indexes (OOTB or custom):

http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D4%2Cname%3D"Lucene+Index+statistics"%2Ctype%3D"LuceneIndex"

To look at the actual indexes, you can stop the AEM / Oak instance and use the oak-run "console" to explore the indexes among various other options:

java -jar oak-1.1.6.jar explorer /path/to/crx-quickstart/repository/segmentstore

(oak repository)

On the resulting console, run the following command, to dump the index into file system:

lc dump path/to/dump /path/to/luceneIndex/inRepository

eg.

lc dump path/to/dump /oak:index/myCustomLuceneIndex

Now analyse the resulting dump using Lucene Index analyser tool Luke:

java -XX:MaxPermSize=512m -classpath luke-with-deps.jar;oak-lucene-1.1.6.jar org.getopt.luke.Luke

The following jars needs to be used:

The luke-with-deps.jar can be downloaded from https://github.com/DmitryKey/luke/releases
The oak-lucene-1.1.6.jar can be downloaded from Maven repository as well.

Guidelines

The basic steps for tuning your queries can be summarized as follows:

Identify the slow queries in your system. This could be the one that takes a lot of time to execute primarily because there are not proper indexes and there are node traversals happening. The system itself can be queries through JMX to identify the slow and popular queries.
Tune your queries to ensure that you can use indexes whenever possible.
Identify properties and nodes that needs to be indexed, based on the slow queries identified, and create Oak indexes for them.
Verify your queries and indexes after creating indexes. The indexes are optimum if the cost of the queries identified are lowest - 0 being the best.
Repeat steps 3 and 4 above (by even creating different indexes for the same query) till the most cost effective index is created.

Identification of Slow Queries

Identify the slow queries by monitoring the error.log. Note the queries which traverses many nodes, where the log may look something like:

org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed 1000 nodes
... consider creating an index or changing the query

You can also identify the slowest and popular queries in the system (which the Jackrabbit Oak keeps track of) using JMX or Adobe AEM Commons as described in the "Tools" section above.

Indexing

For the same query you can create multiple indexes. However choose the one to be promoted to your production instances which prove to be the the best in terms of achieving lower cost to the query at hand.

Consider the following when deciding the type of index to be used:

Full Text Index Types

Lucene & Solr
Aggregate Index - query time aggregation

Lucene Property Index

Since OAK-1.0.9 Lucene Property indexes are preferred indexes over other type of indexes
Async
Combines properties & full-text.
Supports property condition, range conditions, ordering, full-text
Index-time aggregation - performance improvements
Using Lucene & Solr.
Follow instructions for creating the index nodes as in the OAK's Lucene documentation.

Property Index

Default Indexes
Synchronous
Unique or Non-Unique
eg. UUID
property equals query works best

NodeType Index

Internally Uses Property Index
Uses primaryType and mixins to identify nodes

Traversing Index

Traverses Repo
Does not store any data
Used to retrieve the given path or its child nodes (sometimes the parent nodes as well)

Ordered Index

Async
Not recommended of nodes > 1000 nodes

Index Management

Currently, the indexes can be found under /oak:index/ on AEM.
Once you have identified the best indexes, plan to install them as a package on AEM.
Re-indexing is usually not needed. However when you see degrade in system performance, do double check your existing indexes if it would need a re-index.
Re-index is a costly and resource intensive operation. Use it only when any oak upgrade or mentioned explicitly in release doc for any patch or hotfix
Avoid having multiple properties in index , it may slow your query , its recommended to have one index for each property.
Set the system property oak.queryLimitInMemory to 200000 (or something similar). This will limit the traversal of the nodes to that number

Finally...

Jackrabbit Oak is a project under active development, especially around the area of Indexing. Hence please check official documentation for details. Indexing and tuning requires a lot of observation and practice to get perfect.

I would recommend watching a webinar on this topic by Thomas Muller and Marcel Reutegger on AEM Gems, which is a bit old (before OAK-1.0.9 was released), but immensely useful to understand indexing on Jackrabbit Oak.

Happy Indexing!!!

Writing method with dynamic return type without type casting

noreply@blogger.com (Thomas Joseph) — Sat, 13 Jul 2013 08:12:00 +0000

I have often come across this requirement across multiple projects, where we need to write a method, which can potentially return object of any class, and still you would not like to do a typecasting. Something like this:

MyClass myClass = getDynamicClass(MyClass.class);

Instead of :

MyClass myClass = (MyClass) getDynamicClass(MyClass.class);

Common example of such use case is where you would do some lookup of a dynamic service (as in OSGi) and return the object found. However, with Java 5 and above this is easy. So, if your earlier code was:

protected Object getDynamicClass(Class interfaceClass) {
    Object obj = null;
    // code to create / lookup the dynamic object
    return obj;
}

This code can be converted as follows:

protected <ReturnType> ReturnType getDynamicClass(Class<ReturnType> interfaceClass) {
    Object obj = null;
    // code to create / lookup the dynamic object
    if (interfaceClass.isInstance(obj)) {
        return interfaceClass.cast(obj);
    }
    return null;
}

This is very obvious code, but thought that sharing this would be useful. Enjoy!

Installing Java Plugin for Ubuntu / Linux for multiple users

noreply@blogger.com (Thomas Joseph) — Mon, 25 Mar 2013 15:30:00 +0000

If you are a Linux user like me and have encountered websites that require applet, you may be ok installing the default Java that comes with your distro, and most of the time you may be good to go with it.

But at times you may specifically want to install Sun Java on your system. Since Oracle has licensing restrictions, distros like Ubuntu do not have Sun Java in their repositories, and you will have to resort to some "longer" ways of installing Java as explained in here for Ubuntu.

But even by following the "longer" ways, you may be still annoyed when your browser says that Java plugin is missing (it happened for me for my Ubuntu 12.04). Or else, even when it may be working for you, it may not be working for other users on your computer, and you may have to create soft links to Java plugin executable in .mozilla/plugins directory of every user's home.

Assuming that you have already installed Java flavor of your choice, and want to enable Java plugin for your browser for every user of your computer, just create a soft link to the Java plugin executable under /usr/lib/mozilla/plugins/ directory.

cd /usr/lib/mozilla/plugins

ln -s /usr/lib/jvm/jdk1.7.0_17/jre/lib/amd64/libnpjp2.so

...and you are good to go. Now you can refresh your browser containing the applet to see that it is working, or you can test it at the Java's official test page here. For me here is how my browser looked after I configured Java as above:

Know your Geolocation right away with HTML5

noreply@blogger.com (Thomas Joseph) — Fri, 27 Apr 2012 07:44:00 +0000

Wondered how Google Maps tracks you current location? HTML5! Thats what enables most of the modern-day applications to track down your current location.

You can do it too right away!

Open up your JavaScript/Firebug console. You can hit F12 to work on most of the browsers. And paste the following JavaScript code snippet.

navigator.geolocation.getCurrentPosition(function(position){console.log(position);});

navigator.geolocation.getCurrentPosition(function(position){alert(position.coords.longitude + " / " + position.coords.latitude);});

Did you see a confirmation asked on your browser? Allow it! Thats it! It was so simple. You can now see your location in terms of longitude and latitude on the console. You can use this data along with Google Maps API to spot your location on a map.

Earlier and now...

Earlier, before HTML5, the user's IP was determined and so was the location tracked. Some sites provided friendly ways to do this such as http://www.geoplugin.net. With HTML5, if your browser supports HTML5, this should work whether on a mobile, tablet or on a conventional desktop. Although this way of location tracking won't show up much difference to earlier approach on a desktop, it would sure show a difference if you use it on your mobile with GPS enabled.

Developing Java / OSGi applications on Day CQ

noreply@blogger.com (Thomas Joseph) — Sun, 28 Aug 2011 07:51:00 +0000

Day CQ is a very popular enterprise CMS (Content Management System) suite from Adobe. Although most of the time the development on Day CQ is related to CMS and most of the development can be mapped to a proper CMS system, there are times when you would need to develop some logic in your application, or need to develop some feature that may not be available readily within Day CQ. In these cases, you may need to write some application code.

Day CQ is a Java based software, and Java is the most popular way to develop custom features and application logic. To be specific, Day CQ is largely built using OSGi - a technology that enables you to develop truly modular applications. Developing any custom Java application modules also needs to be built on OSGi.

I have noticed that developing OSGi modules is often a pain for a CMS developer, largely because of the lack of full development support (or call it a buggy development support) from the CRXDE (The bundled Eclipse based IDE), which makes OSGi module development to take long time and painful. This post intends to give more insights to a CQ developer on OSGi, and also to do faster development of OSGi modules with OSGi.

Why the development pain?

Why is OSGi development painful with vanilla Day CQ development environment? Day CQ provides an Eclipse based IDE called as CRXDE. It is excellent for typical CMS development, but when it comes to developing Java applications, it lacks a lot of features, where a regular Eclipse (or your favorite IDE) will win.

CRXDE is not a Eclipse plugin, but rather an RCP application. For this reason, we cannot add any plugins to the CRXDE, as in a regular Eclipse. For this reason, it is not possible to create a regular Java/Maven project in CRXDE, and regular Java development is not possible. Although CRXDE allows us to write Java code, it is buggy. I have often encountered unwarranted compilation errors in Java code. Moreover, we cannot enhance the IDE with any plugin, such as Maven, which is useful for dependency resolution.

Utilize the expertise

It would have been the best thing, if CRXDE was available as an Eclipse plugin. But since it is not, we will explore options where we will develop OSGi bundles where we will use the tools for which they are good at. Use CRXDE for regular CMS development - develop templates, components, workflows etc. Use regular Eclipse (or your favorite IDE) for Java / OSGi development. Use Maven for build and dependency resolution.

One may have some workarounds with CRXDE itself, but again, if we follow the approach I give below, it would be save a lot of our development time, apart from the ease of development.

How each of these tools are used independently is beyond the scope of this post, but rather my focus here is to provide an approach to use these tool in a collaborative manner to make CQ development easy. That apart, we will use few more toolis specific to OSGi, that will make our life simpler when developing OSGi bundles.

Developing bundles for Day CQ

Let me outline the approach on how we will develop, test and finally deploy the OSGi bundles on Day CQ and see it working.

Make a Maven project which can create OSGi bundles.
Write code and install the bundles in local maven repository.
Test the bundle
Configure Day CQ's Felix web console to install bundles just created
Test the Day CQ applications for the bundle just created.

Let me demonstrate the approach with the famous "Hello World" example, since it is better than giving abstract instructions and code samples. We will create three bundles:

a hello world API bundle
a hello world Service implementation bundle
a client test bundle which will utilise the hello world service

Note however, that this approach will be more useful when you have complex requirements and more external maven dependencies.

I have made this "Hello World" application available on github.

Create a Maven Project for your OSGi bundle

I suggest to use Maven for your OSGi bundle projects because:

Maven is excellent in managing all the dependencies
Creating OSGi bundles with Maven Bundle Plugin cannot get simpler. It automatically takes care of including any headers required due to the transitive dependencies.
The learning from Maven can always be reused in your other projects as well, since Maven focuses on convention over configuration.

I prefer to use the following configuration for Maven Bundle Plugin:

<plugin>
 <groupId>org.apache.felix</groupId>
 <artifactId>maven-bundle-plugin</artifactId>
 <version>${bundle-plugin.version}</version>
 <extensions>true</extensions>
 <configuration>
  <instructions>
   <Bundle-Name>${project.name}</Bundle-Name>
   <Bundle-Description>${project.description}</Bundle-Description>
   <Export-Package>${bundle.namespace};version="${project.version}"</Export-Package>
   <Private-Package>${bundle.namespace}.internal.*</Private-Package>
   <_removeheaders>Ignore-Package,Include-Resource,Private-Package,Embed-Dependency</_removeheaders>
   <_include>-osgi.bnd</_include>
  </instructions>
 </configuration>
</plugin>

Here I have a osgi.bnd file place besides the pom.xml which has any specific instructions that needs to be provided to the bnd when creating the bundle. This will also override any instructions that are provided at the POM level (eg. some inherited instructions). For more details on the plugin, visit the official documentation for the Maven bundle plugin.

What's next?

I have given a heads up on a less painful and more robust method to develop OSGi bundles for Day CQ. But there are more things, that can make things easier when developing applications on Day CQ. This would require a more deep dive into OSGi, and I will try to illustrate approaches that uses OSGi's best practices while making the OSGi development easy.

So mark this space for more on OSGi development with Day CQ!

Solving eclipse error with SVN 1.6 (loading JavaHL library)

noreply@blogger.com (Thomas Joseph) — Sat, 03 Jul 2010 06:48:00 +0000

If you are using Eclipse and SVN (Subversion), there are high chances that you are using the Eclipse plugin - Subclipse. And if you recently upgraded to subclipse 1.6, there is a good chance that you have encountered the error or loading JavaHL Library, that says something like this:

Problem:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

If this is the case, then few tips to get this working. As always, I will suggest solutions with the least number of steps, and all simple.

When installing/updating subclipse 1.6, use the update site - http://subclipse.tigris.org/update_1.6.x/ from within Eclipse.
The installation screen shows the features. Make sure that all features marked "required" are selected for installation. (You will anyway be not allowed to proceed otherwise).

If it still does not work, it is highly possible that the Java native libraries required for the SVN to work together with Java (thats Java HL) is not available to the Java. Either not installed or just that Java is not able to find it.

1. Install Java HL

You can refer to here for a generic discussion on this topic, especially for Windows and Mac OS. For Linux, I have simpler solutions :).
Install through your package manager, a package named libsvn-java, that contains the JavaHL. You must make sure that it is for SVN 1.6.
For Ubuntu or other Debian based Linux, the command should be handy:

sudo  apt-get install libsvn-java

However, it is possible that the installation is requiring you to install the whole Java binaries itself, although you already have one (an old version, or a manually configured Java for example), and you are not willing to install the Java packages. In that case, just go to your package maintainer's site, and manually download the package, and unzip it to your system (yes, you will need root permissions), so that the dependencies are not installed.

If you had installations fine, without any hacks, restarting Eclipse should get your Subclipse working. If not, follow what is below.

2. Make Eclipse runtime aware of the new JavaHL binaries.

. There are two options for doing this:

Tell JVM/Eclipse about the JavaHL libraries when starting JVM with the system property, that points to the java library path where the JavaHL binaries are installed. eg:
```
   -Djava.library.path=/usr/lib/jni
```
This option can be specified in config.ini as well.

Put the JavaHL binaries in paths that JVM normally searches for native libraries. Normally these paths are listed, in the error above. Above the paths are listed as:

java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

I went for the second option, since that one seemed most non-intrusive and easy to me. Let me show it how I did that:

Choose any of the listed Java library paths. In a terminal, move into that directory:

cd /usr/lib/jvm/java-6-sun/jre/lib/i386

Make softlinks to the JavaHL libraries.

sudo ln -s /usr/lib/jni/libsvnjavahl-1.so libsvnjavahl-1.so
sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0 libsvnjavahl-1.so.0
sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0.0.0 libsvnjavahl-1.so.0.0.0

Now restarting Eclipse should get everything working fine. If it does not, review your steps, and also read in full length about JavaHL from http://subclipse.tigris.org/wiki/JavaHL, and try the steps again.

Running tomcat/webserver on port 80/443 without root permissions on Linux

noreply@blogger.com (Thomas Joseph) — Thu, 24 Jun 2010 18:52:00 +0000

The Problem

One of the concerns for web application developers is to run their web servers on the default port (which is 80 for HTTP, and 443 for HTTPS). However, since these ports are privileged ports (ports less that 1024 are privileged ports in Linux/UNIX), the developers are either required to run the server either as a root, or run on alternative non privileged ports such as 8080 or 8443.

The Solution

While to the same problem, there may be different solutions, the most simplest of all that I found was to do local port forwarding. Again here, many do not like to fiddle with commands every time. An easier, and one time solution to this is to use a program called "guidedog".

Guidedog is a GUI based advanced and powerful networking configuration tool for Linux, that can be used for packet routing, NAT and port-forwarding. Although it is for KDE, your Linux package manager should be intelligent enough to make that work for you even if you do not use KDE. We will here use its port-forwarding feature.

Steps

Launch your favorite package manager tool (Synaptic/Aptitude/YUM etc) and install "guidedog". For Ubuntu I use Synaptics Package manager, or even a command would do.
```
sudo apt-get install guidedog
```
Once installed, launch the application with root permissions (don't worry, you require it to do just once).
```
sudo guidedog
```
On Ubuntu, you can also find it under the menu Applications->Other. You will have to even edit the menu command to gksu guidedog.
Move to the "Forward" tab. Add a new rule, such that TCP traffic on port 80 is forwarded to port 8080 (or whatever port your server runs on). Similarly, add another rule, such that TCP traffic on port 443 is forwarded to port 8443 (or whatever port your server runs on for HTTPS/SSL connections). After you are done, your screen should look like this:
Apply the changes.

You are done. Now just hit on localhost instead of localhost:8080, next time you start your applications.

Tool power

The tool creates a system rule to forward the traffic to a different port(8080/8443) from its original destination (port 80/443) on the same machine. You can see in the tool that it is much more powerful, and can allow even port forwarding for other machines too, or even do IP Masquadering/NAT (Network Address Translation). However, explaining on these topics would outgrow the topic of this blog and would take more of your and my time too.

VNC via SSH tunneling

noreply@blogger.com (Thomas Joseph) — Mon, 24 May 2010 17:04:00 +0000

5 simple steps to connect through VNC to office/work PC via SSH tunneling

Situation:

You have to connect to your office PC from home via VNC, but your office PC does not have a public IP. The only way to access your PC is through one of the access point machines provided by your office that can be reached through Internet.

Concept:

VNC port is 5900. You can ssh tunnel to your office PC through your office access point, and connect to this tunnel to have VNC connection.

Assumption:

Office access point machine: accesspts1.company.com
Internal IP of your office machine: 192.168.1.31

Also assume that your Office PC is switched on, and access point machine can access your office PC via the internal IP, and can access its ports as well.

Steps:

SSH to access point machine. Provide password when prompted
```
ssh username@accesspts1.company.com
```
Now that you are on access point machine, ssh from there to your office PC. Again provide password when prompted.
```
ssh username@192.168.1.31
```
On your office pc, start VNC server. Keep this terminal open.
```
x11vnc
```
On another terminal, make open a tunnel to your office pc via the access point. Provide the password for your access point machine when prompted.
```
ssh -L 0.0.0.0:xxxx:192.168.1.31:5900 username@accesspts1.company.com
```
Use an VNC viewer to connect to your Office PC via the tunnel. Use the following details:

Host: localhost
Port: xxxx

Agility and more - better & faster business solutions with software - part-3

noreply@blogger.com (Thomas Joseph) — Tue, 30 Mar 2010 15:15:00 +0000

In my previous posts, I discussed about practicing agile in real life, and why adapting the agile prescriptions is important. I discussed how the basic values of Agile manifesto narrates it all. This is my last post in this series, where I will explain about practices even beyond the direct agile prescriptions, that can help build faster and quality software with smiling teams.

Some Agility and beyond

While the software creation itself is creative, the process employed to create this software is also creative. Its not just practicing agile, that helps create good software, there are more things to it. The craftsman (developer) who creates software has also to be given importance (Did I mention that before?). Let me discuss few of these things that include things from the Agile practices such as XP and Scrum, as well as those that go beyond them.

XP practices

Focus on producing better software with XP practices such as continuous integration, with focus on unit tests, integration tests, and automated functional/UAT tests. I do not want to go to the depth of CI and unit tests, but it helps you be confident on regressions, as the software evolves. There are few other inherent advices that Martin Fowler has with the CI such as "Keep the Build Fast", "Test in a Clone of the Production Environment", "Make it Easy for Anyone to Get the Latest Executable" etc. I would highly recommend reading his article.

Continuously try to clear up the technical debts. Did a Quick Fix? Create a ticket that says,.. fix it! Removing old legacy and unused code - it helps developers avoid straying away to look at an unused code, to fix problems, and faster induction of new joiners to team. Old code also makes large codebase, maintenance problems, larger build time and lots of confusion.

Equally important is a focus on continuous refactoring for improved code readability reduced complexity and improved the maintainability and extensibility, for one or more of its benefits. This also helps clear our technical debts.

Closely related to refactoring is the CI, that would help us to be sure that code refactoring is going fine. And to be confident in that a lots of test-cases needs to be in, that would ensure that there are no regressions. So develop a habit of writing unit test case for every bug or feature request. From my experience, going fully TDD (Test Driven Development) sounds like crazy as we develop all the tests-cases upfront, while having no test case or very scanty is also not a good practice. My experience is good to stay somewhere between - say at 50-85%, if we assume TDD to be 100% for writing test cases (use your mind and adapt!).

The rule is simple. Start out lean, and consult Scrum and XP for places they are expert in. You need not take all their advice, but taking an advice is always good. Have something very technical - probably XP is good, other places Scrum is good, some other places - you get advice from both.

Know your team velocity

It is very important to know your teams velocity. What is the team capacity to crunch out the story points (or whatever measure you keep) given to them? If the team velocity is not known, it release dates predicted may not be credible. There are various ways to measure your team velocity. For example, you could measure your team velocity by recording the story points finished for a period of time, or measure the amount of original estimated hours that the team could finish for a period of time.

Talking of this, it is important to remember that the efficiency (sometimes called the Focus Factor) of the team is never 100%, since teams lose time doing unplanned items, doing context switches, helping other teams, checking their email, fixing network snag etc. With my experience, even when we had 9-6 working day, we normally counted it as 8 hours working day assuming the 1 hour lunch time as a normal organizational practice. However, when doing plannings, we used to calculate only 7 hours for a day, and hence allot only 6.5 - 7 hours of approximate work per person per day. That would be around 75-80% efficiency. However, you may like to bill your client w.r.t to all the time spent i.e 8 hours per day per person, which is perfectly fine! :)

It may sound ridiculous to count 60% or 70% efficiency, but on real analysis of your team, it is normal to find such statistics. However, such a statistics is helpful to make realistic expectation from your team, and commit to reliable release dates.

Cumulative Flow diagram depicts the team velocity and the lead time required to finish items - helps to plan and minimise queues/waiting times.

(Image reproduced from the InfoQ minibook)

Most organizations want to reduce lead time, and do get things done faster. Unfortunately many try to do this by getting more people in or working overtime. Usually the most effective way to get stuff done faster is to smooth out the flow and limit work to capacity, not add more people or work harder. This type of diagram shows why, and thereby increases the likelihood that the team and management will collaborate effectively.

The velocity does not vary much, however, you will have to regularly measure your team velocity, since velocity may increase with experience and expertise. However, this velocity may not change drastically over successive iterations, except for may be the first few iterations, when you are still stabilising over the agile practices, or when the team is new to a project. Release planning with these statistics can would help achieve timely and reliable releases and yet prevent team burnouts.

Clear goals and minimized wait times

While focusing on individual productivity is important, it is also important to understand that they maintain their productivity when working as a team. One important factor that may hamper individual productivity when working as a team may be lack of clear goals on what to do next. Having a clear-cut road-map for not just for the day, for for the coming week itself may be good.

Minimising wait times increases team throughput

(Image Courtesy: Vype's web album)

Another factor to be considered is to reduce dependence on each other. Plan out the iteration road-map so that there is minimal dependence of one on the other. See that a team-member is not waiting for the other to finish. There may be dependence, but then, the timing can be adjusted for it. In manufacturing, a good importance is given to reduce queues of intermediate products to reduce costs and increase throughput. For software development, there is no silver bullet for this, however providing an execution plan upfront can help in this, since if developer knows what needs to be done for tomorrow or next week and something that is independent of the current work, they can go ahead and start doing that task, so that individual productivity remains high. Well planned iteration backlog, daily stand-ups and scrum principles of removing impediments can help a lot to this.

Automate to Scale

Try to bring automation to places wherever possible, since doing things manually doesn't scale, but machines can scale. That's how Google has achieved its excellence - by relying on algorithms and machines to scale rather than depending on humans. Automation can be brought at many places - CI, intelligent builds for every commits, sending error reports of CI and builds, and other automated testing, smoke testing, functional testing, and even deployments. You will definitely avoid team burnouts!

Reduce complexity of the software

Reducing complexity in your software would help a long way in more than one ways to create good software. Well known agile and modularity guru Kirk Knoernschild explained in his keynote speech during the recent Jax London conference about reducing complexity in the software. But there is no silver bullet to this, though modularising application is a definite direction to this.

Kirk further explains the cost of complexity with the following statistics:

The number of lines of code doubles every 7 years
50% of development time is spent understanding code
90% of software cost is maintenance and evolution

Look at the #2! So if complexity is reduced, time spent in understanding code is reduced, thereby speeding up the overall development process. It is natural for the systems to grow complex over time, and as system evolves, unless efforts are put in to reduce this complexity.

"As a system evolves, its complexity increases unless work is done to maintain or reduce it. "

(Image Courtesy: Flavio's web album)

There is more to reducing complexity in the code:

Write less code
Build modular applications

Write less code

Avoid writing code whenever possible. The mantra is "less is more". Avoid writing code that is not your main line of business. Many teams spend lot of time and energy writing huge chunks of code and configuration for creating "platforms". Platforms are necessary, but try to evolve it from existing and stable code libraries and frameworks as something that has the least glue code, so that your framework can evolve over time. For example, you need to have great amount of integration points in your application, before jumping ahead to write message brokers, message bus, and all those fancy ESB kind of features, consider one of those ESB's and message brokers available out in the open source or its commercial variant. Will the original author be available all through the project life-cycle to explain the code and intent? Who will maintain it? Will the subsequent maintainers fully understand the code and its intent? When we use popular libraries and frameworks, there is more chance of finding developers who already have knowledge in it.

When developing internal frameworks, its imperative that we design for even what requirements may come after 10 years, but the fact is that even after just 4-6 years, you will be hardly using the same system. Technology may have changed, your requirements may have widely changed and you end to design another framework to tackle those, or may have added a bunch of code to your existing system, and many of the capacities of your old framework goes unused. Still if you feel compelled to write some good amount of code for your framework, consider to market it separately or may be consider still better open sourcing it, so that you can early find co-developers, and build a community around that. Of course you can always commercially sell a software that you have open-sourced.

Build modular applications

While writing modular applications is like a very common vocabulary that we use and have this intent when we design our applications, the reality is that achieving "true" modularity is difficult. Writing component based applications may provide some answer. For Java applications considering OSGi may be a good choice, that has been upheld by all the major application server vendors including IBM, Oracle and Sun. Recent JVM modularity to be introduced with Java 7 with Jigsaw, and the Oracle's more recent modularity efforts with Qwylt are also interesting move towards this.

Focus on developer productivity - tools and techniques continuously

To have a faster development and lesser team burnouts, one very important factor is to focus on how effective is the software craftsman in performing the very act of producing software. It is important to look out to improve developer productivity at every act of the software development - project layout, development mode and development cycle time, running tests, IDE integration, build tool effectiveness, conventions over configuration, code and architecture complexity.

Unit tests and integration tests should be easy to run in all possible ways - with regular builds, test builds, CI builds, and even in IDEs. This helps to frequetnly run the tests withing IDE, and be confident on the code from the very beginning of writing code. This helps a lot especially when refactoring code. Running tests should be equally easy and simple from the command line as well.
Focus on conventions over configuration principles on all possible aspect of software development (including build - eg Maven). Although choice of build tool is not a religion, Maven is for one thing a great productivity improving, since once you learn Maven, you know it even when you change your projects. Adopting such practices also brings down the learning curve, and even new joiners can get productive quickly. Applying this principle is not just for project layout, but the overall architecture and day-to-day coding as well.
Watch out and remove impediments required for a fast build-run cycles. Does the build require punching in a lot of key strokes? Also watch out with continual performance tuning. Does running of you app, almost freezes a 3GB memory system? - High time that you lookout for memory leaks, and do application startup tweaks. At production you may be countering this with a massive server configuration, but think if trimming the code would actually save you on team's energy as well as on your server infrastructure money as well! The builds should be easiliy launched from within and outside IDE. If you have multiple IDEs, make sure that all the IDEs work just fine (Maven plugins are available for almost all major IDEs, that can help you do this).
Strive to find tools that would make things easy and remove/ automate / lessen hassles and repetitious tasks. Tools need not be always visible, eg.. scrum and kanban are tools for process improvement, while Spring ROO is also a tool that explicitly focuses on developer productivity. For instance promote JQuery or similar JavaScript library instead of fully hand crafted JavaScripts.
Carefully choose and use issue tracking systems, and wiki software for your requirements:

Choosing issue tracking systems with integration to your SCM. This would be good to track the commits for a particular issue, and ease code reviews.
Use popular solutions:

There is more probability of finding experienced users- less learning curve.
You don't have to maintain that s/w which is not your mainline business.

Consider Distributed SCM soultions like git or mercurial (Remember that this is just a tool, but it helps manage changes rather than versions, and so that branching and merging is simple.).

Have easy read code-bases, easy run executables and easy obtainable project artifacts. Martin Fowler nicely describes this in his article, where I would like to highlight a few items:

Have everything you need to do a build should be in there including: test scripts, properties files, database schema, install scripts, and third party libraries. Note: Maven eases this out, since you have a separate system (a maven repository) to maintain third party library, and you would just put the dependencies in your POM. The basic rule of thumb is that you should be able to walk up to the project with a virgin machine, do a checkout, and be able to fully build the system. Putting in IDE configuration file is also a good practice. However, do not store the build products itself in the SCM., this rather indicates a flaky build technique that cannot reliably recreate builds.
Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)
Make it Easy for Anyone to Get the Latest Executable - To help make this work, anyone involved with a software project should be able to get the latest executable and be able to run it: for demonstrations, exploratory testing, or just to see what changed this week. Doing this is pretty straightforward: make sure there's a well known place where people can find the latest executable. An internal Maven repository for example is a great place.

This discussion is not a comprehensive one describing each and every bottlenecks for faster development and increased productivity, but covers a lot of ground on it. You may share your own experiences and discussion points using the comments section in this blog.

Technology, R&D and more

Encourage R&D within your organization.
Have a dedicated R&D team if possible. Swap people in and out of the R&D team. Encourage people in all teams to spend 20% time for doing something creative, or may be blog to start with. Consider putting people to R&D as an alternative to benching, if you have such a practice. Encourage the R&D team to publish their findings regularly through blogs and research papers. I will blog separately about the need for blogging. You can identify your gems working this way.

The 80:20 principle at Google is there to bring out wild ideas from developer minds to be experimented. Some of the great projects from Google including GMail, Google News etc have been the result of such researching and experimenting mentality of Googlers.

Technology is still important!
As emphasized in the DDD principles - give good focus to the business at hand, but at the same time don't neglect technology. New technologies emerge to solve challenges that the existing ones cannot solve or because they are not efficient in doing that. This underlines the importance of using newer technology to solve our technical problems. Keep a balance however. It would not be recommend to use bleeding edge technology for production use, but encourage R&D learning to use the new technology 'wisely' whenever needed and possible.

Community learning
Encourage your team to get involved with some good and popular open source project. This could also be a project something that you use yourself, or are planning to use in the near future. It has many benefits - when you officially can take part in OSS activities, you are getting better visibility to your company, apart from the learning that your team and company as a whole gains in the process. There are many ways in which the team can participate - participating in forums and discussions, filing bug reports (with test cases preferred), feature requests, subscribing to developer mailing lists if available, and if possible - commit code, help with release etc. The team also gets better knowledge of good standard practices for software development - issue/ticket management, coding, testing, continuous integration, release, deployment etc.

Community learning helps in many ways

(Image Courtesy: Horace Mann School's web album)

Continuous improvement: Always look out for improvement - talk to peers, get ideas from them (that's why retrospectives are there). Encourage team to spot improvements or pain points in your projects and processes and seek solutions from them. Your team will be forced to think creatively, and you may in-fact get something useful! Have an open minded discussions before incorporating such changes (or rejecting suggestions) to the project.

Solving your software development problems

Agile is about iterations and incremental software development. This helps to build software that evolves over time with the customer requirements. Some prescriptions of agile practices such as from XP about CI, unit testing, helps to keep you confident in what you develop for each increment.

That apart, looking at the discussions so far, and in the links there-in, we can deduce that its not just agility, there is more that helps you build better, and quality software. Agile practices with focus on developer productivity helps you release your software in a timely manner, yet maintaining the quality and budgets, while keeping your team happy and in high spirits.

How to Start...?

A difficult part to start anything new, is to get started. Very true for agile too!. Read a lot on agile, Scrum, Kanban and XP. Start with basic minimals on being agile

Start stand-ups - focusing on highly communicated and focused team
Start iterations - smaller the better
Incorporate to retrospect your processes, code and techniques
Never feel complete with improving your process - it should be a continuous process.

Good start would be to start lean, and then add prescriptions that work for you! And,... don't forget to adapt!

Final Thoughts...

This discussion, or few more blogs cannot cover everything related to software development. So much of literature is there around. Get inspired from those. Go ahead and start somewhere. You cannot start a perfect agile anyway! Over the time, you will improve with every iterations.

Update...

Agility and more: better & faster business solutions with software - part-2

noreply@blogger.com (Thomas Joseph) — Mon, 29 Mar 2010 14:30:00 +0000

In my previous blog post, I talked about why software development needs processes, and the need for agile in the processes. Here, I continue with the discussions, focusing on how the basic principles of agile itself has a lot to say, that can help us build better software.

Practicing agile - adapting is more important

As mentioned before, the popular practiced tools for agile process include Scrum, Kanban and XP. But often, the process models/tools are applied as a religion, being strict on what the text prescribes. This has resulted in some rumblings that has questioned the effectiveness of agile processes to deliver quality and creative solutions to business. The InfoQ book on Scrum and Kanban has highlighted the need for the process models to be adapted to their needs.

One size doesn't fit all - adapting is important.

(Image Courtesy: Robin Skyler Tell's photostream)

So what needs to be practiced is taking a lean model, and adapting it to your needs, probably by adding "prescriptions" from the stouter process models, and even removing "prescriptions" that hinder your progress, but with absolute knowledge of what is being removed. This is what Henrik has put in his InfoQ book - "Do not limit yourself to one tool!".

I have my own experience, where we started out with pure Scrum. Scrum iterations help evolve not only the software but our (scrum) process itself with continuous retrospectives. As we practiced it, over the time we hit against the walls of the Scrum, such that we had to eventually remove few prescriptions of the Scrum, effectively making it leaner and the process as "Scrum Inspired". For example, we would feel OK to extend our sprint for 2-3 days and still declare it as a successful sprint! We would find it OK to even entertain new items during the sprint, if it was critical to the client. Sometimes, 1 or 2 team member may work on items from another product, that is planned in advance for the sprint, because we had small teams, and the same team members would work on multiple products/projects. In spite of this, we always knew where the limits were, and we were effective with whatever we did. We were strict on whatever process we followed. If we changed the Scrum rule, we did that knowingly, and probably would be the outcome of the retrospectives we had at the end of sprints. By adapting the process, not only was our output improved (that would make customers happy), the management as well as the team members were happy at what we do, and how we do it and the overall team productivity.

We all were extremely happy on the retrospective day, since it was the day we could blast out our rants, impediments and criticisms openly. We virtually had fights over the retrospectives. But finally, it all would help us improve what we do.

Mix-n-match and adapting is important. For instance, Scrum does not say anything about releases, but XP does, hence combining them together is more compelling. The "Do it" nature of scrum and the "pair programming" and "move people around" principle of XP and a well communicated team also helps in getting a new joiner to be inducted fast into the team as well!. Most of us have done pieces of XP and agile knowingly or unknowingly, if for example we have written unit tests, did iterations of development and the like. But the real benefit for going agile comes from developing and following process models with prescriptions from the agile processes available. Process models such as Kanban actually give us metrics to measure our process against, and take the best of what works for us from all the process models - agile or not. We can choose to start out with Kanban, with probably the regular iteration, daily stand-up meetings, and retrospectives from the Scrum, add to that the software engineering specific practices from XP, such as CI, unit testing and pair programming.

Again, what is more important is to adapt the process models to your needs. For example, even as I mentioned above for practicing prescriptions from Scrum and XP, I am still wondering how a stand-up meeting would happen for a distributed team? Distributed Scrum? Probably, we can have scrum of scrum, but again, this is something I have no experience. So, when the need comes for it, I will give it a try, practiced, adapted and practiced and improved so on.

Responding to change over following a plan

This is one of the four core values of the Agile manifesto. Adapting your process is a definite way to achieve this objective. Agility promotes responding to change inherently because of its iterative nature, that can enable the team to incorporate changes to a software more rapidly that traditional classic waterfall model. Process framework such as Kanban further fosters this changes to be percolated more rapidly into the development.

Try experimenting. Experiment with new and innovative ideas for improving processes. Invite all sort of crazy ideas from your team as well. I am sure, you will not go worse even if your experiments failed. You will learn so much. Next time you always know what not to do :).

Focus on individuals and interactions over processes and tools

This is again one of the four core values of the Agile Manifesto. This is what we have covered already. Let me explain. By focusing more on adapting the process to our needs instead of following the process as a religion, we are focusing more on the people and interactions for sure. The stand-up and the retrospectives for example are effective to give importance to the people who are involved, removing their impediments, and giving importance to their interactions. Once we practice scrum and XP, there is a value that the team learns. It is more important to follow these values, than to follow these practices by book.

Communications and Interactions

As we talk of individuals and interactions, I should emphasize on the need for effective communication and interaction within the team and with the customer (or his representative). Well communicated teams is one of the focus of scrum and XP (and most of the agile practices). The team stake holder should ensure proper communication channels. Small and well knit teams help a lot on this. If a team member has a problem, he knows already to whom this should be addressed, and the turnaround time to address this should be maximum of two meetings - i.e, the impediment should be addressed before the second team meeting after the impediment had been raised.

(Image courtesy: Avish's web album)

To enhance our communication channel, we used enterprise blogging at three levels:

Organizational blog: We used this blog to express new ideas and rants across teams.
Team blog: we used this blog within the team for various purpose

Daily status update - every team member used to put the daily status update for days work under the following headings:

Tasks completed
Tasks pending
Impediments
Things I found new today
Comments

Project specific posts - To discuss and document a project specific problems, and ideas. Many times although, we would promote some of the posts to the organizational blog too.

One-to-one interactions are really good at fostering better communication and interactions. It will be extremely good to have your team work at the same location, if possible within same cubicle. ThoughtWorks practices this with "Dining Table" culture for the team that fosters greater collaboration and communication (and flow of ideas). For distributed teams, this may not be feasible, where better interaction can be provided by having the team members available to each other on IMs. Use of wiki based whiteboards as discussed above can be a good way to go for distributed teams. Well, there can be a lot of talk for this. So I reserve the 'distributed teams' topic to a separate discussion.

Human factor

While focusing on individual developer productivity is important, it is also important to see how the team as a whole can mesh well and be productive. A very important thing is to have well-knit and well communicated team. Focus that they can work well with each other considering the human emotions and social factors.

Frequently changing the context is not a good idea. Everybody may not be equally good at changing contexts. It would be nice to group together similar tasks. Human minds takes time to get adjusted to the new environment and get productive. Again, this is a topic that may itself need a separate discussion.

Tools do not replace process and individual interactions

When we started out with Scrum, we used whiteboard for everything - from daily tracking our burndowns at daily scrums to the retrospectives. But later to keep track of what we do, we moved parts of these into our enterprise wiki. Later we also tried with JIRA's GreenHopper plugin to manage most of the scrum, except the retrospectives. We where still open to change or adapt to tools as would our process be adapted. In short, we used the tools to aid us what do.

Using tools does not mean that we are doing what the tool is meant for. What we do for scrum is more important than the tool used to manage scrum. Using the Agile PLM does not mean that we have gone agile. Tools should be at our disposal to aid us what we do more effectively.

Focus on the developer skills

A craftsman is the center-point of every creation - not just software. From the craftsman of a building to that of a supersonic jet aircraft - the skill of the craftsman involved is very important and crucial to it. Software craftsman is no exception.

The developer should realize the importance of developing skills even beyond his current needs. Developers should proactively try to learn, innovate and come with new ideas. Always try to learn something new at work, and from around your (professional) community. Work proactively to improve the environment you work in. Work towards a more productive process, for getting better output for your company and yourself. No one can teach you or or force you to learn. It has to come from within. Try reading books, blogs and articles, try to keep up with what is happening around. Find how a new technology could help you create better software. Find how a new technique or trick could help you do more with less effort. Attend conferences, and user groups, speak at them, share your learning - good and bad, blog about them. Peter Goodliffe and Dan North have some (pdf) interesting advice (pdf) on this topic that they presented on the recent QCon London 2010 conference.

As software shops that nurture these software craftsmen, it is important to promote them at these things, and provide a conductive atmosphere for the developers to nurture their skills. I will discuss these things in later in this post (series) under the title "Some Agility and beyond".

Focus on working software over comprehensive documentation

This is again one of the principles of Agile manifesto. But this often creates confusion and debates. Since, many times good documentation is called for a good software. Because, at later stages, a software without documentation, may be hard to understand, and maintain. However, this principle, does not mean to avoid documentation. A good discussion on agile documentation is provided here. After all, documentation is all about communication, so as mentioned before - documentation is just a medium of communication, focus instead on the overall communication.This topic alone requires a lot of discussion, but let me discuss few very important topics in this.

Code is your document

Documenting your code - code is your document. Code commenting, with importance in probably what is the intent of the code rather that what the code is doing,.. any good developer will understand what the lines of code do, if they know the intent of the code. At times, for an intent you may have to write a lot of classes and methods, there put the intent at the index/root of where it all begins, and probably give reference to it. See the JDK, or the Spring API documentation,.. its so beautifully documented.

A test-case is worth a thousand words

Developing test case is also a good documentation, to showcase the functionality. The test-cases can be automated unit test cases, that very much is a very good documentation - technically as well as functionally. Tests should run in all possible ways - regular builds, test builds, CI builds, and even in IDEs. I will cover more on this in my next post in this series.

External Documents

You will have to maintain external documents too. For example, a document that discuses your architecture and its intentions. Focus on developing the software first, and writing documentation for the working software instead of writing for speculative design. Managing external documents, should be among the task list during your iterations. And very important of all, keep in a format that is easy to edit and maintain. You may have a document management software to maintain them or it may be as simple as managing all of them in your enterprise wiki.

More to come ...

In the next blog post in this series, I will try to explain practices that stretch beyond agile prescriptions, that can help build better software. While some of the practices are within the agile prescriptions, few others have also been recommended by Agile experts such as Martin Fowler, while few other are practices that have been proved to work, to produce some admirable software, while having a happy team.

Update...

Agility and more: better & faster business solutions with software - part-1

noreply@blogger.com (Thomas Joseph) — Thu, 25 Mar 2010 14:10:00 +0000

Software development is a creative process, where we create software to deliver solutions to business problems (most of the time), and/or to add value to it. Although the holy duty of a software developer is to deliver solutions to business problems, and not always necessarily developing software. But when we have decided to build a software, we work as teams (most of the times), and the business problem is the central focus of the process. But, the overall quality of the developed software also depends on how we are developing the software - from the requirement capturing to coding, testing, and implementation. This highlights the need for a process at all stages of software development.

Note: Software is also developed for fun, some write it to learn programming (we all have did that), some times it is written for personal use (eg,. automating some mundane task). But most of the time, its written for business needs - let it be a game software, and ERP, the OS itself, or the search engine software - most of them eventually turn as a business or a business solution. In this post I am talking mostly about the software that deals with business and hence the software shops find time and money to be spent to develop these software. So whenever I talk of software, I mean it to be the software developed in these software shops, unless otherwise indicated.

These series of posts highlights my understanding and experience on how the Agile processes can be effectively used to improve the overall quality of the software solutions with reasonable time leads and better estimates. Why and how agile processes needs to be adapted? For Software Engineering teams, there is even more than just being lean/agile, that needs to be into the process.

(Image courtesy: Coffish's web album)

This is of course not a comprehensive discussion on any of the process tools/frameworks discussed here. Although I discuss about Scrum, XP and Kanban to certain extent, what I tell here is the basics, a skeleton that every team/organization can give thoughts to, and then adapt them to their needs. You may add whatever more is needed to meet some standards, but just don't overdo it that you have problem remembering your own processes - keep it as lean as possible. There are several literature that talks about these specific techniques. I will not talk about all the aspects of software development too! Here I will talk in general, about faster development cycle and developer productivity while achieving quality and timely software releases with less team burnouts and overtimes.

Why this talk?

I am into the software development profession for over 4 years now and about 7 years overall into the world of software. I have developed software the ugly way, the good, way and still an ever improving way that gets better each passing day. I was very fortunate to have a wide variety of experience in such a short span. I've worked as a junior programmer to leading teams. I have been a Scrum Master for nearly 2 years. I continuously look out to improve myself and the environment in which I work. I try to implement my learning whenever possible, and its a continuous process where the learning and improvement never stops.

Although, I have practiced focused agile processes for almost 2 years now, I do not consider it as too long to be compared to the seasoned agile experts. Nor do I have done any agile/lean consulting for organizations. But I do talk to peers, friends, friends of friends and seek their experiences and suggestions, share mine with them, and advice them. I read a lot, or get to watch conference videos and slides on agile processes and tools, on developer productivity, apart from the efforts on improving my technical skills. I still look forward to attending user-groups and conferences to discuss out these things. This is why that I feel that I need to share my experiences and learning with others, especially for those who have not yet started on a good agile practice, or are trying to find experiences of others to improve their own processes. After-all, process improvement is more to do with experience - learning from our own and from others, sharing our own and getting valuable feedback, and further improving.

What is a good software?

The Economist magazine (Nov 27, 2004 p. 71) cites the Standish Group's estimates that

"...30% of all software projects are canceled, nearly half come in over budget, 60% are considered failures by the organizations that initiated them, and 9 out of 10 come in late."

With this statement, let us have a quick description of a good software:

Solves the business problem at hand
Gives and adds value to the business and hence the customers
Evolves with customer needs (as fast as the business itself): Maintainable
Free of bugs :)
Could be built and deployed with lesser cost
Available to use on time

With this small description of a good software, we can now focus on making good software, where we need people and processes.

Need for agile

Putting it fast, being agile helps foster better output for software shops. However agile is a too general term to be practiced or explained as such, although the general aims and principles of agile is clearly stated at the agile manifesto. Hence, we will need to follow the process tools that have been designed with the agile principles, such as the Scrum, Kanban and XP.

The Agile Hype
Often people get confused on "what is agile"? The Agile Manifesto is the defining document of "agile." It lists 4 fundamental values and 12 guiding principles. That is the full definition of "agile." Based on those values and principles, we are expected to derive processes and practices that are consistent with the agile approach. However, often the term Agile has been over-hyped, that the people often do not recognize the true meaning of agile. The hype has infused software vendors to come up with tools to support agile, that has often increased this chaos, where people believe that they need those tools to go agile. Also, agile practitioners, often refer to a certain school of practice as Agile. With various schools of agile practice, new comers often get confused about Agile.

Dave Nicolette explains why is it so hard to find a clear definition of "agile"?.

Since the overall quality of the software developed also depends on how we are developing the software, the need for a organized process to produce the software becomes important. Process models such as CMMi evolved over years with this focus - "provide organizations with the essential elements of effective processes that ultimately improve their performance". Over the years people found iterative and Agile methods to be more effective in delivering better output. And in the recent times, there has been a considerable shift to adopting leaner agile models such as Scrum and Kanban over heavier and prescriptive process models such as RUP. Prescriptive means that there are more rules to follow, and adaptive ones have lesser rules to follow. Agile methods are lightweight since they are less prescriptive than the traditional methods of software process. Even within agile, the leaner and less prescriptive models are becoming more popular, naturally, because they are more adaptive to one's organizational needs and seem to be more effective and easy to learn and practice than the prescriptive ones.

Process models: both extremes of scale does not seem to work well.
(Image reproduced from InfoQ minibook)

Even with the need for processes to govern the software development, there are more reasons to go for agile process.

Your customers have continually evolving requirements and changing requirements
Your customers are not very sure about their requirements, and they need your help to identify those requirements
Your team needs to respond quickly to change requests, and deliver projects
You need have a major rework of the project just delivered, because of requirement mis-understanding with customer.
You have teams working overtime continually and at other times sit idle waiting for the other team/person to complete
You have chaos in team due to lack of communication (many times you never feel that your teams have this problem!)
Managers are over burdened, and often need to go in to resolving issues between teams that work on same project
Your estimates prove wrong and your projects go over-budget and/or take longer to complete.

Of course, seasoned agile practitioners can find even more points to be added to the above. But this is what was very obvious to me.

More to come...

In the following posts in this series, I will explain about practicing agile in real life, and why adapting the agile prescriptions is important. I will also explain about practices even beyond the direct agile prescriptions, that can help organizations to build faster and quality software.

Update...

Downloading InfoQ videos and slides

noreply@blogger.com (Thomas Joseph) — Wed, 24 Mar 2010 07:52:00 +0000

InfoQ is a portal with lot of interesting and happening stuff for IT people, and especially for the developers. Some of the areas, it specializes includes - Java, .NET, Ruby, SOA, Agile and Architecture. Like many others, I am a regular user of the site, that provides me the "energy supplements" for my developer life.

Like many others, I was always interested to download the videos and slides from the InfoQ website. With some effort I could device a way to achieve that - exclusively for my private usage. I did this because, often my internet connection is slow and then watching the entire presentation/interview would eat much of my time - most of it waiting for the content to get downloaded.

I wanted to share this technique with others on my blog. However, I was concerned on the IP rights and so checked the website for it. However I could not find one that stated a usage policy. So I decided to check the same with the website owners - c4media. I emailed them and in the reply, I was advised not to publish any details on downloading slides and videos. This was the response I got.

Thanks for taking the time to ask us about this. We’d ask that you not provide instructions on how to download or view these videos beyond the means we currently offer.

When we go out and film these interviews or presentations one of the stipulations in the agreement we sign with the conference or individual is that the video not be made available for download. By allowing or even turning a blind eye to people doing this it would not only put us in potential legal hot water but would also “blacklist” us from filming future presentations or interviews.

Due to this we can unfortunately only support people viewing the presentations or interviews in the formats we currently offer; either streaming them online or, in some cases, downloading the MP3.

It was a co-incidence to find that InfoQ was reacting publicly to seek feedback for their site, probably at the same time when I was drafting the mail. The responses will indicated that people are interested to download the videos and slides, and the InfoQ's response to these queries.

Further, I was encouraged by the InfoQ to publish their response to me, showing how transparent InfoQ was. Respecting the InfoQ's legal terms, I have decided not to publish any technique for downloading the InfoQ videos or slides.

InfoQ may change the way it is presenting the videos and slides, to stick to its legal agreements. However, as far a technical feasibility of devising any hacks are concerned, even RSA encryption can be compromised, for that matter, given enough of time and CPU power.

I am however, happy about the fact that InfoQ is redesigning its website, closely working with the users. It may also present direct method to download interviews, which is InfoQ's "original material". I would encourage you to provide your feedback, and help yourself to make the site more useful.

Tech and some...

noreply@blogger.com (Thomas Joseph) — Wed, 17 Mar 2010 16:31:00 +0000

I had just brought a new headphone for making Skype calls. I was trying testing it out, and I recorded my voice to test it. What came in was a movie song that I was chanting over. I played it back to find too much of background noise. I wanted to remove it. Now for any utility for my OS, I do not rely on a search engine,.. I rely on my "Synaptic Package Manager" on my Linux box - Ubuntu 9.10 (Karmic Koala). Quick was it, to get "Audacity" to help me record, and edit audio.

Notice, that I split the audio into multiple tracks easily and mix it at proper timings.

Believe me. Within half an hour I could learn the basic stuff and the edited voice was so good to hear now. I thought, how about mixing some music as well? Soon I got the Karaoke for it as well. Without much practice or effort, I could record that song with the music tack mixed. Hear it to believe it!

Your browser does not support the audio element.

This is a romantic song "Pathira Pullu" from the Malayalam Movie - "Ee Puzhayum Kadannu"(1996). It is one of my favorites songs. Music composed by Johnson (one of my favorite music composers), and originally sung by KJ Yesudas (obviously my favorite singer).

Those who know me personally can recognize my voice. I have just removed some background noise, added some echo and bass. I am just a bathroom singer, but the voice seems to come out pretty well after the treatment. Even as a fist-time sound recordists, the result was pleasing to me, even with the low quality of the microphone and the always present background noise.

I would encourage people to try this out. Linux is not just for geeks. Linux is for all human beings. I am sure there should be more professional tools available for professional people. You can at-least use the tool to record your voice at home and send to participate in your favorite music reality shows :).

PS: I would have preferred to upload flac/ogg format, but then my friends on non-Linux machines would have no way to hear it.

A tribute to my ex-colleagues

noreply@blogger.com (Thomas Joseph) — Tue, 16 Mar 2010 13:33:00 +0000

Moving with the waves of time, I changed my job. So did some of my other colleagues at Miediant Solutions - my previous employer. Now after we all are into our new jobs, happy and looking forward for a pleasant and prosperous future with our new jobs, we all miss our team, the togetherness we shared. We enjoyed the fun-filled, yet productive work environment, that had so much to learn, and had contributed so much to our professional career. While, many love to hate their bosses, we hope to have him again in our team. We all keep our fingers crossed, and hope for the best to happen, and meet each other and work together again some day.

Thank you all for making the work a pleasurable experience, and the office - a home away from home,
a place we wish to stay together. We shared so much professionally, and
we worked like soldiers in the war.

Thank for all the sprints, the daily scrums and the virtually fighting retrospectives,

the lunch, the outings - thanks for all that time.

Thank you all for the professional and personal growth gained together.

Wish you all a prosperous and happy future!

QCon London 2010 slides

noreply@blogger.com (Thomas Joseph) — Mon, 15 Mar 2010 07:02:00 +0000

The QCon London 2010 has just concluded. There was extremely good response from the delegates. Many attendees have blogged on the same.

The conference had 19 tracks spread over 3 days:

Wednesday

Architectures You've Always Wondered About
Software Craftsmanship
Non-Relational DBs & Web Oriented Data
Dev and Ops: A single team
Functional programming
Solution Track: Wednesday

Thursday

2015 Software Development
Irresponsible Architectures and Unusual Architects
Pragmatic Cloud Computing
Agile Evolution
AlphaGeeks on .NET
IT - more than tools and technology
Solutions Track: Performance and Scalability

Friday

The Concurrency Challenge
Cool Stuff with Java
How do you test that?
SOA 2010
Browser as a Platform
Solution Track: Friday

Some great speakers such as Robert C. Martin, Martin Fowler, Rod Johnson, Joe Armstrong, Dan North teamed up for this.

Some of the slides for these are available online on the official QCon London website. Although, the slides by themselves do not ever match for going there yourself, these and the blogs are good for those who could not attend it.

Saving a streaming video / presentation

noreply@blogger.com (Thomas Joseph) — Mon, 21 Dec 2009 17:09:00 +0000

Ever wondered to to save those streaming videos from Youtube, Metacafe, DailyMotion and so many others? I always wanted to save those streams, so that I could watch them again and again even without an Internet connection.

I am a faithful Firefox user, and one of its wonderful plugin - DownloadHelper has helped me for a long time to capture and save Youtube videos as flash video files on my local hard disk. That was quite helpful. But I always wanted to save those InfoQ videos as well, but unfortunately, DownloadHelper could not provide support for InfoQ (among others).

It was only recently, that I used the conversion feature of DownloadHelper, in which I was converting one of those Youtube videos to be converted to wmv format. I noticed, that the conversion started only after the whole of the video stream was played on my browser. And I could readily guess, that the conversion must be using one of the libraries on my machine. So I looked at the process to find that it was using ffmpeg command to do this job. I also noticed, that actually every video was buffered to the /tmp on my Ubuntu. The filename I could see was something like FlashVOZFnS.

Naturally, if a movie plays on your machine, it must be stored somewhere before being played on your browser. So now if you want to save your next streaming video, just identify where in your system does the flash player saves its movie temporarily before being played on your browser, and save that file to your videos folder, and possibly give it a .flv extension too.

Note:

As soon as you close the tab containing the video, the temporary file is discarded.
Not all videos may be fully buffered before playing. Very long videos and webcasts may not be fully buffered.

Converting flash movies to other format (Linux)

And now if you also want to convert it into other formats, its very easy in Linux with the following commands:
FLV to WMV:

ffmpeg -i /home/thomas/Desktop/video.flv -y -v 0 -acodec wmav1 -b 1000kbps -f asf -vcodec wmv1 /home/thomas/Desktop/video.wmv

FLV to MPG:
ffmpeg -i /home/thomas/dwhelper/Desktop/video.flv -y -v 0 -f asf -vcodec mpeg4 /home/thomas/Desktop/video.mpg

There are so many other options with this command, even up to video capturing your screen, but I am not very expert at those. So I leave it to you to experiment with those options.

Update

For any streaming video site, if you cannot see DownloadHelper to your help, make sure that you should have only this one tab containing any video to download. In other words, if you have multiple tabs with video contents, DownloadHelper seems to get confused, so you may not actually get a download link help.

---------------
Edited and removed some content for legal reasons.
---------------

Configuring log4j with XML in OSGi

noreply@blogger.com (Thomas Joseph) — Tue, 15 Dec 2009 14:43:00 +0000

Sometime back, I had to make perf4j work on OSGi. It required log4j as the logging library and purely demanded log4j.xml (it would not take the .properties file since the configuration must attach downstream appenders). I assumed that it should be simple, but unfortunately it was not. In the rest of this post, I describe how I could get around this, and used log4j.xml to configure log4j logging in OSGi.

Some background on logging for OSGi

Logging was something that was looked as one of the earliest thing for building up our OSGi infrastructure libraries. Since most of the bundles and libraries require logging to be there, we were forced to provide its implementation at the early stages itself. What came as the first viable option was to use Pax Logging. It is still the best logging mechanism for OSGi. It can take configurations in a variety of ways, the most pleasing of all the available configuration models was using the ConfigAdmin service that would leverage the OSGi's dynamism. This is again the best was to configure any service, and not just logging. Even without any explicit configuration Pax Logging would just work fine with its default behavior with a DEBUG for all the bundles in the system.

We chose Pax Logging with ConfigAdmin service. At first the ConfigAdmin provider was chosen as Pax ConfMan. But due to its limitation of not providing runtime updates, I chose to opt for Pax Coin which is still under incubation, but has some good features including runtime property updates and capability to consume configuration data from multiple sources. I was happy to make Pax Coin consume configuration from file system. But I could find that it simply did not accept configurations from the xml file.

So, as of now though providing configuration data via ConfigAdmin seems to be a good choice, I could not find any quick solution to this. So I found a temporary hack for this.

Hack

Note: The following instructions are for working on Pax Runner. If you are not already using it, I would highly recommend it as a tool to start your OSGi framework. It would be the most elegant way to get started with OSGi.

Remove any references to pax logging from your provisioning, so that it doesn't appear in your OSGi platform.
Add commons-logging and log4j jars to the platform as bundles. Add the following URLs in your profile/provisioning file:
```
    wrap:mvn:commons-logging/commons-logging/1.1.1
    wrap:mvn:log4j/log4j/1.2.15
```
Point to the log4j.xml file it OSGi classpath via Pax Runner. Its easy to append --cp=<path_to_dir> as a parameter when starting Pax Runner. The directory path_to_dir will now be in classpath, so you can place our log4j.xml there to be picked up by the log4j system.

Future

This was a temporary hack, but the concern is that I need to find a better way to make ConfigAdmin service consume XML.

| Thomas Joseph's technical blog

Don't be a dick technologist

Universal Applications with Universal JavaScript

JavaScript is going Universal - Why should this matter you?

JavaScript going Universal

Digital Marketing, CMS, Ecommerce and Omnichannel

My Experiences with Universal JavaScript

JavaScript - much beyond just rendering

Sling - Who is closing my JCR Session?

Insane sanity (testing)!

Three great advice

Nice is just ordinary! You need to be Extra-ordinary - Amazing - Remarkable!

The secret sauce of wrapping a jar as OSGi bundle

Before you begin

Choosing your tools

For the Maven users

Setting up the lab

Bundle it up!

Realize what you need - single or family pack?

Things to Remember

1. Know what you are doing

2. Verify the vanilla jars versus your OSGified bundle

3. Perfect on the contract - Import and Export packages

4. Ensure no BND warnings

5. Dealing with cyclic dependencies

6. Adjusting your client bundle

7. Classloader Hell

8. Test thoroughly

9. Be license compliant

For AEM / CQ developers

Before you say - I am done (or not)!

References

Tuning your JCR Queries for the AEM & Jackrabbit OAK

Ground Rules (AEM Specific)

Tools

Guidelines

Identification of Slow Queries

Indexing

Index Management

Finally...

Writing method with dynamic return type without type casting

Installing Java Plugin for Ubuntu / Linux for multiple users

Know your Geolocation right away with HTML5

Earlier and now...

More to read...

Developing Java / OSGi applications on Day CQ

Why the development pain?

Utilize the expertise

Developing bundles for Day CQ

Create a Maven Project for your OSGi bundle

What's next?

Solving eclipse error with SVN 1.6 (loading JavaHL library)

1. Install Java HL

2. Make Eclipse runtime aware of the new JavaHL binaries.

Running tomcat/webserver on port 80/443 without root permissions on Linux

The Problem

The Solution

Steps

Tool power

VNC via SSH tunneling

5 simple steps to connect through VNC to office/work PC via SSH tunneling

Situation:

Concept:

Assumption:

Steps:

Agility and more - better & faster business solutions with software - part-3

Some Agility and beyond

XP practices

Know your team velocity

Clear goals and minimized wait times

Automate to Scale

Reduce complexity of the software

Write less code

Build modular applications

Focus on developer productivity - tools and techniques continuously

Technology, R&D and more

Solving your software development problems

How to Start...?

Final Thoughts...

Update...