Thomas Joseph's blog

Solving eclipse error with SVN 1.6 (loading JavaHL library)

noreply@blogger.com (Thomas Joseph) — Sat, 03 Jul 2010 06:48:00 +0000

If you are using Eclipse and SVN (Subversion), there are high chances that you are using the Eclipse plugin - Subclipse. And if you recently upgraded to subclipse 1.6, there is a good chance that you have encountered the error or loading JavaHL Library, that says something like this:

Problem:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

If this is the case, then few tips to get this working. As always, I will suggest solutions with the least number of steps, and all simple.

When installing/updating subclipse 1.6, use the update site - http://subclipse.tigris.org/update_1.6.x/ from within Eclipse.
The installation screen shows the features. Make sure that all features marked "required" are selected for installation. (You will anyway be not allowed to proceed otherwise).

If it still does not work, it is highly possible that the Java native libraries required for the SVN to work together with Java (thats Java HL) is not available to the Java. Either not installed or just that Java is not able to find it.

1. Install Java HL

You can refer to here for a generic discussion on this topic, especially for Windows and Mac OS. For Linux, I have simpler solutions :).
Install through your package manager, a package named libsvn-java, that contains the JavaHL. You must make sure that it is for SVN 1.6.
For Ubuntu or other Debian based Linux, the command should be handy:

sudo  apt-get install libsvn-java

However, it is possible that the installation is requiring you to install the whole Java binaries itself, although you already have one (an old version, or a manually configured Java for example), and you are not willing to install the Java packages. In that case, just go to your package maintainer's site, and manually download the package, and unzip it to your system (yes, you will need root permissions), so that the dependencies are not installed.

If you had installations fine, without any hacks, restarting Eclipse should get your Subclipse working. If not, follow what is below.

2. Make Eclipse runtime aware of the new JavaHL binaries.

. There are two options for doing this:

Tell JVM/Eclipse about the JavaHL libraries when starting JVM with the system property, that points to the java library path where the JavaHL binaries are installed. eg:
```
   -Djava.library.path=/usr/lib/jni
```
This option can be specified in config.ini as well.

Put the JavaHL binaries in paths that JVM normally searches for native libraries. Normally these paths are listed, in the error above. Above the paths are listed as:

java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

I went for the second option, since that one seemed most non-intrusive and easy to me. Let me show it how I did that:

Choose any of the listed Java library paths. In a terminal, move into that directory:

cd /usr/lib/jvm/java-6-sun/jre/lib/i386

Make softlinks to the JavaHL libraries.

sudo ln -s /usr/lib/jni/libsvnjavahl-1.so libsvnjavahl-1.so
sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0 libsvnjavahl-1.so.0
sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0.0.0 libsvnjavahl-1.so.0.0.0

Now restarting Eclipse should get everything working fine. If it does not, review your steps, and also read in full length about JavaHL from http://subclipse.tigris.org/wiki/JavaHL, and try the steps again.

Running tomcat/webserver on port 80/443 without root permissions on Linux

noreply@blogger.com (Thomas Joseph) — Thu, 24 Jun 2010 18:52:00 +0000

The Problem

One of the concerns for web application developers is to run their web servers on the default port (which is 80 for HTTP, and 443 for HTTPS). However, since these ports are privileged ports (ports less that 1024 are privileged ports in Linux/UNIX), the developers are either required to run the server either as a root, or run on alternative non privileged ports such as 8080 or 8443.

The Solution

While to the same problem, there may be different solutions, the most simplest of all that I found was to do local port forwarding. Again here, many do not like to fiddle with commands every time. An easier, and one time solution to this is to use a program called "guidedog".

Guidedog is a GUI based advanced and powerful networking configuration tool for Linux, that can be used for packet routing, NAT and port-forwarding. Although it is for KDE, your Linux package manager should be intelligent enough to make that work for you even if you do not use KDE. We will here use its port-forwarding feature.

Steps

Launch your favorite package manager tool (Synaptic/Aptitude/YUM etc) and install "guidedog". For Ubuntu I use Synaptics Package manager, or even a command would do.
```
sudo apt-get install guidedog
```
Once installed, launch the application with root permissions (don't worry, you require it to do just once).
```
sudo guidedog
```
On Ubuntu, you can also find it under the menu Applications->Other. You will have to even edit the menu command to gksu guidedog.
Move to the "Forward" tab. Add a new rule, such that TCP traffic on port 80 is forwarded to port 8080 (or whatever port your server runs on). Similarly, add another rule, such that TCP traffic on port 443 is forwarded to port 8443 (or whatever port your server runs on for HTTPS/SSL connections). After you are done, your screen should look like this:
Apply the changes.

You are done. Now just hit on localhost instead of localhost:8080, next time you start your applications.

Tool power

The tool creates a system rule to forward the traffic to a different port(8080/8443) from its original destination (port 80/443) on the same machine. You can see in the tool that it is much more powerful, and can allow even port forwarding for other machines too, or even do IP Masquadering/NAT (Network Address Transilation). However, explaining on these topics would outgrow the topic of this blog and would take more of your and my time too.

VNC via SSH tunneling

noreply@blogger.com (Thomas Joseph) — Mon, 24 May 2010 17:04:00 +0000

5 simple steps to connect through VNC to office/work PC via SSH tunneling

Situation:

You have to connect to your office PC from home via VNC, but your office PC does not have a public IP. The only way to access your PC is through one of the access point machines provided by your office that can be reached through Internet.

Concept:

VNC port is 5900. You can ssh tunnel to your office PC through your office access point, and connect to this tunnel to have VNC connection.

Assumption:

Office access point machine: accesspts1.company.com
Internal IP of your office machine: 192.168.1.31

Also assume that your Office PC is switched on, and access point machine can access your office PC via the internal IP, and can access its ports as well.

Steps:

SSH to access point machine. Provide password when prompted
```
ssh username@accesspts1.company.com
```
Now that you are on access point machine, ssh from there to your office PC. Again provide password when prompted.
```
ssh username@192.168.1.31
```
On your office pc, start VNC server. Keep this terminal open.
```
x11vnc
```
On another terminal, make open a tunnel to your office pc via the access point. Provide the password for your access point machine when prompted.
```
ssh -L 0.0.0.0:xxxx:192.168.1.31:5900 username@accesspts1.company.com
```
Use an VNC viewer to connect to your Office PC via the tunnel. Use the following details:

Host: localhost
Port: xxxx

Agility and more - better & faster business solutions with software - part-3

noreply@blogger.com (Thomas Joseph) — Tue, 30 Mar 2010 15:15:00 +0000

In my previous posts, I discussed about practicing agile in real life, and why adapting the agile prescriptions is important. I discussed how the basic values of Agile manifesto narrates it all. This is my last post in this series, where I will explain about practices even beyond the direct agile prescriptions, that can help build faster and quality software with smiling teams.

Some Agility and beyond

While the software creation itself is creative, the process employed to create this software is also creative. Its not just practicing agile, that helps create good software, there are more things to it. The craftsman (developer) who creates software has also to be given importance (Did I mention that before?). Let me discuss few of these things that include things from the Agile practices such as XP and Scrum, as well as those that go beyond them.

XP practices

Focus on producing better software with XP practices such as continuous integration, with focus on unit tests, integration tests, and automated functional/UAT tests. I do not want to go to the depth of CI and unit tests, but it helps you be confident on regressions, as the software evolves. There are few other inherent advices that Martin Fowler has with the CI such as "Keep the Build Fast", "Test in a Clone of the Production Environment", "Make it Easy for Anyone to Get the Latest Executable" etc. I would highly recommend reading his article.

Continuously try to clear up the technical debts. Did a Quick Fix? Create a ticket that says,.. fix it! Removing old legacy and unused code - it helps developers avoid straying away to look at an unused code, to fix problems, and faster induction of new joiners to team. Old code also makes large codebase, maintenance problems, larger build time and lots of confusion.

Equally important is a focus on continuous refactoring for improved code readability reduced complexity and improved the maintainability and extensibility, for one or more of its benefits. This also helps clear our technical debts.

Closely related to refactoring is the CI, that would help us to be sure that code refactoring is going fine. And to be confident in that a lots of test-cases needs to be in, that would ensure that there are no regressions. So develop a habit of writing unit test case for every bug or feature request. From my experience, going fully TDD (Test Driven Development) sounds like crazy as we develop all the tests-cases upfront, while having no test case or very scanty is also not a good practice. My experience is good to stay somewhere between - say at 50-85%, if we assume TDD to be 100% for writing test cases (use your mind and adapt!).

The rule is simple. Start out lean, and consult Scrum and XP for places they are expert in. You need not take all their advice, but taking an advice is always good. Have something very technical - probably XP is good, other places Scrum is good, some other places - you get advice from both.

Know your team velocity

It is very important to know your teams velocity. What is the team capacity to crunch out the story points (or whatever measure you keep) given to them? If the team velocity is not known, it release dates predicted may not be credible. There are various ways to measure your team velocity. For example, you could measure your team velocity by recording the story points finished for a period of time, or measure the amount of original estimated hours that the team could finish for a period of time.

Talking of this, it is important to remember that the efficiency (sometimes called the Focus Factor) of the team is never 100%, since teams lose time doing unplanned items, doing context switches, helping other teams, checking their email, fixing network snag etc. With my experience, even when we had 9-6 working day, we normally counted it as 8 hours working day assuming the 1 hour lunch time as a normal organizational practice. However, when doing plannings, we used to calculate only 7 hours for a day, and hence allot only 6.5 - 7 hours of approximate work per person per day. That would be around 75-80% efficiency. However, you may like to bill your client w.r.t to all the time spent i.e 8 hours per day per person, which is perfectly fine! :)

It may sound ridiculous to count 60% or 70% efficiency, but on real analysis of your team, it is normal to find such statistics. However, such a statistics is helpful to make realistic expectation from your team, and commit to reliable release dates.

Cumulative Flow diagram depicts the team velocity and the lead time required to finish items - helps to plan and minimise queues/waiting times.

(Image reproduced from the InfoQ minibook)

Most organizations want to reduce lead time, and do get things done faster. Unfortunately many try to do this by getting more people in or working overtime. Usually the most effective way to get stuff done faster is to smooth out the flow and limit work to capacity, not add more people or work harder. This type of diagram shows why, and thereby increases the likelihood that the team and management will collaborate effectively.

The velocity does not vary much, however, you will have to regularly measure your team velocity, since velocity may increase with experience and expertise. However, this velocity may not change drastically over successive iterations, except for may be the first few iterations, when you are still stabilising over the agile practices, or when the team is new to a project. Release planning with these statistics can would help achieve timely and reliable releases and yet prevent team burnouts.

Clear goals and minimized wait times

While focusing on individual productivity is important, it is also important to understand that they maintain their productivity when working as a team. One important factor that may hamper individual productivity when working as a team may be lack of clear goals on what to do next. Having a clear-cut road-map for not just for the day, for for the coming week itself may be good.

Minimising wait times increases team throughput

(Image Courtesy: Vype's web album)

Another factor to be considered is to reduce dependence on each other. Plan out the iteration road-map so that there is minimal dependence of one on the other. See that a team-member is not waiting for the other to finish. There may be dependence, but then, the timing can be adjusted for it. In manufacturing, a good importance is given to reduce queues of intermediate products to reduce costs and increase throughput. For software development, there is no silver bullet for this, however providing an execution plan upfront can help in this, since if developer knows what needs to be done for tomorrow or next week and something that is independent of the current work, they can go ahead and start doing that task, so that individual productivity remains high. Well planned iteration backlog, daily stand-ups and scrum principles of removing impediments can help a lot to this.

Automate to Scale

Try to bring automation to places wherever possible, since doing things manually doesn't scale, but machines can scale. That's how Google has achieved its excellence - by relying on algorithms and machines to scale rather than depending on humans. Automation can be brought at many places - CI, intelligent builds for every commits, sending error reports of CI and builds, and other automated testing, smoke testing, functional testing, and even deployments. You will definitely avoid team burnouts!

Reduce complexity of the software

Reducing complexity in your software would help a long way in more than one ways to create good software. Well known agile and modularity guru Kirk Knoernschild explained in his keynote speech during the recent Jax London conference about reducing complexity in the software. But there is no silver bullet to this, though modularising application is a definite direction to this.

Kirk further explains the cost of complexity with the following statistics:

The number of lines of code doubles every 7 years
50% of development time is spent understanding code
90% of software cost is maintenance and evolution

Look at the #2! So if complexity is reduced, time spent in understanding code is reduced, thereby speeding up the overall development process. It is natural for the systems to grow complex over time, and as system evolves, unless efforts are put in to reduce this complexity.

"As a system evolves, its complexity increases unless work is done to maintain or reduce it. "

(Image Courtesy: Flavio's web album)

There is more to reducing complexity in the code:

Write less code
Build modular applications

Write less code

Avoid writing code whenever possible. The mantra is "less is more". Avoid writing code that is not your main line of business. Many teams spend lot of time and energy writing huge chunks of code and configuration for creating "platforms". Platforms are necessary, but try to evolve it from existing and stable code libraries and frameworks as something that has the least glue code, so that your framework can evolve over time. For example, you need to have great amount of integration points in your application, before jumping ahead to write message brokers, message bus, and all those fancy ESB kind of features, consider one of those ESB's and message brokers available out in the open source or its commercial variant. Will the original author be available all through the project life-cycle to explain the code and intent? Who will maintain it? Will the subsequent maintainers fully understand the code and its intent? When we use popular libraries and frameworks, there is more chance of finding developers who already have knowledge in it.

When developing internal frameworks, its imperative that we design for even what requirements may come after 10 years, but the fact is that even after just 4-6 years, you will be hardly using the same system. Technology may have changed, your requirements may have widely changed and you end to design another framework to tackle those, or may have added a bunch of code to your existing system, and many of the capacities of your old framework goes unused. Still if you feel compelled to write some good amount of code for your framework, consider to market it separately or may be consider still better open sourcing it, so that you can early find co-developers, and build a community around that. Of course you can always commercially sell a software that you have open-sourced.

Build modular applications

While writing modular applications is like a very common vocabulary that we use and have this intent when we design our applications, the reality is that achieving "true" modularity is difficult. Writing component based applications may provide some answer. For Java applications considering OSGi may be a good choice, that has been upheld by all the major application server vendors including IBM, Oracle and Sun. Recent JVM modularity to be introduced with Java 7 with Jigsaw, and the Oracle's more recent modularity efforts with Qwylt are also interesting move towards this.

Focus on developer productivity - tools and techniques continuously

To have a faster development and lesser team burnouts, one very important factor is to focus on how effective is the software craftsman in performing the very act of producing software. It is important to look out to improve developer productivity at every act of the software development - project layout, development mode and development cycle time, running tests, IDE integration, build tool effectiveness, conventions over configuration, code and architecture complexity.

Unit tests and integration tests should be easy to run in all possible ways - with regular builds, test builds, CI builds, and even in IDEs. This helps to frequetnly run the tests withing IDE, and be confident on the code from the very beginning of writing code. This helps a lot especially when refactoring code. Running tests should be equally easy and simple from the command line as well.
Focus on conventions over configuration principles on all possible aspect of software development (including build - eg Maven). Although choice of build tool is not a religion, Maven is for one thing a great productivity improving, since once you learn Maven, you know it even when you change your projects. Adopting such practices also brings down the learning curve, and even new joiners can get productive quickly. Applying this principle is not just for project layout, but the overall architecture and day-to-day coding as well.
Watch out and remove impediments required for a fast build-run cycles. Does the build require punching in a lot of key strokes? Also watch out with continual performance tuning. Does running of you app, almost freezes a 3GB memory system? - High time that you lookout for memory leaks, and do application startup tweaks. At production you may be countering this with a massive server configuration, but think if trimming the code would actually save you on team's energy as well as on your server infrastructure money as well! The builds should be easiliy launched from within and outside IDE. If you have multiple IDEs, make sure that all the IDEs work just fine (Maven plugins are available for almost all major IDEs, that can help you do this).
Strive to find tools that would make things easy and remove/ automate / lessen hassles and repetitious tasks. Tools need not be always visible, eg.. scrum and kanban are tools for process improvement, while Spring ROO is also a tool that explicitly focuses on developer productivity. For instance promote JQuery or similar JavaScript library instead of fully hand crafted JavaScripts.
Carefully choose and use issue tracking systems, and wiki software for your requirements:

Choosing issue tracking systems with integration to your SCM. This would be good to track the commits for a particular issue, and ease code reviews.
Use popular solutions:

There is more probability of finding experienced users- less learning curve.
You don't have to maintain that s/w which is not your mainline business.

Consider Distributed SCM soultions like git or mercurial (Remember that this is just a tool, but it helps manage changes rather than versions, and so that branching and merging is simple.).

Have easy read code-bases, easy run executables and easy obtainable project artifacts. Martin Fowler nicely describes this in his article, where I would like to highlight a few items:

Have everything you need to do a build should be in there including: test scripts, properties files, database schema, install scripts, and third party libraries. Note: Maven eases this out, since you have a separate system (a maven repository) to maintain third party library, and you would just put the dependencies in your POM. The basic rule of thumb is that you should be able to walk up to the project with a virgin machine, do a checkout, and be able to fully build the system. Putting in IDE configuration file is also a good practice. However, do not store the build products itself in the SCM., this rather indicates a flaky build technique that cannot reliably recreate builds.
Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)
Make it Easy for Anyone to Get the Latest Executable - To help make this work, anyone involved with a software project should be able to get the latest executable and be able to run it: for demonstrations, exploratory testing, or just to see what changed this week. Doing this is pretty straightforward: make sure there's a well known place where people can find the latest executable. An internal Maven repository for example is a great place.

This discussion is not a comprehensive one describing each and every bottlenecks for faster development and increased productivity, but covers a lot of ground on it. You may share your own experiences and discussion points using the comments section in this blog.

Technology, R&D and more

Encourage R&D within your organization.
Have a dedicated R&D team if possible. Swap people in and out of the R&D team. Encourage people in all teams to spend 20% time for doing something creative, or may be blog to start with. Consider putting people to R&D as an alternative to benching, if you have such a practice. Encourage the R&D team to publish their findings regularly through blogs and research papers. I will blog separately about the need for blogging. You can identify your gems working this way.

The 80:20 principle at Google is there to bring out wild ideas from developer minds to be experimented. Some of the great projects from Google including GMail, Google News etc have been the result of such researching and experimenting mentality of Googlers.

Technology is still important!
As emphasized in the DDD principles - give good focus to the business at hand, but at the same time don't neglect technology. New technologies emerge to solve challenges that the existing ones cannot solve or because they are not efficient in doing that. This underlines the importance of using newer technology to solve our technical problems. Keep a balance however. It would not be recommend to use bleeding edge technology for production use, but encourage R&D learning to use the new technology 'wisely' whenever needed and possible.

Community learning
Encourage your team to get involved with some good and popular open source project. This could also be a project something that you use yourself, or are planning to use in the near future. It has many benefits - when you officially can take part in OSS activities, you are getting better visibility to your company, apart from the learning that your team and company as a whole gains in the process. There are many ways in which the team can participate - participating in forums and discussions, filing bug reports (with test cases preferred), feature requests, subscribing to developer mailing lists if available, and if possible - commit code, help with release etc. The team also gets better knowledge of good standard practices for software development - issue/ticket management, coding, testing, continuous integration, release, deployment etc.

Community learning helps in many ways

(Image Courtesy: Horace Mann School's web album)

Continuous improvement: Always look out for improvement - talk to peers, get ideas from them (that's why retrospectives are there). Encourage team to spot improvements or pain points in your projects and processes and seek solutions from them. Your team will be forced to think creatively, and you may in-fact get something useful! Have an open minded discussions before incorporating such changes (or rejecting suggestions) to the project.

Solving your software development problems

Agile is about iterations and incremental software development. This helps to build software that evolves over time with the customer requirements. Some prescriptions of agile practices such as from XP about CI, unit testing, helps to keep you confident in what you develop for each increment.

That apart, looking at the discussions so far, and in the links there-in, we can deduce that its not just agility, there is more that helps you build better, and quality software. Agile practices with focus on developer productivity helps you release your software in a timely manner, yet maintaining the quality and budgets, while keeping your team happy and in high spirits.

How to Start...?

A difficult part to start anything new, is to get started. Very true for agile too!. Read a lot on agile, Scrum, Kanban and XP. Start with basic minimals on being agile

Start stand-ups - focusing on highly communicated and focused team
Start iterations - smaller the better
Incorporate to retrospect your processes, code and techniques
Never feel complete with improving your process - it should be a continuous process.

Good start would be to start lean, and then add prescriptions that work for you! And,... don't forget to adapt!

Final Thoughts...

This discussion, or few more blogs cannot cover everything related to software development. So much of literature is there around. Get inspired from those. Go ahead and start somewhere. You cannot start a perfect agile anyway! Over the time, you will improve with every iterations.

Update...

Agility and more: better & faster business solutions with software - part-2

noreply@blogger.com (Thomas Joseph) — Mon, 29 Mar 2010 14:30:00 +0000

In my previous blog post, I talked about why software development needs processes, and the need for agile in the processes. Here, I continue with the discussions, focusing on how the basic principles of agile itself has a lot to say, that can help us build better software.

Practicing agile - adapting is more important

As mentioned before, the popular practiced tools for agile process include Scrum, Kanban and XP. But often, the process models/tools are applied as a religion, being strict on what the text prescribes. This has resulted in some rumblings that has questioned the effectiveness of agile processes to deliver quality and creative solutions to business. The InfoQ book on Scrum and Kanban has highlighted the need for the process models to be adapted to their needs.

One size doesn't fit all - adapting is important.

(Image Courtesy: Robin Skyler Tell's photostream)

So what needs to be practiced is taking a lean model, and adapting it to your needs, probably by adding "prescriptions" from the stouter process models, and even removing "prescriptions" that hinder your progress, but with absolute knowledge of what is being removed. This is what Henrik has put in his InfoQ book - "Do not limit yourself to one tool!".

I have my own experience, where we started out with pure Scrum. Scrum iterations help evolve not only the software but our (scrum) process itself with continuous retrospectives. As we practiced it, over the time we hit against the walls of the Scrum, such that we had to eventually remove few prescriptions of the Scrum, effectively making it leaner and the process as "Scrum Inspired". For example, we would feel OK to extend our sprint for 2-3 days and still declare it as a successful sprint! We would find it OK to even entertain new items during the sprint, if it was critical to the client. Sometimes, 1 or 2 team member may work on items from another product, that is planned in advance for the sprint, because we had small teams, and the same team members would work on multiple products/projects. In spite of this, we always knew where the limits were, and we were effective with whatever we did. We were strict on whatever process we followed. If we changed the Scrum rule, we did that knowingly, and probably would be the outcome of the retrospectives we had at the end of sprints. By adapting the process, not only was our output improved (that would make customers happy), the management as well as the team members were happy at what we do, and how we do it and the overall team productivity.

We all were extremely happy on the retrospective day, since it was the day we could blast out our rants, impediments and criticisms openly. We virtually had fights over the retrospectives. But finally, it all would help us improve what we do.

Mix-n-match and adapting is important. For instance, Scrum does not say anything about releases, but XP does, hence combining them together is more compelling. The "Do it" nature of scrum and the "pair programming" and "move people around" principle of XP and a well communicated team also helps in getting a new joiner to be inducted fast into the team as well!. Most of us have done pieces of XP and agile knowingly or unknowingly, if for example we have written unit tests, did iterations of development and the like. But the real benefit for going agile comes from developing and following process models with prescriptions from the agile processes available. Process models such as Kanban actually give us metrics to measure our process against, and take the best of what works for us from all the process models - agile or not. We can choose to start out with Kanban, with probably the regular iteration, daily stand-up meetings, and retrospectives from the Scrum, add to that the software engineering specific practices from XP, such as CI, unit testing and pair programming.

Again, what is more important is to adapt the process models to your needs. For example, even as I mentioned above for practicing prescriptions from Scrum and XP, I am still wondering how a stand-up meeting would happen for a distributed team? Distributed Scrum? Probably, we can have scrum of scrum, but again, this is something I have no experience. So, when the need comes for it, I will give it a try, practiced, adapted and practiced and improved so on.

Responding to change over following a plan

This is one of the four core values of the Agile manifesto. Adapting your process is a definite way to achieve this objective. Agility promotes responding to change inherently because of its iterative nature, that can enable the team to incorporate changes to a software more rapidly that traditional classic waterfall model. Process framework such as Kanban further fosters this changes to be percolated more rapidly into the development.

Try experimenting. Experiment with new and innovative ideas for improving processes. Invite all sort of crazy ideas from your team as well. I am sure, you will not go worse even if your experiments failed. You will learn so much. Next time you always know what not to do :).

Focus on individuals and interactions over processes and tools

This is again one of the four core values of the Agile Manifesto. This is what we have covered already. Let me explain. By focusing more on adapting the process to our needs instead of following the process as a religion, we are focusing more on the people and interactions for sure. The stand-up and the retrospectives for example are effective to give importance to the people who are involved, removing their impediments, and giving importance to their interactions. Once we practice scrum and XP, there is a value that the team learns. It is more important to follow these values, than to follow these practices by book.

Communications and Interactions

As we talk of individuals and interactions, I should emphasize on the need for effective communication and interaction within the team and with the customer (or his representative). Well communicated teams is one of the focus of scrum and XP (and most of the agile practices). The team stake holder should ensure proper communication channels. Small and well knit teams help a lot on this. If a team member has a problem, he knows already to whom this should be addressed, and the turnaround time to address this should be maximum of two meetings - i.e, the impediment should be addressed before the second team meeting after the impediment had been raised.

(Image courtesy: Avish's web album)

To enhance our communication channel, we used enterprise blogging at three levels:

Organizational blog: We used this blog to express new ideas and rants across teams.
Team blog: we used this blog within the team for various purpose

Daily status update - every team member used to put the daily status update for days work under the following headings:

Tasks completed
Tasks pending
Impediments
Things I found new today
Comments

Project specific posts - To discuss and document a project specific problems, and ideas. Many times although, we would promote some of the posts to the organizational blog too.

One-to-one interactions are really good at fostering better communication and interactions. It will be extremely good to have your team work at the same location, if possible within same cubicle. ThoughtWorks practices this with "Dining Table" culture for the team that fosters greater collaboration and communication (and flow of ideas). For distributed teams, this may not be feasible, where better interaction can be provided by having the team members available to each other on IMs. Use of wiki based whiteboards as discussed above can be a good way to go for distributed teams. Well, there can be a lot of talk for this. So I reserve the 'distributed teams' topic to a separate discussion.

Human factor

While focusing on individual developer productivity is important, it is also important to see how the team as a whole can mesh well and be productive. A very important thing is to have well-knit and well communicated team. Focus that they can work well with each other considering the human emotions and social factors.

Frequently changing the context is not a good idea. Everybody may not be equally good at changing contexts. It would be nice to group together similar tasks. Human minds takes time to get adjusted to the new environment and get productive. Again, this is a topic that may itself need a separate discussion.

Tools do not replace process and individual interactions

When we started out with Scrum, we used whiteboard for everything - from daily tracking our burndowns at daily scrums to the retrospectives. But later to keep track of what we do, we moved parts of these into our enterprise wiki. Later we also tried with JIRA's GreenHopper plugin to manage most of the scrum, except the retrospectives. We where still open to change or adapt to tools as would our process be adapted. In short, we used the tools to aid us what do.

Using tools does not mean that we are doing what the tool is meant for. What we do for scrum is more important than the tool used to manage scrum. Using the Agile PLM does not mean that we have gone agile. Tools should be at our disposal to aid us what we do more effectively.

Focus on the developer skills

A craftsman is the center-point of every creation - not just software. From the craftsman of a building to that of a supersonic jet aircraft - the skill of the craftsman involved is very important and crucial to it. Software craftsman is no exception.

The developer should realize the importance of developing skills even beyond his current needs. Developers should proactively try to learn, innovate and come with new ideas. Always try to learn something new at work, and from around your (professional) community. Work proactively to improve the environment you work in. Work towards a more productive process, for getting better output for your company and yourself. No one can teach you or or force you to learn. It has to come from within. Try reading books, blogs and articles, try to keep up with what is happening around. Find how a new technology could help you create better software. Find how a new technique or trick could help you do more with less effort. Attend conferences, and user groups, speak at them, share your learning - good and bad, blog about them. Peter Goodliffe and Dan North have some (pdf) interesting advice (pdf) on this topic that they presented on the recent QCon London 2010 conference.

As software shops that nurture these software craftsmen, it is important to promote them at these things, and provide a conductive atmosphere for the developers to nurture their skills. I will discuss these things in later in this post (series) under the title "Some Agility and beyond".

Focus on working software over comprehensive documentation

This is again one of the principles of Agile manifesto. But this often creates confusion and debates. Since, many times good documentation is called for a good software. Because, at later stages, a software without documentation, may be hard to understand, and maintain. However, this principle, does not mean to avoid documentation. A good discussion on agile documentation is provided here. After all, documentation is all about communication, so as mentioned before - documentation is just a medium of communication, focus instead on the overall communication.This topic alone requires a lot of discussion, but let me discuss few very important topics in this.

Code is your document

Documenting your code - code is your document. Code commenting, with importance in probably what is the intent of the code rather that what the code is doing,.. any good developer will understand what the lines of code do, if they know the intent of the code. At times, for an intent you may have to write a lot of classes and methods, there put the intent at the index/root of where it all begins, and probably give reference to it. See the JDK, or the Spring API documentation,.. its so beautifully documented.

A test-case is worth a thousand words

Developing test case is also a good documentation, to showcase the functionality. The test-cases can be automated unit test cases, that very much is a very good documentation - technically as well as functionally. Tests should run in all possible ways - regular builds, test builds, CI builds, and even in IDEs. I will cover more on this in my next post in this series.

External Documents

You will have to maintain external documents too. For example, a document that discuses your architecture and its intentions. Focus on developing the software first, and writing documentation for the working software instead of writing for speculative design. Managing external documents, should be among the task list during your iterations. And very important of all, keep in a format that is easy to edit and maintain. You may have a document management software to maintain them or it may be as simple as managing all of them in your enterprise wiki.

More to come ...

In the next blog post in this series, I will try to explain practices that stretch beyond agile prescriptions, that can help build better software. While some of the practices are within the agile prescriptions, few others have also been recommended by Agile experts such as Martin Fowler, while few other are practices that have been proved to work, to produce some admirable software, while having a happy team.

Update...

Agility and more: better & faster business solutions with software - part-1

noreply@blogger.com (Thomas Joseph) — Thu, 25 Mar 2010 14:10:00 +0000

Software development is a creative process, where we create software to deliver solutions to business problems (most of the time), and/or to add value to it. Although the holy duty of a software developer is to deliver solutions to business problems, and not always necessarily developing software. But when we have decided to build a software, we work as teams (most of the times), and the business problem is the central focus of the process. But, the overall quality of the developed software also depends on how we are developing the software - from the requirement capturing to coding, testing, and implementation. This highlights the need for a process at all stages of software development.

Note: Software is also developed for fun, some write it to learn programming (we all have did that), some times it is written for personal use (eg,. automating some mundane task). But most of the time, its written for business needs - let it be a game software, and ERP, the OS itself, or the search engine software - most of them eventually turn as a business or a business solution. In this post I am talking mostly about the software that deals with business and hence the software shops find time and money to be spent to develop these software. So whenever I talk of software, I mean it to be the software developed in these software shops, unless otherwise indicated.

These series of posts highlights my understanding and experience on how the Agile processes can be effectively used to improve the overall quality of the software solutions with reasonable time leads and better estimates. Why and how agile processes needs to be adapted? For Software Engineering teams, there is even more than just being lean/agile, that needs to be into the process.

(Image courtesy: Coffish's web album)

This is of course not a comprehensive discussion on any of the process tools/frameworks discussed here. Although I discuss about Scrum, XP and Kanban to certain extent, what I tell here is the basics, a skeleton that every team/organization can give thoughts to, and then adapt them to their needs. You may add whatever more is needed to meet some standards, but just don't overdo it that you have problem remembering your own processes - keep it as lean as possible. There are several literature that talks about these specific techniques. I will not talk about all the aspects of software development too! Here I will talk in general, about faster development cycle and developer productivity while achieving quality and timely software releases with less team burnouts and overtimes.

Why this talk?

I am into the software development profession for over 4 years now and about 7 years overall into the world of software. I have developed software the ugly way, the good, way and still an ever improving way that gets better each passing day. I was very fortunate to have a wide variety of experience in such a short span. I've worked as a junior programmer to leading teams. I have been a Scrum Master for nearly 2 years. I continuously look out to improve myself and the environment in which I work. I try to implement my learning whenever possible, and its a continuous process where the learning and improvement never stops.

Although, I have practiced focused agile processes for almost 2 years now, I do not consider it as too long to be compared to the seasoned agile experts. Nor do I have done any agile/lean consulting for organizations. But I do talk to peers, friends, friends of friends and seek their experiences and suggestions, share mine with them, and advice them. I read a lot, or get to watch conference videos and slides on agile processes and tools, on developer productivity, apart from the efforts on improving my technical skills. I still look forward to attending user-groups and conferences to discuss out these things. This is why that I feel that I need to share my experiences and learning with others, especially for those who have not yet started on a good agile practice, or are trying to find experiences of others to improve their own processes. After-all, process improvement is more to do with experience - learning from our own and from others, sharing our own and getting valuable feedback, and further improving.

What is a good software?

The Economist magazine (Nov 27, 2004 p. 71) cites the Standish Group's estimates that

"...30% of all software projects are canceled, nearly half come in over budget, 60% are considered failures by the organizations that initiated them, and 9 out of 10 come in late."

With this statement, let us have a quick description of a good software:

Solves the business problem at hand
Gives and adds value to the business and hence the customers
Evolves with customer needs (as fast as the business itself): Maintainable
Free of bugs :)
Could be built and deployed with lesser cost
Available to use on time

With this small description of a good software, we can now focus on making good software, where we need people and processes.

Need for agile

Putting it fast, being agile helps foster better output for software shops. However agile is a too general term to be practiced or explained as such, although the general aims and principles of agile is clearly stated at the agile manifesto. Hence, we will need to follow the process tools that have been designed with the agile principles, such as the Scrum, Kanban and XP.

The Agile Hype
Often people get confused on "what is agile"? The Agile Manifesto is the defining document of "agile." It lists 4 fundamental values and 12 guiding principles. That is the full definition of "agile." Based on those values and principles, we are expected to derive processes and practices that are consistent with the agile approach. However, often the term Agile has been over-hyped, that the people often do not recognize the true meaning of agile. The hype has infused software vendors to come up with tools to support agile, that has often increased this chaos, where people believe that they need those tools to go agile. Also, agile practitioners, often refer to a certain school of practice as Agile. With various schools of agile practice, new comers often get confused about Agile.

Dave Nicolette explains why is it so hard to find a clear definition of "agile"?.

Since the overall quality of the software developed also depends on how we are developing the software, the need for a organized process to produce the software becomes important. Process models such as CMMi evolved over years with this focus - "provide organizations with the essential elements of effective processes that ultimately improve their performance". Over the years people found iterative and Agile methods to be more effective in delivering better output. And in the recent times, there has been a considerable shift to adopting leaner agile models such as Scrum and Kanban over heavier and prescriptive process models such as RUP. Prescriptive means that there are more rules to follow, and adaptive ones have lesser rules to follow. Agile methods are lightweight since they are less prescriptive than the traditional methods of software process. Even within agile, the leaner and less prescriptive models are becoming more popular, naturally, because they are more adaptive to one's organizational needs and seem to be more effective and easy to learn and practice than the prescriptive ones.

Process models: both extremes of scale does not seem to work well.
(Image reproduced from InfoQ minibook)

Even with the need for processes to govern the software development, there are more reasons to go for agile process.

Your customers have continually evolving requirements and changing requirements
Your customers are not very sure about their requirements, and they need your help to identify those requirements
Your team needs to respond quickly to change requests, and deliver projects
You need have a major rework of the project just delivered, because of requirement mis-understanding with customer.
You have teams working overtime continually and at other times sit idle waiting for the other team/person to complete
You have chaos in team due to lack of communication (many times you never feel that your teams have this problem!)
Managers are over burdened, and often need to go in to resolving issues between teams that work on same project
Your estimates prove wrong and your projects go over-budget and/or take longer to complete.

Of course, seasoned agile practitioners can find even more points to be added to the above. But this is what was very obvious to me.

More to come...

In the following posts in this series, I will explain about practicing agile in real life, and why adapting the agile prescriptions is important. I will also explain about practices even beyond the direct agile prescriptions, that can help organizations to build faster and quality software.

Update...

Downloading InfoQ videos and slides

noreply@blogger.com (Thomas Joseph) — Wed, 24 Mar 2010 07:52:00 +0000

InfoQ is a portal with lot of interesting and happening stuff for IT people, and especially for the developers. Some of the areas, it specializes includes - Java, .NET, Ruby, SOA, Agile and Architecture. Like many others, I am a regular user of the site, that provides me the "energy supplements" for my developer life.

Like many others, I was always interested to download the videos and slides from the InfoQ website. With some effort I could device a way to achieve that - exclusively for my private usage. I did this because, often my internet connection is slow and then watching the entire presentation/interview would eat much of my time - most of it waiting for the content to get downloaded.

I wanted to share this technique with others on my blog. However, I was concerned on the IP rights and so checked the website for it. However I could not find one that stated a usage policy. So I decided to check the same with the website owners - c4media. I emailed them and in the reply, I was advised not to publish any details on downloading slides and videos. This was the response I got.

Thanks for taking the time to ask us about this. We’d ask that you not provide instructions on how to download or view these videos beyond the means we currently offer.

When we go out and film these interviews or presentations one of the stipulations in the agreement we sign with the conference or individual is that the video not be made available for download. By allowing or even turning a blind eye to people doing this it would not only put us in potential legal hot water but would also “blacklist” us from filming future presentations or interviews.

Due to this we can unfortunately only support people viewing the presentations or interviews in the formats we currently offer; either streaming them online or, in some cases, downloading the MP3.

It was a co-incidence to find that InfoQ was reacting publicly to seek feedback for their site, probably at the same time when I was drafting the mail. The responses will indicated that people are interested to download the videos and slides, and the InfoQ's response to these queries.

Further, I was encouraged by the InfoQ to publish their response to me, showing how transparent InfoQ was. Respecting the InfoQ's legal terms, I have decided not to publish any technique for downloading the InfoQ videos or slides.

InfoQ may change the way it is presenting the videos and slides, to stick to its legal agreements. However, as far a technical feasibility of devising any hacks are concerned, even RSA encryption can be compromised, for that matter, given enough of time and CPU power.

I am however, happy about the fact that InfoQ is redesigning its website, closely working with the users. It may also present direct method to download interviews, which is InfoQ's "original material". I would encourage you to provide your feedback, and help yourself to make the site more useful.

Tech and some...

noreply@blogger.com (Thomas Joseph) — Wed, 17 Mar 2010 16:31:00 +0000

I had just brought a new headphone for making Skype calls. I was trying testing it out, and I recorded my voice to test it. What came in was a movie song that I was chanting over. I played it back to find too much of background noise. I wanted to remove it. Now for any utility for my OS, I do not rely on a search engine,.. I rely on my "Synaptic Package Manager" on my Linux box - Ubuntu 9.10 (Karmic Koala). Quick was it, to get "Audacity" to help me record, and edit audio.

Notice, that I split the audio into multiple tracks easily and mix it at proper timings.

Believe me. Within half an hour I could learn the basic stuff and the edited voice was so good to hear now. I thought, how about mixing some music as well? Soon I got the Karaoke for it as well. Without much practice or effort, I could record that song with the music tack mixed. Hear it to believe it!

Download Song (mp3)

This is a romantic song "Pathira Pullu" from the Malayalam Movie - "Ee Puzhayum Kadannu"(1996). It is one of my favorites songs. Music composed by Johnson (one of my favorite music composers), and originally sung by KJ Yesudas (obviously my favorite singer).

Those who know me personally can recognize my voice. I have just removed some background noise, added some echo and bass. I am just a bathroom singer, but the voice seems to come out pretty well after the treatment. Even as a fist-time sound recordists, the result was pleasing to me, even with the low quality of the microphone and the always present background noise.

I would encourage people to try this out. Linux is not just for geeks. Linux is for all human beings. I am sure there should be more professional tools available for professional people. You can at-least use the tool to record your voice at home and send to participate in your favorite music reality shows :).

PS: I would have preferred to upload flac/ogg format, but then my friends on non-Linux machines would have no way to hear it.

A tribute to my ex-colleagues

noreply@blogger.com (Thomas Joseph) — Tue, 16 Mar 2010 13:33:00 +0000

Moving with the waves of time, I changed my job. So did some of my other colleagues at Miediant Solutions - my previous employer. Now after we all are into our new jobs, happy and looking forward for a pleasant and prosperous future with our new jobs, we all miss our team, the togetherness we shared. We enjoyed the fun-filled, yet productive work environment, that had so much to learn, and had contributed so much to our professional career. While, many love to hate their bosses, we hope to have him again in our team. We all keep our fingers crossed, and hope for the best to happen, and meet each other and work together again some day.

Thank you all for making the work a pleasurable experience, and the office - a home away from home,
a place we wish to stay together. We shared so much professionally, and
we worked like soldiers in the war.

Thank for all the sprints, the daily scrums and the virtually fighting retrospectives,

the lunch, the outings - thanks for all that time.

Thank you all for the professional and personal growth gained together.

Wish you all a prosperous and happy future!

QCon London 2010 slides

noreply@blogger.com (Thomas Joseph) — Mon, 15 Mar 2010 07:02:00 +0000

The QCon London 2010 has just concluded. There was extremely good response from the delegates. Many attendees have blogged on the same.

The conference had 19 tracks spread over 3 days:

Wednesday

Architectures You've Always Wondered About
Software Craftsmanship
Non-Relational DBs & Web Oriented Data
Dev and Ops: A single team
Functional programming
Solution Track: Wednesday

Thursday

2015 Software Development
Irresponsible Architectures and Unusual Architects
Pragmatic Cloud Computing
Agile Evolution
AlphaGeeks on .NET
IT - more than tools and technology
Solutions Track: Performance and Scalability

Friday

The Concurrency Challenge
Cool Stuff with Java
How do you test that?
SOA 2010
Browser as a Platform
Solution Track: Friday

Some great speakers such as Robert C. Martin, Martin Fowler, Rod Johnson, Joe Armstrong, Dan North teamed up for this.

Some of the slides for these are available online on the official QCon London website. Although, the slides by themselves do not ever match for going there yourself, these and the blogs are good for those who could not attend it.

Saving a streaming video / presentation

noreply@blogger.com (Thomas Joseph) — Mon, 21 Dec 2009 17:09:00 +0000

Ever wondered to to save those streaming videos from Youtube, Metacafe, DailyMotion and so many others? I always wanted to save those streams, so that I could watch them again and again even without an Internet connection.

I am a faithful Firefox user, and one of its wonderful plugin - DownloadHelper has helped me for a long time to capture and save Youtube videos as flash video files on my local hard disk. That was quite helpful. But I always wanted to save those InfoQ videos as well, but unfortunately, DownloadHelper could not provide support for InfoQ (among others).

It was only recently, that I used the conversion feature of DownloadHelper, in which I was converting one of those Youtube videos to be converted to wmv format. I noticed, that the conversion started only after the whole of the video stream was played on my browser. And I could readily guess, that the conversion must be using one of the libraries on my machine. So I looked at the process to find that it was using ffmpeg command to do this job. I also noticed, that actually every video was buffered to the /tmp on my Ubuntu. The filename I could see was something like FlashVOZFnS.

Naturally, if a movie plays on your machine, it must be stored somewhere before being played on your browser. So now if you want to save your next streaming video, just identify where in your system does the flash player saves its movie temporarily before being played on your browser, and save that file to your videos folder, and possibly give it a .flv extension too.

Note:

As soon as you close the tab containing the video, the temporary file is discarded.
Not all videos may be fully buffered before playing. Very long videos and webcasts may not be fully buffered.

Converting flash movies to other format (Linux)

And now if you also want to convert it into other formats, its very easy in Linux with the following commands:
FLV to WMV:

ffmpeg -i /home/thomas/Desktop/video.flv -y -v 0 -acodec wmav1 -b 1000kbps -f asf -vcodec wmv1 /home/thomas/Desktop/video.wmv

FLV to MPG:
ffmpeg -i /home/thomas/dwhelper/Desktop/video.flv -y -v 0 -f asf -vcodec mpeg4 /home/thomas/Desktop/video.mpg

There are so many other options with this command, even up to video capturing your screen, but I am not very expert at those. So I leave it to you to experiment with those options.

Update

For any streaming video site, if you cannot see DownloadHelper to your help, make sure that you should have only this one tab containing any video to download. In other words, if you have multiple tabs with video contents, DownloadHelper seems to get confused, so you may not actually get a download link help.

---------------
Edited and removed some content for legal reasons.
---------------

Configuring log4j with XML in OSGi

noreply@blogger.com (Thomas Joseph) — Tue, 15 Dec 2009 14:43:00 +0000

Sometime back, I had to make perf4j work on OSGi. It required log4j as the logging library and purely demanded log4j.xml (it would not take the .properties file since the configuration must attach downstream appenders). I assumed that it should be simple, but unfortunately it was not. In the rest of this post, I describe how I could get around this, and used log4j.xml to configure log4j logging in OSGi.

Some background on logging for OSGi

Logging was something that was looked as one of the earliest thing for building up our OSGi infrastructure libraries. Since most of the bundles and libraries require logging to be there, we were forced to provide its implementation at the early stages itself. What came as the first viable option was to use Pax Logging. It is still the best logging mechanism for OSGi. It can take configurations in a variety of ways, the most pleasing of all the available configuration models was using the ConfigAdmin service that would leverage the OSGi's dynamism. This is again the best was to configure any service, and not just logging. Even without any explicit configuration Pax Logging would just work fine with its default behavior with a DEBUG for all the bundles in the system.

We chose Pax Logging with ConfigAdmin service. At first the ConfigAdmin provider was chosen as Pax ConfMan. But due to its limitation of not providing runtime updates, I chose to opt for Pax Coin which is still under incubation, but has some good features including runtime property updates and capability to consume configuration data from multiple sources. I was happy to make Pax Coin consume configuration from file system. But I could find that it simply did not accept configurations from the xml file.

So, as of now though providing configuration data via ConfigAdmin seems to be a good choice, I could not find any quick solution to this. So I found a temporary hack for this.

Hack

Note: The following instructions are for working on Pax Runner. If you are not already using it, I would highly recommend it as a tool to start your OSGi framework. It would be the most elegant way to get started with OSGi.

Remove any references to pax logging from your provisioning, so that it doesn't appear in your OSGi platform.
Add commons-logging and log4j jars to the platform as bundles. Add the following URLs in your profile/provisioning file:
```
    wrap:mvn:commons-logging/commons-logging/1.1.1
    wrap:mvn:log4j/log4j/1.2.15
```
Point to the log4j.xml file it OSGi classpath via Pax Runner. Its easy to append --cp=<path_to_dir> as a parameter when starting Pax Runner. The directory path_to_dir will now be in classpath, so you can place our log4j.xml there to be picked up by the log4j system.

Future

This was a temporary hack, but the concern is that I need to find a better way to make ConfigAdmin service consume XML.

Microsoft ASP.NET MVC - Why Java web developers should reconsider action frameworks?

noreply@blogger.com (Thomas Joseph) — Wed, 18 Nov 2009 05:53:00 +0000

Microsoft's ASP.NET MVC is a paradigm shift in web development from its pioneering component based development offered with WebForms to action based web framework. This was a natural evolution due to the pains in the component based web development. While the Java community started out with the action based frameworks, the shift of the community seems to be on a higher side for the component based web development led by JavaServer Faces (JSF) in the recent years. The shift of Microsoft towards action frameworks should make the Java web developer's heads again turn towards action frameworks and reconsider their choice towards the paradigm for web development. Perhaps a thinking about polyglot web development is needed, rather than abandoning action frameworks.

A stroll with component web framework

Like many other Java web developers, I also had a chance to use JSF for my web development, after using Struts classic for almost two years. Everything was so exciting as we started with it. There was en enthusiasm to go on with Rapid Application Development (RAD) with drag and drop, the event oriented programming ushered by the component framework. The wide variety of JSF extensions, and widgets that would help us develop rich web applications, added to our excitement. The pain in tinkering with JavaScript and browser quirks would be zeroed.

As we started out, there were some initial hiccups because of the change in the way we think about developing web applications. Now we are less worried about the basics of HTTP, requests, responses, request parameters and its conversion between class properties. We were excited about the wide variety of rich controls that a JSF extension such as Richfaces or ICEFaces could offer.

Now about one year down the line as we started out writing more and more code, I felt at certain places, that I was missing action framework. It was not easy to bookmark pages - the URLs where least indicative about it, they are not search engine friendly. There are a whole lot of problems indicated by people at various blogs and articles. The flow control with JSF was very flaky, unless I used Spring Webflow to conquer my workflow requirements. I could no longer easily play with small tweaks that action frameworks would have always allowed me to. I had lost the flexibility to the framework that was primarily made to hide the details about HTTP. Although there are ways to get around with the limitations, some workarounds are very ugly and the bottomline was - the simplicity had gone. There was a higher learning curve too.

While the problems faced with JSF may not be true for every component frameworks, the higher adoption rate for JSF is causing me to rethink on it, before I could invest more efforts on it.

Looking back at action web framework

While the flexibility problems with JSF were still haunting, I heard about ASP.NET MVC from Microsoft - an action framework for .NET web development. I was looking back to action frameworks now. When I last used Struts classic in 2007, Struts 2 had already hit the popularity headlines. But I did not had a chance to try that out, because of the obvious migration problems associated with a large project. With a mixed response to this, I started looking into Struts 2.

While I am still learning Struts 2, I could realize that the framework is much more easier that its predecessor. It could hide away the HTTP details, but at the same time, it was easy for anyone to play with the intricacies of HTTP if one would like to. Its damn simple, with least learning curve, probably because action framework are the right fit to thinking on how web works. Lead architect of Microsoft - Scott Guthrie has quoted on the action framework - "It enables full control over HTML markup and URL structure, and facilitates unit testing and a test driven development workflow" explains this.

I wondered why even after three years of Struts 2 (and so many other action frameworks for Java), JSF adoption rate was going high when compared to Struts or any other action frameworks. The job boards posted more requirements for JSF jobs when compared to action frameworks like Struts. This made me look into some depth why Microsoft adopted action frameworks, who were the people who introduced component based programming to the web development with ASP. ASP was later evolved with .NET as it became ASP.NET.

A look into why component web frameworks emerged

The significant roots of component based frameworks could be traced back to Microsoft coming out with ASP way back in mid-nineties to late nineties, as web started growing out from serving static HTML pages to creating dynamic web applications using the same HTTP protocol and HTML markups. The programming style in ASP although made some exposure to HTML, HTTP, GET and POST, primarily it was intended for VB6 desktop developers who would like to move on to web development. ASP.NET 1.0 continued this pace within the .NET framework. It was easy for seasoned VB6 developers to drag and drop components from the toolbox to author a page, with the least concern that it was a programming for web. VB6ers felt at home as they had to deal with post-backs and click handlers.

The same period in the Java space was taking a different stride. The Servlet specification was the first attempt from the Java community to deal with dynamic web pages. The pain of writing HTML markup in Java code paved the way to emergence of JSP. This was relaxing to an extent, but there were no formal web frameworks. This made the emergence of Struts by around mid of 2000, that brought in the concept of MVC to the web development. It quickly became the most popular framework in the Java community. Sun finally came out with a web-framework specification in 2004 as JSF1.0. JSF was largely similar to ASP/ASP.NET model programming and aimed at a competition with it. It is component based and even driven, that allowed rapid application developments that aimed the IDEs to give away drag-and-drop UI development combined with post-backs and even handlers in backing beans.

Why Microsoft is making a transition to action framework?

Definitely Microsoft is more experienced in component based development. The developers have always missed an inherent MVC with WebForms. They have also missed the Test Driven Development (TDD) robustness in their applications. The separation of concerns provide extensibility and testability, that was missing in WebForms. Action framework provided by the ASP.NET MVC has been now recognized as a natural fit to developing web applications. The power, simplicity and flexibility of action frameworks was witnessed earlier with MonoRail by the .NET developers. The emergence of ASP.NET MVC is the result of learning from the past and the recognition of needs by the developers for a simple, yet powerful action framework to the mainstream .NET developers. Now ASP.NET developers could enjoy TDD approach to web development in addition to the simplicity and extensibility provided by the action framework.

While the ASP.NET MVC framework was released just this March 2009, the active community around it has already found need for further improvements with an effort already into the development of its 2.0 release. I do not rule out the reasons to attract non .NET web developers into the Microsoft platform as one reason for Microsoft to come out with this plan. A notable point here is the recommendation by Microsoft to move the WebForms to the new action framework. It is already predicted by some to be the norm for ASP.NET development by 2011. ASP.NET MVC will be shipped with Visual Studio 2010 underlines its importance to Microsoft.

Learning from each other

Though late, Microsoft community has identified the importance of action framework for web development. Now when the Microsoft community is moving from component based to action framework based web development, the Java community really needs to re-evaluate its use of JSF for web development. Definitely, each paradigm of web development has its own importance, and probably can and should co-exist. But the larger Java community should not forget the learning of Microsoft community with web development framework.

On the other hand, Microsoft community can benefit from the learning in Java community with action frameworks such as Struts 2. ASP.NET MVC has released under open source license (MS-PL) and working with the community to develop it further, which is a welcome step. It also allows the community to contribute to the growth and higher adoption of the framework.

Rapid Application Development
As far as rapid application development is concerned, Rails principles such as convention over configuration, intelligent defaults, and automatic code generation with templates could be employed. ASP.NET MVC already includes tooling that uses code generation to quickly create a view based on a model object. Ruby on Rails is already a good example that can be followed. Struts 2 is no doubt, a great framework for Java web development with a flexible, simple and pluggable framework with convention over configuration and intelligent defaults that can be emulated by even the Microsoft community in developing its action framework. A good deal already seems to be into the ASP.NET MVC 1.0 inspired from Struts 2.

RAD can also be supported with plugin architecture as in Struts 2 that already offers plugins for GWT, jQuery and dojo among others. More and more code generators like the Spring ROO are also a welcome step. Perhaps we will witness a plethora of such code generators with Maven3 gets ready to be into the wild, that will support such extensions including the IDE integrations.

WYSIWYG editors for component framework can definitely will have its own role to play. We can see such editors for GWT and JSF as well. Better WYSIWYG editors for action frameworks are needed for RAD.

Concluding thoughts

The web frameworks do not end with JSF and Struts or with ASP.NET WebForms or ASP.NET MVC. There are notable advantages with frameworks like GWT and Echo that take a different stride to web development. Its time we need to re-look into our web development and see if we are beating every nail with the same hammer. Today as we speak of polyglot programming and polyglot persistence, probably a thinking about polyglot web frameworks is also needed.

Evolving persistence, SQL and ORMs to the next generation

noreply@blogger.com (Thomas Joseph) — Thu, 29 Oct 2009 12:20:00 +0000

Recently there has been a lot of discussion on the web describing the ORMs as a thing of the past, and its associated discussions that agreed and disagreed with their arguments presented to various degrees. The discussions even took a long conversations across the twitter space too.

Persistence models and application evolution

Persistence is important for any application. Over the years this has been done in various ways. While RDBMS and SQL were the most obvious choice till few years from now, recent times have also seen successful practical usage of other persistence models which include Amazon's SimpleDB, and Google's BigTable among others. These non-relational alternatives fall under the NoSQL (or “post-relational”) persistence models.

Naturally it becomes quite obvious that over the time, relational databases have proven inadequate to satisfy the growing needs of the Web2.0 applications. The advent of the new-generation non-relational databases are a proof to this. Every persistence model has a niche area to which they cater. The newer databases have has also made people to think about "mix-n-match" use of the various persistence models as people talk more and more about polyglot persistence.

The relation-object impedance mismatch

Since RDBMS has a solid theoretical foundation, and has been used across various applications, its existence for another century cannot be ruled out. Yet new advances in persistence like CouchDB, BigTable, SimpleDB, Graph databases, etc are most welcome steps in advancing the persistence mechanism.

Even with newer persistence models, a lot of the enterprise applications still use, and will continue to use relational databases. Applications that use up this data are largely represented as objects. While object model seem to be the most obvious choice for application development, the mismatch between the relational model and the object model comes as the reason for a lot of hues and cries that have given rise to ORMs. The learning curve and the pain in using these ORMs have now given rise to debates questioning the existence of ORMs in the future.

More than debating over the existence of ORMs itself, rather a deep look is needed, to the problem which the ORM is trying to solve. Why does the ORM exist? The most popular answer would be impedance mismatch. But the mismatch is pointed out between relational data representation versus the object model.

Overhauling the persistence mechanisms

Often the data storage mechanism and its interfacing API are seen as an integral part, including the RDBMS data storage and SQL. They are often seen as one, but are actually two separate components of the modern day relational databases. They were created at a time when the existence and usage of OO was questionable. Even now when they are used for a long time, it can be argued that SQL and RDBMS themselves have a mismatch in the paradigms. Now when OO has gained so much importance, it is time that the RDBMS and SQL should be re-looked from the OO perspective.

We can consider two aspects in any persistence mechanism:

The way data is stored (relational, key-value, hierarchical , column, JSON etc) using various data structures and storage mechanisms. Closely associated to this data storage will be the related methods of storing and fetching the stored data (algorithms used to fetch data based on persistent store itself - eg relational algebra) that could be probably exposed through function calls. The function calls will be closely associated to the data store mechanism itself.
Interfacing provided to interact with persistent store (SQL, NoSQL, ObjectQL, programming language API etc).

Often these two aspects are baked together (in our minds most of the time). This thinking should evolve and these two should essentially be seen as separate, and ideally pluggable. We talk about adapter pattern in programming languages, but the same concept could probably be applied to persistence mechanisms as well. The persistent storage and its storing and fetching mechanism should allow widely different interfacing provided to interact with it. DB vendors can provide few interfacing mechanism, and also allow others to write other interfacing mechanism that should be pluggable.

Interfacing will expose the persistent store in various forms - relational model, or object model or even other models used now and may come in future. Mapping of the datastore persistence model to the exposed model should reside with the interfacing mechanism used.

Concluding thoughts

Achieving this practically may not be very easy, but not impossible. Once this is achieved, ORMs will rather shed off the mapping weight and evolve as frameworks that enhance developer productivity on issues like caching, application scaling etc. Rather, ORMs will no longer be ORMs and probably a new breed of softwares will evolve that will take care of the cross-cutting concerns for an application when dealing with persistent stores.

This may look crap to some, but then that is what I wish. I am an application developer presenting my ideas on what a database itself can provide, and how applications can leverage the new capabilities. The issue needs to be looked at by db vendors and research scholars. DB vendors and db developers need to give deeper thoughts to the problems faced by application developers in their need to persist application data. How many of us use the bare databases (using only SQLs probably) without an application? The current forms of persistence may be right in their own theoretical framework, but it should evolve to fit in seamlessly with the application development space, where it is used.

And till that happens, I will have to carefully weigh my choices among NoSQL, SQL and ORMs, and muddle with the problems that have already heated up a lot of discussions in the application development space.

My state of affairs - part 2 (looking for new projects)

noreply@blogger.com (Thomas Joseph) — Tue, 27 Oct 2009 13:44:00 +0000

In my previous blog post, I mentioned about the my state of affairs, of how things have been moving in last 2-3 years, and why I was so irregular at posting blogs. The state of affairs was actually not finished yet. I continue it here.

Looking out for new job/consulting opportunities

My current employer is finding it difficult to sustain its employees. While some employees quit at the start of the turbulent times, some of us including me decided to stay strong with the company. And now more than one year after showing signs of trouble, the company is not yet able to fully recover and get back to normal business. The management is so kind to ask us to look out for other jobs as an open offer to us. We can take our own time to move out. Although the management has plans to bring it up again back to business, it doesn't want its employees to be dragged across the difficult times any longer. As our work is tapering off, I will be looking out to see if some good offers can come along.

So if you are looking out for an experienced developer or consultant, with proven track record in providing effective business solutions with good exposure to various technologies, try me - I will be happy to help you out. I specialize in Java/EE, Spring, Maven and OSGi (thats about building truly modular Java applications). And if you want few more experience hands, may be my company can help you in your IT endeavors, specializing in creating robust modular and scalable Java applications.

In the meanwhile I will be working on some OSS projects and writing blogs. So, keep an eye on this blog. My blog posts will give a good understanding of various technologies and how the technology can be best put to use in solving real-life business problems.

I am also working out on a project (of my own) that is a good mix of some technologies I have already learned, or have scratched the surface. It will provide the ground for me to build depth of my technical knowledge as well as serve as a ground to build up applications for my client that require ground-up work. Some time down the line, I will also give out this code to the OSS space.

If you would like to know more about me, follow the links below:

My state of affairs - part 1

noreply@blogger.com (Thomas Joseph) — Mon, 26 Oct 2009 20:10:00 +0000

Now it has been a while that I have blogged, and I understand that not many people are there who follow my blog. Though some of the topics on which I have blogged in the past have been getting some good hits, I cannot maintain a regular list of people who will visit my blog often, probably because I am not regular at blogging. But thanks to those 3 feed-readers who have been with me, and probably would like to know, what keeps me busy that prevents me to write blogs, and now why is it that I am giving out an explanation. Also, probably in every other serious post that I write, I almost mention that the next post will be coming very soon, and still that does not happen.

Last 3 years

While it is a fact that I have been dealing with so many topics in the past, especially in the last 3 years or so, and every time I feel like posting about it on the blog, and yet that does not happen. all this time, I have been dealing with a vast breadth of topics in the Java space - Rules Engine/Drools, OSGi (Pax, Felix, Equinox, Karaf), Maven, ESB (ServiceMix), Message Routers (Camel), JPA/Hibernate, Spring, Webworks/Struts2, Spring Webflow, JSF, Facelets, RichFaces, REST, SOA, GAE (Google App Engine), GWT, Spring Security and its various tools, implementations and their integrations on how these topics can play with each other. It was during this time that I got more into the OSS space as well. Initially moved by the JBoss professional open sourcing and then more impressed and involved with the OPS4J philosophy. The single factor that made me meet OPS4J was its PAX line of products that dealt with OSGi. And as I started learning OSGi, I was happy to take out some of my personal time to make few commits into the same products in PAX - that made me smile better as I struggled with my initial steps with OSGi.

The journey begins

This journey begins as I complete almost one year with my previous employer, when the client and the employer impressed with my performance handed over the implementation of intelligence in an insurance quoting system to me. Thanks to them, it was a great experience. I implemented the system using Drools and I was able to deliver the system that could be taken out to production systems within 7 months from the date I was assigned the task, with hardly any help from anybody else. Whats more, the system was almost zero on bugs as of my knowledge. Here I started my journey of exploring technologies beyond the regular Java/J2EE developer. Though in the past I had done it on my personal itch, this was a better experience, since could see it working on a real system. Before this, I had experience with the regular Java/J2EE experience like the majority developers with just Servlets, JSP, Struts, SQL, EJB etc etc.

Last 2 years

Now it is almost 2 years that I've been working with my current employer, and most of this time I was into R&D primarily with OSGi and its related aspects. While I personally like to dig past bleeding edge technologies, and playing with it, a similar work profile as my job profile was something that was very exciting. I was always excited about how technology could be used constructively to provide least effort and long term solutions to real-life business problems. I was the prime responsible for bringing up a framework for the company that could be used by its products/services to cater its clients in the coming future. The power and dynamism of OSGi was a major push to this. During this time, I had the opportunity to deal with a variety of technologies to make up the framework as mentioned above. I did made serious advances in setting up the technological stack for my company based on OSGi. But I still feel very sorry that I could do very less on contributing this knowledge to the OSS, as I see people trying to struggle with OSGi and web .

But why should all that prevent from posting blogs?

All this made me so much involved in understanding the technologies that I considered it worthwhile to spend that 2-5 hours to understanding the other technology for integration to the stack, than to write a blog on what I had just learned. I even spent my personal time for understanding aspects of technology that was outside the scope in my official research. Also, its not that I did not write blogs. While a very few appeared here on my public blog, most of them went on to my corporate blog that is not a public one. Also, all this time, my understanding about any technology is something, that I think would be better to blog about as I learn more. But over the time, that never happened as I piled up more and more technologies.

So, now whats there for the future?

Guessed, it right! I am again going to say that I will try to be more regular at blogging. Blogging is a tool to showcase yourself, express your ideas and to market yourself. Now this time, there are reasons too why I should blog, about which I will be explaining in my immediately next blog (mmmh..., you will not have to wait, its ready as I write this. This blog was bloating too much, so I just decided to split it into two).

JavaOne 2009 slides are now out!

noreply@blogger.com (Thomas Joseph) — Thu, 11 Jun 2009 06:26:00 +0000

Now the slides of the sessions at JavaOne 2009 are out and available at Sun Developer Network (SDN).

You need to be a registered user there! (For some, this sucks!)

Modularity and OSGi was one of the discussed topics in the sessions, but worried to find that Sun has not mentioned anything on it (OSGi) on the site, though it has put up the slides :).

More Goodies

Sun has put up some good articles and other stuffs on JavaOne as well on its site.

OSGi DevCon 09 - Component Oriented Development in OSGi with DS, Spring and iPOJO

noreply@blogger.com (Thomas Joseph) — Thu, 11 Jun 2009 05:31:00 +0000

This slide discusses some comparisons on the various component oriented development models with OSGi. Probably there a mention of Peaberry, would have been good. Sure, I will post some in the coming posts.

OSGi DevCon 09 - Component Oriented Development in OSGi with DS, Spring and iPOJO

View more Microsoft Word documents from Heiko Seeberger.

OSGi DevCon 2009 Review

noreply@blogger.com (Thomas Joseph) — Thu, 11 Jun 2009 05:16:00 +0000

Last few weeks I was more into the OSGi updates going around with the EclipseCon and JavaOne. In this and few more posts I will try to put something to capture on the recent developments on OSGi and related fields and technologies.

So here is an interesting slide I found.

OSGi DevCon 2009 Review

View more Microsoft Word documents from njbartlett.

Book Review: Maven: The Definitive Guide

noreply@blogger.com (Thomas Joseph) — Thu, 14 May 2009 11:19:00 +0000

Sonatype recently released its 0.5 version of its Maven Book - Maven: The Definitive Guide.The print edition published by O'Reilly, the printable PDF published by Sonatype is available as a free download! I went thorough the book quickly and here is my first hand impression. The books is also available for online reading.

Cool!

I have been using Maven now for quite sometime, and I was very happy to read the book. The book is a fit-for-all, who wants to use maven in someway or other.

It starts with as simple as downloading and installing maven, to explaining the core basics of maven, its dependency management, plugins etc, to optimizing plugins and POMS. It serves as a tutorial as well as a reference guide for Maven.
The book explains concept with ample examples. It also covers to explain some of the most useful plugins in depth. It explains the maven build cycle in depth, build profiles, maven assemblies, site generation and maven repositories, m2eclipse and nexus at depth. It also explains customizing plugins and also on writing our own plugins.

How to use the book

An extract:

This book is divided into three parts: Introductory Material, Part I, “Maven by Example”, and Part II, “Maven Reference”. The introductory material consists of two chapters: Chapter 1, Introducing Apache Maven and Chapter 2, Installing and Running Maven. Part I, “Maven by Example” introduces Maven by developing some real examples and walking you through the structure of those examples providing motivation and explanation along the way. If you are new to Maven, start with Part I, “Maven by Example”. Part II, “Maven Reference” is less introduction than reference, each chapter in Part II, “Maven Reference” deals with a focused topic and dives into as much detail as possible about each topic. For example, the Chapter 17, Writing Plugins chapter in Part II, “Maven Reference” deals with writing plugins by providing a few examples and a series of lists and tables.

While both Part I, “Maven by Example” and Part II, “Maven Reference” provide explanation, each part takes a different strategy. Where Part I, “Maven by Example” focuses on the context of a Maven project, Part II, “Maven Reference” focuses on a single topic. You can skip around in the book, Part I, “Maven by Example” is by no means a prerequisite for Part II, “Maven Reference”, but you'll have a better appreciation for Part II, “Maven Reference” if you read through Part I, “Maven by Example”. Maven is best learned by example, but once you've gone through the examples, you are going to need a good reference to start customizing Maven for your own environment.

Kudos

The work is excellent and what makes it even better is the fact that the company and the authors do not consider it complete yet! Its a beta 0.5. This is what they have to say on it:

If this book moves out of Beta, we're essentially saying that there is no more content to add and nothing is going to change. I've always thought that good books evolve over time and that they live beyond the confines of the chapters and sections that define them. A good book is an ongoing conversation and a series of interactions not just between authors and readers, but of lateral interactions between readers. That was a fancy way of saying that the book is a community.

Sonatype and the book authors have done an excellent job. I would recommend the book to anyone who uses Maven.

Flex Clients for OSGi apps

noreply@blogger.com (Thomas Joseph) — Tue, 12 May 2009 07:04:00 +0000

As I continued to work on OSGi based web apps, there was rising demands from our clients and partners about supporting Flex clients in future. I took a lead to evaluate our existing architecture consisting mainly of POJOs, supported with Spring and OSGi.

Thanks to the documentation at BlazeDS site and the refcard at DZone for helping me out. I am almost ignorant of Flex programming, but still I could make out few things. Here is my first-cut evaluation.

Flex and Java with POJO

I was happy to learn that with Spring's hailed paradigm of programming with interfaces and POJOs would make integration with Flex very simple, and would hardly require any change in how we program and architect our application.

Flex has strong ties to Java. Its server BlazeDS is based on Java remoting. Good news that with Spring BlazeDS, we can reuse our interfaces and its POJO implementations.

Spring BlazeDS as described by SpringSource:

Spring BlazeDS Integration is a new addition to the Spring portfolio, and a component of the complete Spring Web stack. This project's purpose is to make it easier to build Spring-powered Rich Internet Applications using Adobe Flex as the front-end client. It aims to achieve this purpose by providing first-class support for using the open source Adobe BlazeDS project and its powerful remoting and messaging facilities in combination with the familiar Spring programming model.

Blaze DS is based on Java remoting and web messaging from Adobe. Its primary purpose of existence is easier integration of Adobe Flex/Adobe AIR with existing Java code to create Rich Internet Applications (RIA). Spring BlazeDS leverages more on this end by its dependency injection, and simpler integration platform.

Blaze DS can be set to run on any web container such as Tomcat. It listens to a particular URL for client requests and processes them. Flex clients connect to the Blaze DS server and executes Java methods. The base transport protocol will be HTTP.

The regular client calls with Blaze DS is usually handled by a Servlet from Blaze DS. However, when Spring Blaze DS comes into picture, this is replaced by the Spring's Spring MVC Servlet, so that Spring can intercept those client calls.

Flex works on the concept of message broker. A class that will can handle client calls. The clients calls will be routed to the message broker through additional configuration specific to Blaze DS. We should be happy to expose our interfaces and its implementation classes to work as message brokers.

A sample Spring's application-context.xml file

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:flex="http://www.springframework.org/schema/flex"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/flex
http://www.springframework.org/schema/flex/spring-flex-1.0.xsd">
<!-- Spring Beans's -->
<bean id="myService" class="MyServiceImpl" />
<!-- Simplest possible message broker -->
<flex:message-broker />
<!-- exposes myService as BlazeDS destination -->
<flex:remote-service ref="myService" />
</beans>

The point to be noted above is that myService can be given any reference, including a reference to any OSGi service.

Another notable point about Spring BlazeDS Integration is that Flex can support authentication, that can easily plug in with Spring Security.

Kudos

Spring BlazeDS has not yet had its final release, but indeed a great work from Spring guys to help lift existing POJO based applications to work with Flex clients in most non-intrusive manner.

Guice getting into Java SE?

noreply@blogger.com (Thomas Joseph) — Fri, 08 May 2009 14:00:00 +0000

One day I may use @javax.inject.Inject as my regular Java annotation! Yes, dependency injection with annotations might become a standard Java API. Google Guice and SpringSource have partnered to standardize a proven, non-controversial set of annotations that make injectable classes portable across frameworks. The JSR has been proposed.

Read more about this at the official Google Code Blog.

Submitting form to new window

noreply@blogger.com (Thomas Joseph) — Mon, 20 Apr 2009 11:06:00 +0000

Today one of my co-workers was in an effort to submit a form that would bring up a new popup window. After some Googling, they found a solution.

<form name="datacollector" method="POST" action="new-url.do" target="_blank">
<!--
     ...
     other form elements
     ...
-->
</form>

Though this was getting them a new window, on Firefox it was appearing as a new tab. They soon found a setting in FF, that would bring it to a new window itself. However, they wanted a popup without any toolbars, menus etc.

They approached me. Thanks to one of my student days HTML/JavaScript tutorial. I was able to recollect it and give them a solution.

<script type="text/javascript">
   function submitToNewWindow(frm) {
       window.open('','popupWin','scrollbars=no,menubar=no,height=600,width=800,resizable=yes,toolbar=no,status=no');
       frm.action= "new-url.do";
       frm.target = "popupWin";
       frm.submit();
   }
</script>
...
...
<form name="datacollector" method="POST">
<!--
     ...
     other form elements
     ...
-->
      <input type="button" value"Generate" onclick="javascript:submitToNewWindow(this.form)"/>
</form>

The idea is simple. Open a window with a name (here popupWin) using JavaScript that will have no scrollbar, menu, etc. Now in the form's target, supply this window name. This would bring up the form submission's result on the specified popup window. As far as the "_blank" is concerned, it is a HTML's default name for a new blank window.

This is a basic feature of HTML, that in the target, it accepts the name of a window. Note that if a window with that name does not exist, probably a new one will be created.

This solution was tested to be working in FF as well as IE.

Though this is a simple solution, I thought to put up here on my blog, as my co-workers were not able to get anything like this on Googling.

Cross Platform Java Daemon

noreply@blogger.com (Thomas Joseph) — Thu, 16 Apr 2009 14:49:00 +0000

Running a Java application (non graphical) would normally need a command line for it. This seems OK if we are developing small applications, but today we see enterprise applications that often needs to go and work as a server, that need to start up and go down with the OS. For such applications, the requirement of avoiding a command line gains importance. Such applications are often referred to as Daemons in the *nix space, and services in the Windows space.

Solutions?

Wondering that you have seen so many Java applications that can work without command line? Its not that Java applications without a command line is not possible. Though Java is popular for being platform independent, doing this "daemonising" task has not been of that nature. Many of the Java based server products have been doing that in various different ways, and in most cases using native code. And if your look around for Java Daemon, or Java as Windows Service, you will find separate and widely different solutions.

And there are solutions specific to platforms, for instance - the nohup and screen command in linux can get you an application without terminal. For the nohup, though your solution to stop the application, is probably to kill it.

Recently I had a similar need, I looked around and could find a bunch of solutions:

Do it yourself!

Going through even the surface of these will give you an idea of how widely the implementations vary. These are solutions that help you lift your existing application, if you yourself do not want to provide any implementation of getting daemonised. But if you are happy to learn on this concept, or want to get your code provide the "daemon" power natively with simplicity and lightweight effort, read on...

(Note: I would recommend a reading of the Commons Daemon. Also note that the code below is just ideas and concepts, you may need to evolve it further to make it production ready.)

The concepts

Any Java application needs an entry point for it to start working. This is the public static void main(String[]) method. And the common way to do this invocation is to instruct the JVM and show it the class that contains this method block, with the java command. This command will start the JVM which starts the Java program starting with the main method. The command line is blocked until the Java program completes. This is a valid scenario for small and interactive applications, but for multi-threaded server programs, we do not want the command line.

The daemon solutions have varied way to enter a Java program (other than the main method as well). But here, we will use what we already know: enter a Java program with main method and still remain detached from the command line.

There we need to understand two concepts:
1. Starting a program. Using Java, it is possible to start "any program" using Runtime.exec() and ProcessBuilder.exec(). Wondering? Yes, any program that you can run in a command line can also be launched within Java programs as well. Another notable aspect of this is that the launched program does not have a command line associated with it, and it is optional for the launching Java program to keep track of this process. Well, we will use this concept to launch a Java process inside a Java program!
2. Stopping the program For a Java program, there is no way to get the control of a process running in the system. So if we launched a process from a Java program, and lost control over it, how can we tame it to bring it down? Establish protocol! We will make the launched program to open a port and start listening to it, and when it receives a valid shutdown command, start its shutdown procedures.

Taming the terms...

Below, I will show how to do this. But before, I will introduce some terms, so that further discussions are easy.

TargetApp - the Java application (possibly a server program) that we want to get detached from the command line and go "daemonised".
Daemon - The class that actually goes into daemon mode and listens to shutdown commands. If you are coding the daemon inside your application, this functionality will be inside one of your classes in the TargetApp.
Launcher - This program will kick off the Daemon as a process detached from the command line. Also provides interfacing to provide shutdown command to the Daemon.

Reaching out with Code

1. TargetApp

Since this can be any Java application that usually works in a loop, I will create a simple application that will keep printing a funny text to a file created in your current working directory.

package org.otw.daemon;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Date;

public class FunnyTargetApp {

   private boolean shouldRun = false;
   /**
    * @param args
    */
   public static void main(String[] args) {
       FunnyTargetApp app = new FunnyTargetApp();
       app.start();
   }

   /**
    * Starts the application.
    */
   public void start() {
       shouldRun = true;
       Thread workerThread = createWorkerThread();
       Thread commandTracker = createCommandTrackerThread();
       commandTracker.start();
       workerThread.start();
       try {
           workerThread.join();
       } catch (InterruptedException e) {
           e.printStackTrace();
       }
   }

   /**
    * Stops the application, in a clean way.
    */
   public void stop() {
       System.out.println("Stopping...");
       shouldRun = false;
   }

   /**
    * Creates a thread that mimics a continuous job as that of a server.
    * @return
    */
   private Thread createWorkerThread() {
       return new Thread(
               new Runnable () {
                   public void run() {
                       File file = null;
                       FileWriter wri = null;
                       try {
                           file = new File("test.out");
                           System.out.println("Started with file at ["
                                   + file.getAbsolutePath() + "].");
                           file.createNewFile();
                           wri = new FileWriter(file);
                           while(shouldRun) {
                               wri.write("Funny Text >> "+ new Date());
                               wri.write("\n");
                               wri.flush();
                               Thread.sleep(1000);
                           }
                       } catch(Exception e) {
                           e.printStackTrace();
                       } finally {
                           if (file !=null && wri!=null) {
                               try {
                                   wri.flush();
                                   wri.close();
                               } catch (Exception e) {
                                   e.printStackTrace();
                               }
                           }
                       }
                   }
                  
               }, "Worker Thread"
       );
   }

   /**
    * Creates a thread that will keep track of the commands issued on the command line.
    * @return
    */
   private Thread createCommandTrackerThread() {
       return new Thread(
               new Runnable () {
                       public void run() {
                           System.out.println("Listening to commands from terminal.");
                           while(true) {
                               InputStreamReader isr = new InputStreamReader(System.in);
                               BufferedReader reader =  new BufferedReader(isr);
                               String command = null;
                               try {
                                   command = reader.readLine();
                               } catch (IOException e) {
                                   e.printStackTrace();
                                   break;
                               }
                               if (command.equalsIgnoreCase("stop")) {
                                   stop();
                                   break;
                               } else {
                                   System.out.println("invalid command.");
                               }
                           }
                       }
               }, "Command Tracker"
       );
   }
}

2. Daemon

This is the class that actually gets detached from the terminal and becomes a daemon, while keeping a reference of the TargetApp. So basically it has to deal with 3 functionality:

a. Launch the TargetApp and keep its reference.

The TargetApp can be launched as a thread or as a process. The preferred way should be as a thread. For launching as a process, the TargetApp should have a either a start method or a regular main method. For launching TargetApp as a thread, invoke methods needed to start the TargetApp, if no such specific method is there, just invoke the main method.

//Starting as a Thread (reflecting on the main method)
Class payload = getPayloadStartClass(); // determine the payload class via configurations
Object payloadInstance = payload.newInstance();
Class[] paramTypes = new Class[1];
paramTypes[0] = arguments.getClass();
Object[] param = new Object[1];
param[0] = getPayloadArguments();
Method method = payload.getMethod("main", paramTypes);
method.invoke(payloadInstance, param);

Note: This snippet uses reflection. You may use an interface to make things better, using a startup method.

// Starting as a Process
Class payload = getPayloadStartClass(); // determine the payload class via configurations
String javaOptions = //get it via configurations
String payloadArg = //get it via configurations
String command = "java " + javaOptions + " "
 + payload.getCanonicalName() + " "
 + payloadArg;
log.debug("Starting command line: " + command);
Runtime.getRuntime().exec(command, null);

b. Starts up a shutdown port and waits for a shutdown command to be issued.

ServerSocket serverSocket = ServerSocket(getShutdownPort());
Socket socket = serverSocket.accept();
boolean isLocalConnect = false;
InputStream stream = null;
PrintWriter out = null;
try {
   isLocalConnect = socket.getInetAddress().isLoopbackAddress();
   socket.setSoTimeout(getNetworkTimeout());
   stream = socket.getInputStream();
   out = new PrintWriter(socket.getOutputStream(), true);
} catch (AccessControlException ace) {
   log.warn("StandardServer.accept security exception: "
                      + ace.getMessage(), ace);
   return;
} catch (IOException e) {
   log.error("StandardServer.await: accept: ", e);
   return;
}

c. Stop the TargetApp and then itself for a clean shutdown on receiving valid shutdown command on the shutdown port.

String command = readStream(stream); // reads command from the input stream
boolean match = command.toString().equals(shutdown);
if (match) {
   log.debug("Shutdown command recieved via Telnet.");
   shutdown(); // calling the shutdown procedures
   return;
} else {
   log.warn("Invalid command.");
}

You can telnet to the port and provide shutdown command. The Launcher's stop routine also does the same thing (the code will follow soon).

Remember to register shutdown hooks on the application startup, and remove it explicitly, if the shutdown was not initiated by the shutdown hook. Shutdown hook is a Thread that is not started, and registered with the JVM. When the JVM goes down, this thread will be started. So we can write some cleanup code in this thread. Remember to return from the run of the shutdown hook quickly.

3. Launcher

This class will provide us hooks to start up the Daemon, and stop it based on arguments (from the command line). To launch the Daemon as a Java process and detach it from the command line, we will use the Runtime.exec(). This will be similar to the process mode launching of TargetApp.

String strCmd = "java " + "-classpath " + classpath + javaOptions
   + " org.ops4j.laboratory.users.open4thomas.jdaemon.Carrier";
log.debug("Starting command line: " + strCmd);

Process proc = Runtime.getRuntime().exec(strCmd, builder.getArguments());
Thread hook =  createShutdownHook(proc);// create shutdown hook thread
Runtime.getRuntime().addShutdownHook(hook);

To stop the running Daemon cleanly, the class will set up a socket connection to the Daemon's shutdown port, and supply a shutdown command. This will trigger a shutdown procedure in Daemon, which in turn can trigger shutdown on the TargetApp.

Socket socket = null;
PrintWriter out = null;
try {
   socket = new Socket("localhost", getShutdownPort());
   out = new PrintWriter(socket.getOutputStream(), true);
} catch (UnknownHostException e) {
   log.error("Unknown address: localhost.");
  return;
} catch (IOException e) {
   log.error("Couldn't connect to: localhost.");
   return;
}

out.write("shutdown\n");
out.flush();

Getting around with security holes.

There is one important security hole in this approach. When the Daemon is listening on the shutdown port for shutdown command, it is possible for a hacker to compromise on this feature.
This can be eliminated with few approaches:

1. Provide password protection

The application could ask for a password that could be stored in encrypted format, and before accepting shutdown command, the user will be needed to provide a password.

2. Preventing DoS attack

A malicious user can connect to the shutdown port, and provide junk command continuously effectively doing a DoS attack. This can be dealt by accepting limited amount of failed password, or wrong command attempts. Limit the length of command/password.

More improvements

In the Launcher's stop, after issuing shutdown command to the Daemon, we could check for the existence of a PID file to know if the application is still running. The PID file can be created by the Daemon with "deleteOnExit" flag.

Another improvement area is to provide an interface that the TargetApp should implement, so that it will be easy to call startup and shutdown routines on the TargetApp.

There could be more, but not right now in my mind.

Want more code?

Sure, the code snippet I provided here is just to explain the core concept. There are other things required to get it working. I worked out quickly on this, and the code is available in my lab at OPS4J here. Please note that I may change the code over time. I am writing this article as of revision #14752.

This is a regular SVN repository. So you can get the source code with

svn co https://scm.ops4j.org/repos/ops4j/laboratory/users/open4thomas/jdaemon/jdaemon-main

Exciting future for the web

noreply@blogger.com (Thomas Joseph) — Wed, 08 Apr 2009 10:21:00 +0000

The future of web applications seem to be exiting for every web user. With the drafting and evolution of HTML 5 , and browsers like Firefox coming up with plans on Prism and Weave , its plugins, and evolving nature of supporting applications such as Google Sync (mobile and browser ) and Google Gears , the experience of a web user seems to be in a good light. Some of the fantastic experiences would be:

Inherent support for offline browsing and using web applications by browsers.
Inherent support for rich web UI.
Enhanced experiences with audio and video on web browsers.
Seamless and centralized browser profiles. Browser will seamlessly synchronize your profile including bookmarks, browsing history, saved passwords and tabs across multiple browsers, so that your desktop, laptop, and mobile phone can all work together.

Many of them are already available for a beta preview. These seem to not only enhance the experience of a web user, but also developers like me will find an interesting turn in developing web applications.