Venky - Tech space

Are you settling for mediocre solution?

2012-04-24T16:05:00.001-07:00

"This is just an Intranet application"
"People would hardly use this feature"
"There will be only less than 100 users for this site"
How many times have you heard such arguments during discussions. I have heard similar comments many times and have also made such comments myself. I have just realized that I had been convincing myself with these arguments to settle with mediocre solution for some of the "interesting" problems.

There is one particular requirement in my recent project where I faced similar situation.

Initial approach:

We came up with some sort of a "conventional" solution to address a requirement. This approach was not very clean as it had couple of issues
- We were violating the general rule of the site to support browser back/forward actions
- We were using URL query param to hide/show few links
Even though we had these issues, I was convinced with the solution and wanted to go ahead with that. I had "strong" reasons (read as excuses :) ) for the same.

Reasons (Excuses):

This application doesn't deserve "advanced/complex" features / This requirement is unwanted: This project has only 100 end users. We don't have to spend much time on building this "advanced/complex" features just for these users. I am not convinced yet that this requirement is needed.

Just enough design/This is an exceptional case: The conventional solution for this requirement works and we don't have to spend much time on this. We can violate the back/forward support for this requirement as it is an exception case.

Story points: It is only a 2 (S/something) pointer and we can't spend much time on this story.

This scenario would never happen: NO user would be performing these sequence of steps (Even though I don't have any metrics to support this claim)

But,
Luckily in my project, all mediocre/conventional solutions were challenged and the team was not convinced with my reasons. We had to come up with a better solution.

When we spent a little more time without thinking about excuses, we were able to come up with a better/cleaner solution. This out of the box solution met all our requirements and also made us feel good about. All it required was that little push from others and a little time to overcome the mediocre solution mode.

Now when I look back at my "strong reasons", everything seem to be flawed:

I could not see the difference between unwanted feature and differentiators. Without these features (related to usability), our site would end up being mediocre. It would not be able to get more users on to the system.
Looks like my notion of "Just enough design" also had "taking exceptions" for granted. In this particular case, I had already convinced myself that this requirement can take exceptions (like not supporting browser forward/backward) without exploring some more possibilities.
Story points are just a metrics to track progress and it should not be used for killing innovation/thinking out of the box.
I think, I had made up the "scenario would never happen" argument to support my "this is an exception" argument.

Lessons learnt:

Mediocre solutions come into existence only by convincing ourselves that it is the right thing to do in THIS case/application. All it needs is a little more time to come up with a better alternative.
Team members should be constantly challenging each other on mediocre solutions in order to build a great product. It also helps the team to improve their technical and problem solving skills.

Are you coming up with excuses or a better product?

Physical card wall is injurious to health

2012-04-23T07:40:00.002-07:00

...of the project.

Don't get me wrong, let me finish it. Only if your project uses ALM tools and physical wall at the same time.
NOTE: ALM tools - Agile Lifecycle management tools (like Mingle, Version One, etc)

Physical Wall Vs ALM tools wall

We are not getting into this argument right now. By now most of us would have used both and know the pros & cons of each. Personally, I am OK with any one of these.

Physical Wall & ALM tools - Hybrid approach

This is where I have a problem. In most the projects I have worked on, we have used this hybrid model. We would use ALM tools to create story, prioritize and for discussions with client but use physical wall to track the progress of the sprint/iteration.

Issues (I faced) in this approach

No single source of truth: Few things would be up-to-date in physical wall but not on the tool and vice versa. Consider this case - You find that a story card is 'In Dev' lane in physical wall and 'In QA' lane in ALM wall. Does this mean that:
- Story is 'Dev complete' and moved to 'In QA' by dev in ALM wall but not on Physical wall.
- QA found a bug and moved the story back to 'In Dev' in physical wall but not on ALM.
We cannot be sure and usually we end up chasing the team members working on the story for "correct status". Is it worth maintaining two walls?

Duplication of effort: Duplication is painful even here. Imagine that you just did your team sign up on the physical wall. How motivated would you be to update the exact same information on the ALM tool. So you can't complain the team for not having things in sync.

ALM Tool is not a boon anymore but a burden: In most of the teams I have been, it is a burden for few individuals (like PM, BAs, sometime Devs) to keep the ALM tool in sync with the physical wall. A tool that was supposed to be helping the project becomes a burden.

Invalid metrics: ALM tools provide lot of useful metrics that can be used to track the project and make some useful decision. Since the tool is not used properly, metrics provided is also invalid.

Wrong/Incomplete status on the wall: One of the projects I worked on had distributed team members. The physical wall in offshore would have only the stories that are worked on by offshore team members as it was hard/painful to track the cards worked on by onshore members. This wall was actually projecting wrong state of the project to everyone. It is better not to have such a wall than to have with wrong/incomplete information. (In this case, we stopped having the physical wall)

And there are few more...

It is hard to convince everyone and remove the physical wall from a project team that is used to having it.

Arguments in favor of Physical wall

Here are the few arguments that I have faced in favor of Physical wall:

I am used to doing signups and track things on Physical Wall
I prefer to physically feel story cards than rely on software tools.
Physical wall can promote more collaboration within team.
I have to click through 100 links to get the information I need in the ALM tool.
It is easy to refer to physical wall and get any information almost immediately. I don't want to login to ALM tool every time.
I prefer Physical Wall over ALM tools, it is simple.
It just takes couple of minutes to sync both tool and physical wall. Why do you bother?
Physical wall is an information radiator. Anyone walking-by can know the status of the project
Last but not the least - There are other major issues in our project. Can we look at this later? - :( YOU CAN NEVER FIND AN ANSWER TO THIS.

My take

I could find convincing reply to the above questions except the last two:

Reality: ALM tool has become a necessity in most of the projects because of the benefits and flexibility it brings in. Also, physical wall is not an option for a distributed team. It is the reality and you can't escape from ALM tools in many cases.

Single source of truth: As discussed in the issues section, we all love to have single source of truth than to end up with conflicting information.

Personal preference is not an option: I can't introduce a new language or a framework into a project since I am comfortable in using it. When we are working as a team, we discuss the pros/cons and decide on one tool/language/framework. If ALM tool can't be replaced, try to embrace it rather than fighting it and maintaining a physical wall.

Other benefits does not outweigh the issues: Few people claim that physical wall improves collaboration and some of them prefer to see stories as physical cards & track it through different lanes. On the collaboration part, I am not sure whether there is any fact to back this but I think ALM walls can also have the same effect. On the second benefit, most of the ALM tools provide standup / dashboard view, that mimics the physical wall. Even if we assume that the above benefits cannot be met with a tool, these are only side effects of a physical wall and it would take a back seat considering the issues it brings along.

Access to information immediately: Most of the ALM tools have long user session time, so we can keep a tab open in our browser(just like email/facebook) to refer whenever we need any information. So we can get necessary information in a click instead of the need to login everytime.

ALM tool usability (100 click syndrome): It would take a day or two to get used to any new tool. Most of the product development companies would have usability experts to make the interface as easy as possible to stay in business. If we could just spend that short time to get used to the tool, we won't be exaggerating and complaining about "100 clicks" needed to do an operation in ALM tool.

Information radiator: Like build monitor, if each team can project their card wall on a monitor, it would still act as an information radiator. Even though this cannot exactly replace the physical wall, it is a trade off decision we have to go with to see the benefits of using just the ALM tool.

And last, Does it really worth it: When you can get all the information that is needed from ALM tools dashboard/standup wall, why do we need to replicate the information on physical wall?

Even though we were able to try without physical wall successfully in a small team, I was not able to try it in my 20 member project because of the last argument(10 - Other issues take priority). But I would definitely try it in future.

You could help in refining this by sharing your comments, views and experience. Also, it would be great if you shed some light on how you overcame the issues in the hybrid approach I mentioned earlier.

Builders to setup Unit test data

2010-11-15T18:35:00.000-08:00

Exposing Setter methods or constructors in domain objects for setting up Unit test data should never be done.
Builder pattern is a simple alternate to setup unit test data using reflection without exposing unnecessary setters and constructors in domain objects.

1) Abstract builder:
Create an Abstract builder that will be extended by all domain builder objects.

import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class AbstractBuilder<E> {
    private E domain;

    public AbstractBuilder(E domain) {
        this.domain = domain;
    }

    public E build() {
        try {
            List<Field> domainFields = getAllFields(new ArrayList<Field>(), domain.getClass());
            Map<String, Field> builderFieldsMap = index(getAllFields(new ArrayList<Field>(), this.getClass()));
            for (Field domainField : domainFields) {
                Field builderField = builderFieldsMap.get(domainField.getName());
                if (builderField != null) {
                    copyFieldValue(domainField, builderField);
                }
            }
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return domain;
    }

    private void copyFieldValue(Field domainField, Field builderField) throws IllegalAccessException {
        domainField.setAccessible(true);
        builderField.setAccessible(true);
        domainField.set(domain, builderField.get(this));
    }

    private Map<String, Field> index(List<Field> allFields) {
        Map<String, Field> map = new HashMap<String, Field>();
        for (Field allField : allFields) {
            map.put(allField.getName(), allField);
        }
        return map;
    }

    public static List<Field> getAllFields(List<Field> fields, Class<?> type) {
        for (Field field : type.getDeclaredFields()) {
            fields.add(field);
        }
        if (type.getSuperclass() != null) {
            fields = getAllFields(fields, type.getSuperclass());
        }
        return fields;
    }

    public E getDomain() {
        return domain;
    }
}

2) Domain builder class:
Consider a domain class Person with 4 attributes (firstName, lastName, gender, dateOfBirth). You should create the domain builder object with same 4 attributes.


public class PersonBuilder extends AbstractBuilder<Person> {
    private String firstName;
    private String lastName;
    private String gender;
    private Date dateOfBirth;

    public PersonBuilder() {
        super(new Person());
    }

    public PersonBuilder withFirstName(String firstName) {
        this.firstName = firstName;
        return this;
    }

    public PersonBuilder withLastName(String lastName) {
        this.lastName = lastName;
        return this;
    }

    public PersonBuilder withGender(String gender) {
        this.gender = gender;
        return this;
    }

    public PersonBuilder withDateOfBirth(Date dateOfBirth) {
        this.dateOfBirth = dateOfBirth;
        return this;
    }
}

3) Using Builders in Unit Tests:

You can use the builder as shown below

Person testPerson = new PersonBuilder().withFirstName("hello").withLastName("World").build();

Tip:
If you are using Intellij IDEA as the IDE, use Builder plugin to generate the "with" methods quickly.

Redeploy application to Weblogic Server through Ant script

2010-11-15T16:04:00.000-08:00

The most time consuming task in a Web app development is to redeploy the application into the server for every change and test it.

In Weblogic Workshop IDE, we have an option to redeploy the app into Weblogic server by just right clicking it. But in Intellij Idea, I was not able to find such feature. We spent considerable amount of time to redeploy application using 'Weblogic Console' until we figured out the 'Ant' script to do it.

<project>
<property name="bea.home" value="C:/bea"/>
<property name="wl.home" value="${bea.home}/weblogic92"/>
<property name="wl.domain" value="domainName"/>
<property name="wl.server" value="serverName"/>
<!-- classpath to have weblogic.jar as the first jar -->
<path id="base.path">
<fileset file="${wl.home}/server/lib/weblogic.jar"/>
<fileset dir="${wl.home}/server/lib/">
<include name="**/*.jar"/>
</fileset>
<pathelement path="${java.class.path}/"/>
</path>

<!-- weblogic tasks-->
<taskdef name="wldeploy" classname="weblogic.ant.taskdefs.management.WLDeploy">
<classpath refid="base.path"/>
</taskdef>
<taskdef name="wlserver" classname="weblogic.ant.taskdefs.management.WLServer">
<classpath refid="base.path"/>
</taskdef>

<target name="deployApp">
<wldeploy action="deploy" verbose="true" debug="true"
name="sampleApp" source="z:/app/target/sampleApp.ear"
user="weblogic" password="weblogic"
adminurl="t3://localhost:7001" targets="${wl.server}"/>
</target>

<target name="redeployApp">
<wldeploy action="undeploy" verbose="true" debug="true" name="sampleApp" 
user="weblogic" password="weblogic"
adminurl="t3://localhost:7001" targets="${wl.server}"/>
<wldeploy action="deploy" verbose="true" debug="true"
name="sampleApp" source="z:/app/target/sampleApp.ear"
user="weblogic" password="weblogic"
adminurl="t3://localhost:7001" targets="${wl.server}"/>
</target>
</project>

TIP:
Use SUBST command in Windows to map your project location to a new drive. For eg.,

subst z: d:/venky/projects/utils/sampleApp

Individual developers in the team can checkout the code to any location but using this SUBST command will help the team to refer the project from same location like z:/src. Even though IDE's use relative path, it is nice to do this as the team can use these kinds of file without any customization.

Hibernate @OneToMany annotation filter by Latest date

2010-07-20T18:56:00.000-07:00

When there is a @OneToMany relationship between domain objects, a common requirement is to fetch the latest child record based on some date column or a version number column. This can be easily implemented in Hibernate using @Filter annotation and some SQL as shown below.

Consider a case where a Person has more than one Mobile phone and he always uses the latest mobile phone.

Person object has a @OneToMany relationship to MobilePhone object .

@Entity
@Table(name = "PERSON")
@FilterDef(name = Person.LATEST_PHONE_FILTER)
public class Person{
   public static final String LATEST_PHONE_FILTER = "latestPhoneFilter";

   @Column(name = "PERSON_ID")
   private Long id;

   @Column(name = "FIRST_NAME")   
   private String firstName;

   @Column(name = "LAST_NAME")
   private String lastName;

   @OneToMany(mappedBy = "owner", fetch = FetchType.EAGER)
   @Filter(name = LATEST_PHONE_FILTER, condition = "PURCHASE_DATE =
   (select max(M.PURCHASE_DATE) from MOBILE_PHONE M where M.PERSON_ID= PERSON_ID")
   private List<MobilePhone> mobilePhones;
   
   public MobilePhone getLatestMobilePhone(){
      return CollectionsUtil.isEmpty(mobilePhones) ? null: mobilePhones.get(0);
   }
}
@Entity
@Table(name = "MOBILE_PHONE")
public class MobilePhone{    
    ..............
    ..............
    @Column(name = "PURCHASE_DATE")
    private Date purchaseDate;

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "PERSON_ID")
    private Person owner;
}

If you notice the SQL used in the filter, it has a condition like M.PERSON_ID= PERSON_ID.
PERSON_ID column without alias name refers to the PERSON tables primary key. Hibernate takes care of giving proper alias name while generating SQL.

This configuration will always fetch only the latest mobile phone to the List of MobilePhone and the getLatestMobilePhone() method can return that object.

Recommended Technical Books

2010-05-29T12:09:00.000-07:00

Long time since I wrote any blog..
We recently got new technical books for our office library. ThoughtWorkers pitched in with their books recommendation and we ended up getting a great collection of books. I hope these recommended books will help others too.

Sl No.	Title (Sorted Alphabetically)	Author
1	About Face 3: The Essentials Of Interaction Design	Alan Cooper, Robert Reimann, David Cronin
2	Agile Estimating and Planning	Mike Cohn
3	Agile Software Development 2ed- The Cooperative Game	Cockburn
4	Agile Software Development, Principles, Patterns, And Practices	Robert C. Martin
5	Agile Testing: Practical Guide For Testers	Lisa Crispin
6	Bargaining For Advantage: Negotiation Strategies For Reasonable People 2nd Edition	G. Richard Shell
7	Clean Code: A Handbook Of Agile Software Craftsmanship	Robert C. Martin, Michael C. Feathers, Timothy R. Ottinger
8	Clr Via C#: Applied Microsoft.net Programming 2e	Richter J
9	Cocoa Design Patterns	Erik Buck, Donald Yacktman
10	Cocoa Programming Developer's Handbook (developer's Library)	David Chisnall, Scott Anguish, Erik Buck, Donald A. Yacktman
11	Concepts, Techniques, And Models Of Computer Programming	Haridi Seif
12	Concurrent Programming On Windows	Joe Duffy
13	Core Python Programming 2nd Edition	Chun
14	Design Patterns: Elements Of Reusable Object Oriented Software	Gamma
15	Designing Gestural Interfaces: Touchscreens And Interactive Devices	Dan Saffer
16	Designing Interfaces: Patterns For Effective Interaction Design	Jenifer Tidwell
17	Domain-driven Design	Eric Evans
18	Don't Make Me Think: A Common Sense Approach To Web Usability, 2nd Edition	Steve Krug
19	Emotional Design: Why We Love (or Hate) Everyday Things	Donald A. Norman
20	Enterprise Integration Patterns: Designing, Building, And Deploying Messaging Solutions	Gregor Hohpe, Bobby Woolf
21	Essentials Of Programming Languages,3/ed.	Daniel P Friedman
22	Ext Js 3.0 Cookbook	Jorge Ramon
23	Extreme Programming Explained 2nd Edition	Beck
24	Getting Things Done: The Art Of Stress-free Productivity	David Allen
25	Groovy And Grails Recipes (recipes: A Problem-solution Approach)	Bashar Abdul-jawad
26	Groovy Recipes	Scott Davis
27	Growing Object-oriented Software, Guided By Tests (beck Signature)	Steve Freeman, Nat Pryce
28	Gui Bloopers 2.0, Second Edition: Common User Interface Design Don'ts And Dos (interactive Technologies)	Jeff Johnson
29	Handbook Of Usability Testing: How To Plan, Design And Conduct Effective Tests, 2nd Ed	Jeffrey Rubin, Dana Chisnell
30	Head First Ajax	M. Rebecca Riordan
31	Head First Iphone Development	Dan Pilone, Tracey Pilone
32	Head First Javascript	Michael Morrison
33	Head First Object-oriented Analysis & Design	Brett Mclaughlin, Gary Pollice, David West
34	High Performance Web Sites: Essential Knowledge For Front-end Engineers	Steve Souders
35	Implementation Patterns (addison-wesley Signature)	Kent Beck
36	Influence: The Psychology Of Persuasion (collins Business Essentials)	Robert B. Cialdini
37	Introduction To Algorithms 3e	Thomas H Cormen
38	Iphone For Programmers: An App-driven Approach (deitel Developer)	Paul Deitel, Harvey Deitel, Abbey Deitel, Eric Kern, Michael Morgano
39	Iphone Hacks: Pushing The Iphone And Ipod Touch Beyond Their Limits	Damien Stolarz, Adam Stolarz, William Hurley, David Jurick, Jurick David, Stolarz Adam, Stolarz Damien
40	Iphone In Action: Introduction To Web And Sdk Development	Christopher Allen, Shannon Applecline
41	Iphone Sdk Development (the Pragmatic Programmers)	Bill Dudney, Christopher Adamson, Marcel Molina
42	Jasper Reports For Java Developers : Create.design,format,and Export Reports With The Wor	Heffelfinger
43	Javascript Programmer's Reference	Alexei White
44	Jquery Cookbook	Cody Lindley
45	Lean Software Development: An Agile Toolkit	Mary Poppendieck, Tom Poppendieck
46	Learning Ext JS	Shea Frederick, Colin Ramsay, and Steve 'Cutter' Blades
47	Lessons Learned In Software Testing	Bret Pittichord, James Bach, Cem Kaner
48	Made To Stick: Why Some Ideas Survive And Others Die	Dan Heath Chip Heath
49	Maverick!	Ricardo Semler
50	Modular Web Design: Creating Reusable Components For User Experience Design And Documentation	Nathan Curtis
51	Patterns Of Enterprise Application Architecture	Martin Fowler
52	Planning Extreme Programming	Kent Beck, Martin Fowler, Martin Fowler
53	Practical Common Lisp	Peter Seibel
54	Practices Of An Agile Developer	Venkat Subramaniam, Andy Hunt
55	Pragmatic Programmer, The	Andy Hunt, David Thomas
56	Professional Css: Cascading Style Sheets For Web Design	Christopher Schmitt, Mark Trammell, Ethan Marcotte, Todd Dominey, Dunstan Orchard, Todd Dominey
57	Programming F#: A Comprehensive Guide For Writing Simple Code To Solve Complex Problems	Chris Smith, Glen Smith
58	Programming Google App Engine: Build and Run Scalable Web Apps on Google's Infrastructure (Animal Guide)	Dan Sanderso
59	Programming In Objective-c: A Complete Introduction To The Objective-c Language	Kochan Stephen G.
60	Programming In Scala: A Comprehensive Step-by-step Guide	Martin Odersky, Lex Spoon, Bill Venners
61	Pulling Strings With Puppet: Configuration Management Made Easy	James Turnbull
62	Python For Unix And Linux System Administration	Noah Gift, Jeremy Jones
63	Refactoring Improving The Design Of Existing Code	Fowler Martin
64	Restful Webservices	Leonard Richardson, Sam Ruby, David Heinemeier
65	Selfish Gene 30th Indian Edn.	Richard Dawkins
66	Smalltalk Best Practice Patterns	Kent Beck
67	Software Testing: A Craftsman's Approach, Third Edition	Paul C. Jorgensen
68	Structure And Interpretation Of Computer Programs - 2nd Edition (mit Electrical Engineering And Computer Science)	Harold Abelson, Gerald Jay Sussman
69	Succeeding With Agile: Software Development Using Scrum (addison-wesley Signature)	Mike Cohn
70	Test-driven Development By Example	Beck
71	Testing Computer Software, 2nd Edition	Cem Kaner, Hung Quoc Nguyen, Jack Falk
72	Testing Extreme Programming	Lisa Crispin, Ken S. Rosenthal, Tip House
73	The Art Of Innovation	Tom Kelley
74	The Design Of Everyday Things	Donald A. Norman, Don Norman
75	The Knowledge-creating Company (harvard Business Review Classics)	Ikujiro Nonaka
76	The New Turing Omnibus: Sixty-six Excursions In Computer Science	A. K. Dewdney
77	The Psychology Of Computer Programming: Silver Anniversary Edition	Gerald M. Weinberg
78	What Customers Want: Using Outcome-driven Innovation To Create Breakthrough Products And Services	Anthony Ulwick
79	Windows Presentation Foundation Unleashed (wpf)	Adam Nathan, Daniel Lehenbauer
80	Working Effectively With Legacy Code	Feathers

Parsing Excel Sheet made easy with Annotations and POI

2010-01-28T20:32:00.000-08:00

HSSF - Horrible Spreadsheet Format – not anymore. With few annotations, excel parsing can be done in one line.

We had a requirement in our current project to parse multiple excel sheets and store the information to database. I hope most of the projects involving excel sheet parsing would be doing the same. We built a extensible framework to parse multiple sheets and populate JAVA objects with annotations.

We will discuss the steps to use annotations to parse excel sheet and populate Java objects.

Consider we have an excel sheet with student information.

Parsing Logic:
While parsing this excel sheet, we need to populate one “Section” object and multiple “Student” objects related to a Section. You can see that Student information is available in multiple rows whereas the Section details (Year, Section) is available in column B.

We will have to annotate the above information to the domain class, that can be interpretted by the sheet parser.

1) Annotate Domain Class:
First we will see the steps to annotate Section object:

@ExcelObject(parseType = ParseType.COLUMN, start = 2, end = 2)
public class Section {
 @ExcelField(position = 2)
 private String year;

 @ExcelField(position = 3)
 private String section;
 
 @MappedExcelObject
 private List <Student> students;
}

You can find three different annotation in this class.
i)ExcelObject: This annotation tells the parser about the parse type (Row or Column), number of objects to create (start, end). Based on the above annotation, Section value should be parsed Columnwise and information can be found in Column 2 (“B”) of the Excelsheet.

ii)ExcelField: This annotation tells the parser to fetch “year” information from Row 2 and “section” information from Row 3.

iii)MappedExcelObject: Apart from Simple datatypes like “Double”,”String”, we might also try to populate complex java objects while parsing. In this case, each section has a list of student information to be parsed from excel sheet. This annotation will help the parser in identifying such fields.

Next step is to annotate Student class:

@ExcelObject(parseType = ParseType.ROW, start = 6, end = 8)
public class Student {
 @ExcelField(position = 2)
 private Long roleNumber;
 
 @ExcelField(position = 3)
 private String name;
 
 @ExcelField(position = 4)
 private Date dateOfBirth;
 
 @ExcelField(position = 5)
 private String fatherName;
 
 @ExcelField(position = 6)
 private String motherName;
 
 @ExcelField(position = 7)
 private String address;
 
 @ExcelField(position = 8)
 private Double totalScore;
}

i) ExcelObject: As shown above, this annotation tells parser to parse Rows 6 to 8 (create 3 student objects). NOTE: Optional field “zeroIfNull” , if set to true, will populate Zero to all number fields (Double,Long,Integer) by default if the data is not available in DB.

ii) ExcelField: Student class has 7 values to be parsed and stored in the database. This is denoted in the domain class as annotation.

iii) MappedExcelObject: Student class does not have any complex object, hence this annoation is not used in this domain class.

Once the annotation is done, you have just invoke the parser with the Sheet and the Root class you want to populate.

//Get the sheet using POI API.
String sheetName = "Sheet1";
SheetParser parser = new SheetParser();
InputStream inputStream = getClass().getClassLoader().getResourceAsStream("Student Profile.xls");
Sheet sheet = new HSSFWorkbook(inputStream).getSheet(sheetName);

//Invoke the Sheet parser.
List entityList = parser.createEntity(sheet, sheetName, Section.class);

Thats all it requires. Parser would populate all the fields based on the annotation for you.

Crack PDF password - Java Brute force algorithm

2010-01-07T18:16:00.000-08:00

Recently I was in a big mess when I lost the password for an important PDF document. I ended up using the Brute force algorithm and cracked the PDF password.

PDFBox API is being used in the below code to open the PDF file. Please download the dependent JAR files from here.

Word of caution: Based on the length of the password, it might even take few days to run it.

package com.bruteforce;

import java.io.IOException;

import org.apache.pdfbox.exceptions.CryptographyException;
import org.apache.pdfbox.exceptions.InvalidPasswordException;
import org.apache.pdfbox.pdmodel.PDDocument;

public class BruteForceDecryptPDF {

 private static final String PDF_FILE_PATH = "c:/file2.pdf";
 private static final int PASSWORD_MAX_LENGTH = 4;
 private static final int PASSWORD_MIN_LENGTH = 2;
 // Array with characters to use in Brute Force Algorithm.
 // You can remove or add more characters in this array.
 private static char fCharList[] = { 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h',
   'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u',
   'v', 'w', 'x', 'y', 'z','0','1','2','3','4','5','6','7','8',
   '9'};

 public static void main(String[] args) {
  long startTime=System.currentTimeMillis();
  PDDocument document = null;
  boolean found = false;
  try {
   document = PDDocument.load(PDF_FILE_PATH);

   for (int i = PASSWORD_MIN_LENGTH; i <= PASSWORD_MAX_LENGTH; i++) { 
    try {
     startBruteForce(i, document);
    }catch(PasswordFoundInterrupter ex){
     found=true;
     System.out.println("SUCCESS. PASSWORD: - " + ex.getMessage());
    }
    catch (Exception e) {
     e.printStackTrace();
    }
   }
   if(!found){
    System.out.println("Failure - Password not found");
   }
   document.close();
   long timeTaken=System.currentTimeMillis()-startTime;
   System.out.println("TOTAL TIME TAKEN - " + timeTaken);
  } catch (IOException e) {
   System.out.println("PDF File not found at - "+ PDF_FILE_PATH);
   throw new RuntimeException(e);
  }
 }

 public static void startBruteForce(int length, PDDocument document) {
  StringBuffer sb = new StringBuffer(length);
  char currentChar = fCharList[0];

  for (int i = 1; i <= length; i++) {
   sb.append(currentChar);
  }

  changeCharacters(0, sb, length, document);
 }

 private static StringBuffer changeCharacters(int pos, StringBuffer sb,
   int length, PDDocument document) {
  for (int i = 0; i <= fCharList.length - 1; i++) {
   sb.setCharAt(pos, fCharList[i]);
   if (pos == length - 1) {    
    // Uncomment this to see the generated passwords. It may slow down if you are trying for larger size passwords.
    //System.out.println(sb.toString());
    if(valid(sb.toString(),document)){
     throw new BruteForceDecryptPDF.PasswordFoundInterrupter(sb.toString());
    }
   } else {
    changeCharacters(pos + 1, sb, length, document);
   }
  }

  return sb;
 }
 
 public static boolean valid(String password, PDDocument document) {
  System.out.println(password);
  try {
   document.decrypt(password);
   System.out.println("Success - " + password);
   return true;
  } catch (IOException e) {
   throw new RuntimeException(e);
  } catch (CryptographyException e) {
   return false;
  } catch (InvalidPasswordException e) {
   return false;
  }
 }
 
 private static class PasswordFoundInterrupter extends RuntimeException{
  public PasswordFoundInterrupter(String message) {
   super(message);
  }  
 }
}

Unit testing repository/DAO classes

2009-12-02T09:44:00.000-08:00

In an enterprise application we have many layers. As part of testing a layer, we would Mock the underlying layer objects and test it. But to test the DAO layer, we need to write integration test to perform DB operations.

1) Declare Spring context aware Abstract Test class:

@ContextConfiguration(locations = {"classpath:data-applicationContext.xml"})  
 @TransactionConfiguration(transactionManager = "transactionManager", defaultRollback = true)  
 @Transactional  
 public abstract class SpringTransactionTestContext extends AbstractTransactionalJUnit4SpringContextTests {  
 }

2) Extend this class in all DAOTest and Autowire DAO:

 public class StudentDaoImplTest extends SpringTransactionTestContext {  
   @Autowired  
   private StudentDaoImpl personDao;  
 }

3) Setup Test data:
If Student class is dependent on some School table, to test this class we need to insert School record.
Options:
i) Using Session factory:
Use SessionFactory instance and save School object as shown below:

@Autowired  
 private SessionFactory sessionFactory;  
 School school= new School("School Name");  
 sessionFactory.getCurrentSession().save(school);

Also, builders can be used to create objects with required test data. Look at this post on how to use builder pattern in Unit tests.

ii) Using SimpleJDBCTemplate - Less preferred:
SimpleJDBCTemplate object is found in . This can be directly used to run SQL scripts to insert into tables.

String schoolInsertQuery = "Insert into SCHOOL(ID,NAME) values (?,?)";  
 this.simpleJdbcTemplate.update(schoolInsertQuery, schoolId, schoolName);

Once the Test execution is completed, it would automatically rollback the transaction (As defaultRollback is set true).

Picking up latest/highest record from a group using rank() function

2009-11-24T18:26:00.000-08:00

Consider a example where we have Student, Subject and Mark information stored in different tables.

If we need to retrieve the toppers in every subject for a given year through SQL, the usual way is to use a Subquery, sort the result descending and pick up the ROWNUM=1 record.

Instead of doing all that, Oracle's rank() function can help us to achieve the same in a simple way.

Table Structure:
- STUDENT (ID, Name, Class)
- SUBJECT (ID, Name, Year)
- MARK (Subject_ID, Student_ID, Score)

SQL

 SELECT STUDENT.*, MARK.Mark, SUBJECT.Name  
 FROM STUDENT, SUBJECT,  
 (SELECT *, rank() OVER( partition by Subject_ID order by Score desc) rank FROM MARK )MARK  
 WHERE MARK.Student_ID=STUDENT.ID  
 AND MARK.Subject_ID=SUBJECT.ID  
 AND SUBJECT.Year=10  
 AND rank=1

In the above example, instead of joining MARK table directly, we use Oracle rank() function to get the rank column also.

Here we partition the table by Subject_Id, apply the descending order by Score and then rank the result. When we add the WHERE clause on rank=1 it would fetch the toppers in every Subject.

This rank() function is also handy to get the latest record based on date from a given group.

Testing Hibernate list() DAO method which fetches all records.

2009-11-24T17:45:00.000-08:00

Consider the example where you have a static table "VERSION" and the DAO method that fetches all records from VERSION table through Hibernate.

DAO Method:

   public List getAll() {  
     Criteria crit = getSession().createCriteria(Version.class);  
     crit.addOrder(Order.asc("name"));  
     return crit.list();  
   }

Poor Test case:
Many occasions I have noticed Test cases that directly assert on the number of records being returned.

 public class VersionDaoImplTest extends AbstractTransactionalJUnit4SpringContextTests{  
   @Autowired  
   private VersionDao versionDao;  
   @Test  
   public void shouldReturnTheListOfVersions() {  
     List versions = versionDao.getAll();  
     assertEquals(5, versions.size());  
   }  
 }

As seen above, this test case tries to assert Only 5 records are there in the Version Table. When a new record is added to this table, this test case would fail.

Proper test case:
Other way of testing this method is given below.

 @Test  
   public void shouldReturnTheListOfVersions() {    
   List versions = ratingsDao.versionDao.getAll();  
     assertNotNull(ratings);  
     int prevSize = ratings.size();  
     String insertSQL = "Insert into VERSION (NAME,VERSION_NO) values (?,?)";  
     this.simpleJdbcTemplate.update(insertSQL, "TST+", 95);  
     this.simpleJdbcTemplate.update(insertSQL, "TST", 96);  
     this.simpleJdbcTemplate.update(insertSQL, "TST-", 97);  
     versions = versionDao.getAll();  
     assertEquals(prevSize + 3, ratings.size());  
   }

In this test case, we get the number of records currently in the database. We then insert 3 new records into the table and try to assert whether the number of records being returned has increased by 3.

Testing Sort Order:
Also to test the Criteria.addOrder(Order.asc("name")) you can enhance the test case with the following:

 for (int i = 1; i < versions.size(); i++) {  
       assertTrue(versions.get(i - 1).getName().compareTo(versions.get(i).getName()) < 0);  
 }

This would assert that records are sorted by the NAME.

Spring MVC - Dynamic list binding and validating the input data

2009-10-05T11:00:00.000-07:00

It took quiet some time to figure out how to display a dynamic list of data on screen and get user input information back. We also had to perform validation on this dynamic list input data and display error message on the corresponding row.
Here is the simple example to achieve the above requirement.

Consider that you need to display Parent and list of child information on a screen and get the Age and Sex information of each child. Also you need to show error message on the screen if the user has not entered the age or sex information.

1) Form bean

public class MotherFormBean{  
   private String name;  
   private List<Child> children;  
   public MotherFormBean() {  
     children= LazyList.decorate(new ArrayList(),  
       FactoryUtils.instantiateFactory(Child.class));  
   }  
 }  
 public class Child{  
   private String name;  
   private String age;  
   private String sex;  
 }

Instead of initializing the List as normal ArrayList, use the LazyList class (Commons-collection package). This is to avoid "IndexOutOfBoundException" when the data is submitted back to the controller.

2) JSP

<form:form action="${formAction}" name="childrenDetails">  
 <table><thead>  
  <tr>  
   <th><b>Name</b></th>  
   <th><b>Sex</b></th>  
   <th><b>Age</b></th>  
  </tr>  
  </thead>   
  <tbody>   
   <c:foreach items="${command.children}" var="child" varStatus="loop">   
    <tr>   
     <td><c:out value="${child.name}"></c:out>  
       <form:hidden path="children[${loop.index}].name"/>   
     </td>  
     <td><form:select path="children[${loop.index}].sex">  
        <option value="">--Please Select--</option>  
        <option value="M">Male</option>  
        <option value="F">Female</option>   
       </form:select>  
     </td>  
     <td><form:input path="children[${loop.index}].age"/></td>   
    </tr>  
   </c:foreach>   
  </tbody>   
  </table>  
 </form:form>

This code will print all the children names and show input fields for Age and Sex field.

3) Validator Class

When the List data is submitted back, Spring constructs the data back to the Form bean with help of the LazyList collection. Validation and raising error on these fields is very simple.

 public class MotherBeanValidator implements org.springframework.validation.Validator {  
 .......   
   public void validate(Object target, Errors errors) {  
     MotherFormBean bean = (MotherFormBean) target;  
     List<Child> children = bean.getChildren();  
     for (int i = 0; i < children.size(); i++) {  
       if (StringUtils.isEmpty(children.get(i).getSex()))   
        errors.rejectValue("children[" + i + "].sex", "select.sex");  
       if (StringUtils.isEmpty(children.get(i).getAge()))   
        errors.rejectValue("children[" + i + "].age", "select.age");  
     }  
   }  
 }

JSP Changes:
Include <form:errors> tag after the sex and age input fields.
eg.)

 <form:errors path="children[${loop.index}].sex"/>

Hope this helps you in displaying the Collection data using Spring MVC and validating it.

Thanks to Matt Flemings blog for shedding some light on the LazyList.

Google Code Jam 2009 experience

2009-09-13T07:26:00.000-07:00

   I participated in Google code Jam 2009 and tried to solve such problems for the first time in my life. As expected I could go only till Round 1 but it has been a great experience to participate in this event.

   It was amazing to see people solving the problems within few minutes which I had no idea where and how to start. I was particularly impressed by "ACRush" (Winner of Google Code Jam 2008) - before I could even understand the "The next number" problem in Round 1B he submitted the solution for it in 6 mins. Google has again amazed me by providing such a platform for people to compete. The best part of it was making the solution submitted by all users available for download once the competition is over.


   Having considerable professional experience in IT, I was hoping to fair better. But while attending the contest I felt that it had its own deterring factors. I was trying to visualize everything as Java objects and wasting time in modularizing code and naming it properly. Also having no exposure to such problems took its toll while racing against time. When I looked at the solutions from other users, most of them have used multi-dimensional arrays to solve the problems. Apart from my college days, I had almost never used multi-dimensional arrays in any of the professional projects. "Complex business logic" would typically involve complex queries and triggering different actions when an event occurs. We don't write algorithms or solve problems like these in a typical web application. Also maintaining state in multi-dimensional arrays is more prone to defect which would explain my lack of exposure to it.

Another interesting thing about Google Code Jam is the competition statistics - by Country and Programming language. I was surprised to find that only 1220 contestants from India competed in the qualifiers second to China which had 1324 contestants. But in round one, only 6 from India had the "Perfect Score". I assume most of the university students are not aware of this competition. Google should promote this event better as it is a great platform for students to compete against the world's top coders.
Programming language: Java - 499 people but C++ - 1200 people. C++ has comfortably outnumbered Java as the preferred programming language in the code jam competition. Not happy to see this as a java developer :)

I would never miss Google Code Jam competition again. I hope that I would fair better in 2010 competition. Congratulations to all the participants of the code jam and thanks to Google for providing this opportunity. Keep going.

Consultant - "Doing things quickly" - Not really

2009-08-26T12:51:00.000-07:00

I was working as a consultant from an Indian IT company at a client location in Melbourne for almost three years. It was an exiting place to work as I was given opportunity try out new technologies for the project.

As a consultant, I had the following perspective on my role and obliged with it:

I had to quickly learn the tools and technologies used at my client place (like., Weblogic Portal, Spring, IBatis, Clearcase, etc)
I had to do my job quickly to add value to my customer.
I had to understand the project functionalities quickly.
I had to adapt to new technologies quickly and start contributing.

But only when my clients hired three consultants from ThoughtWorks, I realized how little was my value add to the customer apart from my "doing things quickly".

I had gravely missed to think on these lines as a consultant...

1. Are we using the right tools? Are there better tools out there? Clearcase version control - it is being used at my client place for more than 5 years. After the initial hiccups, I got used to it and tried to solve our problems within the tools limits. I never had a clue that this could be challenged. But TW, proposed Subversion as an alternate for development environment and integrated with Subversion to Clearcase behind the scene. Introduction of this SVN to the Test driven development environment is claimed to have helped the developers is refactoring the code at ease.

2. Are we using the right technology? Can it be done in a simple way with other technology?
Weblogic Portal was traditionally used at my client place for more than five years. It is a good technology to work on if you are familiar with it. It had so many features and good IDE to develop things.
But we had a big learning curve with Weblogic portal in our first project and it has always been difficult for new developers coming into project. Few developers tried proposing alternate technologies but was not able to convince everyone.
Again TW consultants, annoyed with the time taken to learn,build and deploy portal applications, proposed Spring MVC as the alternate technology. But this time they presented it with a case study comparing the effort involved in developing a functionality in Weblogic portal and Spring MVC. As it directly translated to $$$ terms, it caught the attention of everyone and Spring MVC was inducted into clients technology stack.

3. Are we up-to-date with technology?
Being up-to-date with technology not only helps in doing things in a smarter way but also leads to doing things quicker. I am still wondering how few people are so familiar with the latest open source technologies. For example, which tool will you use for XML processing?

Do not choose the opensource library just because you have used it earlier.
Look out for better opensource libraries which will help you to achieve the result in a smart way.
Be up-to-date about the new libraries.

I never imagined that a technology like "Weblogic Portal" which was considered as the standard for all applications could be replaced. I never thought a tool like "Clearcase" which is being used for 5 years can have an alternate.The lesson I learnt from this experience is "Never take anything for granted".

I always thought that I had to learn the technologies used by the client but I have now realized that it is also my duty to identify what is right for the client.