The Art and Craft of Great Software Development

My Ideal Greenfield Development Platform: Now vs. 5 Years ago

2013-01-30T09:39:00.000-05:00

As I've grown as a developer (sometimes two steps forward and one step back - ha ha!) I've had the privilege to work with some very smart people - not just "IQ" intelligent but "savvy" people who saw where things were going technology wise. So I've learned a lot in the last five years about my technology preferences - sometimes by choice, sometimes by necessity - in some cases just playing catch up (e.g. JavaEE vs. Spring) and bleeding edge in others (e.g. Memcached, JBehave).

As a result of that experience, every so often I dream of starting a project from scratch and imagine what technologies I would choose to use for my Java stack based on what I know now. Admittedly what's below is a very Java centric stack and I need to work on looking into Ruby on Rails / Python & Django to broaden my skill set too etc.

Anyway I hope you find the following table an interesting comparison of technology choices now vs. 5 years ago. In many cases the "Then" technology is still around and still viable. Typically what makes the "Now" technology more attractive is a fantastic combination of lower price and better / faster (and free) support - after that features and speed of execution are winning attributes.

Component	Then (5 Years Ago)	Now (2013)	Why?
Middle-Tier Framework	JavaEE (JBoss)	Spring	In 2005-2006 I went from WebSphere to preferring JBoss because IBM could not move fast and the App Serve was slow. Then I found JBoss also had some issues - less of them - but enough. Just figuring out which JMS messaging solution they would implement in a release was a chore (JBossMQ? JBoss Messaging? HornetQ?). Spring is so much faster in terms of performance, has fewer issues and has faster release schedules and no "two tier" system GA vs. supported
Deployment Environment	Your Environment / Your Data Center	The Cloud (AWS)	You need to scale on demand these days and pay only for what you need - and that includes Ops folks, DBAs etc. And why AWS as opposed to someone else - simple - maturity of their offering.
Build Tool	Ant	Maven	Why? "Convention over configuration" although I have found Maven and its plugins a bit more buggy than those for Ant. So its not clear cut.
Build Environment	Local Builds	Jenkins	What do you live under a rock? If you aren't doing continuous builds with all your tests automated and hooked in (unit, integration, acceptance, performance) you're crazy! :-)
Relational Database	Oracle	MySQL	There's no reason to pay for a database anymore - Facebook runs on MySQL for crying out loud - InnoDB too!
Key-Value Store / NoSQL	None	MySQL or DynamoDB (or Couchbase)	Sometimes you need a Relational DB and sometimes you just need a KV Store. Well since I'm all about AWS you gotta start with DynamoDB. But with AWS you gotta implement your inter-region replication yourself - however a up and comer that impressed me and one to look at is Couchbase.
Caching Tier	Roll your own / Terracotta	Memcached	Fast, cheap and great support on the web.
Web Container	Tomcat	Tomcat or NGINX	Tomcat still rocks and I haven't built an app yet where Tomcat was the bottleneck but NGINX is getting some play and is worth a look
Unit Testing	Junit	JUnit	JUnit always rocks! Always will!
Functional Testing	WinRunner/LoadRunner etc.	JBehave	Again in the spirit of "Fast, Cheap" and hooks into JUnit JBehave is emerging as a great BDD tool. Hopefully it will keep emerging and develop some snazzy reports.
Bug Tracking	Roll Your Own / ClearQuest	JIRA	JIRA just works . . . very well. I wish the Scrum interactions were a bit better (Grasshopper I find non-intuitive at times).
Source Code Control	Perforce	Git	Git is free, fast and branches are easy. It's a bit hard to get used to but once you do you'll never look back. Oh and GitHub.
IDE	Eclipse	Eclipse	As much as I'd love to switch to IntelliJ the plugins and support just don't match Eclipse (esp. for AWS)
UI	Web (HTML/Javascript/CSS)	Web AND Native Mobile	I still love the web and Javascript but HTML5 is still in its infancy. With Mobile web and app usage skyrocketing the best way to get good performance is (for now) to go Native. Still perhaps in 3 or 4 years HTML5 support will be better.
Messaging Platform	JMS	SQS	Again since I love AWS I gotta go with SQS but what I really love with it is it's failover capabilities (backed by 3 copies in S3) and it's separation of read vs. delete. Brilliant. Their Pub-Sub solution (SNS) is equally great.
Testing	50% Manual, 50% Automated	100% Automated	Speed matters - automation + continuous integration (and ideally continuous release) is critical to that end.

I could go into more stuff like Team communication / projects (e.g. Sharepoint vs. Wikis vs. SaaS providers) but I haven't seen anything that makes me go "Wow" - although after my experience with JIRA I'd probably start with Atlassian stuff.

I'm sure in 5 years time I will be making newer and more informed choices. The great thing about Software is all parts of it are always on the move. The hard thing about Software is trying to keep up with the same! :-)

I'd like to hear people's thoughts on the above, their own personal experiences, preferences and if they are aware of anything I've forgotten or if they have questions about technology X?

I wish Steve McConnell was on Twitter . . . . Book review of "Software Estimation"

2012-09-16T16:42:00.001-04:00

Published over 6 years ago "Software Estimation" by Steve McConnell is a great read.
As a practitioner of the agile arts I must say in reading it now this book seems like the last great attempt to "fix" waterfall and "big design up front" (BDUF) methodologies which were known for their very distinct big phases of requirements, design, development, testing and release. The kicker for these techniques was often that the development and testing estimates were VERY far off. If they followed McConnell's advice as he laid out in this book they'd have had more success.

Agile basically works around many of the problems McConnell tries to solve by focusing on short iterations (of < 4 weeks) with new releasable and functional software produced at the end. Basically it avoids many of the risks inherent in Waterfall/BDUF by making the software cycle "too small to fail".

That said there are a great many things still to be learned from Steve's great book. Waterfall and BDUF are by no means dead and even so there are some lessons here about the inherent nature of (errors in) estimations by developers. Even in agile I have experienced serious under-estimation (by a factor of 2x or 3x) by developers - where a story that should have taken 1 sprint takes 3 or more. So we still have much to learn. However the key theme this book drove home to me was pretty much that "Software estimation is so hard that we pretty much gave up and are doing short iterations because that's the most we can estimate". I am sure that wasn't Steve's point but that was my inference because since the book's release estimation has taken the back-burner to story points, burn down charts, stand-ups and sprints.
More people are becoming Scrum masters and fewer are taking PMP

Agile, PMP Job Trends

Agile jobs - PMP jobs

In this long blog article I try to capture some of the key learnings I made from this book

Part 1: Critical estimation concepts

CHAPTER 1: What Is an "Estimate"?

Tip #1: Distinguish between Estimates, Targets and Commitments

When business people are asking for an "estimate", they're really often asking for a commitment or a plan to meet a target
Estimation is not planning
When you see a "single point estimate" ask whether the number is an estimate or whether its really a target
A common assumption is that the distribution of software project outcomes follows a bell curve. The reality is much more skewed.
What is a good estimate?
The approach should provide estimates that are withing 25% of the actual results 75% of the time
Events that happen during the project always invalidate the assumptions that were used in the estimate.
The primary purpose of software estimation is to determine whether a project's targets are realistic enough to allow the project to be controlled to meet them. The executives want a plan to deliver as many features as possible by a certain date.
"A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets"

CHAPTER 2: How Good an Estimator are you?

Studies have confirmed that most people's intuitive sense of "90% confident" is really comparable to something closer to "30% confident"
Where does the pressure to use narrow ranges come from? You or an external source?

CHAPTER 3: Value of Accurate Estimates?

Is it better to overestimate or underestimate?
If a project is overestimated, stakeholders fear that Parkinson's law will kick in - work will expand to fill the time allotted.
Another concern is given too much time, developers will procrastinate until late in the project.
A related motivation for underestimating is the desire to instill a sense or urgency
Figure 3.1: The penalties for underestimation are more severe than the penalties for overestimation
The Software Industry's Estimation Track Record

Failure rates
1000 LOC 2%
10,000 LOC 7%
100,000 LOC 20%
1,000,000 LOC 48%
10,000,000 LOC 65%

The Software industry has an underestimation problem.
What top executives value most is predictability - business need to make commitments to customers, investors, suppliers, the marketplace and other stakeholders

CHAPTER 4: Where Does Estimation Error Come From

Four Generic sources

Inaccurate information about the project being estimate
Inaccurate information about the capabilities of the organization that will perform the project
Too much chaos IN the project to support accurate estimation (i.e. try to estimate a moving target)
Inaccuracies arising from the estimation process itself

Simple example of a Telephone Number checker and the requirements questions / uncertainties that could result in very different design approaches.
The cone of uncertainty

Initial Concept 0.25x to 4x (Range = 16x)
Approved Product Definition 0.5x to 2x (Range = 4x)
Requirements Complete 0.67x to 1.5x
UI Design Complete 0.8x to 1.25x
Detailed Design Complete 0.9x to 1.1x

The cone of uncertainty is the BEST-case accuracy possible to have. It isn't possible to be more accurate - it's only possible to be more lucky.
The cone does not narrow itself - if a project is not well controlled you can end up with a cloud of uncertainty that contains even more estimation error.
"What you give up with approaches that leave requirements undefined until the beginning of each iteration if long-range predictability"
Sources of project chaos

Requirements that were not investigated very well
Poor designs leading to lots of code rewrite
Poor coding practices leading to extensive bug fixing
Inexperienced personnel
Incomplete or unskilled project planning
Prima Donna team members
Abandoning planning under pressure
Developer gold-plating
Lack of source code control software

In practice, project managers often neglect to update their cost and schedule assumptions as requirements change.
Omitted Activities (pp.44)

Missing Requirements

Non functional requirements: Accuracy, modifiability, Performance, Scalability, Security, Usability etc.

Missing software-development activities

Ramp-up time for team members
Mentoring
Build & Smoke Test support
Requirements clarification
Creating test data
Beta program management
Technical reviews
Integration work
Attendance at meetings
Performance tuning
Learning new tools
Answering questions
Reviewing technical documentation etc.

Missing non-software-development activities

Vacations, Holidays, Sick days, Training, Weekends(!?!?)
Company meetings, department meetings, setting up new workstations

Developer estimates tend to contain an optimism factor of 20 to 30%. Although managers complain that developers sandbag their estimates - the reverse is true. Boehm also found a "fantasy factor" of 1.33

CHAPTER 5: Estimate Influences

The obvious one: Project Size
Diseconomies of scale a 1M LOC project takes more than 10x the effort of a 100k LOC project.
The basic issues is that in larger projects coordination among larger groups of people requires more communication. As Project size increases, the number of communication paths among people increases as a SQUARED function of the number of people on the project.

Lines of code per staff per year

10k LOC project --> 2k to 25k
100k LOC project --> 1k to 20k
1M LOC project --> 700 to 10k
10M LOC project --> 350 to 5k
Other influences: The kind of software being developed
Personnel factors

According to Cocomo II on a 100k LOC project the combined effect of personnel factors can swing a project estimate by as much as a factor of 22!
The KEY personnel decision: Requirements Analyst Capability and only THEN the programmer
The magnitude of these factors has been confirmed in numerous other studies

Other influences: Programming Language
Lots of other adjustment factors: See table 5-4 on page 66
Key Learning: Small and Medium-sized projects can succeed largely on the basis of strong individuals. Large projects however still need strong individuals but project management, organizational maturity and how well the team coalesces are just as significant.

PART II: Fundamental Estimation Techniques

CHAPTER 6: Introduction to Estimation Techniques

Considerations in choosing estimation techniques

What's being estimated - features, schedule, effort
Project Size

Small: < 5 total technical staff. Best estimates are usually "botom-up" techniques created by individuals who will do the actual work
Large: 25+ people that lasts 6 to 12 months or more. For these teams the best estimation approaches tend to be "top-down" approaches in the early stages. As the project progresses more bottom-up techniques are introduced and the projects own historical data will provide more accurate estimates.
Medium: 5 to 25 people lasting 3 to 12 months. Can use any of the techniques above.

Software Development Style: Iterative vs. Sequential

Evolutionary Prototyping
Extreme Programming
Evolutionary Delivery
Staged Delivery
RUP
Scrum

CHAPTER 7: Count, Compute, Judge

Count first
Count if at all possible, compute when you can't cout. Use judgement alone ONLY as a last resort
What to count? Find something to count that's highly correlated with the size of the software you are estimating. And find something to count that is available sooner rather than later in the development.
Historical data

Average effort hours per requirement for development
Average total effort hours per use case / story
Average dev/test/doc effort per change request

CHAPTER 8: Calibration and Historical Data

Used to convert counts to estimates - lines of code to effort, user stories to calendar time, requirements to number of test cases
Your estimates can be calibrated using any of three kinds of data

Industry data
Historical data
Project data

Using data helps avoid subjectivity, unfounded optimism and some biases.
It also helps reduce estimation politics
Start with a small set of data

Size (LOC)
Effort (Staff months)
Time (Calendar months)
Defects (classified by severity)

Be careful how you measure e.g. 8 hour work days? How about vacations? Overtime?
It is surprisingly difficult in many organizations to determine how long a particular project lasted

CHAPTER 9: Individual Expert Judgment

To create the task-level estimates, have the people who will actually do the work create the estimates
When estimating at the task level decompose estimates that will require no more than about 2 days of effort. Tasks larger than that will contain too many places that unexpected work can hide. Ending up with estimates that are 0.25 to 0.5d of granularity is appropriate.
Use Ranges to help identify risks and where things can (and often do) go wrong

Best Case
Most Likely Case
Worst Case
Expected Case

Estimate Checklist

Is what's being estimated clearly defined?
Does the estimate include all the KINDS of work needed to complete the task?
Does the estimate include all the FUNCTIONALITY AREAS needed to complete the task?
Is the estimate broken down into enough detail to expose hidden work?
Have you looked at notes from past work rather than estimating from pure memory?
Is the estimate approved by the person who will actually do the work?
Is the productivity assumed in the estimate similar to what has been achieved on similar assignments in the past
Does the estimate include a Best Case, Worst Case and Expected Case?
Have the assumptions in the estimate been documented?
Has the situation changed since the estimate was prepared?

Compare actual performance to estimated performance so that you can improve estimates over time.

CHAPTER 10: Decomposition and Recomposition

The key is if you create several smaller estimates some of the estimation errors will be on the high side and some will be on the low side. The errors will tend to cancel each other out to some extent. Research has found that summing task durations was negatively correlated with cost and schedule overruns.
Since developers tend to give near-Best case estimates, schedule overruns often compound on one another since the chance of each of the estimates coming in as scheduled is so very low.

CHAPTER 11: Estimation by Analogy

Get detailed size, effort and cost results for a similar previous project
Compare the size of the new project to a similar past project
Build up the estimate for the new project's size as a percentage of the old project's size
Create an effort estimate based on the size of the new project compared to the previous project
Check for consistent assumptions across the old and new projects

CHAPTER 12: Proxy-based Estimates

Fuzzy Logic

Very Small
Small
Medium
Large
Very Large

As a rule of thumb the differences in size between adjacent categories should be at least a factor of 2
Story Points e.g. Fibonacci sequence.
Cautions about rating scales - the use of a numeric scale implies that you can perform numeric options on the numbers: multiplication, addition, subtraction and so on. But if the underlying relationships aren't valid - that is a story worth 13 points doesn't really require 13/3 as much effort as a story worth 3 points - then performing numeric operations on the 13 isn't any more valid than performing numeric operations on "Large" or "Very Large"
T-Shirt Sizing

Remember that the goal of software estimation is not pinpoint accuracy but estimates that are accurate enough to support effective project control
In this approach developers classify each feature's size relative to other features as Small, Medium, Large, XL etc.
This allows the business to trade-off and look for features with the most business value and lowest development cost.

CHAPTER 13: Expert Judgment in Groups

Group Reviews

Have each team member estimate pieces of the project individually, and then meet to compare estimates
Don't just average estimates - discuss the differences
Arrive at a consensus estimate that the whole group accepts

Individual estimates have a Magnitude of Relative Error (MRE) of 55%.
Group-reviewed estimates average an error of only 30%
Studies have found that the use of 3 to 5 experts with different backgrounds seems to be sufficient.
Wideband-Delphi Technique

CHAPTER 14: Software Estimation Tools

Allows you to simulate different project outcomes
Data you'll need to calibrate tools

Effort in staff months
Schedule, in elapsed months
Size, in lines of code

Summary of available tools - see pp.163 (valid as of 2006)

CHAPTER 15: Use of Multiple Approaches

Use multiple estimation techniques and look for convergence or spread among the results

CHAPTER 16: Flow of Software Estimates on a Well-Estimated Project

When you reestimate in response to a missed deadline base the new estimate on the project's ACTUAL progress not on the project's planned progress.

CHAPTER 17: Standardized Estimation Procedures

Estimation should be fit into a Stage-Gate process

Discovery

Approved preliminary business case

Scoping

Approved product vision
Approved marketing requirements

Planning

Approved software development plans
Approved budget
Approved final business case

Development

Approved software release plan
Approved marketing launch plan and operations plan
Approved software test plan
Pass release criteria

Testing and Validation

Pass release criteria

Launch

The process should

Emphasize counting and computing rather than use of judgement
Calls for the use of multiple estimation approaches
Communicates a plan at predefined points
Contains a clear description of an estimate's inaccuracy
Defines when an estimate can be used as the basis for a project budget
Defined when as estimate can be used as the basis for internal or external commitments.

PART III: Specific Estimation Challenges

CHAPTER 18: Special Issues in Estimating Size

Using Lines of Code in Size estimation (data is easily collected but translation into "staff months" of effort is error prone)
Function-Point Estimation

The number of function points in a program is based on the number and complexity of
External inputs (e.g. screens, forms, dialog boxes)
External outputs (e.g. screens, reports, graphs etc)
External queries
Internal Logical files
External interface files

CHAPTER 19: Special Issues in Estimating Effort

Productivity variations among different kinds of software projects can show very different effort estimates (per LOC) and cost (per LOC)

CHAPTER 20: Special Issues in Estimating Schedule

Basic Schedule Equation

Schedule In Months = 3.0 x cubeRoot(StaffMonths)

Schedules compress and the shorted schedule

If the feature set is not flexible, shortening the schedule depends on adding staff to do more work in less time
Numerous estimation researchers have investigated the effects of compressing a nominal schedule.
All researchers have concluded that shortening the nominal schedule will increase total development effort.
There is also an impossible zone and you can't beat it - the consensus of researchers is that schedule compression of more than 25% nominal is not possible
Similarly you can reduce costs by lengthening the schedule and conducting the project with a smaller team
Lawrence Putnam conducted fascinating research on the relationship between team size, schedule and productivity.
Schedule decreases (and effort increases) as you add team members - until you hit 5-7 on a team. After that the effort goes up very much more quickly and schedule ALSO starts to get longer.
Thus a team size of 5 to 7 people appears to be economically optimal for medium-sized business system projects.

CHAPTER 21: Estimating Planning Parameters

Estimating Architecture, Requirements, Management effort for projects of different sizes. The larger the project the more the architecture, test, requirements and management costs.
Developer-to-test ratio is settled more by planning than by estimation - that is it is determined more by what you think you SHOULD do than by what you predict you will do.
Good analogy about ideal time and planned time: football game - 60 minutes vs. 2 to 4 hours elapsed time.
Defect Removal

Formal Design Inspections: 55% rate of removal (mode)
Informal design review: 35%
Formal code inspection: 60%
Informal code review: 25%
Low Volume (< 10 sites) Beta Test: 35%
High Volume (> 1,000 sites): 75%
System Test: 40%

Other rules of thumb

To go from one-company, one-campus development to multi-company, multi-cit: allow for 25% increase in effort.
To go from one-company, one campus development to international outsource, allow for a 40% increase in effort.

CHAPTER 22: Estimate Presentation Style

Communicating Estimate Assumptions

Which features are in scope
Which features are out of scope
Availability of resources
Dependencies on 3rd-parties (and their performance)
Unknowns

Expressing Uncertainty
Try to present your estimate in units that are consisten with the estimate's underlying accuracy
Ranges are the most accurate way to reflect the inherent uncertainty in estimates at various points in the Cone of uncertainty.
Do not present a commitment as a range, a commitment needs to be specific

CHAPTER 23: Politics, Negotiation and Problem Solving

Estimate negotiations tend to be between introverted and more junior technical staff and seasons professional negotiators.
Understand that executives are assertive by nature and by job description and plan your estimation discussions accordingly.
You can negotiate the commitment but do NOT negotiate the estimate
Educate nontechnical stakeholders about effective software estimation practices
Treat estimation discussions as problem solving, not negotiations. Recognize that all project stakeholders are on the same side of the table. Everyone wins, or everyone loses.
Getting to Yes

Separate the people from the problem
Focus on interests, not positions
Invent options for mutual gain
Insist on using object criteria

Frank's Summary

This is a great book I should have read a few years ago. Everyone should. Even if you are doing agile development there are tons of great tips and tricks (e.g. effectiveness of design & code inspections, using best/worst case estimates, negotiation techniques) that are useful regardless.

Like I said above I think the rise of agile techniques pretty much indicates, at least to me, that most software practitioners do not have the patience, determination and doggedness to follow the practices McConnell outlines above. Because of that their estimates (and thus their perceived performance by their customers) is poor. Agile methods especially Scrum and Kanban have achieved success by trying to limit the cognitive planning and estimating load - keeping the process simple and light and "result" focused (ship!). What I like to call "Too Small To Fail". Even so a lot of organizations have trouble adopting agile and need help. There are various reasons for this but they are the same reasons their other processes were flawed - the problem is not in the process itself but how it is being executed and the ability of those trying to do the execution.

I just wish Steve McConnell was on twitter though - I could do with a daily dose of the knowledge and wisdom he puts into his books.

Securing your data in the cloud- key management hell

2012-09-01T10:47:00.000-04:00

Invariably when you start migrating to the cloud you'll find you need to encrypt some or all of your data that you store there. Apart from the performance hit this seems easy right?
Except when you start to think of how you are going to manage keys used to encrypt / decrypt.
Because your Data and your business logic (app servers) are no longer in your control you can't just leave your keys on your app server (EC2) instances. If a hacker compromises those data keys then they can access your data. The same is true in normal in-house environments but at least you can trust the folks who run your data center - or at least there is direct accountability - you can fire them or pursue them legally. If an AWS person in Tokyo for example goes rogue what is your recourse?

So your data encryption keys THEMSELVES need to be encrypted. OK not a big deal.
But now where do you store the "master" keys to decrypt the data keys . . . . and so on. Maybe you store the master keys in your non-cloud environment and call out from EC2 to get them (but now you could be subject to another type of attack). So far I haven't heard a good architectural solution (barring something like human based Two factor authentication required when starting up an EC2 instance? But now your auto-scaling is hosed).

Anyone have any ideas or see any workable reasonable solutions?

Architecting in the Cloud

2012-03-20T16:03:00.000-04:00

So I am starting to investigate architecting an app for AWS. I have the basics of EC2, S3, SQS, RDS, Simple DB (now replaced by DynamoDB). But I had some lingering questions about architecting solutions for the cloud as I am always a believer that "There ain't no such thing as a free lunch"

So I came across the following wonderful talk on YouTube and the associated slides on Slideshare.
I figured I'd share all the links here for easy access. Enjoy!

Architecting in the Cloud by Simone Brunozzi

AWS Cloud Tour, July 2011

Part 1

http://www.youtube.com/watch?v=vJ6XvQ94UnM

Part 2

http://www.youtube.com/watch?NR=1&feature=endscreen&v=KL7KI1JQliY

Part 3

http://www.youtube.com/watch?feature=endscreen&NR=1&v=5hXnoNGnlFQ

Part 4

http://www.youtube.com/watch?NR=1&feature=endscreen&v=RHVFARV5mrc

Part 5

http://www.youtube.com/watch?NR=1&feature=endscreen&v=k84spxyTQ_s

Slides on Slideshare

http://www.slideshare.net/AmazonWebServices/2011-aws-tour-australia-architecting-for-the-cloud-demo-and-best-practices-by-simone-brunozzi

The Perils of Asynchrony

2012-01-29T17:01:00.005-05:00

Every time you come across anything more than a rudimentary system that has some moderately serious performance needs, someone somewhere on the team considers using asynchronous processing to help reduce (perceived) response time. That person needs to be identified and quickly locked in a padded room . . . . Just kidding!

Often that person is me and often I end up writing the code and relearning why doing things in an asynchronous way (that also meets a certain "near guarantee" SLA including DR and HA needs) is very very hard.

So it was with some chagrin that I was tasked with coding some infrastructure components to implement a Task Queue for my current team. The goal was to satisfy a need to improve response time of the application when it was doing some back-end tasks that required 100ms or more. The tasks themselves aren't time critical but they are important e.g. replicating data to a remote site.

Fortunately this time I'm not writing financial systems software - there you typically need to guarantee that although a task is asynchronous that it will be done within some SLA (e.g. 2-3 seconds). So at least I didn't have that worry. That said, for most apps asynchronous doesn't mean performing the task "whenever" - it means a little bit later than now. In addition you still have to consider that your solution is Highly Available and has built in Disaster (or just plain VM crash) Recovery.

Anyway the default solutions in Java for Asynchronous processing are

1) Threads (and java.util.concurrent - which is awesome)

2) JMS (Java Message Service)

Why Threads aren't the solution

Java.util.concurrent is great - a really great step up from Threads and writing your own infrastructure to start, stop and generally manage and handle threads and thread pools. If you haven't dug into this package yet do so now. It's a life saver.

The downside to Threads / Java.util.concurrent is that when you launch a job to be executed in the same JVM and same VM slide then

1) If your JVM or your VM die without some persistence mechanism your job is lost forever.

2) You never have just one asynchronous job and you'd like to be able to balance the load

of asynchronous jobs across your entire farm of virtual machines and their JVMs.

JMS in a Nutshell

So that's where people start to invoke JMS (Java Message Service). JMS proposes two

core models of asynchronous communication.

1) Queues - which are a point-to-point communication system. VM #3 writes a message to a queue and some other thread in some other VM reads that message from the queue. Only one writer - one reader.

2) Topics - which are a "broadcast" communication system. VM #3 broadcasts a message to a topic and multiple threads on multiple VMs (each with potentially different processing goals) read the thread.

Here come the problems . . . .

The key problem with Queues is ensuring that although there is only one reader that the job, once it is read from the queue, actually gets executed.

The key problem with Topics is often ensuring that although there are multiple readers - every individual type of processing (e.g. store a record in a database table) happens only once - otherwise you are wasting resources.

There are other problems too. JMS doesn't have any built-in specification for Disaster recovery or High availability. Therefore each JMS provider (and there are many e.g. MQSeries, Progress Sonic, Active MQ) provides their own mechanisms.

Persistence

The first other problem is ensuring that if a message is written (to a topic or queue) and one of the JMS processes (a broker) crashes or there's a network hiccup that the message is not lost. For that to occur you need a persistence mechanism. The persistence mechanisms either use the local filesystem of the broker or a relational DB. Either way you are just pushing your DR problem a bit further away from your JMS broker to now worrying about the local disk or your database. IMO it's not exactly a solution to replace one DR nightmare with another one. In addition to doing this another MAJOR downside to persistence is that it typically reduces throughput by an order of magnitude or more. See for example this link for Active MQ alone but this link comparing ActiveMQ and JBoss MQ and JBoss Messaging..

Guaranteed Messaging - who's guarantee?

The other problem is ensuring that once a message is read by a thread that it executes the task successfully. That guarantee is impossible - VMs crash, exceptions are thrown etc. Ideally if the thread can't handle the task you would want to put it back on the queue for someone else to attempt to handle it. This is hard with JMS although there is a way to "trick" JMS into supporting this that involves using message acknowledgement. That is you do not use the AUTO_ACKNOWLEDGE default - only acknowledging receipt of the message to the broker after the message has been successfully process. However please check how the Broker does this as there is some evidence that doing this just blocks the broker - giving you more guaranteed message handling at a huge cost in scalability.

Another problem with this approach is that you need to handle the possibility of messages being handled more than once. The reason for this is that the thread that originally picked up the message may have processed it successfully but failed JUST before it was going to acknowledge the message. Thus hopefully your message processing is either idempotent or can handle duplicate processing of messages without too much trouble.

High Availability

The other side of the coin for persistence is high availability. Although you want a broker storing data in a file system or to database for persistence, you need some number of other brokers (at least one) up and running and ready to take over. Ideally they are on a different VM, a different blade and perhaps in a different data center. None of which is easy.

Just a side note real quick - one thing I really liked about Amazon's Simple Queue Service (SQS) is how it handles persistence. Unlike JMS which has simple "Send Message" and "Read Message" semantics. SQS has "Send Message" and "Read Message" and "Delete Message". Unlike in JMS, reading a message in SQS does not remove the message from the queue. It "locks" it temporarily and if the message consumer does not call to SQS to "delete" the message after a certain time (30 seconds is the default), SQS itself puts the message back on the queue and assumes the original consumer failed to process it. So why not use SQS? Well frankly all my processing is local and I'd rather not take the Boston -> Virginia network hit (on the send and the receive). But it would make sense if my app was deployed entirely inside an EC2 instance.

Another problem: message ordering

Oh wait here's another Gem for you to wonder. Say your app has some basic CRUD features - Create, Read, Update, Delete. Now you start to put CRUD related activities on the queue. Queues do not implicitly guarantee ordering. So a thread may begin processing an Update event BEFORE the related Create event was finished. Awesome eh!!! So what do you do in this case?

Sometimes you can detect that failure (you can't update something that wasn't created) and just put the message back on the queue or retry a small number of times before failing. Others make the choice to have a single reader for these critical cases to ensure proper ordering.

The downside to THAT of course is

1) Scalability sucks

2) How are you going to handle Disaster recovery / failover.

For #2 I've seen folks configure 1 thread in N separate JVMs across a linux VM farm.

Only one thread is truly active at a time - the others are on "hot standby" using some form of ping / heartbeat functionality to detect if the "one true" thread dies and then some mechanism for become the active thread and telling all the other threads. MAN was that complicated and buggy code.

Oh just one more problem - queue backups!

Even after solving all these problems

- Persistence

- High Availability

- Messages handled more than once

- Messages arrive out of order.

You get the problem of what if your asynchronous task processing slows down and your thread pool becomes maxed out and your queue starts to back up? In one case we have to replicate data from the Europe to Asia. Although it's over a high speed network, the latency is at LEAST 250 ms. And it could be worse. You can't just add threads - you might max out your JVM's memory. I don't live in an EC2 world (yet) where I can just spin up new Linux VMs - and

even if I did the cost might be prohibitive at some point.

I could start to persist some of the queue to somewhere - disk / DB but again I'm going to have to write code to handle all of this plus testing is going to be a bear and the edge cases are killer.

And have a thread read from that storage area at a later point and retry.

Asynchronous - don't unless you have to!

So here's the problem with asynchronous solutions - they are powerful but introduce HUGE complexities in testing, scalability, disaster recovery, failover and high availability.

Even if you get the code right the first time - you better automate all your scenarios to ensure any future "fix" doesn't regress on your edge case handling.

In general my advice is avoid asynchronous solutions unless ABSOLUTELY necessary. And when you do watch for the "tail wagging the dog" - when you design and architect solutions to handle asynchronous processing cases that start to overwhelm your normal design/architecture. Beware introducing unnecessary complexity - but with asynchronous processing much of this complexity is sadly necessary.

If you are reading this and are aware of any good solutions to these asynchronous problems please let me know. Thanks!

References

Some good references I use when designing asynchronous solutions (to make sure I'm not reinventing the wheel)

Java Concurrency in Practice

Enterprise Integration Patterns

The Technology Hype Cycle and why you should bet your Software Career on the Cloud

2011-07-28T09:54:00.006-04:00

You'd have to be living under a rock not to be well aware of the hype cycle in technology. Here is a list of hype cycles I've personally lived through and been affected by. I've tried to identify the year where expectations were either rising the fastest or reached their peak

1999 - Y2K

2000 - Y2K (Yes it was still pretty hot)

2001 - Java / J2EE

2002 - Linux (also Enterprise Application Integration was hot here too)

2003 - .Net Framework

2004 - Open Source & Web Services & BPEL

2005 - SOA (Service Oriented Architecture)

2006 - ESB (Enterprise Service Bus)

2007 - SaaS

2008 - Web 2.0 and Mashups

2009 - Virtualization

2010 - Cloud Computing

2011 - SmartPhones

I'm probably off by a year or two in some cases but as you can see each and every year has something new.

Of all of those hyped technologies very few of them delivered on the (marketing) promise. However I think some of them really were a net positive. On a scale of "mostly helpful" to "mostly hype" here are the highlights:

1) Open Source and Linux definitely helped reduced a small chunk of development costs. The problem is most of the costs in development are people - that said any reduction was useful. As a (Java) developer my first preference in most technology choices is Open Source because most of these products "just work" and there is enough documentation and forum help to dig you out of most holes.

2) Java was probably a net drag initially (especially J2EE in its first few incarnations) but after JavaEE 5 things are getting better.

3) SOA was a slight evolution (XML + HTTP?) however if not for the advent of REST, SOA would reside more towards hype than helpful as REST dispelled the myth (pushed by consultants and software firms) that we needed a bunch of heavyweight WS-* protocols to make SOA viable.

4) Virtualization was definitely a net positive for IT organizations and data centers - definitely more helpful than hype saving organizations lots of money.

So what about the cloud, cloud is getting hyped a lot lately. A lot of people don't fully understand it so this may surprise people: the advent of the Cloud is, in my opinion, the biggest event in software development since Tim Berners-Lee proposed the world wide web .

Take that in for a minute. I am willing to bet my public reputation that the Cloud is going to be a HUGE sea change for IT organizations and for developers. Bigger than anything in the past 12+ years.

As a developer if you are not getting on board with this you are gonna be left behind. This is like being a N-Tier C++ programmer in 1994 as the web was starting to gain traction. Get in with the cloud NOW!

Why? Cloud solutions (computing, storage etc.) allows companies (especially large ones) to reduce and perhaps eliminate many of their data centers and supporting IT staff. For smaller companies the "Pay as you go" model (converting CapEx costs to OpEx) and the "elasticity" support to allow you to scale-on-demand are exciting. Venture Capital companies are now demanding the companies they invest in to use the cloud - it helps to reduce costs now and scale more later without large outlays. So who is NOT going to get on board this trend? I can't think of any organization that cares about costs / growth who will not ultimately want to adopt this model.

Also as the number of devices per person (Tablets, laptops, smartphones) continues to accelerate the "elasticity" demands will become ever more prevalent for large enterprises as well as Web 2.0 companies. True you can't use more than one device at a time but devices are getting smarter and creating work themselves (to keep your email, rss feeds, facebook status all up to date etc.).

This is like when Electricity distribution took off with the advent of AC (alternating current) it was critical for economic growth. True the Cloud may take 3 or 4 years to REALLY take off (1994 in the web wasn't nearly as exciting as 1999) but when it does . . . .

And when stuff is hot (like cloud is, or is going to be) the pay goes up up up and the firms that drive value in this space are going to be worth billions.

There are a number of Cloud providers and it's growing almost daily - but if you want to get on board I suggest you start taking a look at Amazon EC2 and Amazon's Elastic Beanstalk (for Java folks). Back in the nineties they used to say "No one was ever fired for choosing IBM" - not that IBM was flawless but it was hard to argue they weren't top of their game for Enterprise software (back then). Today I think the same is true of Amazon - not perfect but compared to the other providers they are clearly on top of their game.

Getting knowledge of AWS and understanding the very different architecture, design and coding strategies (e.g. security, privacy, failover concerns and NoSQL, Hadoop etc.) will help you transfer more quickly over to whatever is "next" when invariably Amazon gets surpassed by someone with a better mousetrap.

I still have a few doubts. Invariably there will some mini-Bubble in cloud company valuations - in the hype cycle there will be a "Peak of expectations" and a "Trough of disillusionment", but in the end the "Plateau of productivity" is going to be a very good (but very different) place to be. And when a large wave of change is coming - it's better to be ready with a surfboard and paddling ahead of the wave, than sitting on the beach waiting for it to come ashore.

Scrum Purity Test

2011-07-13T13:56:00.007-04:00

OK so just for fun (malicious fun!) I decided to create a Scrum purity test. The reason is I am pretty sure a LOT of people out there SAY they are using Scrum but aren't really - or they aren't using all the tools and techniques involved. Fair to say, I have never fully utilized Scrum in any place I have worked - mostly for political reasons - usually because of lack of buy-in from Senior Management. That said, we've made it work and I'm sure others have too.

But let me say that I don't think you HAVE to adopt Scrum 100% to get value from it. Small iterations with defined deliverables and constant communication and accountability can go a long way regardless of what you call it.

Anyway here is my take on a "Scrum Purity Test". There is another take here.

FIRST AND FOREMOST

Q: Do you have Sprints at the end of which you deliver (roughly!) shippable functionality?

- No - Then don't do this test - you've already lost. I don't think you can consider yourself agile without small iterations.

- Yes, but the Sprints are 4 weeks long and the code has lots of bugs - zero points

- Yes, the Sprints are under 3 weeks and the code has some bugs - 5 points

- Yes, the Sprints are under 2 weeks and the code is tested (but has some small issues) - 10 points

- Yes, the Sprints are under 2 weeks and we release to production - 20 points

MEETINGS

Q: Do you have a Daily Scrum Standup meeting?

- No - zero points

- Yes but it goes over 15 minutes - 5 points

- Yes, it's under 15 minutes and we answer the three questions - 10 points

Q: Do you have a Sprint Planning meeting?

- No - zero points

- Yes kind of, it's kind of ad hoc - 5 points

- Yes Ken Schwaber would be proud - 10 points

Q: Do you have a Sprint review meeting at the end of a Sprint?

- No - zero points

- Yes - 5 points

Q: Do you have a Sprint Retrospective to identify what can be improved?

- No - zero points

- Yes - 5 points

ARTIFACTS

Q: Does your team maintain a Product backlog?

- No - zero points

- Yes, kind of, it's ad hoc and it lacks priority- 5 points

- Yes its prioritized, updated regularly, visible to all - 10 points

Q: Does your team maintain a Sprint Backlog?

- No - zero points

- Yes, but features aren't broken into tasks - 5 points

- Yes, features are broken down and well understood before beginning - 10 points

Q: Does your team maintain a Burndown chart

- No - zero points

- Yes, kind of, we track it roughly in an ad hoc fashion - 5 points

- Yes, we use some tools such as Greenhopper or Excel and its visible to all 24 x 7 - 10 points

ROLES

Q: Does your team have a Scrummaster?

- No and we could use one - zero points

- No we don't need the protection / impediment removal (e.g. in a startup) - 5 points

- Yes we have one and we also report to her (e.g. a Project Manager) - 5 points

- Yes we have one and it's not a person we report to - 10 points

Q: Does your team have a Product Owner?

- No - zero points

- Yes but they really don't understand the user - 5 points

- Yes and they know our user base inside and out - 10 points

SCRUM-BUTS

Q: Does your team do Estimates?

- No - zero points

- Yes and we estimate using hours and days using MS Project - 1 point

- Yes and we use planning poker and our estimates are within 15-30% - 5 points

- Yes, we use planning poker and our estimates are within 10% - 10 points

Q: What are your Specifications like?

- Big requirements docs - zero points

- Good user stories - 5 points

- Just in time requirements / stories - 10 points

Q: How is testing performed?

- We have No dedicated testers on team - zero points

- Developers Unit test only - points

- The feature is tested soon after Sprint is complete - 3 points

- Software passes acceptance testing as Sprint ends - 8 points

- Software was tested and is deployed to Production - 10 points

Now what is your total - out of 130 points? What grade would you give yourself on the following scale?

110 - 130: A - you are doing a great job following Scrum (just remember to make sure your users and stake holders are happy)

90 - 110: B

70 - 90 : C

50 - 70 : D

< 50: F

I have to say I thought we were doing pretty good but when I scored my team we got a 59 a D! Yeouch! That said we ship on time and everyone is pretty happy but clearly we are not doing great following Scrum.

What do you think? Are there any critical aspects of Scrum that I missed? Any items that deserve more points (or less)? I am being unfair?

POSTSCRIPT

Some folks have been kind enough to get me searching for "the Nokia test" which has thrown up the following

The Nokia Test and its history

2-Way SSL with Java: The Keystore Strikes Back - Part 1

2011-03-10T20:46:00.013-05:00

You'd think with how prevalent SSL/TLS on the web that SSL with Java is easy, well it's not. In addition to having to get to grips with understanding how private/public key, certificates work there's all the networking that goes on in Java.

If you start off trying to get to grips with the in-built Java Secure Sockets Extension you're gonna be stunned by the complexity of it all.

Clear as mud eh? The JSSE guide alone is 73 pages long - but I hate to say that its REALLY a must read. Print it out (double-sided to save some trees) grab a Venti decaff or two and wrestle the information into your brain.

Thankfully Jakarta Commons HTTP client makes it relatively easy code-wise (Side Note: Why the heck does HTTP Components v4.0 seem so much more complicated than v3.0? The v4 docs seems much less clear!). For example:

HttpClient httpclient = new HttpClient();

GetMethod httpget = new GetMethod("https://www.bob.com/");

try {

httpclient.executeMethod(httpget);

System.out.println(httpget.getStatusLine());

}

finally {

httpget.releaseConnection();

}

In any event it's pretty straightforward code-wise with HTTP Client until you have to start to get to grips with three funky things - they are

1) the keystore

2) the truststore

3) keytool to access both

Truststore

Lets start off with a truststore. In a normal 1-Way SSL situation, say on a browser, you need your bank's server to authenticate itself to assure you that you are hitting the REAL bank server and not some spoofed site. So in the 1-Way SSL case the bank server sends your browser a certificate signed by a CA(Certificate Authority) then your browser compares that CA signature against an in-built list of CAs. In a similar way your truststore is your secure store of trusted CA entries or self-signed certs provided from third parties you trust. So essentially the truststore contains the certificates you use to authenticate other servers / processes. Every Java install comes with it's own truststore located in $JAVA_HOME/lib/security/cacerts

Keystore

A keystore is, as you can imagine, is a secure store for keys used in the SSL protocol. Essentially it's where you will put private keys and their associated certificates used to authenticate YOU as client to a server. Typically in a normal web browser transaction you use 1-Way SSL to authenticate the server then you use a login/password combo to authenticate you. Well in B2B systems you can use 2-Way SSL where client and server authenticate each other - and this is where the keystore is critical to authenticate your Java program, the client, to the server.

Typically for client authenticate you create either a self-signed or you purchase a CA signed cert. You store your private key and public cert in the keystore and provide your B2B partner with the self-signed cert - or in the case of a CA-signed cert typically your partner's truststore will trust that CA and you are good to go.

keytool

keytool is the command line tool to maintain these two beasts. And let's just say it's not easy to use keytool. It's like snowboarding - its good when you're used to it but getting started with keytool you're gonna spend a lot of time on your ass or face first! If you're just getting started with keystores and truststores I strongly recommend using Portecle - an open source GUI tool which makes keytool obsolete and should really ship with Java someday.

Then you have to tell java where they keystore and truststore are and the passwords to these stores. Well you can do that with System properties e.g.

 -Djavax.net.ssl.keyStore=keystore
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStore=truststore
-Djavax.net.ssl.trustStorePassword=trustword

If standards are good, more is better right?

Getting tired yet? Well this is gonna put you over the edge . . . . when you get keys and certs from your B2B partners how many types of files do you think there are - two? three? four? Let's list the ones I've had to deal with lately.

It's madness! If you want a good description overview of the file formats then this link is a good place to start. This is when openssl becomes your friend when you need to convert from different file formats into formats compatible with keytool - oh and good luck finding documentation on which file formats keytool takes!

In any event so far things are tough but not insurmountable - well so I thought until two B2B partners decided to provide me each with a key for my keystore to authenticate me. A key of the same type (RSA), signed by the same CA. Then all hell broke loose - cos Java don't support that without some major surgery . . . . and that's the topic of Part II.

Migrating from JBoss 4 to JBoss 5

2011-01-07T15:37:00.004-05:00

[ Yes I know it has been a while - two years, two very busy years but hey I'm back - for now ]

Anyway I have been using JBoss 4.2.3 for over a year right now and really like it (although the clustering / JMS stuff seems WAY overcomplicated - and needing 80 config files - not fun!). But anyway it is time to upgrade to JBoss 5 and the path has been marred by many stops and starts and has been surprisingly difficult.

In any event I found the following links to be a life saver and thought I would share them

The real kickers have to be

1) Increased adherence to the Java spec that cause WARs / JARs to no longer deploy

2) Changed location of XML files

3) Changed XML filenames

4) Changed XML file contents

Man they don't make it easy do they?

How to Motivate Developers - A three step framework

2008-12-09T19:49:00.015-05:00

No . . . . not THOSE three steps!

Anyway, now that we're approaching the end of the calendar year, invariably many folks are filling out self-evaluations and managers are presiding over year-end reviews, and choosing raises and bonuses for their staff.

Managers should also be considering the year ahead and one of the most important aspects - developer productivity. A key element of productivity is motivation and this article tries to answer the question "How to motivate developers".

But first some comments on prior work . . . .

There have been quite a few blog articles written by others on this topic (see some references below). They're pretty solid but they're mostly written from the perspective of
1) Developers or
2) Managers who don't have a major technical background.

The problem with #1 is that self-reporting of what a person wants is very unreliable. Often people don't really know what they want. Or they know what they want but don't truly understand how it will make them feel or for how long (see this really great article on happiness for some more insights)

The issue with #2 is that, without knowing the perspective of the other side (developers understanding what it means to be a manager and vice-versa) the full picture is often lacking.

One thing I noticed in my research for this post is that many of the articles below try to ask "What do developer's want?". That's important but a bit misleading. For example take a more common situation - housework. I don't want to do the dishes - I'd rather be playing Call of Duty 4 - but I *have* to. For many reasons - the sanitary one is obvious - the not so obvious one . . . . my wife will kill me otherwise :-) So what's my motivation - good health and staying alive :-) What motivates me isn't what I want.

There's also another aspect to this that I feel people haven't considered. I am going to venture (without citation . . . I couldn't find one) that the majority of a person's motivation is intrinsically determined. That is - their motivation level is largely driven by who they are, their personalities and things in their lives you have no control over.

So please don't get the idea from this that I believe one can control 100% of a developer's motivation. You can influence it, but only so much. In many respects motivation should be a key aspect when interviewing new team members - because once they're hired you can only impact a relatively small aspect of their motivational make-up. Again that's based on experience - I'm not aware of any scientific studies to confirm this hypothesis.

To repeat, I feel that, to a large extent, people are already largely motivated or not. You can improve things quite a bit though. Say you could make a developer 5-10% more effective - that's like having an extra 2 weeks to a month of their development time each year.

That's a lot of extra bugs fixed or new features produced. Do that across a team of 6-8 developers and you can make a very noticeable improvement.

Anyway after thinking about it - from both a managerial and developer perspective, here's what I've come up with for my framework.

STEP 1) Compensate Well
This is one of those "necessary but not sufficient" conditions you often hear about. You can address every other motivational item but if you don't pay well and give good benefits you're shooting yourself in the foot and worst of all people are going to go elsewhere. Probably most of all you're just not going to be able to attract and retain really good (i.e. already motivated) developers.

If you think of the famous Maslow's Hierarchy of needs, this is one of those fundamental "lower" or more basic needs.

STEP 2) Remove or reduce demotivating aspects
You can pay all the money in the world to people but if you have non-existent specs, unrealistic deadlines, micro-manage etc. people aren't going to be motivated for long and they will quickly become unmotivated or leave.

As I just stated there are several things that many developers experience that drive them batty.
i) Poor requirements - there's nothing worse than building junk but consequently (for me at least) there's nothing better than great (detailed, unambiguous) requirements and the resulting happy users.

ii) Unrealistic deadlines - whether this results in 90 hour+ weeks or buggy software - this is very much a drain on developers, their ego and sense of pride. Developers want to be proud of their work and go home and see their families. Is that too much to ask?

Yes everyone has crunch times and there's invariably some extra unforeseen work but hopefully it's infrequent and as a manager hopefully you don't have to "go to the bank" too often.

iii) Too many meetings - I've found that the more formal a place is (or the less friendly it is) the more meetings you need to just get people to talk to each other. But many meetings are a bad sign - ideally developers would/should be able to talk to each other and meetings should be "organic" and not forced.

iv) Disrespectful or uncooperative colleagues. "Bad apples" can ruin a whole batch. The problem is exacerbated when management is unaware of them or worse, is aware, but refuses to do what it takes to remedy the situation.

v) Processes, Tools or Techniques that slow you down e.g. slow builds, slow unit tests, slow desktops etc. They suck productivity and reduce enthusiasm.

vi) A poor work environment e.g. a place where it's too loud, there's no privacy, you get lots of interruptions etc.. A bad environment can destroy motivation or at the very least be a distraction. And distractions are TERRIBLE for a developer's focus on the problem at hand.

Either way, my recommendation is that before focusing on MOTIVATING factors, I would focus on removal or amelioration of demotivating factors. There's also some evidence that there is some psychological difference between demotivating aspects and motivational factors.

The good thing is by focusing on these e.g. improving requirements, reducing unnecessary meetings, speeding up processes etc. you are also improving team productivity even if there's no motivational boost to be had. But that's unlikely - removing some of those impediments should buy you some good will - but frequently that fades over time. Either way, for the productivity boost alone that's makes good business sense.

STEP 3) Focus on positive motivational aspects.

Here are the things that you really are hoping will translate into more motivated (and more productive and happy) developers

i) Support Developer Learning
Developers love to learn - new languages, new tools, new frameworks. Spring, Ajax, Hibernate, Ruby - that gets their juices flowing. Now the challenge is how do you direct that to produce the product you're paid to . . . . . the key here is to align the new item with key gaps needing filling. Need an Object-Relational Mapping system - Hibernate! Need a rapid prototyping language - Ruby! etc. You also need to factor in the productivity dip as they learn.

ii) Provide Challenges & Problem Solving Opportunities
Developers in general love to solve problems - whether it's how to integrate new features into a legacy system, how best to compare two lists of strings or how to make a system more scalable developers love such problems.

In my experience however there are sub-types - different developers who prefer different kinds of challenges - ideally again you would align the challenge to the person who wants it!

iii) Provide Feedback
Developers want to learn and grow - beyond the technical learning / challenge they also need feedback in other ways to help them grow as individuals and adapt to their organization. Often I find this is where management falls down. Who could blame them / us - we often say we're open to hearing about areas needing improving but not so good in practice. I've found the Sandwich Feedback Technique works quite well though.

iv) Other Ideas
See the section below for other really good ideas from other articles

These are all the things that "developers" want, in the sense of the higher "self actualizing" levels of Maslow's hierarchy. Again, these factors should not be detached from "conditions on the ground" as it were.

For example I've known a few developer's whose idea of self-actualizing was deciding what to eat for lunch that day. They didn't want to learn ANYTHING new - no new languages, no new environments - they know what they have and don't want to change. Their motivation isn't going to take a boost from being asked to learn something - on the contrary.

STEP 4) The Secret Step - Fear as motivator
OK I said there were three steps, but here's that "magic" 4th step. The "nuclear option". Go here only when you must e.g. you or someone on your team might be fired unless some action is taken, a client could lose a lot of money etc.

As with the "doing the dishes" analogy, there comes a point when you have to do what you don't want to do. You do it because the ramifications of not doing it are worse - MUCH worse.

Often if you're seen as the "good cop", you need a "bad cop" to make this work. Sometimes you might need to be the "bad cop". Most folks know the analogy having seen it many times on the various CSI or Law & Order shows on TV.

Basically the line here is "Do this or you'll lose your bonus / won't get a raise / be fired".

As I said you can't rely on using this too often, but you need to be sure also that people know the chance is there. Developers are smart people - if you don't follow through on what you promise you lose credibility. This lesson on follow-through was something that my two year old taught me. "If you don't get down from that chair, you'll get a timeout". Guess what, unless you start handing out "timeouts" they just will continue to stand on that chair.

Again you don't want to do it too often and when you do you need to explain the reasons, it can't be seen to be arbitrary. For example - fire too many people, and people will start updating their resumes and looking elsewhere rather than live in fear. Don't fire anybody and you send the message that there are no consequences and people want to leave because they tire of supporting the slackers. It's a fine line but most of your strong developers already can identify who isn't pulling their weight and will thank you for pulling the trigger and finding someone better.

Anyway there's my thoughts on developer motivation - I look forward to your comments.

REFERENCES TO RELATED WORK

Nine things developers want more than money
1) Being Set up to Succeed
- Good requirements
- Realistic Deadlines
"Being forced to build crap is one of the worst things you can do to a craftsman"
2) Having Excellent Management
3) Learning New Things
4) Exercising Creativity and Solving the right kind of problems
5) Having a voice
"When a developer speaks, someone should listen. When several developers are saying the same thing, someone should listen and act . . .quickly"
6) Being Recognized for Hard Work
7) Building Something that Matters
8) Building Software without an Act of Congress
9) Having Few Legacy Constraints

Top 10 ways to motivate geeks
1) Geeks are curious. Let them feed their desire to learn things
2) Geeks like to be self-sustaining. Let them figure things out on their own.
3) Geeks are creative even if they don't know it. Give them a chance.
4) Geeks need tools, good ones. Give them more than they need.
5) Private, yet collaborative. Geeks need to be left alone, but not too alone.
6) Free stuff. T-shirts, food, desktop widgets, whatever.
7) Control
8) Geeks need recognition
9) Freedom
10) Compensation - Saved this for last, but geeks gotta live too

Motivating Software Developers
1) Allow me to focus
2) Allow me to feel that I am making progress
3) Allow me to make something I can take pride in
4) Allow me to do it for me, not for the money
5) Allow me to work on interesting problems
6) Allow me to be part of a team
7) Allow my ideas to be taken seriously

What motivates software developers?
1) Introduce new technologies and techniques
2) Practice pair programming to encourage communication, sharing of skills and team building
3) Encourage participation in community developer events (user groups, code camps), blogs (share links across the team), books (monthly bookshelf anyone?) and conferences.
4) Avoid generalized training - in my opinion this tends to serve the paycheck programmer more than the dedicated ones.
5) Interesting projects
6) Satisfy your customer

Motivating Software Developers

To Motivate or not to demotivate
"Only taking away the things that make people dissatisfied, will simply result in people having neutral feelings towards their jobs."

Some programming principles: People

Managing and Motivating Developers: Tips for Management Cluefulness

What makes people tick - 10 motivation factors in the workplace

"The first thing to note here is that most people are motivated by:
1. doing interesting stuff
2. feeling recognized and appreciated
3. making an impact"

What motivates programmers?

11 ways to motivate geeks

Five things your manager could be doing better

To Beta or not to Beta - That is the question

2008-12-01T19:34:00.008-05:00

"Whether 'tis nobler in the mind to suffer the slings and arrows of outrageous bugs, Or to take arms against a sea of defects, And by opposing end them." (with props to Bill S.)

In the past few years it's become more and more acceptable to release "Beta" software to the public - almost as if it was a production release.

The reasons for that I believe are manifold but boil down to
1) Gain user feedback
2) Release early to gain "mindshare" and/or get first to market.
3) Your QA process or team isn't that great (or your unwilling to spend money on them) and you'd rather have users do your testing for you.
4) You need to gain the confidence of your customers / investors in what you're building.

The number one proponent of the Beta, I must say is Google. I've been a Google Docs user for well over a year now and I love it - but it's still in Beta. The one thing I don't love is the FREQUENT UI changes in direction though - but hey it's a Beta and it hasn't lost a Document or Spreadsheet of mine yet.

But it's clear from using what Google call a "Beta" that their goals are really predominantly #1 (user feedback) and probably to a lesser extent #2 (mindshare). It's been very well tested (internally) before it hits the public. You can read more of what Google think about Betas here.

So the quality of their Betas is quite high (see also gmail) and a Beta for them is probably a production release for 90% of everyone else.

Here's the problem though - in making Beta's "popular" I think they (and others) have lowered the bar at least insofar as management can now look at it and say "Well if Google did it - why can't we?". So there's a tendency now to ship lower quality software and at the last minute people decide to slap a "Beta" label on it if it's not up to snuff so that they can declare victory. Then they hope people will use it as if it was production quality.

Personally I think software professionals should be using a public "Beta" ONLY as a last resort. If you need to gain user feedback - try not to be cheap about it. Get a bunch of potential users in a room and have them use your product - try not to prime them - just watch them. Video them. Understand what are they trying to do - how does that align with your product? Can you make your application so easy to use it doesn't need a manual? That should be the goal (although that might be ultimately unattainable any progress in making your application easy to use is good).

If the reason is #2 (to be first to market) then I think you don't understand technology trends. Neither Google or Internet Explorer (or Firefox) were first to market - look where Yahoo! and Netscape are now.

I guess I never really "got" the whole first-mover advantage thing. Think about it:

Apple vs. Windows (and back again)
Ford vs. Toyota
Betamax vs. VHS
Atari vs. Nintendo
WordPerfect / WordStar vs. Office
Friendster vs. Facebook vs. MySpace

The anecdotal lesson I got from these cases was to NOT be first mover and learn from the other guy and where he fails.

Anyway if you're in the position of #3 (you're not investing in your QA team/process) then that's a sorry state of affairs. Although we live in a world of finite resources - finite dollars and hard-to-find tech experts so I guess it can be excusable sometimes. But if you just punted to force your users to do the bulk of your testing work then shame on you.

If you're in position #4 - releasing a Beta to the public to gain confidence well it's hard to argue too much with that given the sorry state of "production" software these days (*cough* Vista *cough*) people's expectations are sadly low so its understandable people (especially investors, customers) want some reassurance.

Don't get me wrong on this though - I'm not the type of guy to wait until the product is perfect. It's more a question of what are your motivations and are you being lazy in releasing a Beta? If you want user feedback - GET USERS and GET THEIR FEEDBACK! Don't ship junk!

After doing some Googling I came across a blog article with similar sentiments
A call for revolution against Beta Culture

"I'm mostly tired about the fact that it seems that we all have given up. Tired because . . . . in reality, it's laziness and a poor job on the manufacturer part that we have accepted without questioning. Instead of calling foul play and refusing to participate, we keep buying."

"That's the key: We have surrendered in the name of progress and marketing and product cycles and consumerism."

Right on man - it just doesn't feel right. It feels lazy and short-sighted.

Here are some other thoughts by other folks

So here's my question to you, dear reader, what is your opinion on Betas? Useful software engineering technique or cop-out?

JarFinder.com - Where have you been all my life?

2008-11-13T21:03:00.003-05:00

Ever have that moment where you discover a great little tool and you wonder "Where have you been all my life?"

Well I had that moment today when I finally came across http://www.jarfinder.com/

Ever have a hard time tracking down which Jar a class belongs to?

My bane is org.apache.xml.serializer.Serializer

It turns out it's in Xalan of all places (and I can never seem to remember that).

Now with http://www.jarfinder.com/ you don't have to struggle anymore to find the jar that contains that class file.

Turns out this has been out there since at least early 2006 - wow.

Book Review: The Definitive Guide to Terracotta

2008-08-25T08:55:00.021-04:00

(Please Note: Apress, the publishers, kindy offered me a Java book to review for my blog. I accepted and out of > 100 options I chose this one due to my love of all things scalability and because Terracotta seems to address a hard problem of distributed cooperation within clusters. Anyway I guess the point is, I got this book for free, some may consider this places me in a slight position of a conflict-of-interest, some may not care. I certainly don't believe it has colored my view of this book - but you, dear reader, should be aware nonetheless. In any event I am certainly appreciative to Apress.)

Over the past few years as principal engineer / architect (and back again) I've faced a couple of hard problems which I'm sure many of you have too. Once you get to an architecture where you have more than one JVM or server, how do you quickly and easily share information across them? Yes the database is one easy solution but the performance / scalability hit in "always going to the database" becomes quickly onerous.

Then one decides to cache to reduce this load but now how to you keep caches in synch?

Then one might go down the path of deciding to use some messaging system (e.g. JMS, Tibco, WebSphereMQ etc.) to ease intercommunication but there the amount of coding, testing and debugging needed due to the complexity of this roll-your-own solution becomes quite problematic.

Another similar problem comes where you have a load balancer in front of your web / app servers for performance, DR and HA reasons and you have some stateful application (like most web sites are today). How can you share easily session information across servers to make for a seamless experience?

So for all of these problems that's where something like Terracotta comes in. It basically sits between your application and the JVM and allows you to declaratively define what information is to be shared across JVMs. It promises to do so without any code changes (nice!).

A compelling product to say the least. Anyway, in a bid to learn more about this product I decided to review this book.

TITLE: The Definitive Guide to Terracotta: Cluster the JVM for Spring, Hibernate and POJO Scalability
AUTHORS: Terracotta, Inc.
ADDALL: Link

CHAPTER 1 Theory and Foundation: Forming a Common Understanding
The first chapter starts by laying some foundation and defining terms - always a good start. "Terracotta is a transparent clustering service for Java applications" is the mantra and they go on to explain what this term means. They proceed to talk about how the underlying memory model that Terracotta gives you (across JVMs) is now transparent to the application.

They discuss at a high level how the service provides you advantages such as high availability, scalability (scale-out) and improved performance (by not requiring a DB hit to share information).

As examples, they cover the classic use-cases of Terracotta
1) Distributed Caching
2) Database Offload
3) Session Replication
4) Workload partitioning

CHAPTER 2 History of Terracotta
Chapter two discusses the forces that resulted in the creation of Terracotta - from the forces to scale out rather than scale-up (e.g., preference for loose coupling, availability of cheap commodity hardware, cheapness of linux etc.).

But with scale-out comes the problem of keeping JVMs / Servers in synch. The solutions such as
1) Scale the Database
2) In-memory replication
3) Partitioning the data

each came with their own problems.

And so Terracotta came into being. Whereas folks such as Amazon (and eBay) took the approach of "eventual correctness" (aka "eventual consistency") where each application instance could complete transactions to local disk and eventually flush to the database, Terracotta's founders chose another solution as their business folks were "not prepared to discuss the ramifications of an eventually correct architecture, where users might be told that a previously confirmed purchase order could not be completed because of miscalculations in inventory long after checkout completed".

And so they sought to effectively create a "General Purpose L2" cache. The original implementation was too intrusive where developers would often forget to serialize and replicate changes to L2-based objects to keep things in synch and this led to regressions and eventually to a significant slow down in the pace of development.

With Scalability and Availability often becoming opposing forces it was refreshing that their solution aided both. The transparency of the solution also does not necessitate the need of one programming model over another e.g. EJB vs. Spring or JPA vs. Hibernate vs. iBatis.

CHAPTER 3 Jumping Into Terracotta
Ah here we go - C O D E! They literally start off with a simple (clustered) "Hello World!" example and start to get into how to configure Terracotta. I wish they had spent some more time here, perhaps a whole chapter, helping someone set-up a REALLY good environment (say multi-machine, or at least an env for multiple programs operating simultaneously) - a lot of this is left up to the user to figure out, let alone perform. That I think dilutes the message of Terracotta and doesn't give the reader a good "WOW" factor when they see this in operation.

In any event we start to get into the "meat" here and discover how Terracotta ensures all changes to shared data have been applied before a read is performed. And just after a write, Terracotta ensures that all memory changes are made available to other Java processes that might need them.

CHAPTER 4 POJO Clustering
So having seen a quick example they correctly now need to dive in to expain how Terracotta handles Java objects and the virtual heap and secondly explain how Terracotta manages thread coordination between JVMs.

They define a "root" - a field in any class that you declare as being clustered. Terracotta traverses the graph of object references from that root to cluster those objects too. Since these objects are clustered and durable beyond the scope of a single JVM they are sometimes referred to as "superstatic" - having the same lifecycle as the virtual heap.

Typically data structures like Map, List and other Collection objects are chosen as root objects.

Terracotta is pretty smart since not EVERY object on the virtual heap needs to be instantiated in every JVM. Terracotta can load an object as needed. Just like the virtual memory subsystem of a modern OS swaps contents to and from physical memory and disk, Terracotta lets your application behave as if there was an almost unlimited physical Java heap.

Then comes information on such topics as Distributed Garbage Collection and how threads are coordinated - again the details are such that there are no real surprises or "tricks" here - the folks at Terracotta have really taken Transparency to heart. "Terracotta provides exactly the same access serialization, coordination and visibility guarantees to threads in different JVMs as the JVM itself provides to threads within the same JVM".

Then we get into more meaty topics of locks in Terracotta and how Terracotta extends the concept of locking. Again by using declarative methods they help keep much of the messy coding inherent in locking out of the developers hands and keep it where it belongs - in the Terracotta infrastructure.

From there we leap to Terracotta "transactions" wherein Terracotta batches changes made to objects into sets, helping to ensure that threads always see a consistent view of clustered objects.
From there we delve into what kinds of objects are not "Portable" (cluster-able with Terracotta) e.g. file-system related classes such as java.io.FileDescriptor (host-machine specific) and instances of java.lang.Thread (JVM-specific) are some examples.

Interestingly there is also the concept of "Physically vs. Logically managed objects" . The former are objects wherein their field data values are distributed to the Terracotta server and from there to the other cluster members. The later (Logically managed) are clustered by Terracotta by recording the method calls on those objects and their arguments and replaying them on the other members of the cluster.

Examples of logically-managed objects are Hashtable, HashMap and HashSet (spot the common theme?) - yes that's because the Hashcodes used to create the internal structure of the object are JVM-specific.

From there we get more into understanding Clustered POJOs but personally I felt much of this information was repeated either earlier or later in the book. But after that there's a more fully formed example used to elucidate much of what was discussed earlier in the chapter.

CHAPTER 5 Caching
We begin this chapter with a discussion of caching and the trade-offs and problems it incurs

Space for time
Freshness
Complexity
Durability
Duplication of data across caches

Then a quick example shows how Terracotta can be used to quickly provide a distributed cache.
From there we delve into which of the Map structures are best for such data structures within Terracotta. Interestingly ConcurrentHashMap is generally the best choice when sharing maps but sadly LinkedHashMap is not supported by Terracotta. Harumph!

Then we get into some of the gory details of caching that we all need to know

Eviction and Expiration policies
Persistence
Distributed Caching (again I felt this was repetitive)
Use of partitioned data

Interestingly they claim that it is largely by avoiding serialization (typical of transmitting updates through a cluster using more traditional methods) they gain a very significant part of the performance improvement of their product.

From there we get a quick example with Ehcache (Easy Hibernate cache) and then onto chapter 6.

CHAPTER 6 Hibernate with Terracotta
I found this to be my favorite chapter - quite a bit more details in it than other chapters, good solid examples, and the benefits of the product become abundantly clear.

We start off with a great overview of Hibernate and how Terracotta can be used to improve it - by clustering the second-level Hibernate cache and also by using it to cluster Hibernate session objects.

From there we get a good example of how Hibernate and Terracotta together can be used to save on DB hits. Hibernate's cache runs the risk of staleness if another JVM updates data in the Db and so Terracotta helps fill this gap by preventing such staleness issues.

Then we get some great stuff lacking in other books - HARD NUMBERS!
Hibernate clustered with Terracotta gave a 4x boost over Hibernate with second-level cache alone when focused on DB updates. When focused on DB reads, we get a > 250x boost. Naturally your mileage may vary but at least we're getting some good ideas of what to expect.

We now get into the details of configuring Hibernate to be aware of Terracotta and vice-versa. All straightforward and relatively simple stuff.

CHAPTER 7 Extending HTTP Sessions with Terracotta
This was another good chapter on how to share HTTP Session information across JVMs, servers using Terracotta. A very useful feature that helps avoid most of the problems associated with persisting HTTP session information to afford your cluster the ability to scale-out, be HA etc.

Yet again we see the Transparency of Terracotta as it transparently (to your web app) hooks into the servlet container. From there we get a few nice examples to see all of this works with Tomcat. Fortunately Terracotta supports the following web / app servers

WebLogic
WebSphere
Jetty
Apache Geronimo
JBoss

and the following frameworks

Struts 1.1
RIFE
Wicket

CHAPTER 8 Clustering Spring
Here we get a pretty short chapter where basically the point is you can point Terracotta at Spring beans rather than declare each class / field yet again in your Terracotta config file.

CHAPTER 9 Integration Modules
Terracotta supports the idea of external configuration for a component you might be shipping that takes advantage of Terracotta's features. This allows you to ship a Jar and the user of that jar then does not need to include Terracotta config information for this component into their own Terracotta config.

This feature is called a "Terracotta Integration Module" or TIM. It consists of config info and perhaps code that specifies how the component should be clustered, how locking is performed etc. They then go on to describe how TIMs are created, used and configured.

CHAPTER 10 Thread Coordination
This chapter seemed like it should have been more up front, also it's quite short and I thought there would be more here. They get into some of the details of thread coordination in relation to Terracotta. I got something out of this chapter - I'm just not sure exactly what it was.

CHAPTER 11 Grid Computing Using Terracotta
Naturally Terracotta lends itself to Grid computing i.e. supporting the splitting of a workload across nodes. From there we get into the "Master/Worker" pattern and an implementation in Java and then into how to refactor the original example for improved performance / scalability by reducing contention, batching work, multiple work queues, addressing fault tolerance.

CHAPTER 12 Visualizing Applications
Finally in chapter 12 we learn about visualization techniques and tools to help you comprehend what a cluster is doing and why it is going slow or fast. They show many metrics the tools capture and what they reveal and how they can be used to tune your application.

REVIEW SUMMARY
This is a rock-solid book with a solid introduction. I wouldn't agree that it's a "Definitive Guide" - but I guess that's just an Apress standard naming convention (the same way Manning has the "In Action" series).

I'd like to have seem more help up front in getting your environment set-up for the examples, some case-studies of how Terracotta has been used, more benchmarks, perhaps even benchmark code. But given the fact that it's the ONLY book I can find on Terracotta it's fortunately pretty good and gets you "in the game".

Clearly Terracotta can't have infinite scalability - Terracotta must communicate between JVMs over the network so a guide of best practices, practical limits etc. would have been useful on things such as how to optimize network architecture, data structure design / optimization etc. would have been great.

PROS: Relatively Short and to-the-point. Examples are simple and straightforward.

CONS: Not enough examples. Not enough code. Would be nice to have them help you set up an environment. Would have been nice to see some more hard numbers of performance / scalability boost over other solutions. Quite a bit of repetition - typical of books written by teams without a clear "separation of concerns".

GRADE: B+

BUY IT? If you're using starting to use Terracotta - absolutely. If you're interested in making your application faster or scaling-out in general - probably.

A sad passing

2008-07-25T17:41:00.003-04:00

Randy Pausch sadly lost his life to Pancreatic Cancer. He was 47.

Link

His life, his work and his brave struggle against indomitable odds (Pancreatic cancer has a 5% survival rate after 5 years) was an inspiration to many. He is survived by his wife and three young children - 6, 3 and 2.

Wherever he is, his spirit I'm sure continues that quest for knowledge and to share that knowledge with others.

To quote T. S. Eliot

"We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time. "

Automated Regression Testing: Why, What and How

2008-06-22T09:53:00.015-04:00

Having to wear the hat of three different roles (Architect / Manager / Developer) you find there are few things in software development where there isn't a significant trade-off depending on your role and it's viewpoint.

For example when I'm managing, things like refactoring take a back seat to rolling out new product - even though I'm aware of the positive impact of refactoring. When being an architect I'd rather minimize code duplication (or triplication) - a long-term gain to team productivity but at the cost of product in the short-term. When developer I'd rather be focused on developing the cool new product than bug fixing. There's few things I can wholeheartedly stand behind when having to wear all three hats.

However good automated regression testing is something that I'm pretty sure everyone can agree on. It's usually not a huge investment of time but the payoff is large and grows over time - like a savings account.

From each point of view

Developer --> Less Time bug fixing. I can go home before 7pm. Yeah!
Manager --> Better quality product, risks identified earlier. Less screaming customers! Yeah!
Architect --> Great way to support refactoring with minimal product risk.
QA --> Less manual drudge work.

Probably several of you are asking - "OK but what is a regression test?". Put simply it's testing that finds bugs in code that used to work in past versions.

WHY
As mentioned above, regression testing is first and foremost a "risk reduction" exercise - you want to find bugs in code as fast as possible with minimal effort (automated!). This becomes more important the longer your product has been in production - no-one wants a release to be "three steps forward and two steps back". You want your product to always be increasing it's value to your user base. Bugs will happen - you just want them to either be minor ones or stuff in your new code - not the old stuff that people rely on to get their job done.

In any event, people's expectations for software these days are sadly low (Thanks Microsoft!) so delivering anything reasonably reliable gets noticed.

It also acts to reduce the "drudge" work of manual testing. That work is also subject to human memory - unless you have all the test cases written somewhere - are they all up to date? You sure? For the most part the answer to that is no.

Similarly automated regression tests also act to codify and formalize one's experience so, God forbid, you lose a QA person, a product manager, developer etc. You don't lose the entirety of their knowledge.

Although some would argue this makes those roles more replaceable - in my experience it acts to free up QA, Developers, Product managers to do more valuable - and LESS replaceable work.

It also helps your team be "more proactive and less reactive" (forgive the management speak). But as we all know - the more your team spends fighting fires the harder it is to have a truly enjoyable work place. Or at least that's my opinion.

HOW
So how do you "regression test"?

0) Stop Here!
First I'm going to assume some things - first and foremost that you have a source control system. If you don't then that's the first order of business - get one! SVN, CVS or whatever (personally I prefer Perforce - actually I love it).

I'm also going to assume your team or organization cares about product quality - no problem right? Think about it though - think about your past work experiences. I've been in several situations where there was no will to do any regression testing of any real merit - I tried to fight the good fight but sooner or later you get labeled as obstructionist. True you can't spend all your time on this - but if you can't spend a few hours per week per developer you're in for trouble. I'm sure I'm not alone in this experience.

1) Unit Tests
Well JUnit test cases are an obvious place to start. So obvious I won't spend time on them - the benefits and myriad and have been discussed ad infinitum (nauseum) here and elsewhere.

The key though is to remember to also focus on unit testing from a product point of view as much as one can - not solely a code point of view. Although code coverage tools are helpful too - 100% code coverage doesn't really mean as much as one would hope (see this great article by Andy Glover).

Also remember you don't just want to test functionality (both positive and negative cases) but also, if you can, performance / scalability and security.

2) Automated GUI tools
Assuming your product has a GUI you'll probably find yourself or your QA automation team writing scripts in some tool such as Segue Silk, HP/Mercury WinRunner, Quick Test etc.

These tools are good because it's very hard to unit test a GUI. The number of paths through the GUI are myriad for anything more than the simplest UI.

However these tests are often much slower than unit tests and require significant resources - both people and machine to run.

Again don't just focus on tests for functionality but also look at testing performance/scalability and security (and I'm sure other things too)

3) Projects-specific ways
Often you'll find ways that are specific to your software or project. Some examples are below

EXAMPLES
Two examples I'm very proud to have been a part of are described here.

The first was a product where my firm transmitted data to another firm - financial information about products we had for sale. The products were then advertised on the other firm's web site.
The key problem was that the data was constantly changing (think stock prices) so the information had to be pretty current (not totally real-time but a gap in minutes was about the worst we would tolerate).

Fortunately this customer had a test system which we could attach to - upload data and verify that any changes we made would still would work. How to tell it all worked as it was supposed to? I wrote a Java program using HttpUnit that would login to their web site - find the appropriate product pages, parse the HTML and pull out the prices. Then it would compare that data against what was in my local DB (the data I was supposed to send them).

Once I had run that for an entire day without issues I was extremely confident that all was right with the world.

Also a nice upside was I could run that SAME program against our production system (it's read-only after all) to verify that production was fine.

Now the REALLY cool thing was when something happened in production e.g. network outage or some remote problem typically I could inform my user base and my management very quickly. Often they would know before our business partners were aware that THEIR system was having problems. It's really nice to be able to have that level of confidence in your product and your users and management notice too let me tell you.

Another good example was more typical to software product firms where one has many customers running the product. Typically as we're solving problems with this product we would get copies of their databases to debug issues. In turn we would take several of these customer databases and run them through nightly regression tests.

The test would compare the current release vs. the last known release (considered the "gold copy"). Not only would we diff results but also we'd compare execution times. And although the environment was not totally locked down for a true performance comparison - a 20%+ discrepancy of performance for > 1 day was typically a sign of a potential performance issue in something in our code or the data model, indexes etc.

OK enough bragging . . . let's talk about some best practices.

BEST PRACTICES

#1) Find and Test the Gaps
Testing is never done and neither will your regression testing. Naturally you will (or should) want to test the riskiest pieces first - places that have largest impact if they break and/or break frequently. Often too you should use customer feedback and bug reports to help guide your list of functionality to add to the test. Look for patterns - key areas of misunderstanding or bugs. Code coverage tools can also be helpful but don't rely on them to tell you when you're done.

Keep a prioritized list of areas to cover - start at the top and work your way down.

#2) If you're not finding bugs ask why?
One way you're regression test can stop finding a lot of bugs is if it's out of date - if you haven't been spending time adding to your test suite (you're adding more functionality say?) then you're short-changing yourself. With every release of your code - your regression test should grow or improve.

Another is if the tests being created aren't aligned with what's known for risky / bug-prone areas.

Another is when your developers are so damn good they don't have bugs anymore - but we all know that's not all that common. But often you'll find that's because they run the regression themselves when they realize how valuable it is.

#3) Have dedicated hardware
You don't need the biggest, baddest machine - often a recent late-model desktop will suffice in most cases. But you need this to do #4

#4) Run them all the time or at least nightly
Your tests should be running all the time or at worst - once per night. Once your tests are big enough there may not be enough time in the day on on existing hardware - so you may wish to keep some long running lower priority tests for the weekend or alternate test suites every other day.

#5) When you max out - throw hardware at it
Hardware is a commodity - it's cheaper to buy some high end Solaris hardware for $10k than it is to take two months of a developer who earns $100k per year plus benefits to speed things up. So when you start to max out your current regression test machines throw hardware at it first - then improve the performance (unless there's some blindingly obvious quick fix in software).

#6) Critical tests first
In the spirit of "fail fast" and risk mitigation - you first want to perform the tests that will find the most critical items or test bug-prone pieces of code.

#7) Automate Result Interpretation
Once the results from your regression test are ready it should email relevant people and provide a synopsis of the results. A daily regression isn't very useful if it takes an hour to figure out if you passed or not. Eventually no-one will read it.

#8) Add tests continuously
Don't wait for a particular "phase" of the project to add more tests - do so continuously otherwise you'll find that time suddenly disappears and then you're playing catch-up.

#9) Don't reinvent the wheel - use tools
If you're going to automate don't waste time - use the all the tools you can. There are so many great tools out there from open source to commercial (although I must say the commercial tools I find VERY expensive compared to their utility).

Selenium
Canoo WebTest
JUnitPerf
JMeter
HttpUnit
AutoHotKey -- if you don't know this already - check this out ASAP!
Apache POI especially if you produce reports in Word Doc / Excel format
Good ole unix diff, pipe and sort (this reminds me why I dislike Windows as a dev platform - Unix tools rock so thank God for CygWin)

And there's probably a half-dozen I'm not aware of that I hope you, dear reader, will tell me about.

Don't forget static code analysis tools too (See "Analyze this - put your code on the couch!") to help find bugs (e.g. multi-threading) that your regression tests might miss.

Having a great suite of regression tests brings great confidence for a team of developers - who are often frankly a pessimistic lot - that can mean you and your team will stand out in the crowd and separate the true engineers (who care about delivering a quality product) from the cowboys.

(p.s. Thanks to Rick for the push!)

It does not take a Superstar - Developer types enumerated

2008-03-29T06:23:00.009-04:00

As someone who more recently has taken on more of team lead duties (*cough* project manager *cough*) I've had to reflect on how to dole out tasks and my success (and some failure) at this has really caused me to refine my understanding of different developer types I've known over the years.

So I've tried to enumerate different types that probably many of you are familiar with. These types probably don't 100% encapsulate somebody - often you'll see a number of these types in one person, or one type might dominate at any one time.

Also I'll try to give each type a rough score on a few dimensions. Dimensions that help me to assign tasks or pair people together.

1) Gets-it-done: Can be relied on to get the job done, fix the critical bug and will work with whoever it takes to do so.
2) Plays-well-with-others: the phrase "team player" is over used. I'm just talking about someone who gets along with others, who people like to work with.
3) Bugs: How many bugs in their code? How serious?
4) Code Quality: Is their code readable? Extensible? Good unit tests?

and I'll rank each types on these dimensions as: fair, good or excellent

Ok so here goes

Type #1: The Communicator
This is the developer that has great communication skills - speaks really well, listens and writes well. They can communicate complex topics easily and also they're great to pair up with business analysts/product managers/business folks to help tease out requirements. Very valuable when you need to have someone work with other teams. Good code but not the greatest but often it's readable and extensible and well thought through.

Gets-it-done: Good
Plays-well-with-others: Good to Excellent
Bugs: Good
Code Quality: Good to Excellent

Pairs well with "The Grouch" (see below).

#2 The Diva
This is the person who can just blow you away with the amazing product they create but boy do you have to massage their egos and keep them happy. In a crunch when things *must* get done you're just not sure which side of them is going to show up.

Very valuable often in the creative design or prototyping phase when they can come up with fabulous things no-one can think of or no-one thinks is possible. On the downside, they often have a surprisingly high number of bugs often which they attribute to the (lack of) intelligence of the user.

Not someone that works well with other teams! And if they don't respect your technical acumen - BOY! will they try to run roughshod over you. And God forbid you give them a deadline or pressure - the moaning and complaining and drama! AAGGHH!

Gets-it-done: Fair-to-Good
Plays-well-with-others: Fair
Bugs: Good
Code Quality: Good (not as good as they think though!)

Often Associated with: The Genius.

Side-Note: What's really funny is when you've got TWO divas on your team. They really tend not to like each other. Ironically because they think the other one's insufferable. Oh boy!

#3 The Genius
This is the one that knows the technology platform inside and out. They can come up with great solutions. But too often they come up with great solutions that no-one needs and/or the solution just doesn't work well with other teams' work.

They have great code - unit tests, readable, extensible, little duplication. Low numbers of bugs but when there are bugs it's usually a *huge* one that suggests they really didn't think through how their work would be used.

Their throughput isn't great as they tend to overthink things - so those huge bugs might take a week to address even if it's blocking everyone. They're similar to the "Diva" just without the drama and they're pretty level headed. Also similar to the Diva they don't work well with deadlines. They also have the nasty habit of checking in major things right before everyone else's deadlines - and then the fire drill that ensues . . . . . . . sigh.

Gets-it-done: Good
Plays-well-with-others: Good
Bugs: Good
Code Quality: Excellent

#4 The Fedex Guy or Gal (a.k.a. "The Savior")
When it "absolutely, positively has to be there overnight". Reliable, solid people with good code, little drama. Just gets it DONE. Period. This person is the savior for the team lead or project managers worst moments.

Often plays well with others, may not be the best communicator and has a fair few bugs but you don't really care because they fix them so fast. Also when it's fixed it's FIXED! May not be the smartest one or the most creative but they don't need to be - they lack the drama and often are good learners too. Good to pair with a "Diva" or "Genius" as this person is often more practical and will fix critical bugs fast.

Gets-it-done: Excellent
Plays-well-with-others: Good
Bugs: Good
Code Quality: Good

#5 The Grouch
Entrenched in the organization often through many years experience. They have some knowledge or skill that you just don't think you can replace. Not necessarily a bad or nasty person. It's just they're grouchy - you don't want to ask them a question or give them a task.

They get the job done and don't cause too much hassle outside your team. They fix bugs reasonably fast but often the code is unmaintainable or unreadable (can you say "Job Security"?) and they don't want to listen to you about FindBugs or Unit Tests or things like Refactoring - yeah that class with 40,000 lines of code looks GREAT!

Gets-it-done: Good
Plays-well-with-others: Fair to Good
Bugs: Good
Code Quality: Fair to Good

Good to pair with "The Communicator" who might be able to help smooth out rough edges.

And then there are a few types that we'd all wish we didn't have to talk about but often must deal with . . . . . .

The not-so-smart one
Ouch, you wouldn't have hired this person if you had interviewed them but often you inherited them. They don't want to learn. try-catch-finally blocks scare them. They use arrays because they learned them in C and learning about a Linked List seems unnecessary (and they just love fixing ArrayIndexOutOfBoundsException).

They're just not interested in staying current or learning from others. Usually every problem is someone else's fault or someone else's responsibility. Their solutions lack any creativity or planning. They're just not cut out for this.

Gets-it-done: Fair
Plays-well-with-others: Fair
Bugs: Good
Code Quality: Fair

The A-hole or The Bully
Yes this happens. This is the developer that throws everyone else under the bus. Often associated with "The Genius" or "The Diva" this is the one to be very wary of. If you share a boss in common you're going to need to keep your ear to the ground to make sure they're not throwing YOU under the bus.

They're often tolerated because they're really smart or they work in a culture of blame and backbiting where it's tolerated. Often it's wise to have a network of friends in your workplace who look out for each other - usually this guy (and let's be honest it's usually a guy) can be sidelined if lots of folks stick together. Another approach is to befriend this person. Really listen to them and try to bite your tongue. Get their opinion on things (as they're often high on themselves) and pretend like you care about it :-) Then they'll often go after someone else.

Also don't email them anything - they'll twist it and use it against you. However you *must* reply to them if they attack. Which they do. The hard part is when they attack without you knowing.

As the team lead I will go 110% out of my way to defend myself and my team from these ones. Boy do they irk me. Best of all they try to throw their intellectual weight around. I *love* when they try to do it with me - try, just try! Oh and when they screw up (no-one's perfect remember) the payback from all the other teams - WOW!

When they leave or are fired there's a palpable sense of relief across the organization. Often your boss knows all this well enough - they're just looking for the right opportunity to replace or neuter this one.

Gets-it-done: See Diva/Genius.
Plays-well-with-others: Awful
Bugs: See Diva/Genius
Code Quality: See Diva/Genius

Clearly this one generates a *lot* of emotion with me - people deserve to work in a respectful work environment. It's hard to hire a good team and then people like this make developer's lives miserable and want to leave. That's bad for the team and bad for the organization as a whole.

Wrap-up
The biggest conclusion I've taken - and a big surprise to me - it doesn't take a Superstar! In fact I don't think such a person exists. You've got different people with different skills and personalities. No-one's got it all. Although you might think you do *cough* Diva *cough* :-)

Often you get great productivity and great product by the right pairings - the Diva with the Fedex guy/gal. The Grouch with the Communicator. As long as they're WILLING to get along you can really pull together a great diverse team of people whom it becomes a pleasure to work alongside.

How to file a good bug report

2008-03-15T13:16:00.007-04:00

One of the first lessons, hard lessons, you learn coming into the world of software development out of college is how hard it can be to understand bug reports.

Variously they can be vague, misleading, inaccurate, confusing or best of all - misunderstanding a feature for a bug (the latter often points to usability issues).

Having endured many of these and now doing a fair bit of testing of my current application, I try to "be the change I wish to see in the world". So I've come up with the following scheme for bug reports.

#1 Start with a good short bug summary
For example:
"Data Entry screen disappears after entering certain unusual keystrokes"

The good part of this is a manager / team lead gets a quick idea of how serious this issue is (screen disappearing - bad!) where it occurs (data entry screen) and what caused it (unusual keystrokes). Useful in understanding just how serious it is and how soon it should be reviewed and/or fixed.

Next up you need to focus on the needs of the developer - basically instructions so the developer can reproduce the issue

#2 The *exact* build you are using
e.g.
March 27th 18:05 PM

Sometimes developers get dupe issues with related if not similar symptoms and having the exact build they can often realize "Hey that's that same issue I fixed yesterday" and they can then tell the tester - try the 28th build because it's related to issue X we resolved.

#3 Environmental Aspects
Including
- What machine the different parts of the system were running on e.g. web server, GUI etc.
- What database you were pointing at and DB login if appropriate
- How you installed the build e.g. options
- Your GUI login if appropriate
- If you can, it's really awesome to leave that build around and up-and-running so a developer can come over to your area and check it out for themselves. Not always possible but for those big critical issues it's a real time saver.

#4 Instructions to see the bug
The steps need to be detailed, step-by-step.

Often when I report a bug I spend a few minutes trying to get the least number of clear steps to reproduce an issue. This is important not only for the developer but for the person who subsequently tests the fix perhaps minutes, day or even months later.

You don't want to be too detailed "point your mouse to the 'Send' button and left-click" or too vague "enter some data and hit 'Send'"

1) Login to the GUI
2) Go to 'Tools > Data Entry > Advanced'
3) Select the 'Schedule' tab
4) Enter '03/04/08' and then without saving try to tab off
5) You get a System error (see attached)

5 Supporting Documentation
Screenshots are great 'proof' - especially for those developers you may know, who try to reproduce the issue for 5 seconds and then say "Unable to reproduce" and cancel your bug which you spent quite a bit of time entering. Very annoying!

Some folks who have long complicated steps often choose to add video capture too which is great (but bulky).

Logs are great too especially when they contain more useful errors related to things the developer can grasp e.g. Malformed SQL on line 223 of DataEntry.java.

Often these are attachments and as a technical lead I often trawl through the logs to find the most important lines e.g. often the first stack trace and then copy-and-paste that into the main bug description.

Other things you need in a bug report
- A unique identifier for this bug report
- Priority - Stick to High, Medium and Low - beyond that lies madness! :-)
- Severity is optional - often I find it confused with Priority for various reasons- again three levels should suffice - major, moderate and minor/cosmetic
- Who created the issue and when?
- Who is assigned the item?
- Who is the tester going to be?

Good bug reports take time. But it's worth it - I figure for every minute I put in to the report to make it clear and concise I'm probably saving 5 to 10 minutes of a developer's time and then more for the person who might have to test it (might be me 6 months from now).

As a team lead also I'll reject reports that lack these details. It's funny I've known some experienced QA folks who just can't seem to get this right - it's very frustrating to start a back and forth - which builds is this - please add logs - can you add a screenshot?- and then there are some who write them so well it just becomes a slam-dunk for the developer. Often they're the ones sitting right beside the developer - they've each learned how each other works - how to keep each other productive and keep frustrations to a minimum. That's good all around.

No surprise here - offshoring losing popularity

2008-02-27T19:59:00.006-05:00

So the worm has turned eh? Offshoring of IT jobs is losing it's popularity.

"Among respondents whose firms previously had ceased offshore operations (61 respondents), the most popular reason cited for doing so was that the operations required too much management and close oversight. Additionally, 30 percent of these CIOs believed that they weren't realizing expected cost savings."

I can certainly attest to that - software development is a very interactive profession - so much of what we communicate never gets written down, much is communicated informally at the water cooler etc. When a person is not "on site"- whether they're in Boise or Bangalore the job becomes more difficult.

I've been on both sides of this - having a client in Ohio while my team and I worked in Boston was pretty tough - being in person was so much easier. Also I worked for a large mutual fund firm on a project where half the team was in India. We did weekly video conferences and bi-weekly 6.30am phone calls. It was hard! With the shortage of IT workers now in India (since everyone pretty much outsourced there at the same time) we found it very hard to hire (and impossible to retain) senior developers. Being the architect I felt I had to step up and that was fine but the other local developers found it a burden.

The really hard parts were cultural differences and inevitable misunderstandings but most of all the time difference was the most frustrating. With almost 24 hours between responses (you write an email at 9am EST, they won't answer it until, say, 10pm but you won't read it until 9am the following day) it's hard to keep momentum especially on critical issues.

More than ever these experiences taught me that keeping development teams close together - ideally customers with management with developers with architects helps get to success quickly.

Offshoring won't go away and I don't feel it should - I really enjoyed the intercultural and travel opportunities - but this just points to those "bean counters" who think software development is like "brick laying" or something that scales well when you add people - you just need bodies in seats - the cheaper the better.

It's NOT! Perhaps we should buy them copies of The Mythical Man-Month

10 things Project Managers wish Developers understood

2008-02-18T12:38:00.023-05:00

Well I've been very busy the past few months - most of it adapting to a new role as a Project Manager - technically I'm called a "team lead" or a "director" or a "senior manager" - depends who you ask, but when it comes down to it I manage the dev team to a project plan in cooperation with a product manager.

Being on the "other side" now for a while it's been a real eye-opener. It's definitely true that it's hard to relate to someone until you've walked in their shoes for a good bit. Having worked with project managers several times before, I've always wondered about why things were a certain way - ways I thought I would have done things differently.

Well not anymore - now having "walked the walk" I thought I'd blog about the things that most developers on a team won't realize until they are the person in charge.

10) Information Overload: I wish I could focus on one thing at a time but I've got a million people coming at me from all directions.
There's product managers, support, management, customers with their needs, developers, other team's manager, architects and developers. That's a lot of relationships and information to manage so don't be surprised when you encounter the following type of interaction.

Developer: So, you remember that issue with the GUI last week?
(subtext: that really critical issue I finally figured out the solution to?)

Manager: Which issue is this?
(subtext: I just spent three hours replying to emails to 14 different people and I'm wiped)

9) Work Pressures: The Manager can get too excited or stressed by the pressures and try to "tell you the solution". Push back (gently) if it doesn't make sense.
When you're the manager and don't code - you get all the responsibility (worry) about meeting the deadlines but very little control over it.

Those managers who have spent time as developers sometimes like to think we can just give other developers the answer. But that doesn't work - developers are a creative and (for better or worse) egotistical bunch (me included) and we want to impress our bosses rather than just take orders.

I've learned to ask guiding and information-seeking questions rather than suggest solutions e.g.

"So how many files are impacted by this change?"
"How will this SQL change impact performance?"
"How will the GUI change impact our key user flows?"
"So if we fix this bug, what are the odds we might break something else?"

8) Project Risks: New technology = Risk
Anytime you kick off a new big project, developers love to use new technology - but the effort can't all be about new technology - with new technologies come risks to the project. There may be efficiency gains but do they outweigh the risks?

I've had managers who just said "No" to new technologies, but they didn't understand the desire of most developers to improve themselves and learn the hottest new technology (e.g. Ajax) / framework (e.g. Spring). It was so demotivating. Also there's often some productivity gains to be had from trying new technologies so it's a short-sighted idea to adopt a blanket "no".

I've also had managers who only said "Yes" and then the project got bogged down in technical issue after technical issue and didn't deliver the functionality anywhere near on time as the developers had a large learning curve (or several curves) to overcome.

As a developer I understand how developers love to learn - but as a manager I have to balance the motivation of my developers with the goal to deliver a functional product.

7) Productivity: The best managers insulate developers from the craziness that goes on in middle management - bad specs, crazy ideas, changes in priority, too many meetings.
You think your pointy haired boss is nuts - you should see the people he/she works for! :-) The problem with modern day software is there's so many teams involved - the dev team, the QA team, support, client relationship, consulting and everyone's got their own viewpoint, their own goals etc. They see either bigger pictures or completely different pictures entirely.

Some developers have some idea that this goes on - if you do thank your boss - they're buying you time to be productive and do the things you love - Code!

6) Capitalism: The reason this company exists is to make money - sometimes you have to do some apparently silly things to get the money.
Customer's aren't always logical, they have their own goals, their own viewpoints (seems to be a theme here). Sometimes the Sales folks have to commit to things to get the deal. They sound reasonable enough unless of course you have to implement them!

Yes you have a right to be ticked off at folks who oversell - but then again the company you work for is (hopefully) there to make money. The Sales folks are the "rainmakers" so the best you can hope for is to build a constructive relationship whereby they consult with you about what's possible and what's a stretch.

5) Doing the things we don't like: We have meetings because they're needed
I know meetings interrupt your favorite activity - coding - but they really can't be helped in some cases. I need to know that you're aware of what's going on - frankly I have very little insight into how things are going until things are nearly done.

Fortunately as a now sometimes-developer I use tools like FindBugs, PMD, Checkstyle etc. to check on the code and use Unit Tests and Code Coverage to see how's the quality roughly holding up. But all those are proxies for the real thing - the quality of the product from a user's perspective.

So how else can I quickly get to know how things are going - meetings. Sorry!

4) Information Overload II: I can't be expected to remember EVERYTHING.
I forgot we promised to meet to talk about "that" question. I wanted to put a reminder in Outlook but I got interrupted twice and then I was called to my boss's office - I forgot. Be gentle on your manager - they're really trying (or should be) - if I make a mistake let me know. Keep us in the loop - just find the right moment.

3) Information Overload III: Hold off on the details - First tell me why I should be concerned
Developers come in all shapes and sizes - there are the ones who bottle up all information, stay heads down in the cube and only come out when they're done. The scary thing there is you don't know they're done until . . . . well . . . they're done. I can manage that one pretty easily - I try to "ping" people and check-in. Hopefully not too often.

The other side are the developers who think you want to know every piece of minutiae and that you understand the impact - for example:

Developer: "So you know that new Finder class team X implemented? Well they created a new Factory method which isn't thread-safe. I mean they're nuts don't they know that our thread pool will etc. etc. . "

Manager (who was deep in figuring out critical dependencies in the project plan): (thinking) is he gonna think I'm a moron if I say "Huh?"

It's better to say what the impact is and hold off on the details e.g.
"I think our new multi-user scenarios are going to have some problems with the new code from team X".

That gets to the point (there's a problem) and stops me from having to think too hard (why should I care about thread safety in a particular class) - you should be able to tell from the smoke already coming out of my ears? :-)

2) It's not all about you (the developer)
I was guilty of this one for a while when I was just a devleoper. I really thought that of all the issues faced by the team mine were the most significant. I'd get peeved when issues I felt were important were being ignored. But the reality was I wasn't aware of many of the points listed above. Wow, perspective is everything.

1) It's good to be the King
No it's REALLY good :-) Previously as an Architect or lead software engineer I had a position of influence but little power. Now I have people reporting to directly to me. I control their reviews, their raises, how much they work and how late, which project is next. I still consider myself an "Architect" at heart and I love being in the code - but it's nice; knowing the technology, how developers are often motivated etc. and now to finally REALLY have a big say in what gets done and how.

For example, there's some really complex code we've written lately - quite fragile unfortunately due to it's (necessary?) complexity. As the developer I probably would be too lazy to write the full set of unit tests or wouldn't have the time. As the architect I'd know that we need to do that but very often I could never convince the project manager to let the developer spend the time. Now I'm project manager and can read the code coverage numbers as well as know many of the user scenarios.

So I had the developer write about 40 unit tests covering a very comprehensive range of scenarios. It took about a month to get them all done (ouch!). But in the process we actually found three new bugs. Ultimately though we've mitigated a lot of risks that were in that component.

Now when new bugs come in from the field or from QA, we fix them more quickly and then we can quickly re-run the unit test suite. We also add unit tests for that new test case we just discovered. Now I as a manager I know when the unit tests run green - that despite the code complexity, we're in pretty good shape from a risk/quality point of view.

It's a balance of course - we've discovered more unit tests we need to write for this component (based on how our users are using it in beta) but we're under the gun for the next few weeks in another area so we'll get to it - but not yet.

It's also good when it comes to hiring time - too few managers know how to spot a good developer - having been a developer and spent a good deal of time on screening questions and interview techniques I feel that I and the team are pretty good at identifying strong folks. I usually do the screening so that the developer's on my team only spend time interviewing really strong candidates who don't just "talk the talk" or have keyword-heavy resumes and don't really have the talent to back it up.

Postscript: I'll also write an article coming from the other perspective - things that developers wish managers knew.

Agile Programming in a nutshell

2007-11-27T19:37:00.000-05:00

See this Link

When Dilbert mocks you, you know you're mainstream! :-)

My Favorite Java and Software Engineering Articles

2007-08-24T22:26:00.001-04:00

Here they are - a collection of quite a number (250+) of the many online articles on Java, Software engineering (architecture, the development process), XML etc. that I've read over the last years - probably going back to 2001. I've included a URL and a personal score (on a scale of 1 to 5 with 5 being awesome).

I hope you'll find this useful - some of these articles took quite a bit of digging to find but were well worth the effort.

http://spreadsheets.google.com/pub?key=prCSPRhj1PCGuYZvjEITgXw&output=html

Please feel free to shoot any comments, edits, suggested additions (or deletions?). Because the underlying spreadsheet is on Google Docs (love it!) it will get updated on a semi-regular basis as the reading list continues to expand.

4 reasons it sucks to be in QA

2007-08-06T20:46:00.000-04:00

Well that's my take on it folks - of all the roles in the Software Development Life Cycle probably QA has it worst. Managers, Architects, Developers, Business analysts - no-one takes as much BS or has such a thankless role as the valiant folks in your average QA team.

Now I hope most of you reading this will disagree but I'm betting there's a lot of nodding heads out there (probably mostly from folks in QA).

What are the main reasons for this you say? There are four as I see it:

1) Respect
QA tends not to get respect from people in other roles - but most especially and most oddly the least respect (most disrespect) comes from software developers. Personally I find this strange that developers have such disrespect (sometimes even open and outright disdain) for those tasked with ensuring the quality of the product being delivered.

I think a lot of developer frustration comes from the questions and problems that arise out of the QA team/group. Yes, often QA folks don't understand the ins and outs of an application. If that's because someone in QA doesn't want to learn then that's a worthwhile beef but more often than not the QA team are STARVING for information.

Often they're brought in at the last minute, given a vague, incomplete and out-of-date spec and told "Hey go test this!". Yes the developers had the same spec and were told "Hey go code this!" but they're finished building out the app so they've (hopefully) fleshed it all out at least in their heads and in the software they've built.

Developers deride the QA role - "What do those guys know? They're just testers!".

That's just the wrong attitude. These guys and gals are the ones who, at the least are helping to make the product (not code folks - product!) better, and at best saving your ass from being fired for shoddy work. You and they are on the same team though you may not realize it. Try to remember that.

Another reason that I feel developers beat on QA is that the developers get beat on themselves (think Dilbert) and like some bullied kids they look to take it out on the only one's they can - someone weaker - not managers, not architects, not the business analysts, not other developers - who's left? QA!

I've seen it many-a-day - the self-righteous attitude developers take in correcting QA or better yet informing them that their "bug" won't be fixed because it "works as designed" or they spent 5 seconds trying to reproduce the bug and could not.

I'm not saying "works as designed" or "unable to reproduce" are a bad reason to kill a bug - just the attitude that goes with it. Call the tester - explain why it works that way - should the docs be updated to make that clear? Possibly! If you can't reproduce a bug maybe there's some context or other missing information - maybe it's particular to the tester's environment. Just blowing things like that off might make a developer feel good but it diminishes the overall product and the team too. Take the time to do it right - a phone-call or walk-by will take 5 minutes at most.

2) Responsibility vs. Power
The QA team have a very important responsibility - they are the gatekeepers of product quality. At least in theory anyway! Also quality is a hard thing to identify or quantify. Is it the overall user experience, the number of bugs, the number of critical bugs, how do you define "critical"? Just too many subjective things. Oh and usually they're understaffed and/or under-tooled.

However for all this responsibility they have the least power. In the hierarchy of management structures QA comes pretty much last.

Everyone else thinks they can do the QA job - how hard can it be? But the reality is very few people can do QA right. And especially not developers! We are our own worst enemies in this case - we use the application like we coded it not like it was meant to be used.

3) Last to be hired and First to be fired
In the boom and bust cycle of software QA folks seem to be the last to be hired - usually towards the end of the boom when all your new customers start to complain about the bugs from all the newbies you hired at the start of the boom. But then when the bust inevitably comes QA folks are first on the chopping block followed by middle managers. No wonder there aren't many experienced QA folks about - again it's a widespread lack of understanding of the importance of QA and testing in the software process.

4) Paradox of excellence
The "Paradox of excellence" is something we're all familiar with but may not know it. Since the art of software (it ain't engineering or science yet) is not well understood, few outside of software can understand how hard it is to do software "right" - with few bugs. It takes a long time. Therefore when software is done right people don't perceive the absence of bugs - they just see that the process seemed to take forever to complete.

Basically the "paradox of excellence" states that you don't get credit for problems that never happened. Typically in the normal software shop when bugs happen and they are fixed fast by developers the developer gets kudos. Who gets kudos for the lack of bugs in a release? No-one.

So if QA does their job right - no-one notices. The same is true for developers to an extent but they can usually redeem themselves if and when bugs do show up. QA have no such opportunity - if they *do* spot issues they becomes "spoilers" for the party giving the bad news no-one wants to hear.

With those four issues facing you as a person in QA - lack of respect, little power, no praise, fear of being laid off, how much motivation would you have every day?

In my opinion a rock solid QA person is one of the most critical roles on a project. In fact in my experience the best QA people know the application better than the developers, the managers, the analysts and even the users. They know the ins and outs - breadth and depth and know where most of the bugs pop-up! I wish most developers were so aware.

Solutions?
How do we resolve this problem for folks in QA? There are two steps - the first easy and the second not so much:

1) Co-locate QA and Developers
Have developers and QA sit side-by-side and work together closely throughout the entirety of the software lifecycle. Make them truly part of the team! Have lunch together!

Physical separation of QA from the dev team is a bad idea. Having separate reporting structures for Dev and QA might be good since having same reporting structure creates a little "conflict of interest" - management understandably want good news, QA are rarely bearers of good news - but it's news that needs to be heard!

2) Developers and management need to snap out of it!
Frankly this treatment of folks in QA is one of those little "secrets" in software that we should be ashamed of if we consider ourselves to be "professionals" in the true sense of the word.

Developers and management need to respect the critical role of QA, the complexity of their job and most of all to drop by their cubes and say thank you for all their hard work they do without much praise - day-in, day-out. Your product is better for it, the industry is better for it (and could do with much more of it) and thus the rest of us managers, architects and developers are better for it.

Another Google first - missed estimates - hiring to slow - share price drops 7%

2007-07-19T21:39:00.000-04:00

Looks like Google just missed analysts estimates on EPS (see article here).

Key Quote

In the conference call, CEO Eric Schmidt admitted that one area the company exceeded its expense plan was in hiring, which added more than 1,500 workers to the payroll during the quarter. "We are very pleased with the talent that we've brought on board, but going forward we will watch this area very closely," he said.

This is one of the key downsides to being a public company it's very hard to "surge" and take some time to build now for the future by hiring as many great people as you can.

That said with every other measure (revenue, operating income etc.) on the up-and-up Google is the still the one to beat.

In other news it looks like Yahoo! is having a very tough time and now there is talk (again) of Microsoft buying them.

Are Agile methods detrimental to your design?

2007-07-19T21:11:00.000-04:00

InfoQ has an interesting article entitled Are Agile Development Practices Detrimental to Architecture and Design?

I've said it before and will say it again TANSTAAFL - There Ain't No Such Thing As A Free Lunch (with thanks to Robert Heinlein). You can't expect to suddenly change to run an Agile project (e.g. using Scrum / XP) and expect miracles - that is a gain to your project without any associated pain or cost.

My feeling has always been that with Agile development you need quite senior people who can run the project fine without the heavier and better known "structure" of traditional processes. Also you'll hope that they'll pay attention to key architectural and design issues without a traditional BDUF architectural or design phase.

But regardless of the type of process you choose (BDUF vs Agile) there are three people (or three roles) you need on your project for long-term success that is release-after-release year-after-year.

You need
1) Someone who knows either the vertical area (e.g. Financial Trading Systems or the Retail sector or Cell Phone advertising) and where that field of expertise is going (e.g. more analytics for trading systems, SaaS / ASP models for Retail).

You also need
2) Someone who knows the users and their likes / dislikes ( e.g. they like responsiveness and dislike overly flashy apps but may be amenable to Web 2.0 technologies such as twitter more easily) also if there are any key users to watch for (e.g. head traders or your CEO's best friend!)

And finally
3) Someone who understands what all of that means for the Technical Architecture and Design
and who can create a roadmap to get there . . . . .

Where do you find such folks on your team?

#1 and #2 are typically Product / Project Managers / Business Analysts
#3 is typically the "Architect" / Technical Team Lead for the project

The key value to Agile is getting to market quickly - combine that with talented senior people who are good at numbers 1,2 , and 3 and you've got this key architecture/design issue mitigated to a large degree. That said, it does depend on the little issue about predicting the future accurately! But so does all software architecture and design.

Forget SQL Injection you need to watch for XML Injection

2007-07-19T21:06:00.000-04:00

With all the XML flying around these days (SOA, ESB and Ajax) you need to be ever more vigilant.

Check out this article from DeveloperWorks entitled Avoid the dangers of XPath Injection

Now more than ever validate your inputs!