David's Software Development Survival Guide

Technical Design Essentials

2008-09-15T22:07:00.000-07:00

As far as the current state of software technology, pretty much everything has been written on technical design - but then again, it looks like Search Engines, with their current flaws, want us to write about the topic - and don't seem to lead to anything much relevant.

So here is a list of 3 books that are essential to better designing software and creating solid architectures.
These are classics that are worth much more than what they're being sold for.
With one exception, the list weights heavily towards pure Object Oriented design and concepts. This doesn't mean that highly-structured OO designs are invariably better than the light-structured ones - it just means that in order to make a technical choice, you need to truly understand what your options are.
Onto the list:

. UML Distilled - by Martin Fowler. The focus of this book is as much on UML (which is a critical tool) as on the development and technical design process (where Fowler truly shines). Fowler is able to articulate in a very concise manner the process, challenges, and best practices of creating great OO designs.
Here are two great quotes that convey well Fowler's tone and philosophy:
. "No user is going to help you for pretty pictures; what a user wants is software that executes"
. "A hard deadline works well in concentrating minds"

. Design Patterns - by Gamma, Helm, Johnson, and Vlissides; aka the Gang of Four
This is an absolute classic.
The book contains 23 software design patterns.
There again, the actual patterns are interesting, but what is truly unique and invaluable is the description of the thought process underneath them.
The authors also lay critical principles of good OO designs - including these two:
. "Program to an interface, not an implementation"
. "Favor object composition over class inheritance"

. Building Scalable Web Sites - by Cal Henderson
Cal Henderson played a key role in building the Flickr architecture; so he knows a thing or two about building scalable web applications.
There is a couple of things that make this book extremely compelling:
1. It reflects on a much less formal approach to build solid architectures - the technology stack is LAMP; with PHP intrinsically putting less focus on the high-structure OO principles that Fowler or the Gang of Four are outlining
2. It touches on all the elements required to build high-traffic web applications - this includes general architecture options; authentication and security requirements; localization and internationalization; hosting; partitioning, distribution and clustering; performance and scalability.

As a bonus - here are 3 websites that are worth looking at:
. OO Tips, by Yonat Sharon, contains a great collection of notes on Object Oriented design. The content is sometimes too deeply rooted into Object Oriented dogma - but there again, knowledge is power.
. Mapping objects to relational databases, by Scott Ambler, is a great paper on the key design issue of Object-relational mapping. The article balances depth and clarity, and perfectly articulate how design choices are a question of trade-offs.
.The Anti-pattern catalog, on Ward Cunningham's web site, has an extensive list of design traps, that you can get familiar with - in order to be better able to identify and avoid them.

From Flip-Flop to Kung Fu via the Red Army - an alternative maturity model

2008-09-06T19:49:00.000-07:00

A number of years ago, the Capability Maturity Model was introduced as a way to measure the maturity of development organizations.
In a nutshell, the concept is that the least mature organizations are relying on undocumented processes, practices and knowledge; and that their success -if any- is the sole result of individuals' heroic prowess.
Grossly oversimplified, the CMM states that organizations mature by gradually developing repeatable processes; then measuring their efficiency, and finally optimizing the way they operate.
Somewhat of a Maslow pyramid of needs applied to organizations - from survival to ongoing self-improvement.
You can read here and there for a more detailed -and clearer- explanation.

The CMM is definitely a good way to "read" a development organization - and address its weaknesses.

Looking back at the development shops I've worked at, and the ones I've worked with; it seems that the maturity of these organizations could also be categorized along an alternative framework, based on 3 very distinct types:

. The Flip-Flop organization:
Flip-Flop organizations are on the lower end of the scale, and are a too common type.
Their essence and commonality lies in the fact that they are lacking focus and a sense of purpose.
Some employ very formal and repeatable processes, others are very Agile; yet they all achieve the same thing: not much, if anything.
A majority of the projects they engage in never come to fruition: they're either shut down in flight, experience a slow death, or do come to completion but with an outcome that has no significant impact.
The management style of these organizations could pretty much be anything - except focused, deliberate, and resolved.
The pattern is self-reinforcing as people quickly understand that there is no point in pushing their limits, that they won't be held accountable for slipping a date, and that they might as well focus on Resume Driven Development.

The common cure for fixing a Flip-Flop organization is to come down with a hammer, set objectives that you can't mess with, and transform it into...

. The Red Army development team
[Apologies to veterans who fought in Stalingrad and freed-up death camps; it's the Management I have in mind...]
The central concept in this type of organization is that it's a crime not to meet a goal or to slip a date - and you might get shot (or fired) for doing so.
Projects need to have aggressive completion dates before starting, and once initiated, there is no turning back unless an Act of God strikes.
There again, processes may or may not be formal/agile/repeatable - although processes do tend to develop very quickly when people need to set boundaries around their responsibility and accountability.
The overall throughput is much greater than in the Flip-Flop organization; the organization does get from point A to point B; but there are 2 major issues:
1. Efficiency is on the same order as driving a military-grade truck to go to the post office (a practice that was popular until recently). Mileage is poor and you need deep pockets in order to operate this way.
2. In software development, the point B defined when you were at point A, rarely has any relevance when you're about reach the said point B. Development is a process of discovery that requires constant adjustments, and markets are typically volatile and demand an ongoing validation of the strategy and objectives.

In order to fix a Red Army organization you need the organization and its leaders to develop a deep concern for both economy and impact - i.e. a focus on getting maximum results without expanding energy unnecessarily.

You then get...

. The Kung Fu organization:
If you're not a fan of Stephen Chow's movies or haven't read Sun Tsu the concept might sound obscure.
What lies within it is a constant focus on quickly assessing your current position, leveraging your strengths and the opportunities presented by the environment, and creating maximum impact with minimal effort.
This requires the organization to be deliberate and disciplined, yet flexible and creative.
In this type of organizations, you'll typically find:
- very lightweight processes
- very seniors engineers that have the respect of the entire organization, but yet are not unquestioned
- a leadership team with a constant focus on performance and results

In a way, Peter Drucker, might have been much closer to Kung Fu than one would have thought.

Persistence

2008-09-03T20:38:00.000-07:00

I'm talking about perseverance here -i.e. "to persist in a state, enterprise, or undertaking in spite of counter-influences, opposition, or discouragement" [m-w definition] - not about data storage...

I recently came across 2 great pieces that are uplifting for any individual or team facing obstacles, and in particular the lack of understanding or empathy from the external world.

The first one is coming from Guy Kawasaki: in a guest post on Sun's blog titled "Five most important lessons I've learned as an entrepreneur" - item 4 reads:
"
Ignore schmexperts. Schmexperts are the totally bad combination of schmucks who are experts--or experts who are schmucks. When you first launch a product or service, they'll tell you it isn't necessary, can't really work, or faces too much competition. If you succeed, then they'll say they knew you would succeed. In other words, they don't know jack shiitake. If you believe, try it. If you don't believe, listen to the schmexperts and stay on the porch."

The other piece is proof to Kawasaki's point - it's a letter from the NY MOMA that was sent to Andy Warhol in 1956.
It reads:
"
Dear Mr. Warhol:

Last week our Committee on the Museum Collections held its first meeting of the fall season and had a chance to study your drawing entitled Shoe which you so generously offered as a gift to the Museum.

I regret that I must report to you that the Committee decided, after careful consideration, that they ought not to accept it for our Collection.

Let me explain that because of our severely limited gallery and storage space we must turn down many gifts offered, since we feel it is not fair to accept as a gift a work which may be shown only infrequently.

Nevertheless, the Committee has asked me to pass on to you their thanks for your generous expression of interest in our Collection.

Sincerely,
Alfred H. Barr, Jr.
Director of the Museum Collections"

The letter ends with a P.S.:
"The drawing may be picked up from the Museum at your convenience."

The 5 Easy Steps to Creating a Bullshit Management Methodology

2008-07-27T21:39:00.000-07:00

I recently went through another 3 day management training class - and it was just excruciating; a real punishment for being a manager.
One of my lead, who was also attending the class, had me promise not to use this stuff at the office.

The management field is plagued by a slew of witch doctors that are selling management books and trainings, in the hopes of making millions, while attempting to impersonate true management theorists - such as Drucker, Humphrey, or De Marco.

There are 2 questions that keep on hitting me when I go through these types of trainings -I've stopped going through the books a while ago, but I don't always get, or choose, the option to skip the trainings-:
. What are the components of BS management methodologies?
. Why people eat the stuff?

I only have a partial answer to the second question and it probably has to do with why there are so many variations of Fast Food chains in the World.

I may have a more complete answer as to what is the fabric -if you will- of these management sub-methodologies.

So here are 5 steps to creating a Bullshit Management Methodology:

1. Pick a good, wide-reaching, and difficult problem - e.g.
. quality sucks in most products;
. communicating with people is difficult;
. change is hard;
. we rarely get to retire before age 30;
you name it...

2. Identify common-sense concepts and ideas that, if applied, do address the problem - BUT introduce a thick layer of theories, concepts, and acronyms on top of it:
Don't just say that Quality must be an overall process that needs to be constantly measured and improved; or that when discussing difficult topics with people, you should assess objectives, values, background, and emotions.
Instead, introduce concepts with unique names and acronyms: "method 3", "level 4", SSX, "black belts", whatever...

This step is absolutely critical and has 2 major benefits:
1. No one wants to buy a book that just says common-sense stuff. It's boring.
2. You put your audience on the edge - they need to memorize stuff, make an effort to follow you.
The idea here is that there should be just enough complexity for the audience to follow you, and yet prevent them from analyzing and questioning the virtues of what you're saying.
In other words, they should have just enough to chew on and digest.
And if it makes people hopeful and upbeat at the prospect of solving a pain or addressing a need - they'll ask for more.

3. Ignore other theories and authored work
You should avoid referring to other authors; especially if they're original, critical, thought-provoking, and articulate - they are your competition.
One caveat - it's OK, and actually recommended, to quote people who've given their names to a theory but haven't published much other than specialized research papers. So Maslow is always a great addition.
In any case, you should never list your sources and references in your books / training materials - unless your publisher forces you to mention other published authors.

4. Be assertive as if you had seen the Light and found the Truth
Do not invite self-criticism or doubt.
Push it as far as you can without ever showing an ounce of doubt.
It's OK to call your work a 'philosophy', but you should instead define a dogma.
The bulk of your audience isn't expecting to read Kant or Hobbes when picking your book. They're looking for a ready-made solution that can give them a comforting faith; not for some uncertain philosophical journey raising more questions than they have answers.

5. Be universal
The Management Methodology you're advocating should have the ambition to apply to all aspects of the audience's life.
The 7 Epsilons Quality Management theory is relevant to your kitchen, and Situated Power Communication should work on your spouse, kids, and dog.
And if it doesn't work; it means the reader didn't get it - and she should know it.

This raises the stakes -the Methodology is much more powerful and critical than the audience initially imagined- and simplifies the reader's problem: just buy this book, follow these steps, and all of your problems will be fixed. And you should sign-up for my seminars if they're not.

And when you're done creating your very own Power Management Methodology for the Enterprise - if you've ended up corrupting yourself, feel lost, burned out, or need to get back to the basics - you can always reflect on what Peter Drucker's was writing in the early '70s:
"An employer has no business with a man's personality. Employment is a specific contract calling for a specific performance... Any attempt to go beyond that is usurpation. It is immoral as well as an illegal intrusion of privacy. It is abuse of power. An employee owes no "loyalty," he owes no "love" and no "attitudes"--he owes performance and nothing else."

Measuring programming languages adoption trends

2008-06-10T15:09:00.000-07:00

I talked a while ago about perceived market trends in the programming language world: Java's stronghold seems to remain untouched, PHP is all the craze despite its intrinsic lack of structure, and the Ruby fad seems to be losing momentum - I actually thought it was pretty much dead until I started discussing with friends and was told Ruby is in fact alive and kicking.

How can this easily be validated?
Beyond formal polling -which is quite involved, expensive, and not necessarily accurate-, there are a number of ways to measure language adoption trends: analysis of book sales volumes, result counts in general-purpose search engines as well as blog-centered ones, or number of related job postings.

One of the key to understand strengths and limitations of each of these approaches is to put them in the context of the Diffusion of Innovation theory. In a nutshell, this theory states that 1. technologies spread by gradually addressing the needs of 4 types of users (innovators, early adopters, early majority, and late majority); and that 2. they do so in a non-linear manner, by typically expanding slowly until reaching the early majority, at which point they start expanding at a much faster rate (if and when the technology is successful).

Here is a rundown of these trend analysis methods, followed by actual results:

. Book sales volumes is an interesting source - as introduced by Tim O'Reilly.
One limitation is that the numbers could reflect interest more than actual usage - buying a book on Ruby programming doesn't necessarily mean that I'm actually using it.
From a Diffusion of Innovation standpoint, books would also tend to be purchased not so much by innovators or early adopters (books are typically not published when these categories start adopting a technology) and more by the early and late majorities - so sales numbers would probably better represent technologies that are already in the mainstream as opposed to the emerging ones.
From a practical standpoint, a major issue with this approach is that the numbers are not published on a regular basis.

. Volume returned by search engines is probably the simplest and most accessible method - I've tried to use it in my low-key analysis of web services adoption trends, and TIOBE publishes indexes established based on this.
One objection with TIOBE's index is that -as with any index- numbers can be taken at face value, with the audience ignoring their actual source and its intrinsic limitations:
1. You need to pick your search keywords wisely; and you'll always get a level of noise in the results. TIOBE uses " programming" as their keywords (as in "java programming" or "PHP programming") which might not be the best choice for say Actionscript (the more relevant keyword here being "Flex"). I use the language name as the search keyword - and I'm not sure how many of my results are related to jewelry when searching on "Ruby".
2. Web count probably tends to favor languages on which there is a large established volume of data -i.e. those who've reached the majority-, or those who are creating an unusual amount of buzz (and thus generating a large number of references).

. Technorati, or other similar blog-centered tools, is a useful variation on search engines - these probably tend to favor languages generating a lot of debate and discussions.
The volume is likely to be generated by innovators and early adopters, and is therefore expected to tend to over-amplify early technologies rather than those in the mainstream.

. Job postings are another place to look at.
In fact, the job market could be one of the best indicators of the penetration of a technology: this is where the data is not affected by the volume of interest, publicity, or discussions; but in fact represents where organizations actually stand in terms of executing on a technology.
One major caveat, is that organizations could more heavily recruit skills they don't have rather than the one they already have on staff – i.e. technologies entering the mainstream rather than established ones.
The location of the job offering could also say a lot in terms of the diffusion of the technology: in tech-focused areas (e.g. the SF bay area), the numbers probably tend to favor technologies that are at the stage where they access the early adopters and early majority; in more traditional areas, they would reflect more or less on the early vs. late majorities, depending on how conservative the area is from a tech standpoint (e.g. NYC vs. Washington DC).

To validate my hunch, here are the results of the above 3 approaches (excluding book sales because of lack of current data) for Java, PHP, Ruby, and Flex:
I'm using craigslist for job postings, and also normalizing results to 100% to help the comparison across sources.

. Google:
Java 487 millions; php 12,230m; ruby 130m; flex 95m
[4%, 94%, 1%, 0.7%]

. Yahoo
Java 908 millions; php 2590m; ruby 267m; flex 195m
[23%, 65%, 7%, 5%]

. Technorati:
Java 228k; php 141k; ruby 51k; flex 60k
[47%, 29%, 11%, 13%]

. Craigslist – SF Bay Area (all jobs)
Java 1125; php 773; ruby 257; flex 250
[47%, 32%, 11%, 10%]

. Craigslist - NYC (all jobs)
Java 406; php 606; ruby 143; flex 218
[30%, 44%, 10%, 16%]

. Craigslist - DC (all jobs)
Java 228; php 267; ruby 47; flex 98
[36%, 42%, 7%, 15%]

What does this say? Probably that I need to do some more digging...

- The google results seem to be out of control.
I initially thought this was because of all the pages ending in .php, but the results are not different by much if searching on text only. So I went onto Yahoo, which is different, but still weighted towards php beyond what I was expecting.
Plausible explanations:
1. Php is indeed predominantly used
2. Because of its nature and audience, php is much more "documented" on the web than java is

- The Technorati weighting is indeed quite different from the regular search engines and instead closely aligned with the SF bay area job posting distribution - but I'm not quite sure what to make of it except that I'm not really seeing the amplification of emerging technologies that I was initially expecting.

- In terms of job postings, the balance is leaning towards java in the Bay Area and PHP elsewhere.
Is it really about being conservative, or does it have to do with the nature of the projects in different locations?
Note also that there are less job postings in NYC and DC combined than in the Bay Area - so the question of what constitutes the "majority" might not be that straightforward.

-As far as Ruby is concerned; there is not much that the numbers reveal - is it still in it's infancy, or slowly winding down?
The only way to tell would be to look at the trending over time - which is a little bit more involved.

I might take another checkpoint in 6 months to see where we are.

Guidelines for good technical design documents

2008-05-05T00:02:00.000-07:00

I recently realized that Yahoo! search gives me a great ranking on these keywords - but then doesn't link to anything relevant.
That's a great question nonetheless... So here are some guidelines.

What makes a good technical design doc varies greatly based on the environment, the level of uncertainty and changes expected, the level of formalism expected or required, the audience and stakeholders, and how the document fits into your SDLC.

There are some guidelines that are good to keep in mind in any case:

Only be as formal as you need to be - sometimes you don't need a document; discussions and brainstorming on a whiteboard might be sufficient
Start on the right foot by giving your document a relevant and clear title, a file name consistent with its content, a version history table, and a table of content
Don't write a Victorian novel - use bullet points and tables, and keep sentences short
When documenting object oriented designs, make sure readers can quickly find and grasp each class' name, responsibility, and relationships to other classes (see CRC for more details)
A picture is indeed worth a thousand words - block diagrams, class diagrams, ER diagrams, and interaction diagrams are great tools that are easy to create. Keep them simple (their intent is to give the big picture), and make it easy on yourself (if you're good at MS Visio great; otherwise paper and scanner might be sufficient)
Make it as short and condensed as can be - the objective is not to write as much as you can; the objective is to describe the technical design as precisely and concisely as possible - most of us are not great writers, and most of us hate reading verbose tech docs
Do yourself a favor, read the Elements of Style by Strunk and White - it's short and to the point, and will help make the writing process less painful for you

Happy writing.

Fixing performance reviews

2008-04-06T22:22:00.000-07:00

Performance reviews can be a great tool - both for managers and contributors - but most often, their intent is missed and their essence diluted in corporate formalism.

Here is a rundown on some of the issues, and some thoughts on how to fix them.

In most organizations performance reviews are comprised of:
- a written self-review, where employees can reflect on their own performance
- a written manager's review, where managers should structure and formalize their evaluation
- a review meeting, where managers can formally deliver their feedback, and discuss with employees

I believe these 3 components are essential, as they allow proper reflection and communication - on both ends.

To spice things up, performance reviews are often driving, and followed by, salary adjustments.
This sounds quite reasonable as you do want to compensate people based on their performance; but this is also putting a lot of weight on the exercise - most employees get less open to feedback when they think that the perception on how they performed on such or such project is solely guiding their paychecks
[People should instead be compensated based on the overall value they provide to an organization in relation to market rates, as opposed to strictly on short-term performance]

The crux of the problem often starts with the manager's written part of the performance review. These written reviews are either too un-structured -consisting in a loosely defined "Manager's Review" section which will eventually be followed by "Employee's Comments"; or conversely, too formalized - I've had to deal with "systems" where managers needed to rate their staff based on over 100+ questions after which a score would be given.
These type of written performance evaluations result in review meetings with similar attributes - either a set of random recollections and thoughts, or a discussion around a metric system that employees try to game in order to get the highest possible score; hoping that their pay raise will follow.

To be truly effective, to both the reviewer and the receiver, performance reviews need to focus of performance, results and growth - and just that.

In order to accomplish this, I structure both self-reviews as well as reviews around 4 questions:
- What were your major accomplishments over the review period?
- What are the strengths you need to build on?
- What are the areas you need to develop?
- In which ways can we help you become more productive?

While the relevance of the first question is obvious; the remaining questions are meant to help both the manager and the employee understand what they're good at (performance and results are obtained by building on strength, not weaknesses); where they need to improve in order to get greater leverage on their skills; and how can they make a greater contribution.

The last key principle to better performance reviews is that the feedback provided during the review should be no surprise to the reviewee.
People can only get better at their game if they receive constant feedback on their performance throughout the year.

Modeling development capacity in offshore teams

2008-03-09T11:55:00.000-07:00

When discussing roadmap options -estimates, plans and development capacity- with Executives, I'm often challenged by their assumption that x staff-weeks of effort here (in the San Francisco bay area) is equivalent to x-staff weeks of effort for our offshore team (located in India).
The challenge for me has been as much to understand what variables affect offshore development productivity, as to communicate it to upper management - in a short and clear manner.

Before going into further details, I should say that
- Our overall development team is medium-sized (20-30 developers)
- Our offshore team represents less than half of that size
- Members of the offshore team are generally very solid and competent engineers; but
. They're on average 5 years more junior that the SF folks
. They have 6 to 18 months tenure with us, when the majority of the SF folks have been working with the product for over 5 years

When talking about differences in productivity, I'm therefore NOT referring to a potential difference due to one of the team being intrinsically less competent or talented than the other; but differences related to experience, background, and distance.

When I have to explain it in less than 10 seconds I usually say that because of the overhead created by the distance as well as the difference in seniority and tenure, we're typically able to get 90+% productivity (compared to the SF team) on 1/3rd of our projects.
For the remaining projects, the overhead is typically so important that it doesn't make any sense to engage on the project at all if it has to be done offshore.

So far, I've identified 3 major factors that affect offshore productivity:

1. Technical competence - meaning abilities, skills, and experience. In my situation, we're mostly affected by the difference in experience. In other cases, you might be affected as well by engineering abilities, meaning having the offshore team not being "as good" as your local engineering team.
The difference in technical competence can be modeled as a percentage of productivity of your local team.

2. Distance - no matter what you do, you will get some overhead related to distance: communication is more challenging when engineers cannot just walk to the next cube to discuss an issue or validate a solution.
The best way I've found to mitigate this is
a. To have the offshore team work on their own set of projects – i.e. minimize the amount of cross-continent communication by allowing the offshore team to work on the same issues and resolve them internally
b. To have an engineer on the local team being dedicated to be the point of contact for the offshore team. This helps understanding where they stand, what their progress and challenges are, and channel and follow-up on communications between the 2 teams
c. To have proper communication solutions in place such as conference calling and/or video conferencing, instant messaging, servers for file transfers, and the like
d. To have regular checkpoint meetings (at least twice a week) with the remote team

While you can very significantly reduce the distance overhead, some will remain.
This can also be modeled as a percentage of productivity of your local team.

3. Nature of work. This is where it gets tricky. There are a number of components that make the nature of the job more or less appropriate to do offshore.
This includes factors such as
. Amount of background info required to do the job. - e.g. Rewriting some SQL code for performance would require close to no new context for a qualified developer to do the job. Conversely, writing a new piece of functionality at the core of your business domain, and using some proprietary framework, would require some significant ramp-up effort as well as ongoing knowledge transfer.
. Amount of iterations required or expected - how much design reviews and other cross-team validations are going to be required
. Size of the job (i.e. small vs. large projects) - this might greatly affect the above factors; e.g. a significant training/ramp-up effort might be more appropriate on a large project

Ultimately, the nature of the work might require both a fixed-cost increment of effort to start the job, as well as an ongoing increment of effort, that could also be modeled as a percentage of your baseline productivity.

To sum it up; if we had to model the above, it would give something like

Offshore Effort = (Baseline Effort x Skill Percentage Adjustment x Communication Percent Adjustment x Nature of Work Percent Adjustment) + Nature of Work Fixed Cost Adjustment

That's probably a little over-the-top for managing day-to-day R&D with an offshore team; but it helps understanding the challenges.
This is one of these cases where modeling is more interesting than the model.

The most interesting part of this is that the productivity of offshore development efforts can vary widely based on these –and potentially other- factors.

What this also says is that offshore development initiatives are bound to be unsuccessful if the distribution and execution of work is not properly thought out.

Management Styles - Beyond X and Y

2008-02-07T12:18:00.000-08:00

I was mentioning in my last post 2 widespread -and related- management theories: MBO on one hand, and Theory X / Theory Y on the other.

Both theories are quite general and high level.
They're both deeply rooted in assumptions about what motivate people.
Theory X, which is authoritarian and directive, assumes people on the job are primarily motivated by money, fear, and/or the need for protection.
On the other hand, MBO as well as Theory Y, are centered on fostering a collaborative environment based on trust, commitment and reciprocity where the focus is on end-goals and results.
MBO/Theory Y assume that people on the job are primarily motivated by self-development and the search for personal achievement.

A number of additional management theories exist.
Some of them are bringing fresh perspectives both in terms of styles and methodology, while at the same time providing different spins on underlying motivational assumptions.
These ultimately give us more keys, tools, and options to do better jobs and bring the best out of our teams.
There are 2 of these I was familiar with -Joel Spolky's as well as Ken Blanchard's-.
I also took a look at Wikipedia's entry on Management Styles.
Here is a rundown.

Wikipedia's entry lists 4 styles:
- autocratic; where the leader makes all the decisions and drives them down the organization - the closest thing to Theory X
- paternalistic; where the leader also makes all the decision, but is more concerned about the well-being of employees - a softer variation on Theory X
- democratic; where decisions are made collectively - probably the closest thing to Theory Y; for as long as you understand that corporations' primary goal is not the well being of their employees, and as such are not meant to be democratic organizations
- and laissez-faire; where decisions are not made in a concerted effort - and which is probably Theory Y applied with no drive; or simply the absence of Management.
This nomenclature is interesting because it relates management styles to a form of political government.
Beyond the fact that, in my view, governments and corporations should have radically different goals [the protection and well being of their citizen on one hand vs. making a profit on the other], I can see a couple of limitations here:
The first one is that most of us put a judgment value on each of these forms of government -from unacceptable to desirable- with democracy being the only mature or reasonable system for the vast majority of us.
The second limitation is that these form of management -and the way they're described- seem to be primarily a by-product of the personality traits of the person in charge. The type of work at hand as well as the attributes of the team being managed -size, skill level, or seniority- seem irrelevant to the choice or application of these styles; when in fact they should be key drivers.

Joel Spolsky has his own -very original- way of describing 3 distinct Management styles.
. Econ 101 Management - this style is assuming that behaviors and performance can be driven by the prospect of financial rewards; and therefore assumes that the main motivation for people to work is money.
It uses compensation mechanisms as the main leverage to get things done; and ignores other motivating factors.
As pointed out by Spolsky, this style is both expensive and inefficient.

. Command and Control - this style consists in applying general infantry military methods in the workplace.
It is relying on imposed authority; and is based on the assumption that the subjects are ultimately motivated by fear, or a deep sense of duty.
If you want to play Drill Sergeant on a team of developers, you'll notice that it's quite inefficient and you'll probably soon end up on your own.
[While on the topic; I should mention that in a number of elite defense organizations, where members are highly trained and skilled, the management style can actually get quite sophisticated - see the book from David H. Friedman, Corps Business]

Spolsky' preferred management strategy is what he calls the Identity Management Method - it consists in encouraging people to identify with the organization, and implicitly assumes they will be primarily motivated by the need for Group Belonging.
Team building exercises as well as treating people with genuine care are critical as this strategy heavily relies on reciprocity to foster performance and loyalty.

The identity management method is a great approach that is virtuous and efficient when applied to development teams and other knowledge workers.
The only limitation that could be objected is that it is somewhat uni-dimensional - it doesn't tell me if and how I should vary my management style between junior and senior developers, or how to deal with an upbeat team after an IPO vs. a depressed and disgruntled team going through multiple rounds of layoffs.

To be most efficient, management styles should not only be applied at the overall level of a team, but should also vary based on general circumstances and individual situations.

I've found the most interesting bit on this from management seminars' Ken Blanchard
The method, called one-on-one leadership, consists in identifying where each individual stands in terms of
1. Competence - i.e. knowledge, skills, experience and capabilities; and
2. Motivation - i.e. determination, confidence, and commitment
Blanchard posits that management style needs to be adjusted considerably based on these 2 variables.
More specifically, they need to be adjusted in terms of the amount of direction as well as support being given:
Adjusting direction consists in providing different levels of granularity when stating objectives and giving instructions. A "low competence" individual typically needs a high degree of direction, with a clear and detailed description of what and how things need to be done. Conversely, a superstar developer would only need measurable goals with a description of the desired outcome and underlying intent -providing specific instructions, such as mingling into her code and telling her how to use such or such language feature, would instead be counter-productive.

Management Support consists in the degree of encouragement and validation that is given to an individual.
There again, this should vary significantly based on the level of motivation of the individual: an ecstatic new-comer might need to be paced, a developer going through the challenges of learning and growing needs a lot of encouragement and feedback; and our superstar developer might not need anything but a soft reinforcement of how much she is valued and appreciated.

To take it further, good management style should not only factor levels of competence and motivation, but also what specifically motivates each individual what they like, do best, and aspire to do better or grow into.

Management Styles - MBO, X, and Y

2008-01-13T19:38:00.000-08:00

[Ed Note:
1. Apologies to my regular readers for not having posted in so long - I got sidetracked... should be back to a 2 to 4 weeks schedule
2. The email subscription has been fixed - you can now use the feedburner email widget on the right to get notified whenever a new post is published]

Shortly after being promoted a manager, someone asked me which management style I was using.
I did find the question a little odd for, at that time, there didn't seem to be many choices in "style" given the field and the environment - technical, volatile, and with most people hating the idea of being "managed".
There are in fact quite a few approaches to managing technical teams; and understanding them is a good step in becoming a better manager.
Here is a tentative rundown:

One of the most known style of management is Management By Objective (MBO).
Popularized by Peter Drucker in the late '50s, its key concept consists in getting buy-in from employees on defined objectives and specific tasks, and asking them when they'll be able to complete their assigned tasks.
Commitment is here essential, as the method relies on:
1. the assumption that the contributor knows better about the task at hand than their managers
2. Once a commitment is made, the contributor will make every effort to deliver on her promise -see the concept of cognitive dissonance for a deeper explanation-

MBO was somewhat of a revolutionary concept at the time, for it was in stark contrast with the generally accepted idea that managers should get results by either knowing more about the job than the people they managed or by having the ability to coerce subordinates into doing things -Management By Intimidation if you will-.
These two approaches are completely irrelevant in the tech field where, if your team members are indeed solid contributors, they'll know more about their jobs than you do; and if you try to "intimidate" them they'll eventually raise their finger -not the one you want- and find a job elsewhere.

Another spin on MBO is Theory X vs. Theory Y, established in the '60s by Douglas Mc Gregor.
Applying theory X consists in managing people by putting emphasis on controlling -meaning not trusting and instead directing and measuring.
Theory Y on the other hand relies on empowering employees and giving them proper level of ownership to get their job done - which is very close with Drucker's MBO.

The key to both theories, and when they would be relevant and effective, lies in what motivates employees in their lives and on the job.
More specifically, theories X and Y are deeply rooted in the Maslow Pyramid of needs:
Theory X can only be appropriate for jobs that only satisfy the lower levels of the pyramid -i.e. "hygiene" needs / jobs that pay the rent.
Theory Y is conversely only relevant in environments where the higher levels of the pyramid (need for group belonging and "self-actualization") can be satisfied:
The need for group belonging makes it compelling for a contributor to deliver on her commitment to her peers, and not let her team down.
The need for self-actualization drives people to learn new technologies, apply new skills, and try to improve their performance.

In my next post, I'll go over a couple more theories on Management styles.

Lumberjacks, hackers, scientists, and engineers

2007-11-05T21:31:00.000-08:00

I was rambling in my previous post about 2 types of developers you might think you want on your team but you really don't.
Conversely, there are a number of profiles that are great assets to any team.

Developer's types is a larger question that has been addressed in a number of ways:
The Myers-Briggs Type Indicator (MBTI) is probably the most accepted way to categorize people and interpret behaviors. This is a great tool to better "read" people, understand what they might enjoy or dislike, and where they might excel.

In the Psychology of Computer Programming, Gerald Weinberg describes a number of personality traits required for being a great programmer:
- ability to tolerate stressful situations
- adaptability to rapid change
- neatness
- humility
- assertiveness
- sense of humor
These are all good attributes to look for when recruiting, or to develop for oneself.

In Managing Technical People Watts Humphrey makes a couple of distinctions among developer types:
- locals vs. cosmopolitans (grossly oversimplified: commitment to employer vs. technology or personal development)
- scientists vs. engineers (I'll talk about these in a bit)

The distinction between scientists and engineers is a great one.
Looking at the folks I have worked with, I see 2 additional profiles that are great assets to a team: lumberjacks and hackers.

Here is a shot at describing these 4 profiles:

Lumberjacks enjoy getting things done, and doing it well.
Lumberjacks are not necessarily great at learning, trouble-shooting, designing, or problem-solving but they're great at executing - often on a fairly established path.
There are a lot of people who do enjoy marking things done, so to be clear, if being done is the name of the game, we're only interested in those who mark things done when they're truly done and done properly.

Scientists enjoy learning, discovering, and inventing.
They enjoy finding order and simplicity from chaos and complexity; or in Humphrey's words "Scientists strive for knowledge and personal meaning. They see themselves as discoverers searching for hidden order and simplicity"
Scientists can appreciate the aesthetics of a solution - often more than the practical application of it.

Hackers enjoy getting problems fixed.
Aesthetics of a solution is typically secondary to end result, and any mean is valid to reach the end.
Hackers are typically great at learning and problem-solving but do not put as much emphasis on structure and meaning as scientists do – which can result in messy solutions if some discipline is not enforced.

Engineers enjoy building solutions and "seek to create their own monuments" (Humphrey).
The emphasis here is on structure and outcome, as opposed to the method or approach (as might be the case for scientists).
There is not necessarily a craving for learning new things, excelling at trouble-shouting, or for elaborating sophisticated solutions, but these qualities are important to make a great engineer.

So which one is best?
In a very small team (3 developers or less) versatility is key, and "engineers" tend to make things more efficient and predictable.
In larger teams, the mix and balance of these types is probably the greatest asset to an organization.
In fact, the challenge for recruiting managers is to make sure that not everyone is formatted along the same mold.
Diversity is what makes an ecosystem strive.

I have a java certification and I'm not afraid to use it

2007-10-14T23:27:00.000-07:00

Every now and then, my team and I embark into a large-scale recruiting effort in order to bring in a few good developers.
The effort is only "large scale" in the context of the size of the current team, the deliverables we're working on, and the fact that the market is hot right now for tech folks - which means it's tough when you're on the recruiting side.
At any rate, I believe we ended-up interviewing over 100 candidates.
Our requirements are simple:
1. can code
2. can solve problems
3. not a jerk

I've outlined in a previous post some guidelines for recruiting developers.

Folks who just don't have the right skill-set or experience are typically easy to identify.

There are however 2 types of candidates that can be particularly deceiving and -I hate to say it- you do not want on your team.

I have a java certification and I'm not afraid to use it
In retrospect, the thing that stroked me most on this recruiting round was the number of candidates who knew programming syntax and libraries by heart, and quickly jumped into coding solutions; only to produce a massive amount of code that didn't solve anything.
The embodiment of maintenance nightmare.
They're the "I have a java certification and I'm not afraid to use it" class.
As the name implies, these candidates typically have one -or even a few- certifications.
Certifications were great back in the 90s. They did give you an edge, and the folks who were carrying them were typically in the top percentile.
Nowadays, certifications are the easiest way for anyone who has some good memory capabilities to get into the programming field.
The problem is that programming is about solving problems.
Not pissing code.

Chatterboxes
The second type of candidate you need to be watching for is the "chatterbox".
Chatterboxes are very articulate, and often very educated.
They can talk in great details about what they worked on; and it sounds really great.
Complex problems; brilliant solutions.
If your interview process is properly setup - you then move into the programming / problem-solving part of the interview, and then nothing much happens.
You may be puzzled at the end of your interview round - the candidate made a great first impression.
Don't be.
When there is a doubt, there is no doubt.

The evolution of Web Services - Darwinism

2007-09-23T23:31:00.000-07:00

There are a couple of largely accepted theories that model or predict technology lifecycle and adoption patterns:
- The Diffusion of Innovations theory offers a model for how a given technology gets accepted and spreads through markets. Its central point is that technologies spread by gradually addressing the needs of 4 types of users: innovators, early adopters, the early majority, and the late majority (a fifth category, the laggards, might just never get it)

- The Technology Acceptance Model (TAM) offers some prediction to End User adoption. The key concept here is that individual users adopt a given technology based on its perceived usefulness and its perceived ease of use.

To my knowledge, there isn't an established theory or framework that models evolution trends of Technologies.
When looking at the history and evolution of web services, we seem to be in front of species that are spreading, adapting, and diverging much like finches in the Galapagos.

The immediate thought that then comes to mind is whether Darwin's Theory of Evolution has some or any relevance to Technology.

The theory of evolution defines three basic mechanisms of evolutionary change:
. Natural Selection is a process by which traits that are more useful in a given environment become more common over time (because they give better chances of survival), while traits that are harmful become rarer
. Gene Flow is the exchange of genes within and between populations, which translates in traits being transferred between populations and species.
. Genetic Drift is a purely random shift of the frequency of traits within a population - traits become more or less common in a population because of the long-term statistical effect of the random distribution of genes in each generation

How could these mechanisms apply to technology?

- Natural Selection is probably the mechanism most relevant to technology trending.
The fitter a technology is to the needs of its market, the more likely it is to stick around, and potentially supersede other technologies
This is why PCs are more likely to be found today than mainframes, why java is more often used than Fortran, and why soap-based web services have replaced xml-rpc.

- Gene Flow is also common in the tech field (although we'd probably want to call it something else).
Features and concepts are constantly exchanged between complementary or competing technologies.
That's how C# got a memory garbage collection mechanism similar to the one in java, and how row-level locking made it in MS SQL Server after years of Oracle claiming it as a key differentiator.
Gene flow is also at the root of hybridization, where traits of different species end-up being combined. This is what might be truly going on right now with REST - which is applying concepts of simpler web protocols, most notably HTTP and RSS, onto Web Services.

- Genetic Drift seems at first least relevant to the tech field, but might in fact be the most interesting bit.
The core concept in genetic drift is that the random distribution of genes in each generation can have a long-term effect on the frequency of traits in a population (because of the statistical law of large numbers, genetic drift is less likely to occur in large population than in smaller ones).
What, if anything, could have a similar impact in the evolution of technologies? What type of mechanisms, if any, can have an effect on the evolution and adoption of a technology, without being connected to its intrinsic fit or value?
Obviously there are a lot more forces that dictate the success or demise of technologies than just their core virtues.
A strategic alliance with IBM propelled MS-DOS into market dominance; technology companies like Oracle spend millions trying to influence the market; and there is a whole ecosystem of media, analysts, and venture capitalists who strive on generating buzz (PointCast or Twitter come to mind).

Who knows - if LISP had been able to be more hip, we might all be using more parenthesis today.

The evolution of Web Services - Measuring Trends

2007-08-28T21:55:00.000-07:00

In my last post, I've tried to summarize a history of web services; the most recent developments being around RPC-style vs. Document style SOAP-based web services, as well as SOAP-based web services vs. RESTful web services.
I'd like here to try quantifying how each of these trends is progressing - the larger question being about finding efficient ways to measure the evolution and adoption of technologies.

Measuring trends
There are a number of heavy-duty tools for measuring technology adoption; the most obvious one is probably to poll organizations that might adopt them.
This is quite involved, can be expensive, and is not necessarily reliable.
Other -more or less reliable- tools include assembling expert panels, gathering feedback from your clients, discussing with analysts, reviewing book sales, or trusting your own gutfeel.

Web publication volume and search trends
Google is another tool that might be effective - not to find information, but to measure adoption levels and trends.
The main advantage is that it's simple and cheap to use.
The first step consists in identifying keywords relevant to the trends you're looking at and want to compare - in the case of web services, that might be "xml rpc", "soap web services", and "rest web services".
Once keywords are identified, a simple query on Google tells us how many documents are found - the number is approximate and is actually more or less reflecting the number of indexed documents that were deemed appropriate, but it should give a good order of magnitude of the volume of information out there.
Separately, Google Trends tells us the history of search volume on these keywords, i.e. how many searches were executed on these keywords over time.

Web Services publication volume:
3.7 million documents are retrieved for "xml rpc", 28 million for "soap web services", and 133 million for "rest web services".
If we assume that these terms are equally reflective of the amount of publication on the Web for their associated technologies (and that's a bold assumption); what this means is that rest is leading soap - at least in terms of publications and discussions on the web- and that xml rpc is marginal compared to REST or SOAP based web services.

Search frequency:
Here is the compared search frequency graph returned on the 3 sets of keywords:

It's important to remember here as well that we're making another major assumption in terms of accuracy of these results - which is challenged by the major blips we're seeing on the graph.

What this seems to say is that there is still interest in xml rpc, that soap web services are still going strong, but that there isn't enough search history and/or volume on REST web services to show any data.

Innovators, early adopters, and mainstream users:
The combined results seem to indicate that xml rpc is still being used but is on the decline, and that SOAP based web services are alive and well.
The more interesting bit is that there is a huge volume of content out there on REST web services but few actual people looking for that information - especially in comparison to the ratio of content vs. search volume on soap based web services.

Other than challenging the use of Google for measuring trends altogether - one interpretation is that REST web services are indeed not mainstream at this point, but they are generating a lot of buzz.
Innovators and early adopters are the most content-prolific type of contributors on the Net, and by virtue of being a hot topic, a lot of content is generated and discussions are occurring over REST on the Web.
Separately, because the technology is not fully embraced by the Masses, little search volume is generated compared to soap-based web services.

This is in line with the theory of diffusion of innovation, which I'll try to summarize in my next post along with other theories of adoption, before taking a shot at looking at the relevance of the Theory of Evolution applied to Technology.

The evolution of Web Services

2007-08-14T09:00:00.000-07:00

There are a lot of changes going on right now in the area of Web Services technologies.
The transition from RPC to document style, and the recent buzz around REST, raise the question as to what the technology is evolving into, why, and along which patterns - if any.

I'll try here to do a recap on the history of web services.
In my next posts I'll try to take a look at the trends and review them against existing technology diffusion, acceptance, and evolution models.

A short history of web services
Distributed computing and interoperability is nothing new and the origins of web services should probably include efforts by DARPA to interconnect distributed computers.
In the enterprise world, flat file exchange via shared directories was a popular technique for exchanging information between different systems.
In the banking world, SWIFT -a private financial messaging network established in the '70s, with its own protocol and message format- is still a very much established standard.

Serializing the object model - CORBA, RMI, and DCOM
In the late '90s, a number of technologies allowing remote procedure calls (RPC) from one system to another became popular:
. CORBA allowed distributed software components to collaborate with a language and platform-neutral remote procedure call specification.
. RMI was essentially offering something similar, except it was entirely java based. As such it was somewhat easier to implement (at least for java programmers) but was also restricted to interoperability between java programs.
. On the Microsoft side, DCOM allowed native Windows programs to interact.

All of these competing standards had at least a couple major limitations:
1. They were restricted in terms of platforms they could interoperate with
2. It was a major challenge to have them work securely over Internet firewalls.

XML-RPC
The momentum java was gaining at that time, and its main value proposition to run anywhere, seemed to have prompted Microsoft to come up with a cross-platform interoperability standard, which could potentially help them get out of the corner they were slowly being pushed into.
In 1998, XML-RPC was advocated by Microsoft as a truly open technology leveraging the web infrastructure.
XML-RPC is essentially a remote procedure call protocol which uses XML format to encode its calls, and HTTP as a transport mechanism.
XML-RPC is a very simple protocol, defining only a handful of data types and commands - and this simplicity made it quite popular across major platforms.

SOAP
The XML-RPC standard quickly evolved into the more elaborate SOAP specification.
SOAP is a natural extension of XML-RPC that provides a greater level of structure and support for data types and operations semantic.

During the same timeframe, the need arose to provide more automation around message parsing, code generation, and discovery of web services over the network.
WSDL became the standard for defining web services interfaces; and UDDI became the standard for registering and finding web services on the Web.
There is today a considerable amount of tooling available for generating web services interfaces (i.e. WSDL) based on existing classes and interfaces, or for generating client libraries based on a WSDL.

Together, SOAP, WSDL, and UDDI form what is commonly referred as Web Services.

From RPC-style to Document-style
As Web Services usage gained momentum, a gradual transition occurred from RPC-style to Document-style.
In the RPC pattern, a client sends a request to a server, and the server immediately sends a response back to the client.
The RPC model is similar to existing programming models, and RPC-style Web Services are often implemented by simply generating services interfaces that map directly to existing language-specific function or method calls.
A major issue with this approach is the stability of the object model that these interfaces are based on: as classes’ attributes and operations change throughout their lifecycle, corresponding web service interfaces need to be re-generated accordingly, resulting in brittle and complex interfaces. A better approach is to provide a layer of abstraction by first defining the interface - a.k.a. Façade.
Another issue is that the RPC-style inherently requires synchronous processing – The RPC-style mandates the client end-point to be expecting the remote procedure call to return from the server-side.

Document-style Web Services offer a more radical alternative by putting the emphasis on publishing a schema, and exchanging messages, rather than on defining a rigid set of operations.
The Document-style approach is also known as the messaging model, or the SOA style (although the later term is overly abused, as can be seen on its Wikipedia definition page)

Unlike RPC Web services, loose coupling is more likely, because the focus is on the document "contract" that the WSDL provides, rather than on the underlying implementation details of operations.
As such Document-style services can be more agnostic to how, when, and by whom they are going to be consumed.
[A good technical article on Document-style web services can be found here]

Even when properly designed, SOAP-based web services often end up being complex to the point where directly coding against them would be a daunting task; and it is quite common to see enterprise web services packages heavily rely on the client-side libraries they’re delivered with - essentially reducing web services to some kind of middleware messaging protocol.

REST
In reaction to this complexity, RESTful web services are now becoming quite popular -at least in discussion forums-.
REST is a concept that was presented in 2000 and lays out a number of principles. The concept is still quite new when applied to web services, and a number of divergent definitions of what constitute RESTful web services are available.
The general focus is on interacting with stateful resources, rather than messages or operations.
Grossly oversimplified, RESTful web services heavily rely on standard operations already defined by the http protocol: GET, POST, but also PUT and DELETE. These operations allow to interact with stateful resources uniquely identified on the network, and to specify whether a given resource should be created, updated, or deleted.

The prototypical example of a RESTful type of "web service" is RSS, which consists of an http GET operation that returns an XML formatted representation of the state of resources exposed by the server.

A good programmatical comparison of XML-RPC, SOAP, and REST can be found here.

While there is a great level of antagonism in forums between proponents of REST vs. SOAP (the REST folks are typically more vocal), it's not clear that both technologies conflict: SOAP is well suited for supporting complex business operations, while REST is most appropriate for implementing simpler operations on simpler schema. In other words, REST seems to be well suited for broadly distributed, consumer-focused, web services (e.g. that of amazon.com or Google) while SOAP seems to be more suited for the enterprise world, where the challenges are around exposing complex legacy services, with requirements on security and transactionality.

In my next post, I'll try to look at some sources and data points for further analyzing the adoption trends.

Next Year's Budget

2007-07-29T22:35:00.000-07:00

I've mentioned in my last entry Scott Berkun's post on Asshole Driven Development and other software engineering antipatterns.
The post generated comments listing 60 additional variations of anti-patterns and a number of them touched on software development issues that are budget-related; i.e. where the focus on expenses and/or revenues negatively impacts the development process.

Some of them were great, others didn't seem as valid; but they all seem to reflect a number of frustrations expressed by developers.

One of the most interesting one was NY$DM; Next Year's Dollars Don't Matter:
"When following this approach, every dumb decision that creates a potential maintenance or portability nightmare is elevated to "essential" because maintenance dollars are in next year's budget, and don't matter for this development project.
Indeed, the absolute worst crud can be preserved because it is an "investment". Rewriting and replacing the crud would only spend this years dollars on an effort that wouldn't have any tangible ROI until several years in the future. The idea that maintenance is usually forever (the service life of in-house software appears to be measured in decades) has no impact at all, since that gigantic maintenance budget is only spent one year at a time, and it's made up of next year's worthless dollars. "

NY$DM describes a situation where development decisions do not factor the long term cost of building a piece of software (or not building it in the proper way), implicitly mortgaging future development budgets.

Variations on NY$DM
The anti-pattern has a number of variations based on size, role of the development organization (e.g. IT vs. R&D), and age / maturity of the organization (e.g. early-stage startup vs. my bank's IT shop).
The context in which NY$DM is expressed seems to be one of an in-house IT shop, where the focus is on not over-spending the yearly allocated budget (a.k.a a cost-driven organization); but similar patterns and variations can found at software vendors, where the focus is on generating value and revenues (a.k.a. revenue-driven organizations) .

In the later case, it is not uncommon to divert development efforts onto some poorly thought-out feature in order to get this one sale that might help meet this year's revenue target. A pattern which easily results in building a set of disconnected features that are too shallow, too specific to a customer, or not taking the product in the right direction.

Along with poor feature investment decision, exclusive focus on this year's revenue numbers and operating costs can also prevent long term technology investments necessary to achieve the product's long-term vision.

Generating Value
To be clear, the goal of commercial software development is to generate more value than is invested - and ultimately make a profit.
As such, trade-offs do need to be made, and there is no point in engaging in the development of a grand feature with a grand design if it means the organization cannot survive.

There is also only so much that can be predicted in any business, and sometimes not making a move is more detrimental than putting something out - even if imperfect.

Software is a process of discovery, and part of it consists in failing early, fast, and often.

The challenge resides in making sure that a rational and conscious decision is being made when such a trade-off is being made, and that the known consequences are collectively understood by the stakeholders and decision-makers.

In other words, there is nothing wrong in building a feature that is more shallow than it seems - as long as it's internally understood that more investments will be needed if taking that feature to where it needs to be.

Cognitive Dissonance and Communication Challenges

The frustration from the developers' side is multi-fold. It generally has its roots in a mix of cognitive dissonance and difficulties in articulating and raising awareness of the impact of the decisions being made at the top.

Engineers strive to build great things, not crap that can get us by. It's also not really exciting to develop something knowing that upcoming development efforts will involve fixing the shortcuts being taken right now.
Understanding and reinforcing that these constraints and trade-offs are an intrinsic part of the commercial development effort can help make things easier and put things back in perspective.

The other great challenge and frustration technical people face is being able to articulate the cost of maintenance of ill-designed functionality or the need -and associated cost- of refactoring a piece of code.
It is often extremely difficult to quantify the long term cost of maintaining some old component built on antiquated technology, or to articulate the future impacts of taking some design infrastructure shortcuts in order to get a feature out earlier.

Non-technical people typically have a hard time understanding -or trusting- that a piece of code needs to be cleaned up -or worst torn apart and rebuilt- in order to create something greater.
You can always use a construction metaphor (e.g. I can't build a skyscraper on top of a shack) but it will only get you so far. Feng shui will also probably only have limited success.

Your main options are constant education and re-enforcement on the issues of total costs of software development and maintenance; as well as possibly leveraging more senior and/or influential technical people within your organization and use their greater authority/credibility to help you articulate your views.

ADD, Bullying, and the Art of War

2007-07-16T10:13:00.000-07:00

Scott Berkun recently wrote an awesome post on Asshole Driven Development and other software engineering antipatterns.

Berkun defines Asshole Driven Development (ADD) as follow:
Any team where the biggest jerk makes all the big decisions is asshole driven development. All wisdom, logic or process goes out the window when Mr. Asshole is in the room, doing whatever idiotic, selfish thing he thinks is best. There may rules and processes, but Mr. A breaks them and people follow anyway.

"Asshole" is the trendiest way these days to describe a bully.
Robert Sutton recently made a hit with his book, The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn’t [read Guy Kawasaki's great review here]
The term works great for generating attention and calling things out, but is not really practical when you want to discuss and resolve a situation where one of your co-worker is being a jerk: calling someone an asshole is somewhat antagonizing... and is reflecting a judgment on the person rather than on his or her conduct. And in most cases, you'll be far more successful building a case against a behavior rather than against a specific individual.

Beyond the individual(s) with an inclination for bullying, ADD can typically only develop in an environment where there is either:
1. a lack of management leadership,
2. a lack of awareness of the situation from the management team,
3. a corporate culture of bullying -including at the senior management level
4. a combination of the above.

Possible remedies very much depend on the situation, but in any case, even if you're great at being an asshole, the recommendation would be to avoid playing back the game: power play typically fires back and is inefficient both for the organization as well as -most important in this case- for you.
Instead, you'd probably want to take it to a different field by constantly calling on rational decision-making and critical thinking, and questioning individual and/or emotional decisions.

If you feel powerless about corporate politics, i.e. how power flows through the organization and how you can influence decisions, a good read is the Art of War from Sun Tzu.
This is not a book about how to get rid of people, but one about finding the most efficient ways to achieve your own goals: know yourself, know your opponent(s), know your terrain, and avoid expanding energy unnecessarily...

Out of the Sandbox

2007-07-01T23:55:00.000-07:00

In my last post, I wrote about Offline Application platforms in general, and Adobe Air and Google Gears in particular.
The technologies are very nice - the larger question is which problemscan they solve, and what type of opportunities are they opening.
In other words, are these just gadgets, some incremental technology additions, or do they have the potential to introduce a set of not-seen-before applications?

From thick to rich
"Thick clients" was how desktop client-server applications were described in the late '90s when web applications started to take over.
While desktop client-server applications offered much more sophisticated User Interfaces, they were a pain to deploy and as such were quickly outpaced by web applications - the later introduced new levels of deployability and accessibility because their logic reside on the server.

For the past few years, the goal of technology players such as Adobe and Google was to develop technologies that would bring desktop-level User Interfaces onto web applications - a.k.a "Rich Internet Applications" - most notably Ajax and Flex.

It only seems natural that once this goal is somewhat achieved, these players complete their technological expansion by bringing web UI onto the desktop.

While long-term strategic market positioning is one way to look at it, it's not clear that the intent here is to attack Microsoft by making Microsoft Visual Studio and native Windows application irrelevant - there's less and less native Windows application development anyway, and the next killer application is unlikely to outcast the MicrosoftOffice suite.

Do you really need gmail on your desktop?
One of the most prominent value proposition claimed by Air/Gears technologies is to allow users to take their favorite Web Applications offline.

But does anyone really need offline desktop applications?
People switched to gmail precisely because it was on the web -i.e. anywhere, and not on a designated computer.
If users wanted to easily save online maps, they'd probably want to save them to an online repository and be able to access them from any network-connected location - not on some random desktop computer.

The need to be able to work while disconnected is irrelevant to 90% ofUsers:
How often are you disconnected?
How often will you be 5 years from now?

This doesn't mean that 80% of the Users may not be asking for this type of features - asking is cheap.
If and when users do get this type of functionality, they realize that offline applications intrinsically hold a number of data synchronization issues - the "which version am I working off?" syndrome -, and often abandon the idea .

Out of the Sandbox
The greater value of the technologies offered by Adobe and Google might be in being able to lift the sandbox restrictions imposed by browsers, ActiveX controls, or the Flash Player.

These sandbox restrictions mainly prevent 2 functions: client-level http services mashups and cross-application integration:
Browser-based applications can't mashup http services from multiple web servers, unless the traffic is proxied by the server serving the said web-based app.
In other words, you can't develop a web application that pulls data from eBay and Google - unless the data transits through your own web server.
These sandbox restrictions also prevent to easily exchange data between applications - be they web-based or desktop-.

Fine points?
May be, may be not.
In essence, the features offered by Adobe Air or Google Gears allow the development of applications that can extract, process, compose, and render information from multiple sources in a way that can fit a user's specific goals - i.e. some sort of ultra-specialized "browsers" that are web-application smart -: Grab data from here and there, transform it in such a format, and display it this way - replace the "here and there" by your favorite web and/or desktop apps - Gmail, linkedIn, eBay, MS Office, et al-; the list grows as fast as the number of services offered on the Web.

The target market for such applications might initially look like somewhat of a niche, but whoever comes up with a more general application might have the potential of challenging some of the largest Internet players by commoditizing, enhancing and controlling the user experience around the services they offer.

Air and Gears

2007-06-17T23:24:00.000-07:00

2 major technologies are being released which are targeted at allowing web developers to build and deploy web applications to the desktop.
Some background and a quick summary - I'll follow-up with some thoughts in my next post:

Background:
When Users start relying heavily on a given web application, there typically comes a point when they start asking to be able to continue using the application while they are disconnected.
This is true for enterprise applications (e.g. being able to enter my expenses while being disconnected from the enterprise Expense Reporting web app) or for consumer web applications (e.g. being to consult my Gmail and draft emails when on the plane).

While Users understand that an offline application would provide them with only a subset of data or functionality (constrained to the amount of data previously synchronized on the desktop); the unconscious requirement is often that the User Interface should be similar -if not identical- to the online web application.

The offline application should allow data to be synch'ed from the server, work offline, and synch back to the server.

In the past, deploying a java-based web application as a Windows desktop application was not the most straightforward thing to do:
You typically needed to have a windows executable embedding and managing a low footprint web/servlet server as well as a browser instance pointing to the embedded web server.
If you wanted to add persistance on top of that, you could either serialize the object graph into local files, or use an embedded database server.
The fun wouldn't stop there if you needed a smart synchronization mechanism between the desktop app and the app server: the challenge there wouldn't be in establishing a connection to the server and exchanging data, but rather in resolving conflicts - i.e. resolving concurrent changes to data made offline and online.

Air and Gears
There pretty much hasn't been much -if any- traffic on the Offline Application Framework front since web applications became mainstream. If you wanted to deploy a web-based application as a Windows desktop application, the only solution so far has been to develop a tremendous amount of infrastructure for being able to deliver a web UI into a desktop application.

Just recently, both Google and Adobe released beta versions of application frameworks allowing to easily create such apps.

Gears is an open source browser extension that allows Html/Ajax-based apps to run on the desktop.
When accessing a Gears-enabled web application, Users will need to install the Gears extension (but only the first time) as well as allow the website to use Gears.
Users would then be able to use whatever desktop-based features the web application provides, and access back the application and its features while disconnected.
From an architectural standpoint, the Gears browser extension has three components that can be accessed via JavaScript APIs:
- a local Web server, which runs in the browser - for storing and accessing application pages offline
- the open-source database SQLite - for storing and accessing application data on the user's computer
- browser extensions that allow multiple JavaScript jobs to run in parallel - the WorkerPool object that allows long-running tasks, such as the code that synchronizes data between your server and users' computers
The Gears browser extension should be under 1 megabyte in size.
More on the architecture can be found here, and a good tutorial can be found there.

Air (Adobe Integrated Runtime) was called Apollo as of last week when it was still in alpha.
Air a cross-operating system runtime that runs outside of the browser (as opposed to Gears, which is a browser extension).
The runtime needs to be installed only once and can be distributed either independently or within any Adobe AIR application.
Users can then run desktop applications that leverage rich Internet applications technologies such as Flash, Flex, HTML, JavaScript and Ajax.
The intent is to allow the development of connected applications that work while offline; but any desktop application could be built on the platform.
Developers should not need to re-write any code to have an existing Flex application run within Adobe AIR, but they may need to write additional code to take advantage of Adobe AIR specific APIs and controls.

Like Gears, Air embeds SQLite as a native database; and much like Gears, AIR 1.0 will not have built-in support for communicating directly to 3rd party databases.
In theory though, it should be possible to write Database drivers in ActionScript and allow AIR applications to communicate directly with local or remote databases.

Another nice bit is that Adobe intends to incorporate a mechanism to make it easy to automatically update applications.

Some cool samples are available online: a desktop RSS reader; an application that allows to save and manage Yahoo Maps locally; a mashup of Google Maps with VCard; an application that allows you to drag and drop files between Amazon Simple Storage Service (S3) storage services and your local file system.

It's interesting to see that there was practically no development in this area; and all of the sudden, 2 of the largest players in web technologies are releasing some major platforms.

In my next post, I'll try to formalize some thoughts as to what is the problem we're trying to solve, and whether being able to solve it might open-up some new possibilities.