XPloring around

The point of Object Orientation

2010-08-17T16:59:00.001+05:30

A common characteristic of large codebases is that it takes disproportionate effort to add or extend functionality. Couple of common reasons:

Hamstrung team - The original developers may have moved on, development beyond the first release may have been transferred to a less capable team or handover may have been ineffective.
Poor codebase - Test quality and coverage may be lacking, encapsulation may be broken, or the build may be inordinately long.

Bad encapsulation is a common culprit. All too often, OO-capable languages like Java and C# are used to author transaction script1 applications. The business logic is use case driven rather than domain driven. Use case driven code is characterized by global code acting on global data. Well encapsulated codebases, on the other hand, consist of collaborating objects, each object encapsulating its local data and behaviour.

For a typical business app, everything else remaining constant, good encapsulation will lead to constant or decreasing delta effort for delta functionality. The point of object orientation is that it is meant to help manage complexity of the business domain. This can only happen when we encapsulate along the grain2 of the domain.

1:http://martinfowler.com/eaaCatalog/transactionScript.html
2:Cutting with the grain: the rhythms of design - Kent Beck

Agile anti-pattern summary

2010-05-02T19:13:00.000+05:30

Here is a summary of anti-patterns I have written about so far:

How to do XYZ in Agile?

2010-03-24T14:18:00.003+05:30

Clients sometimes ask, "How do you do estimation in Agile?" or, in general "How do you do XYZ in Agile?". Typically these are people making a transition from non-Agile methods. They often want to hold on to some existing ways of functioning. They would like to blend Agile into their existing processes. There is a problem here. To be agile is not about following a different set of prescribed processes or practices. The only things that matter are:

Continuous delivery of valuable functionality
Happy team (team includes client)

The agile manifesto starts off by saying:

We are uncovering better ways of developing
software by doing it and helping others do it.

The practices codified under XP (or Scrum) is just documentation of how a bunch of practitioners were able to achieve continuous delivery and happy team. The question, "How do you do XYZ in Agile?" misses the point. It is a relic of "process conformance" mentality. What's more, I was once asked, "Is it ok to ask for a number of tailorings or deviations from the master process template for agile?" I was speechless. Turned out that the organization still retained the services of a group called SEPG (software engineering process group, a relic of CMM) to define a master process template for agile. Every project was supposed to conform to the template and ask approval for tailorings (tweaking a process/practice) or deviations (omitting a process/practice)!

I think this is a case where ends justify means. If you are achieving continuous delivery and happy team, you are obviously doing something right. It doesn't matter how Agile it is. If you aren't achieving continuous delivery and happy team then again it doesn't matter how Agile your processes are. One might argue that this is watered-down agile. Big deal. Granted, it is definitely wise to go by the book first. It is arrogant/foolish to assume that we are smarter than the book before we begin. After all, the book represents distilled wisdom of practitioners. But it is important to keep an eye on the outcomes. All advice is contextual. It is no use wailing that you have done everything by the book and not getting results. It is dogma to stick to the book in the face of contrary results.

Agile maintenance in a devops world

2010-03-15T13:13:00.003+05:30

"What is an agile way of doing maintenance projects?", is a common question. An answer like the following is par for the course:
"It isn't very different from development projects. Write tests to reproduce bugs before you fix them. Treat enhancements as one or more stories. Keep the CI server running. Make sure that changes are occasionally merged from the maintenance branch into trunk. Pairing is probably not as important."

In the context of this post, a 'maintenance' project is one where a client outsources ongoing bug-fixes and enhancements for a live application to an IT vendor (often offshore). Conventional wisdom has it that development is investment (capex) and maintenance is cost (opex). Hence, clients look for much lower billing rates when it comes to maintenance. This results in separate contracts/teams/vendors for development and maintenance. Some clients go a step further and outsource even IT operations to a different team/vendor under a separate contract. The rationale is to stick to core 'business competency' and outsource everything else (let them vendors compete with each other for our slice of IT).

Depending on how critical the said application is for revenue generation, this strategy of 'divide-and-conquer-IT' can be frustrating at best and suicidal at worst. The best of the businesses that make most of their money off their websites outsource little to none of their IT. This is simply because you cannot go to market fast enough if you have to orchestrate between three teams/vendors for every new feature. Equally importantly, the feedback loop gets badly constricted at contractual boundaries. Designing formal, SLA driven protocols of communication between business, development, operations and maintenance is a recipe for bureaucracy and indifference.

Not everyone can afford not to outsource. It is very difficult to put a good IT team in place. A lot of outsourcing has been a reaction to uncooperative in-house IT. But the solution has gone overboard in trying to compartmentalize development, operations and maintenance. You don't want to be overly reliant on a single vendor? Fair enough, consider outsourcing product A to vendor X, B to vendor Y and C to vendor Z. This is better than outsourcing the development of A, B, C to vendor X, operations to vendor Y and maintenance to vendor Z. The latter model requires well defined, stable interfaces between different constituencies. Not suitable where business is changing every week. It is a repeat of the layered-team anti-pattern at a higher level.

What if we want to deliver a grand SOA? Once the team of architects have specified every service, surely it should be possible to outsource a bunch of services to different vendors and the consuming applications to yet others. Splitting up integration work is extremely risky, in my opinion. Service contracts are rarely cast in stone. On the contrary, if you like the idea of evolutionary, guerrilla SOA, you no longer plan to have stable interfaces - the service evolves in tandem with the needs of its consumers. So, try to give all integration work and important applications to your primary vendor. Better yet, don't do big bang top down integration. Evolve bottom up.

On the other hand, it may be that your business doesn't change that often or your application doesn't generate revenue. If so, it is useful to ask "Why build? Why not buy?" every once in a while. SaaS is growing fast. It is likely that someone is offering your bespoke application as a service. At the cost of some tweaks to your business process, you might end up with a better application at lower cost. The SaaS vendor in turn is likely running a full in-house IT.

The world has changed yet again. Devops is gaining currency. These days, a common way of testing the uptake of new features is to divert, say 5% of your traffic to a new deployment and see how it goes. A separate maintenance team is a dinosaur in an age of frequent deployment (blue-green or otherwise) and continuous delivery.

Q. What is an agile way of doing maintenance projects?
A. Don't do maintenance as a separate project.

There is one situation where a maintenance project might make sense. End-of-life support. Basically, you pay for a team to keep an old app running while a new replacement gets built. Other than that, it is all about breaking down silos.

Of course, you will never hear Indian IT majors singing this tune. Their recruitment and business model is based on economies of scale. 'Changing businesses' are anathema to their ossified processes. However, in response to changing realities of the marketplace, some of them have begun to sing an agile tune. Some of them even claim to have proprietary, hybrid, high-performance methodologies that "synergize best of breed practices from CMMi, ISO, Six Sigma and Scrum". I'll explore a certain aspect of this 'synergy' in my next post.

Pair Programming Payoff

2010-03-04T12:49:00.004+05:30

For a project that runs for, say six months or more, there should no extra development cost on account of pair programming. Period. If there is extra cost, it means pairing is not paying off for you. Pairing should pay off in the following difficult-to-measure ways:

Lesser rework during development on account of misunderstood/ill-defined requirements. These things surface quickly when pairs talk to each other.
Slight reduction in maintenance cost on account of clearer code. Because every line is now considered readable by at least two people. (more in case of pair rotation)
A better detail level design results when each pair acts as a sounding board for the other. Good design reduces cost of future change.
Faster/better process of bringing new team members up to speed.
Lesser impact of people taking vacations.
Lesser bugs in QA, UAT
More hours of focused work per day - however professional someone may be it is natural for concentration to wax and wane. Pairing almost always helps here.
Unlike the case of no-pairing, there is no separate code-review activity required

How do you figure if there is extra cost? Estimate both ways. The total 'release-lifecycle effort' for the case of pairing should not exceed that of no-pairing. Individual stories may indicate greater effort but that is okay. It is very difficult to do controlled experiments to nail this down. There are some studies but your mileage may vary.

A word of caution when estimating for such comparisons. Remember that on all real projects, actuals mostly exceed estimates. Some of the difference is borne by the client (change control etc) and the rest by the vendor (unpaid overtime etc). You need to have a realistic idea of effort overrun for the case of no-pairing to be able to compare it with the overrun for the case of pairing.

If pairing is resulting in net higher development cost on a long running project, then it simply means (to paraphrase Kent Beck) that the client is getting XP'd on :)

Label activities, not people

2010-02-25T12:19:00.003+05:30

Ideally, an XP team consists of all talented and experienced people. Every person is assumed to have good communication skills, technical skills, ability to analyze and synthesize, understanding of business drivers, ability to make trade-offs based on business constraints and the ability to work as part of a team. Indeed, there probably are thousands of people who qualify on all these counts scattered around the world. But IT is a pervasive beast. It touches almost every aspect of modern business. IT is also labor intensive. The demand for really good people outstrips supply by a factor of hundred if not more. IT vendors who practice XP are therefore forced to move away from the ideal of more or less homogeneous poly-skilled teams.

Instead, we hire people who have a subset of the above skills and assign them to specific roles in a team. So we have roles like developer, analyst, QA, manager, UX dude/dudette etc. It is essential to recognize that this arrangement is a departure from the ideal - a compromise driven by market reality. How we look at roles depends on our model of the ideal world. If you agree with the XP ideal, then roles are labels attached to the activities on a project. e.g. performance tuning activity needs developer role, activity of scope negotiation needs manager role.

But all too often, we lose sight of the ideal and think of roles as labels attached to the people on a project. e.g. person A is a manager, anything she says outside her labeled area of competency should be taken with a pinch of salt, person B is a developer, anything she says outside her labeled area of competency should be taken with a pinch of salt, and so on. We forget that the labeled area of competency is merely a mutual agreement at the time of recruitment. We end up discounting the credibility of people outside their labeled domains.

This kind of thinking also magnifies capacity constraints in a team. Say we have 4 people with label QA and 12 people with label developer on a team. What happens when stories start piling up in QA? We are less likely to see if some developers can add temporary capacity to QA. Oh no, that would never do. Developers can't test. They are not wired to test. Mass stereotyping sets in.

Soon people start buying into their labels. Developers refuse to test. If you can communicate well, you are suspect as a developer. Managers shy away from writing code, or (shudder) gaining even basic user level technical competence. They go about saying "I can't handle technology", as if it were a feather in their cap. We become more and more entrenched in the silos created by our labels. Truly ploy-skilled people get disheartened in the process. A person with a label of analyst may have a great aptitude for domain modeling. But her inputs may be sidelined by developers. After a while she either starts believing in her artificial limitations or she starts looking for another job where her skills are more broadly appreciated.

It is no good to call everyone consultant and then use a different labeling scheme when it comes to staffing projects. To the extent of their skills, people should get to play multiple roles within a single project. Oh, that complicates staffing. So? Should the tail wag the dog?

Organized religion came about partly as a result of our need for identity and self-worth. It is probably reassuring to say "I belong to this group" and then move on to thinking "My group is better than the other group". We may have outgrown traditional organized religion but we seem to be falling for newer versions.

Fail fast

2010-02-12T16:20:00.005+05:30

When failure is a possibility, design to fail fast rather than slowly. Doing so reduces the cost/impact of failure. What is equally important, failing fast makes further attempts feasible. Learning from previous failures makes future attempts more likely to succeed. This principle is widely applicable in software development:

Methodologies that have fail fast mechanisms baked in are more likely to generate greater ROI. More on this later.
Guerilla SOA is arguably a fail fast take on big up-front SOA.
Code that is written to fail fast is likely to be more reliable in production.
Small, frequent check-ins are likely to cause less overall rework than big, infrequent ones.

Verification: When and How?

But how do you decide at any given checkpoint if we have a success or failure at hand? The quality of verification is crucial. Verification by peer review, while valuable, is prone to oversight. The proof of the pudding is in the eating. The more times you get to eat the better. The analog of eating here is testing functionality. A truly iterative process of software development where functionality gets tested iteratively is likely to achieve better ROI (everything else remaining constant).

Okay, so no one uses waterfall anymore. But we still have projects where big up-front analysis and design are the norm and continuous integration means weekly build. In such cases, we only have limited verification (peer reviews of requirements, design and code) till the very end. Failures (if any) are slow and horrible.

Incremental agile is what almost all XP and Scrum teams follow. They run through the stories for a release doing just enough/just in time analysis, design, coding per story. The boundaries between design and code are often blurred but that is not material to this illustration. Truer verification now becomes possible at the end of every story (QA/customer testing/sign-off). However, each story still gets only attempt. Any changes (learnings?) after that go back into the backlog to be prioritized and taken up with everything else.

But as Jeff Patton points out, it is possible to view each story as a series of progressive enhancements:

Necessity - core functionality (e.g. user registration)
Safety - validations etc. (e.g. confirm via email, add a captcha)
Flexibility (e.g. support openID)
Luxury (e.g. add ajaxy feedback on available userids, password strength)

Yeah the example is a bit contrived (openID would mostly be another story even in incremental mode) but you get my drift. We can now design a release plan that allows the team to iterate on a story, progressively enhancing it. The story sponsor reviews (tests) each story multiple times. Failures (if any) are faster and cheaper. The team learns better. I like this line from a Werner Vogels interview:

With a new radical service, you try to go into prototype mode pretty quickly, and then you start iterating on that until you feel that you understand your business problem.

Stories in regular business applications may not qualify as radically new but very often the team is new to the application in question. "You only understand it when you do it" is a much under-appreciated truth of all knowledge activity (if not all activity).

Expected cost/time

It may be argued that for a given chunk of functionality, iterating increases overall cost/time as compared to doing it in one go. This is only true when the risk of failure is zero. For situations of non-zero risk, the expected cost/time can often be lower with an iterative approach. The table below shows this for a hypothetical but realistic risk profile where risk decreases as learning increases. Your mileage will vary depending on the risk profile of your team and functionality.

Failure is not an option

Sometimes you get to hear sponsors saying they don't care about downside risk because failure is not an option. Some of these projects run into a death march followed by a blow out followed by movement of key people. Then a new team and a new IT partner get to do it all over again.

When Agile Estimation becomes pointless

2010-02-05T08:52:00.002+05:30

If a team has to deliver a certain amount of functionality by a certain date, then the act of estimation becomes an act of commitment. The team has essentially committed to cost, schedule, quality and scope. There are no levers left. In such a situation, velocity becomes a target rather than just a measurement. Targeting velocity makes points based estimation a charade. The team is better off estimating in real days in this case. It helps the cause of commitment. Points based estimation is useful when you agree to the following:

Plan the story list for a release with only about 70% must-have functionality and the rest as negotiable. This implies it is potentially okay to release (go into production) with just that 70% functionality.
By definition, estimates are approximations. Software estimates are more so.
It is counterproductive to insist that your IT vendor/team deliver x points of functionality by date y. Scope negotiation does not mean x remains x. Yes, sometimes only the contents of x need change. At other times, x may become x ± delta. This is not a rip-off. It doesn't make business sense for your IT vendor to rip you off because you won't give repeat business if you are ripped off. Without repeat business, your IT vendor is likely to become uncompetitive because the cost of new business development is very high.

"Embrace change" is a two way street. IT delivery team should embrace changes to requirements and priorities. Clients should also embrace changes to scope when delivery work proceeds at a different pace than originally expected.

The Tragedy of Commons based Peer Production

2010-01-29T08:55:00.003+05:30

In my previous post, I mentioned how authors of small but useful open source projects often struggle for compensation. Using a FOSS/commercial license is one option but it could go either way. They wouldn't really have to resort to compulsory payment if donations were forthcoming from users/organizations who commercially benefit from it. Therein lies the tragedy of 'Commons based Peer Production'. Only a tiny minority donate. This is detrimental to the growth of all forms of free digitial distribution (indy music, books, software etc). It also makes for an unhealthy society. Big leap? Bear with me.

The bulk of financial transactions in G20 economies happen between:

Corporations (B2B)
Corporation and citizen (retail - citizens pays corporation, wages - corporation pays citizen)
Corporation and state (taxes - corporation pays state, state funded projects - state pays corporation)
Citizen and state (taxes - citizen pays state, pension/welfare - state pays citizen)

But what about transactions between citizens? When this goes towards zero, we get a unhealthy society. Centralized services, mostly helpless consumers (citizens). Donating money to the creators of 'digital stuff' that we enjoy is a great way of changing the status quo. The internet has provided a great platform for disintermediation but it will only work if we choose to participate in the process.

We could also try to build momentum within our organizations towards this. Companies that commercially benefit (however indirectly) from free software could set aside some money annually for donations. The beneficiaries could be decided by a poll within the company. After all, this is also part of CSR. Plus it will make for good PR copy.

Open Source authors' struggle for compensation

2010-01-25T16:37:00.006+05:30

I guess we all know the difference between free as in speech (freedom) and free as in lunch (gratis). All open source software confer certain freedoms of use, modification and redistribution. None of them are required to be made available to users at zero cost. Yet, I don't know of a single significant project that is open source and fully paid, i.e. no version available at zero cost. Reasons typically offered are:

It is the freemium business model - Give away the basic software and charge for enhanced functionality or support.
It is not enforceable - Users will simply build the software from the source code and there is no way you can get them to pay.

#1 is okay for projects run by companies. It is often not feasible for projects run by individuals. This a big category of small useful pieces of software (think libaries, plugins, utilities). These people try (in vain?) to make some money via advertisements or by appealing for donations. It is a sad reality that they get no compensation from commercial software/organizations using their work. These people should relook at #2.

I suspect payment is very enforcable against commercial enterprises. Just add a line to your EULA saying "It is illegal to use this software for commercial purposes without paying for it". Most companies would pay a small fee (say $50 for a full version but without support) for a useful piece of software. Either that or their lawyers would blacklist the software. At least you wouldn't have legions of freeloaders just using the software without contributing anything (money, bug reports/fixes, documentation) back. The argument that even freeloaders help spread the word to someone who might eventually buy something is a trifle bleak. Of course, all power to you if are doing this altruistically or if you are happy with the other rewards (fun, learning, reputation).

Giving software away at zero cost means you have to depend on services for income. You either offer your services as an employee (day job) or as support (adding value to what is given away). Either way, it is a model of pricing input instead of output. Scales only with people. Or you have to fall into a category where you can be adopted by a big foundation like Eclipse or Apache or Mozilla. Good luck with that.

No software patents versus patent reform

2009-11-17T00:08:00.005+05:30

We often cry ourselves hoarse trying to justify non-waterfall methods of software development saying that software is fundamentally different from manufacturing or civil construction. There is substance to this. Code is design. Compilation (build) is zero cost for all practical purposes. If we agree that compilation is the equivalent of a manufacturing assembly line or brick-laying, then the economics of software development are very different indeed. The economics of software make it particularly vulnerable to the ill-effects ot the current patent regime.

I used to find it surprising when highly credible people said there was nothing fundamentally different about software patents - "If you're against software patents, you're against patents in general." Until I realized that software patents aren't inherently evil - the devil, as usual, lurks in the details.

Patents exist to balance competing objectives:

Encourage innovation
Give innovators a chance to profit from their success
Improve the body of knowledge in the public domain (eventually)

In practice, the second objective is generally met by conferring a twenty year period of exclusive usage rights. Thus, patents are economic devices. Won't the efficacy of an economic device depend on the underlying economics of the industry it is meant for? Do all industries require twenty years to recoup investment and monetize an innovation? It certainly is ridiculous for software. Technology cycles keep getting shorter - twenty years is several generations in most industries today. A two year patent might be more reasonable for software. Of course, that still leaves jokes like the one-click-checkout patent untouched.

When patents were first conceived, they were meant for flesh and blood patent holders, not for mere legal persons (Corporations). Corporations gained personhood much later. However, the majority of patents today are held by corporations rather than individuals. This has lead to barely legal behaviors that end up stifling innovation in the industry as a whole.

What's really needed is a nimble international body that can tweak the terms of patenting for different industries. But then, international co-operation is hard to come by even for life threatening issues such as climate change. No wonder people push for elimination of software patents instead of push for a better patent regime.

Being explict is better than being implicit - only when everything else is constant

2009-09-30T08:00:00.003+05:30

The benefits of dependency injection are:

Separating configuration of dependencies from their use.
Improving unit testability.
Making dependencies explicit.

As I finished explaining these points to some new hires, I followed it up with a generalisation, "And in general, being explicit is better than being implicit - it reduces ambiguity and improves communication, always welcome in software development." Immediately I was struck by a few incongruities. After all, we have come to appreciate:

The absence of Java-esque checked exceptions in C#.
The flexibility afforded by dynamically typed languages.
The evolution friendliness and interoperability of loosely typed SOAP web service contracts (documents, coarse-grained interfaces, restricting ourselves to the simplest of XSD types).

In the above cases, being explicit comes with a stiff cost. We trade it off for flexibility. So the generalisation still holds. In general, being explicit is better than being implicit. But there is a caveat implicit in there - "In general, being explicit is better than being implicit provided everything else remains constant." If there are additional benefits to be gained from going implicit, then being explicit is questionable.

It helps to make this caveat explicit. A case of making the generalisation practise what it preached.

SOA in India

2009-08-21T22:27:00.003+05:30

businesstechnology.in run by S&S Media (people behind JAX conference) interviewed me on the state of SOA in general and in India in particular.

How applications learn. What happens after they are built?

2009-06-18T00:47:00.002+05:30

I have been reading Stewart Brand's much celebrated book over some time now. Took some inspiration from it for a local KM-community talk on innovation. In particular, I was tickled by his characterization of how ideas spread:

Perhaps culture is driven by just such flea-market ideas in a vast network of uncredited influence.

Replace culture by 'innovation in software' and it still makes perfect sense. Our aggregated blog page is a good example of a flea-market of ideas.

Stewart argues for evolutionary design over visionary design - now where have I heard that before? It seems he once suggested to an architect that he go back to his building to see how the users were finding it. The architect said, "Oh no. You never go back. It's too discouraging." Just reinforces that we need to be on our guard when we choose technologies to build applications. Will the eventual maintainers of the application be comfortable with them? Is the app designed for change? Does the roof (read abstractions) leak?

It's true that construction industry (or any other industry for that matter) isn't a good analogy for software development. In particular, coding is not analogous to construction (compilation probably is). But the life of a building seems to have striking similarities with that of an application. Though buildings are expected to have much longer lives than applications, the total churn that they are subjected to is probably comparable. Buildings are subjected to form-over-function pressures of the marketplace - software apps are less so. Both suffer every now and then from architect hubris.

Wired covered the book long ago. The BBC documentaries based on the book are still available.

Part 1: Flow
Part 2: The Low Road
Part 3: Built for change
Part 4: Unreal estate
Part 5: The romance of maintenance
Part 6: Shearing Layers

There is also a good summary up at: http://www.gyford.com/phil/writing/2004/10/24/how_buildings_le.php

ThoughtWorks on App Engine for Java: Google I/O Enterprise track

2009-06-10T19:06:00.006+05:30

ThoughtWorks' CTO, Rebecca Parsons and Martin Fowler presented (with usual panache) this overview of ThoughtWorks' findings on Google App Engine (and the cloud in general) at Google I/O. Being part of the enterprise track, their talk focussed on enterprise readiness/applicability.

A summary for the busy technologist:

Google App Engine

A reasonable option for a number of applications.
Sets the bar in terms of low barrier to entry
Technical things to consider before taking the plunge:

Testing - development environment not (yet) quite the same as production.
Persistence - Different idioms needed than when persisting to relational databases.
Concurrency - The deployment sandbox is effectively single threaded. A whole bunch of Java libraries that spawn their own threads will need to be ported over before they can be used on the app engine. Moving away from shared memory concurrency is probably good anyway.

Advice for the enterprise

There is a spectrum from infrastructure (Amazon) to applications (Salesforce) with platform (app engine) being in the middle but closer to infrastructure. Domain specific play (statistical analysis cloud) is possible in the space between platform and applications. Of course, considering the venue, the speakers did not emit competing names like Amazon or Microsoft.
Standard potential advantages - dynamic scaling, low entry barrier (therefore low downside risk), pay as you go (op-ex over cap-ex)
Economies of scale (in terms of skilled people and infrastructure) mean that it is much harder to do-it-yourself (private clouds).
Security, privacy and IP - SLAs are ok but not nearly as good as the ability to fire your CIO when things go horribly wrong.
Is it really cheaper? - Unambiguous yes only if you move everything and close down your own data center.
Need new IT org? - probably yes - hopefully one that has better relations with the business.
Vendor lock-in? - probably not much worse than Oracle lock-in.
Other use cases - experiments, spiky one-time computational needs.
This is another of those things that lets you focus on core competencies.
Has the potential to solve the last-mile problem in agile software development - frequent and incremental deployment to production.

Fallacies of Knowledge Management: Summary

2009-05-07T08:21:00.005+05:30

This post summarizes my series on fallacies of knowledge management.

Plug: If you think your organization could use some help on KM, feel free to reach out to us.

KM fallacies: Put usable tools in place, adoption will follow

2009-05-01T19:04:00.002+05:30

The effort of running a typical KM initiative almost makes us want to believe that mass adoption will follow. Unfortunately, Ms.Adoption is not easily wooed. The launch of these initiatives are usually broadcast via company wide email. An increasing percentage of employees seem to be immune to such broadcasts (for good reason?). The early adopters, a smail percentage, use it for a few days and then their activity begins to dwindle. Adoption proceeds at an uneven pace and sometimes fails to reach critical mass needed for positive network effects. A year later, it's time for yet another initiative.

There are often genuine reasons why adoption doesn't take off. For example, the applications may not be easily accessible outside the company intranet. Or the applications may be slow to access from remote offices. Maybe people are just overworked. I am begining to wonder if there is another factor at play. I have observed it on the web and I suspect similar forces are at play within the enterprise: poor citizenship. Too many of us prefer to be passive consumers of public content on the internet. We don't produce or perhaps more importantly, curate existing content. Granted, there are some prolific producers of mediocre content via blogs, comments, tweets, and posts to public mailing lists or discussion groups (and a few prolific producers of really good content via the same channels) but they are more the exception than the rule.

Some examples of poor citizenship with respect to curating content:

Not clicking "I found this review helpful/unhelpful" on a website that carries reviews (e.g. Amazon) or "This documentation was/was not useful to me" on a website that actively solicits feedback on documentation (e.g. many of Google's help pages)
Not adding appropriate tags to images found on Flickr
Not posting answers to unanswered questions that you ask in a public forum and later figure out the answer/workaround for.
There are times when programmers struggle with a cryptic stacktrace at work and a web search points them to the exact cause. All because someone took the effort to blog about that issue. You often see a number of grateful comments at the bottom of such posts. Yet, when I overcome such a problem by myself, I seldom try to write a post about it.

It's not like we don't do it because we are against digital sharecropping. It might be that trying to be a good cyber citizen leads to yak shaving. I suspect that at least some it just boils down to a I-can't-be-bothered attitude. This attitude has tragic consequences for commons like public forums and wikis. Not many people seem to want to take the effort to contribute. But a system that fully relies on user generated content cannot succeed in the face of indifferent users. To some extent, cultural legacies determine our ability to effectively participate in a system that more resembles a gift economy. Incentive systems may help with adoption but they need to be tailored to the specific dynamics of the user community. All in all, adoption is a tough nut to crack.

Encapsulation is not security

2009-04-28T15:06:00.005+05:30

I came across this discussion on the lack of really private methods in Python. Time and again, some of us feel violated that a method that is supposed to be private is somehow accessible to callers (via permissive reflection or otherwise). However, this reaction misses the point of encapsulation.

Encapsulating an object is not about protecting (or securing) the said object. If you must think in terms of protection, then it is about protecting the consumers (callers) of the object. Encapsulation is about freeing your consumers from the knowledge of your implementation details. By doing so, you insulate your consumers from any ripple effect of changes to your implementation.

The term 'access qualifier' for describing public, protected, private doesn't help. Something like 'visibility indicator' might help avoid the notion of access protection.

Knowledge Management Fallacies: make it easy to import and attach documents

2009-04-15T00:05:00.003+05:30

In order to migrate legacy content, one might resort to importing or attaching old documents into a newly fangled KM repository. This is a slippery slope. Importing/attaching becomes a crutch for users who are just too lazy to get used to authoring content in-line (using rich text editors). They continue to author all documents off-line. So what? Is there is a problem?

As Google has so successfully demonstrated, it is often the hyperlinks between documents that are more valuable than the documents themselves. Imported/attached content almost always lacks relevant hyperlinks to other documents in the repository. They hang about like dead limbs of a tree. What is worse, the offline versions often aren't discarded after import. They continue to evolve offline, unbeknownst to other users. They get attached to emails and soon no one cares about the orphaned version in the repository. The Google docs video captures this sitution well.

But of course, we love the power of our offline office tools and will continue to have powerful spreadsheets and presentations created using them. Sure, just don't tack them on as appendages to the one big repository. There is another repository better suited for offline content. It is called a file server. Better yet, use a web-enabled version control system backed by a file server. This gives us permalinks to the latest version of every document. Get them indexed by an enterprise search product. Oh, but our users aren't version control savvy. Come on. It takes fifteen minutes of training to understand update, add/edit, commit. After all, we all claim to be organisations of savvy knowledge workers, don't we?

GWT and JDO on the Java App Engine

2009-04-08T09:19:00.005+05:30

I have been trying to build an application using GWT and JDO on the Java App Engine. This post captures some things I discovered along the way.

GWT-JDO compatibility
The JDO interface is provided by implementing extensions to DataNucleus. It provides a (not so seamless) wrapper around low level DataStore (BigTable) API. Before running the application, JDO annotated classes need to be bytecode-enhanced by the DataNucleus enhancer. (The appengine Eclipse plugin does this automatically). If you are writing a GWT app, then typically, these very JDO classes are also your GWT domain classes (myapp.client.DomainClass). So these classes need to be compatible with respect to gwt-compiler (e.g. Serializable) and data nucleus enhancer. An example of where this isn't obvious is in the choice of datatype for primary key. We can't simply use com.google.appengine.api.datastore.Key because the gwt-compiler cannot serialize Key (seems this is now possible) and we don't have the source for Key. Instead, we use an encoded String form of Key.


@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class SomeDomainClass implements Serializable {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
@Extension(vendorName = "datanucleus", key = "gae.encoded-pk", value = "true")
String id;

User API integration with GWT
The appengine user API lets us authenticate Google users. However com.google.appengine.api.users.User cannot be used as a GWT domain object. gwt-compiler would need its source code. I ended up creating another class myapp.client.User for this purpose. This is probably good anyway because it lets me add app specific behavior to my User.

Development environment
GWT hosted mode gets going the quickest. Custom build file can be written to run local server mode. But neither of these are quite the same as deploying to the real appengine servers at appspot (or your domain). The local environment don't mimic all the characteristics of actual deployment. Post deployment testing is highly recommended for real apps. For purposes of unit testing, the datastore calls can be routed to an in-memory proxy. Also, Fred Sauer's gwt-log is quite handy.

JDO Quirks
Obviously there is no concept of joins. Mappings are also constrained. Pure one-to-one associations aren't supported. Instead, we have a form of composition that the appengine docs refer to as owned one-to-one relationship (e.g. Employee's ContactInfo). I am begining to wonder if the JDO interface is actually a hinderance to thinking non-relationally. It might be better to use the low level datastore API directly.

Overall, it has been fun. Java appengine seems quite promising. ThoughtWorks plug: Yes, we offer app dev and consulting services for all things cloudy.

Update: Coverage by fellow ThoughtWorkers:

Ola Bini
Paul Hammant
John Hume
Philip Calcado

SaaS is a distribution model, Cloud is a hosting model

2009-03-30T00:41:00.003+05:30

The series on fallacies of knowledge management will continue after this post.

I attended the first CloudCamp in India at IIMB today. It was well organised and well attended. I had only planned to network and attend the talks but ended up onstage in true unconference style. Dr. Srinivas P was kind enough to invite me to a panel discussion on certain aspects of cloud computing. While the panel discussions were mostly illuminating, I noticed some mixing of terminology around SaaS. This post is meant to highlight that SaaS is a distribution model for software and is independent of the hosting model (severs under a table, private data center, Google AppEngine, Amazon etc).

Yeah, one could argue that the appengine is more than a pure hosting model but that is not material to this post. It was agreed during the course of the sessions that Amazon's offerings could be described as infrastructure-as-a-service (IaaS) and that of the Google App Engine as PaaS: platform-as-a-service (for lack of a better term).

Now let's take the example of a business that sells tax-return filing software. If it chooses to sell the software by shipping CDs or offering installable downloads, then its distribution model may be termed as software-as-a-shrink-wrapped-product. On the other hand, if it chooses to host the software somewhere and let users connect to it for filling out their returns, then the distribution model is termed as software-as-a-service. Note that we haven't said anything so far about where the software is hosted in the second case. All we know is that the customer is not required to install it in her premises. So, from the POV of a citizen filing returns, it is either shrink wrapped software or software-as-a-service. The end user doesn't care about whether the hosting is IaaS, PaaS or a private data center.

If you really want to complete the circle, you could view IaaS and PaaS as an exteme form of SaaS, viz: system-software-as-a-service - since the cloud service providers basically offer the use of (dynamically scalable) virtual machines as a service to the application developer.

Knowledge management fallacies: access control is key

2009-03-27T12:14:00.001+05:30

Organizations that are straitlaced about providing employees with unrestricted access to information cannot expect them to have a different attitude towards sharing knowledge with each other. A number of organizations share information with employees only on a need-to-know basis. For any given employee, very little (relatively) information falls in the need-to-know category. She is cut off from information that she might just be curious about. But wait, isn't this a good thing? Why distract employees with useless information? Fair enough, don't push unnecessary information to everyone. But don't hide it behind access control either. Someone might just find a serendipitous use for the information. A better approach might be to just hide the information that absolutely needs to be hidden and free everything else up. Rather than share on a need-to-know basis, protect on a need-to-hide basis.

Wikis are a great example of a technology that turns access control on its head. In a typical wiki, not only can everyone read everything by default, they can even edit anything. This feature used to invite ridicule in traditional departments. But the adopters mostly thrived. Hell did not break loose. Wikis have a robust cure for mischief. It is called "revert to earlier version". Authentication is essential. Authorization is less so. Author traceability discourages frivolous edits.

There is also the issue of scale. Preventive access control doesn't scale. What scales instead are mechanisms that offer cheap cures in case of problems. This is commonly accepted when we build applications for the web. Client server applications used to rely on a mechanism called pessimistic concurrency that tries to prevent problems while web scale applications rely on optimistic concurrency, i.e. taking corrective action in case of problems.

In a fast paced world you can't wait to ask for permission at every turn. We have an unwritten code that helps move things along at ThoughtWorks: Ask for forgiveness, not for permission. A friendly access control regime lowers barriers to participation. And participation is absolutely key to the success of any knowledge management effort.

Knowledege Management fallacies: Good organization is key

2009-03-20T13:26:00.009+05:30

Even though keyword based search doesn't yield satisfactory results all the time, it often delivers better than manual organization.

Search trumps Navigation

Sophisticated Users	Others
Use application launchers like Launchy, GnomeDo or Quicksilver	Use hierarchical navigation from start menu or equivalent
(Coders) Use Ctrl-N (IntelliJ) or Ctrl-Shift-T (Eclipse) to open a class	Use hierarchical navigation of the Package/Project explorer
Use search engines to find information on the internet	Might prefer using something like Yahoo Directory to get to the information they seek
Use desktop search to find files on their hard disk	Always use the hierarchical file system explorer

Traditional hierarchical organization requires users to navigate through the hierarchy to get to whatever they are looking for. Such navigation is unnecessary in cyberspace and is painful for seasoned knowledge workers. Reliance on taxonomies is a relic of physical world thinking. Their importance is overrated in a digital medium.

Folksonomies don't replace search either. Being multidimensional, they are useful for finding 'similar content' after you have found the first one. Even then, it requires very good participation by the user community to be relevant.

Comprehensive organization is expensive and time consuming
It takes humans to put things in their place. :-) Correct classification requires some understanding of the subject. At a time when the quantity of content is growing rapidly, it is expensive to hire people with understanding of the subject to maintain a organization hierarchy. Relying on users to do this themselves doesn't work out well in my experience.

Lightweight, deferred organization is easy
One of the problems with upfront organization is that you have to come up with a scheme that is acceptable to most (if not all) users. Instead, it is easy to slap on a starting page or a table of contents later if you are using something like a wiki. When my project wiki began to get unweildy, someone just added a page called "Day one reading material" with links to several other informative pages (thus bypassing all notions of hierarchical navigation). New joiners to the project were simply directed to this page.

Just Ask Around
Asking people for information has never been easier. Corporate microblogging (micro-messaging) tools like Yammer (or just a company wide always-on chatroom) make it very effective and non-intrusive to ask questions like: "Can sometime point me to existing collateral for xyz service offering?"

Knowledge Management Fallacies: Repositories ensure that knowledge lasts longer than employees

2009-03-11T12:48:00.003+05:30

My employer aims to be a home for the best knowledge workers. Several companies operating in the knowledge economy have similar aspirations. Yet, some of them fall shy of calling themselves a people dependent company. They would rather be process dependent (and people independent). Why? Because a process won't tender resignation and leave. Valuable knowledge may be lost when an employee leaves. Knowledge that is codified into a process is safe from attrition. However, this doesn't work in practice.

Knowledge keeps evolving. Codifying it into a process merely takes a (context-free) snapshot of evolution. Knowledge evolution can only happen in people's heads. The only way to preserve knowledge when someone leaves is to make sure that someone else knows the stuff well enough. However this knowledge transfer will fail if only attempted as a handover activity. Why? Because context is severely restricted during a planned knowledge transfer session. Knowledge sharing and collaboration has to be part of the work ethic of an organization. People need to buy into the fact that knowledge multiplies by sharing. In such a collaborative environment, knowledge silos are minimized if not eliminated.

'People dependence' is much more robust than 'person dependence'. A sharing-friendly culture helps move the dependency from person to people. A process and documentation driven culture moves the dependency from person to repository and knowledge atrophies in due course.

Sharing is effective via conversations, not so much via artifacts. This is because of the lossy nature of communication. Signal to noise ratio decreases as bandwidth decreases from face-to-face to phone conversation to instant messaging to email to finally an artifact in a repository.

Knowledge Management Fallacies: Well begun is half done, so let's begin by collecting content

2009-03-09T20:21:00.004+05:30

This is the start of a series of posts on what I consider to be fallacies of knowledge management.

Some ten years ago, I used to be in the habit of printing good articles I came across on the internet. Sometimes, fifty pages at one go. But I wasn't able to keep up with reading all that I was printing. Yet, the act of printing and possessing the content seemed to offer some kind of reassurance. After all, I could read it anytime I wanted to. I realized the folly of my ways when I had to relocate for my next job. I had fifteen kilos of mostly unread, somewhat dusty, printed material. I was faced with the choice of dumping them (with all the associated guilt of wasted resources) or paying to transport them in the hope of catching up some day.

Then I grew a little wise. I started merely saving stuff to my hard disk instead of printing. I created a folder '2bRead' (still have it) and started growing my repository of wisdom there (or so I thought) - whitepapers, presentations, podcasts, videos, you name it. As the folder grew in size, I began to forget what was in there. I used to search on the internet for stuff I already had. Desktop search only partially solved the problem. The indexes began to grow huge and it didn't keep track of deletes well - I used to get false results. It also didn't help that I used to switch between operating systems.

Then came del.icio.us and I thought an end to my woes. Several bookmarks and tags later, I realized that I was spending more time bookmarking and tagging than actually reading and assimilating. It wasn't because I was an obsessive compulsive organiser (far from it). It was just that I often didn't have the time to study something when I came across it - it was much easier (and reassuring) to just bookmark and tag it.

Now I use a much harsher approach - read it then and there or forget it. You might be more disciplined than I am and might find this absurd but I suspect I am not alone. Individuals and organisations are struggling to keep up with the assimilation of information. So they tend to resort to what seems to be the next best thing - the collecting of information. However, this tends to be mostly a futile exercise. It might be more useful to focus on assimilation of information whenever 'information events' happen - talks, presentations, meetings. Focussing on recording and distributing (or storing centrally) in the hope of future assimilation is going to be increasingly ineffective.

Favour individuals and interactions over artifacts and repositories.
Favour context rich conversations over mostly context free collateral.

After all, we don't manage knowledge within digital repositories. We can only hope to curate information within them. Knowledge can only reside in people's heads. Efforts to improve knowledge should therefore focus on assimilation via collaboration rather than on curating information.