Scott Bellware

The Myth of Developer Productivity

2009-07-12T00:43:00.002-05:00

There are a couple of predictable approaches to an imminent car crash. You could throw your hands in the air, scream, and hope for the best, or you could keep your hands right where they are and try to pilot your way through it to the last responsible moment. If you're the pilot of a software team, offloading the responsibility for productivity is like taking your hands off the wheel before you've left your driveway.

The quickest way to shut the door to productivity is to try to solve it exclusively as a tools and automation problem with tools that promise "Developer Productivity". Tools and automation are essential parts of getting software done, but unless you have a firm grip on why you have productivity problems, taking someone else's word for why a tool or automation will solve your problems amounts to little more than a shot in the dark. And since my word is inevitably "someone else's word", please look more deeply into the issue beyond this article.

Productivity that Matters

There's productivity that matters and productivity that doesn't matter. That probably seems nonsensical - after all, if you could gain productivity, wouldn't it matter? It depends on wether the productivity you gain causes a commensurate obstruction at some other point in your software development bucket brigade.

One heck of a lot of tools that are sold specifically as "Developer Productivity" tools create productivity increases for the developers in your development pipeline, and kill productivity for downstream work centers, like testing, packaging, shipping, installation, configuration, and operations.

Focusing on developer productivity without considering the effects of developer productivity efforts on the rest of the whole pipeline will usually create momentary productivity; productivity that only developers will feel, and often only for a short period of unsustainable time.

Software teams and organizations continue to fail to realize sustainable productivity by continuing to make improvements in one area of the pipeline without realizing the cause-and-effect relationship between the localized improvements and resulting degradations in other parts of the pipeline.

Sustainable productivity is the only productivity that matters, and it's the only productivity that can withstand ever more continuous improvements.

Productivity is about product. It's the activity quality of product, or producing, or production - the "ivity" of product. It's about producing and the production of the only thing that really matters - the final product that can be used to achieve the business's goals.

If the goal of software development is just to create developer artifacts without ever delivering them, then developer productivity itself in isolation would matter a whole lot more. The problem with "developer productivity" is that it is inherently productivity in isolation. Productivity in isolation is often naive local optima.

Local Optima

Local optima problems happen all the time in software organizations. They happen when we try to make improvements in one area of the whole software development workflow without understanding the effects on other areas of the workflow.

Developer productivity as we know it colloquially is inherently a local optima concern. My first concern is the productivity of the entire fire brigade. The moment that there's some obstruction anywhere in the whole workflow, the problem will spread outward and poison adjacent work centers, rippling outward, and sometimes making quantum leaps into parts of the pipeline that are, on the surface, seemingly disconnected.

It's not enough to just understand that a change in one area of team's workflow will have effects on another area, it's also critical to understand which kinds of effects will create obstructions for the whole effort.

Greater productivity often come less from increasing the speed that we can do the work itself, and more from recognizing and decreasing the obstructions to production and producing. One of the most obvious ways to attack the productivity problem is to reduce rework and the things that lead to it.

Reducing rework means undertaking organizational and cultural changes. It means that the making of software and the proving that it's right can't be allowed to work at dramatically different paces, which is usually the case with most software teams (even teams using Agile methods).

Every time that testers get backed up in their work, untested software piles up in front of testing. That pile of untested software is the unproven foundation that developers will continue to build upon wether or not it’s sound. As the pile grows, the likelihood of rework grows geometrically, proportionate to the size of the pile, and the near certainty that structural design is insufficiently precise grows along with it.

When the making of software and the testing are disconnected from each other, and software makers and software testers work at full speed regardless of whether they're building up piles of risky inventory, then productivity is going to degrade.

In this kind of situation, doing things that increase programmers' speed isn't going to help productivity at all. In fact, the right thing to do is to either slow the programmers down until the testing inventory is cleared, or to have the developers change hats and work with testing to clear the obstruction. Increasing a developer's speed will only exasperate the problem.

Optimized developer productivity without simultaneously optimizing the entire pipeline is local optima.

Developer Productivity Myths

There are a number of tools and libraries that are sold under the "Developer Productivity" banner. These tools actually deliver developer efficiency rather than developer productivity.

Unless a tool's productivity proposition takes local optima into account, it flirts with negligence. If it does so knowingly and willfully, then it flirts with corruption; verily stealing value from its users and customers.

Consider Microsoft's re-interpretations of "Rapid Application Development" that drive its developer tools design:

Microsoft's Visual Studio enables developers to create input forms and data processing visually with little expense of time and effort. Developers get this work done very quickly, but the resulting systems features are inordinately difficult to test. Microsoft also packages tools that are specifically geared for "testers" in a separate product package, perpetuating the conclusion that doing work and proving that the work is right is the jobs of different people, creating the handoff boundaries that invite unproven work to collect as piles of costly inventory.

While this allows developers to be very efficient in the work of creating the code and artifacts that go into a feature, that efficiency isn’t realized as productivity. The software designs created by developer efficiency tools are unnecessarily and excessively difficult to test. This discourages developers from preventing the defects that pile up in front of the testing work center which also steals productivity from testers, exacerbating the problem of basing today's work on last week's unproven decisions.

Software organizations who are seeking and realizing higher productivity - that is, they produce end product to business sustainably and timely - come to understand that the inventories built up around handoffs and segregated work centers must be decreased and ultimately eliminated in order to really reclaim lost productivity. This has a profound effect on the shape of software teams and organizations, and in retrospect we come to see that our beliefs about organizational mechanics and culture were what kept us trapped in superstitious dark ages of software development productivity.

Regardless of whether tools like Microsoft Visual Studio (among others) seem impressive on the surface, failure to assess the effects of these tools on an entire pipeline will inevitably lead to mere momentary efficiencies and local optima. Worse yet, they will obstruct the learning that does in fact lead to productivity that matters, and will pull managers away from valuable work like teaching and facilitating and force them to become inventory managers and expediters.

It's not just the younger teams using Microsoft's visual tools who are subject to this problem either. Even advanced "agile" teams indulge in developer efficiency local optima like the hypercoding enabled by a higher order of developers tools. In some cases, hypercoding has even motivated urgent and wide-ranging changes to frameworks to optimize for a tool's use even in the face of compelling evidence that more meaningful sources of productivity are likely elsewhere. The compelling productivity of Ruby on Rails programmers despite the unavailability of extensive hypercoding tooling is a good example.

There's no one true answer to whether a tool is going to contribute to productivity. Sometimes just the creature comforts afforded by a tool are significant sources of productivity. But then, not all indulgences in creature comforts can be considered productivity enhancers either.

Ultimately, any of these tools can be used in a balanced, leveled software development workflow. However, until a meaningful understanding and representation of productivity takes root in software development, any team at any level of maturity is going to trade productivity that matters for mere efficiency.

Fixing the Problem

We can affect the things we control. When software projects go awry, we exert control. Any time we exert control without considering the impact on the entire fire brigade of software development workflow, we're going to create efficiencies at the expensive of our ability to produce.

The further away you are from the whole of the team and the workflow, the less likely you are to exert control constructively. As a senior manager, you might prescribe a suite of "Developer Productivity" tools after seeing a compelling presentation from a vendor that is specifically geared to affect your sensibilities from your perspective. If you're a developer, you might convince your team to adopt a "Developer Productivity" tool that demonstrably makes you a more efficient coding machine.

The decisions made by people who are too far from the whole are often a coin toss. There's little telling whether a team will perform better in getting products into the hands of the people who need them, and it’s often difficult to connect the decisions with the ultimate outcomes.

If you're encouraging or enforcing a team organization that disconnects the work and workers from the validation of their work, whether analysis work, design work, construction work, construction inspection work, packaging work, installation work, or operations work, you will inevitably create the conditions that encourage inventory build up and the subsequent obstructions, rework and general degradation of productivity. The shallow pursuit or mere localized efficiencies are more likely to happen when work centers fail to be shaped to productivity goals.

Making any of a number of mistakes that trade local efficiencies for productivity not only degrades productivity, but creates a cycle where the degradation accumulates, leading to the typical software cost curve that is ultimately a reflection of the degrading productivity curve.

Fixing the problem isn't trivial because no single local optimization will have a predictable effect, and a set of disconnected local optimizations degrade productivity even faster and even more unpredictably.

There's no doubt that we need to act on all levels, and ultimately this means decomposing the problem and working on different levels of an organization an at different work centers. But the local things we do to fix the problem have to extend from a holistic understanding of the software development system.

If you want to fix your software development system, find the problems with the system and then understand how the parts contribute to the problem. Evaluate the success of each effort to fix the parts by the effect that it has on the system.

Developer productivity can just as easily be a reality as a myth. Developers are obviously significant contributors to producing software. But an approach to "developer productivity" that isn't also an approach to organizational productivity is often not likely to do more than transfer value from your organization’s treasury to the coffers of a vendor who is more than happy to assume ownership of your precious resources.

We can’t get software done without software tools – this is true - but choose wisely. Your whole team’s productivity is at stake.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Relearning: The Productivity Problem that We're Not Supposed To Talk About

2009-07-08T00:42:00.013-05:00

Imagine that you had no memory; that everything you learned had to be re-learned again and again as you did your work. If you worked in software development, you wouldn't have to stretch too far to imagine it. Re-learning is so much a part of the moment by moment work of software development that it's considered normal. In fact, it's the unrecognized backdrop that software development plays out in front of.

Because relearning is not recognized as a problem as software development, it's almost never talked about it. And frankly, it's not a welcome subject in polite programmer society. Nonetheless, relearning accounts for a lot of unnecessary lost productivity and waste in software development. It can account for wide swings in productivity loss on most software teams, affecting the manageability of software projects and a truthful and meaningful assessment of programmers' real abilities as designers, analysts, problem solvers, and real contributors.

I'll offer an observation from my own experience and from the anecdotal evidence of others in my professional circle: I would be comfortable saying that half of the lost productivity on software teams comes from re-learning. It can come in the form of poor testability - the inability to easily provide proofs that expectations are met - and it can come in the form of source code that can't be understood at a glance. Even teams who have solved the testability problem still largely suffer from the lack of usability of their code, and the inevitable mass of relearning that comes of it.

First We Scan, and then We Read

Text in a text editor is interactive media; subject to the same fundamental usability principles that apply to a web page, a desktop app, or even a billboard on the side of the highway.

Before users read the content of interactive media they scan the content. Programmers scan
code before they read it. This is the singular human behavior that programmers consistently fail to recognize. It's the fundamental behavior that when recognized, becomes one of the pillars that code usability efforts can be built upon, and the starting point for recouping losses from relearning.

A singular focus on readability in code is a concern that, while encouraging, still misses the point. It's the same point that programmers missed as a human-computer interaction industry grew from the ashes of our continued failed attempts to create productive user experiences.

Usable code dissolves into understanding at a glance. It doesn't need to be coerced into understanding. I like to call this kind of code "soluble" code for the image it brings to mind of program text readily dissolving into awareness and understanding. Soluble code is likely readable code, but readable code isn't soluble unless it's written to be soluble.

Reading code isn't like reading a good article, where you start at the top and read to the bottom, enjoying the experience and being fulfilled by it. Granted there is beautiful code in the world, but the typical reasons for reading code are not the reasons that we read articles, books, papers, and the like.

In fact, the nature of the media that this article is published in, and the context that this media is typically consumed in, is such that you are more than likely to start jumping around the text with your eyes and with your mouse, looking for the nuggets and pearls while avoiding having to consume the entire thing linearly as it is written.

The first thing we do with code is ascertain whether it is indeed the code that we need to be working with in order to accomplish whatever task we've taken on. This even happens when we're reading code for the pleasure of it. The first eyes-on experience with some code usually involves rapid scanning of the code to ascertain if we're in the right place; if
we're at the worksite. Much of this happens pre-cognitively as it does with any media that we've been called to act upon. First we scan to get our bearings, and then we read.

We take high-level structural scans, followed by smaller, more detailed scans, followed by reading. If the code that we're scanning is written so that it only yields its meaning from a detailed read, then not only are we not optimizing for the natural human interaction patterns, we're also not taking advantage of providing the incidental knowledge that can be yielded to someone while they're scanning. This incidental knowledge accumulates and becomes a significant part of the material invested in the metal map of a codebase that a programmer accumulates through exposure to code that yields its meaning at a glance. We retain a codebase's textual geography only when we absorb its meaning.

If we don't code for solubility, we force programmers to take detailed reads in order to tease knowledge and understanding from the text. Forcing these detailed reads doesn't lead to greater advantage down the road. Less meaning is retained when forcing detailed reading into a context where a programmer is instinctively trying to scan.

Scanning is as much about a process of elimination as it is a process of accumulation. Both are happening at once during scanning unless code isn't amenable to these processes. If we fail to take advantage of scanning by failing to write soluble code, we're stealing productivity from our team mates, and from ourselves. On the surface, this seems like a negligible issue, but lack of solubility is responsible for a tremendous amount relearning and degraded productivity.

Solubility as a code style permeates a codebase. It's a pervasive quality; it has a constant, pervasive effect. Small improvements add up to significant advances when they have a pervasive effect.

Making Code Soluble

There are no cookie cutter patterns for making code soluble. Some things are obvious: meaningful symbol names (class names, method names, variable names, etc), and higher-level methods that tell the story of some process that call lower-level methods that contain the details. Both of these go a long way to create soluble code. Soluble code yields its meaning immediately.

The particular refinements that a team will make depends on the team, the product being built, the technology, and a host of other conditions. Pushing further for an canonical definition of soluble code patterns might lead to just as much productivity loss as productivity gain. Further practices, beyond meaningful names and anecdotal methods, are contextual and shouldn't be dropped into the code as if they are interchangeable parts.

Soluble code is an experience like the table of contents in a novel. It offers and allows multiple levels of reading, with each deeper level yielding ever greater detail. A table of contents is an affordance to the reader that acts as a navigational aid, or map, of the text. But a novel isn't program code. A table of contents at the beginning of a novel can be sufficient for that kind of media with its implicit user experiences, whereas program code itself must be it's own table of contents, right down to the very small, five-line, composable methods. The user of a novel is typically well-served by a table of contents whose resolution goes no further than mapping out chapters, expecting the reader to start reading linearly from the beginning of the chapter. But this isn't the case with program code.

When we're scanning, we're mapping the code by the what of the code; what the code does, what it's responsibilities and behaviors are. Soluble code allows a reader to immediately understand what is does before forcing the reader to understand how it does it. Soluble code serves both modes: scanning for what, and reading the how. Code that isn't styled this way largely deprives a user of the process of elimination, the incidental knowledge, and the mapping that can be had from scanning.

One of the worst mistakes that programmers make in writing code is in failing to recognize that more productivity will be spent over the life time of code navigating through the code than will be spent writing the code. It doesn't take much more effort to write soluble code that serves the elimination of relearning. Choosing to not write soluble code means choosing to keep the relearning waste well-entrenched, but there's more to this problem than mere choice.

Resistance to Code Usability

There are reasonable objections to styling code for solubility and usability. The how
of the system ends up spread over many small methods rather than concentrating it in fewer, larger locations. The root cause of the aggravation is often not that the code is factored in
small semantic units, but that the semantic units are not the right ones, or that the factoring is just not good. Yes, this is the you're not doing it right response, and as unfashionable is this response is, it can nonetheless end up being the root cause. Nonetheless, some programmers are just not going to want to get used to soluble code style, and there will be inevitable grumblings from people who prefer to use more traditional, procedural structure. It's not hard to bring usability to code, but it can be discomforting at first - like transitioning to a new programming language.

Programmers aren't traditionally the folks on a software team who have their heads in the usability game. And we've gotten to an unfortunate point in programmer culture where the answer to many subsequent problems with programmer ineffectiveness has been to create further specializations and allow programmers to be responsible for a narrower and narrower set of expectations rather than deal with knowledge problems as organizational and cultural problems.

Expecting programmers to be considerate of code usability creates friction. For many programmers it's going to be as comfortable as thawing out frozen, front-bitten fingers. But it's not just a programmer responsibility. A good chunk of the responsibility for change rests squarely with the surrounding and supporting organization and its protocols and mechanics. There's more to rehabilitating software development productivity than introducing programmers to new coding pattens. Organizations that have a commitment to learning cultures will do much better at this.

The staunchest resistance to efforts to reclaim lost productivity due to relearning will come from hero programmers. Hero programmers are those guys in any organization that can get the job done with any code in any state. They are typically blessed with what seems like a supernaturally high-definition mental map of a codebase. This is their best and worst quality. It's their best quality because they often know where to fix a problem in a codebase and have a reasonable grasp of the myriad side effects that might result. It can be their worst quality because it's often an effect of mild Asperger Syndrome common to programmers, engineers, and jobs that require extended, intense focus. It's often accompanied by the lack of awareness and empathy toward peers typical to the condition.

Hero programmers can suboptimize the efforts of a team by not having to rely on soluble code, often navigating a codebase entirely from memory. The ability can be incredibly useful, but the need to write soluble code rarely manifests because it's not a personal need, and the typical lack of empathy obstructs the ability to recognize how this advantage undermines their team mates' efforts to be as effective.

The code created by hero programmers isn't soluble because the heroes rely on uncommon facilities that preclude the need for solubility. They don't notice the design smells because they rely almost exclusively on echo location for navigation. The resulting work product can often only be as effectively worked on by the heroes themselves, which inevitably leads to predictable resource bottlenecks, bus factor risks, excessive specialization, and the general malaise on a team as the effects of code toxicity spill over into the human realm.

The analogy to the baseball shortstop who was famous for making great plays applies: his coach often pointed out that he was out of position to begin with.

Usability is rooted in an ability to have empathy for users. That empathy gives designers the pause to consider whether they got the user experience right. It's the empathy that leads to the questioning that leads to the recognition of interaction design problems in the form of cognitive obstacles that undermine the user's productivity. In an environment with traditional and institutionalized lack of awareness and lack of empathy, dysfunction can drive unconscionable waste.

Hero programmers rarely stop to question whether they've left behind a good experience for others on the team who need to navigate, then understand, then make changes to the code. And most programmers will fail to recognize the two distinct mindsets that are in play when working with code.

Writer's Mind and Reader's Mind

Without ever doubting whether code is usable, programmers will presume that the right thing has been done. Programmers who are good at creating soluble code have learned to doubt every line of code written. It's not that they have all of the answers. More importantly, they've learned to be constantly questioning.

In the words of the anthropologist Claude Levi-Strauss (not the guy who invented blue jeans), "The scientific mind does not so much provide the the right answers as ask the right questions."

Asking the right questions means constantly switching from the writer's mind to the reader's mind. That means that after each bit of code is written, a programmer switches mental contexts and assesses the code from the perspective of someone who has never seen the code before, asking, "What have I done to undermine the immediacy of someone else's understanding of my work? What unrecognized presumptions have I made about their context as a reader that only applies to my context as a writer?"

That's quite a trick, and it's not uncommon to hear the complaint that it can't be done, but it's what interaction designers do all the time. It's not uncommon for the deleterious effects of programmer autism to cause programmers to fail to have awareness sufficient enough to break out of the laser-like focus on writing code and switch back into questioning.

Code that doesn't incur the cost of relearning is code that can be immediately understood by someone who hasn't seen it before with minimal orientation to the code and the problems it solves. It's code that can be understood at a glance. That code is rarely if ever produced by a mind that has lost its awareness of its context and its mode. It isn't produced by a mind that fails to concede that code is written to be read, that the readers are other humans, and that the reader's context and needs are not the context of the writer - at lest not until the reader has found the worksite in the code, and gained sufficient understanding of the worksite to begin to make the necessary changes that they're tasked with.

The writer's mind is a context that often fails to recognize that the focused and relatively linear mechanics of writing code is quite different than the mechanics of consuming code as a reader. And this is where the misconception over readability comes from. The writer's mind, working relatively linearly is also consuming code in that mode. Readability is a quality that pertains to the linear consumption of code as text, and that kind of optimization of experience is only relevant some of the time in some user scenarios.

Break the Habit

The autistic mind is lulled by the hypnotic cadence of constantly pumping out code. It will complain that the constant switching between the writer's perspective and the reader's perspective is ruining their ability to get in the zone. And in truth, it is, but it's that particular zone that is causing a lot of relearning debt to mount up. There are other zones to get into with equally pleasing effects, but it is a matter of breaking some habits and replacing them with new ones. There are a few tricks and techniques that programmers can use to break out of the fog and get into the zone.

The most important thing to practice is the constant self-questioning of whether the code just created is soluble; that it can be understood at a glance and yields enough meaning while scanning to contribute to the mental map. Instead of presuming that everything I do is made of gold, I presume instead that it's made of fools gold. From that perspective, I can usually gain the right amount of objectivity to assess solubility.

Pair programming and test-driven development are two techniques from Extreme Programming that are extremely effective at clearing the fog. When these techniques are practiced together, it becomes very difficult to be lulled into the unexamined mind that produces unexamined code.

If you don't like pair programming, try using some kind of timer on an interval that is just short enough to be uncomfortable that will remind you to come back up for some critical thinking.

And lastly (for this article anyway; there are more tricks out there), Context Specification is a form of Test-Driven Development that recognizes solubility, usability, reader's mind, and authorship, and forces the issue of contextual analysis to bring more practice of awareness into programming.

Commit to Learning

Ultimately, there's no where left to hide. The anti-productivity that comes from inviting and accepting relearning continues to accumulate. We've pushed it into the lowest level of software development where only programmers can (or may) see it, but it still affects everyone touched by the software project or the product.

Relearning waste is a fundamental organizational behavior. Until we shape organizations around dealing with waste and learning, we continue to fail to see in the range of the spectrum where the sheer magnitude of the relearning waste is visible. Relearning is inculcated into software organizations. Shifting from re-learning organizations to learning organizations and learning culture is how this problem is ultimately solved.

It's self-evident: counter-act relearning with learning. The term learning organization isn't the trite and trivial perspectives that see learning as something external to team and organization; a destination where people are sent once in a while to be "trained". Learning is no more about receiving training than quality assurance is about giving click recorders to test monkeys. The emphasis we put on "training" in the software industry undermines our ability to see the "learning" side of the same issue, and to build truly meaningful teaching/learning experiences in our organizations, and to subordinate organizational mechanics and protocols to these imperatives to counteract the constant, tireless forces that drag us back into relearning.

The value we throw away on relearning is recouped when we counter it with meaningful learning. Learning gets real when it's expressed in every organizational protocol and business process, and importantly for software development, when it is expressed in every single line of code written by everyone involved in bringing a solution to life.

The Problem with Big Design Up Front is the "Big" not the "Up Front"

2009-07-03T22:40:00.000-05:00

The risks of Big Design Up Front isn't the "Up Front" part, it's the "Big" part. Doing too much design without validating it inevitably drives a good bit of the productivity loss that continues to hamper software projects.

It's an issue of large batch sizes - the "Big" in "Big Design Up Front". The larger batch sizes mean that we're always building today's software on yesterday's work and yesterday's decisions before proving that yesterday's work and decisions are sound. The larger the batch size, the larger the risk that the incorrectness of design will lead to ever more expensive countermeasures. And in many cases, design flaws are too subtle to be seen immediately, and collect negative potential energy in the form of on-going degradation in productivity that came to be seen as "normal".

Big Design Up Front has often come to be interpreted as "No Design Up Front", and has led to a lot of mediocrity in design by inappropriately democratizing the responsibility for design quality. This is often done in the name of cross-training, but the best way to teach potential software designers to be good software designers is to constantly expose them to good software design, rather than the mediocre design that can come from a misinterpretation of Big Design Up Front.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Designing the Work

2009-07-03T22:27:00.004-05:00

More than just designing the software, technical leadership must also design the work of implementing those designs. Taking the shape of the work into consideration along with the shaping of software modules serves the goal of predictability that comes from leveled production, and the awareness of trouble spots. Designing the work serves human and organizational concerns, and predictable manageability.

Decomposing requirements, be they user stories, features, or what ever you use in your process, is inherently a design activity. Division of labor follows the division of software modules, and the division of software modules also follows the division of labor. These design activities have to be done in consideration of each other, likely by the same people and at the same time.

The design should be communicated to the team, and at the very least, to the members of a team who might be tasked with the work, but it isn't necessary to have the whole team involved in creating the design. The best designers should be involved in doing the design work. This should go without saying, but some of the interpretations of "self-organizing team" can contribute to obscuring the obvious: individuals on teams do indeed have strengths, and some are stronger than others.

Communicating design and expectations is a great side effect of all-hands planning and estimation, but it can create poorer designs that are normalized to the average skill on the team while the team learns how to design software. Becoming a competent software designer takes many years of work and the uncanny circumstances that create the precursors of design awareness that, when complimented with experience, creates the talented and mature abstractionists, analysts, implementers, and engineers who lead design efforts. Planning and estimation and teaching and design do share some common ground, but teaching design and growing designers should be an explicit goal with specific work, rather than a magical side effect.

Decomposition, task breakdown, and scheduling are all aspects of design. When we communicate design to the people doing the implementation, we're setting expectations for their work. When we set expectations for people, we (hopefully) activate their critical minds. They inevitably cross-check a work plan and inevitably cross-check the design as they learn about the plan. That's not exactly the same thing as the design-by-committee exercises that are too frequently the result of consensus estimation design, where decomposition is often done as part of whole team sessions that are concerned with dialing in agreements on story points.

Designers front-load the design, bringing their experience and aptitude for design to bear, and provide structure for the ensuing conversations. The tension between the design and the cross-checking can lead to new insights. It can lead to changes in the scheduling of the work and it can lead to changes in the design, but it's not a consensus-based free-for-all that leads to the mediocratization of design. The front-loaded design done by competent designers produces a more balanced equation, especially when they are also designing the work.

This isn't Big Design Up Front or phased-based SDLC though. Designing the work simply means that there's necessary front-loading involved in producing software. Front-loading doesn't preclude inspecting and adapting, responding to change, or emergent design.

The balance between good work that can be continually built upon, work that creates lower standards of productivity, and work that can be managed to expectations hinges on more than good software design. Software development productivity lives at the intersection between well-designed software and well-designed software work. Agile estimation is a good start down the road to manageable software development. Designing the work closes the gap between the beginnings of manageable work, and work that can be managed.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Flow, Leveling, and User Stories

2009-05-25T00:59:00.009-05:00

Flow and leveling are two sides or the same multi-sided shape. The goal is an understood and controllable cycle time. Without leveling, sustained flow is unlikely. Without flow, improvement is less observable and manageable. Improvement efforts might be rooted more in medieval software superstitions than something closer to a science.

David Anderson talks about the core of Kanban as an agreement that the team has a capacity; that they can only work on so much at a time; and a limit set around that work. Implicit in Lean work management styles is the right-sizing of work items. Reducing the broad variability in work item sizes is a part of the work that goes into flow and into controlling cycle time.

Typically, in a method like Scrum, a team organizes its work efforts around user stories. The user stories that are worked on during an iteration (or sprint) can be any of a number of sizes. Using the Fibonacci estimation technique the variation in story size can be almost infinite.

User stories are a way to communicate expectations, but they're a less effective way to manage work and improvement because of the amount of variation allowed in user story sizes. You could decompose stories into smaller stories so that they're more manageable, but then you'd risk fragmenting the communication value of stories by breaking them into separate chunks that might not carry as much context as the original larger stories.

User stories can be of variable size because they inherently are of variable size. Let user stories be the size that makes sense for the kind of artifact that user stories are: an artifact intended to communicate context and expectations to the development effort. Use right-sized work items to manage work. Decompose user stories into work items and schedule those work items for development in way that takes both customer priority and technical constraints into consideration.

The work items can certainly be linked back the stories that they are decomposed from, but it's the work items rather than stories that go through the development pipeline.

I've sat through enough all-hands Agile planning and estimation sessions to know that while this practice is useful for bring a new agile team together, it's often a terribly ineffective way to establish design. It's a good way to socialize design, but there are alternatives that don't imply the wastefulness of traditional Agile planning and estimation. Using design as a team-building exercise can put the design at risk.

In the worst cases, the all-hands planning and estimation that is common to Agile development methods can hurt the team. It's just as likely that adversarial conditions can break out in the team if an expectation is set that suggests that product design and implementation design is egalitarian and democratic within a team that is necessarily staffed by people of varying experience and ability.

Design should be done by those who are most capable of doing design. Story decomposition should be done by the person or people who are best equipped to do that work. This doesn't preclude the socialization of that design to the team and the proving of that design by the team.

Rather than all-hands estimation and planning, design and planning can be done in short, effective sessions with the team's designers. Since designers are decomposing to work items that are intended to be of a similar size, the team no longer estimates work, but reviews the decomposition and seeks clarification and raises concerns. The collective intelligence of the team is still leveraged, but the potential waste and ineffectiveness of traditional Agile planning is avoided.

The planning and estimation activities change from larger-scale chunks of time, effort, and team participation to smaller units of tiered work and cooperation that can be done just-in-time rather than on those specific days of the week when the whole team's attention can be coordinated and directed to a fixed-schedule meeting. Consequently, any all-hands team meetings can be scheduled on an as-needed basis as well, eliminating any waste that comes from institutionalizing all-hands meetings according to a fixed timebox.

With planning, design, and scheduling being done in smaller units on a just-in-time basis, some of the fixed time boxes in the development process aren't needed. Features can be queued for packaging and deployment on an opportunistic basis as well. There may still be some fixed rhythms at-play in the effort - customer demos and major market events, for example - but these synchronization events don't have to leak their scheduling mechanics into aspects of the work that aren't dependent upon or necessarily coupled to those rhythms.

Work items aren't just same-sized, they're also small-sized. The larger the work item, the more variability there is between the estimate of the work and the actual work. We understand larger work items with less certainty than smaller work items. That's because decomposing work into smaller items means that we have to think about the actual units of design and work that go into delivering those bigger features. Because we're trying to achieve flow in pull-based systems, the variability inherent in larger work items is a likely obstruction.

The specific amount of work that defines what "small" is depends on the kind of work, the team, and a handful of factors. And the actual size of "small" will likely change over time. In some cases, it might just be impossible to get all work items to be of similar size. Consider whether the work can be scheduled so that groups of similarly-sized work items can to be done together.

Without leveling, flow will continue to be difficult, and Kanban a Japanese word that has come to mean "story wall".

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Pull Harder to Find Problems with Flow, and Improve From There

2009-05-24T16:14:00.002-05:00

A focus on flow that leads to continual observation of the health of flow lets us use flow to improve organizational performance. Once flow is happening in a pull system, pull harder. Pulling harder will show you where the next level of obstructions is in your organization that keeps you from the next level of performance.

Lean and Kanban are geared to performance improvement, and their principles are applied regardless of the development process in-play, and in concert with the existing process. As opposed to something like Scrum, Lean doesn't require the wholesale, instantaneous replacement of existing processes with a prescriptive process template like Scrum (or others).

Using Lean principles and Kanban will show you the hotspots in your process where you might need to focus your improvement efforts. The software process mechanics that you end up with could very likely end up looking a whole lot like a prescriptive Agile development process, but the adoption and use of specific practices and mechanics are driven by tangible need, based on the repercussions of several instances of pulling harder and deploying and observing the countermeasures that clear out the chaotic oscillations that result immediately from pulling harder.

Pulling harder means increasing the expectations for the capacity of the system. Lean organizations aren't sitting on their laurels. Successions of pulling harder, solving the resulting organizational, process, and mechanical problems, and understanding and working with resulting performance improvements is Lean's continuous improvement. Continuous improvement means an inculcated value system that doesn't allow its past achievements to distract from the next level of goals, and the inevitable ardors of reaching them.

The organization is there to serve the goal. A Lean organization isn't a goal in and of itself. The organization bends to serve the nexus of value creation at the center of the vaue-add work. An obstruction to the ability to successfully pull harder might be found in a part of the organization outside of the immediate product development team. If that's the case, it's understood that the system as a whole must be optimized in order to continue to improve performance.

Once you've got your bicycle up to speed, pedal harder and see what might get in the way of being able to. And then devise and manage a well-considered plan to adapt the process and the organization to get those obstructions out of the way.

Pulling harder creates process improvement simply by setting expectations that are designed to clearly and brightly illuminate the ugly obstructions the lurk in organizations and processes and even in the tools being used. Simply: improve by improving. Set new expectations for the well-understood process and team and take a principled approach to understanding and working with the team's capacity.

Merely overlaying new process templates on a team isn't really the most reasonable and effective approach to improving performance. A change in process should be goal-driven and executed methodically. You might make some sweeping changes in the process, but the wholesale adoption of something like Scrum's agile project management prescriptions isn't likely necessary, and in some cases such a course might exasperate the effort for performance improvement.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Flaccid Kanban

2009-05-22T22:56:00.006-05:00

A few shops in my professional community have started flying the Kanban flag. This is potentially a great step forward in opening the agile software development dialog to a more rigorous understanding of value, especially in how value is lost and how to defend development efforts from the pervasive value leaks in end-to-end workflows.

XP and Scrum shops - especially those with people with significant XP and Scrum experience - might trip over their unrecognized biases at first.

Iterations and Rolling Wave

Kanban doesn't require iterations. Scrum and XP work management doesn't magically transist into Kanban when we remove iterations. What we get from iteration-less Scrum and XP is Scrum or XP with rolling wave planning. Regardless of the benefit of rolling wave, it's not really Kanban until you're asking (and answering) some fundamental questions about work management, organizational performance, and value.

What is the purpose of Kanban?

The purpose of Kanban in software isn't to merely ditch the overhead of iterations. That overhead is a form of waste that might be avoided when iterations are taken out of the picture, but depending on the surrounding context, just as much compensatory waste might be created when iterations are removed.

The purpose of Kanban - other than visual control systems - is to limit work-in-progress in relation to a team's capacity in order to become sensitive to opportunities for improvement and to see waste and value loss (attribution: david anderson). But it's not enough to just announce in a team meeting that context-switching is now verboten. There's a point to limiting work in-progress, and there are a good number of inter-related practices and principles that benefit the work and the management of the work that come into play.

Flaccid Kanban

Martin Fowler recently wrote an article entitled Flaccid Scrum. The article calls attention to a Scrum implementations and adoptions that try to achieve agile software development's promise without implementing anything but agile project management, ignoring the engineering practices that allow agile project management to yield the promises of agile software development as a whole.

As Lean gains popularity and adoption, it becomes susceptible to the degradations that come of mainstream opportunism. Kanban adoption into relatively mature XP teams seems to be creating a sort of Flaccid Kanban - implementations of Kanban that more or less disregard the underlying reasons and mechanics of Kanban.

The real loss here is that we tend to stop looking for something once we believe we've found it. What has been frequently adopted lately that has been called Kanban appears to be merely rolling wave planning. Not that there's anything wrong with rolling wave, it's likely a necessary component of Kanban in software development, but there's nothing good about reaching for Kanban and coming up short because of the temptation to use Kanban in-name-only merely for the fashionista value.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

A Metaphor for Flow

2009-05-22T21:54:00.003-05:00

Kanban is concerned with flow. Flow brings a group of inter-related concerns into play. Flow isn't as readily-understood in software production as it is in other production industries. Here's a simple metaphor for flow:

Flow is like pedaling a bicycle. As long as you're pedaling, you can continue to pedal, and maybe even pedal faster.

If there's any obstruction to pedaling your bicycle, you certainly won't be able to pedal faster. You'll be pre-occupied with just getting the pedaling happening again. If the chain breaks, you certainly won't be pedaling at anything even approaching your optimum.

So much software development effort is spent trying to get the chain back on the bicycle that we're practically inured to stagnant flow and verily accept it as inevitable.

Waste is only inevitable if you throw your hands up in the air and give up on the field of study and effort that moves software as a production industry into the realm of maturity that other production industries enjoy right now.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Specialization in Recruiting in Hard Times

2009-03-20T20:54:00.002-05:00

Recruiting is only one part of the human resources game, but it’s a self-contained sector when businesses are clamoring for talent. It’s a specialization that has a hard time remaining sustainable when the economy isn’t vibrant enough to support specialization.

Recruiters who are having a hard time have an opportunity to un-specialize, and to turn the business into something that is aligned with circumstances in the here and now. That doesn’t mean that they’re guaranteed to survive, but if they do, they’ll be well-placed to emerge quickly from the slump when it ultimately comes to an end, and to do so with more diverse and compelling services. As difficult as things are right now, dealing effectively and creatively with constraints and surviving them is a darned good way to build a foundation that will support a bigger and better business when the economic tide finally turns.

Recruiting is a brokerage business. In good times, recruiters build well-tended relationships with buyers and often terse and purely functional relationships with recruits. The recruiter brokers the relationship between a business and the talent, collecting fees for acting as an effective middleman. At least that’s how it works when there’s flow through the system.

Without demand from buyers, the need for middlemen essentially collapses, but that doesn’t mean that the human resources business simply disappears. It means that the relationship broker’s role has to change to account for the deficient flow. When the market is down, smart recruiters can play the down market by investing!

Sooner of later, businesses will come looking for talent again. A recruiter that emerges from the downturn with a strong portfolio of relationships with good talent to broker with buyers will emerge strong, and can remain strong for the duration.

Here are a handful of things that recruiters can do now to help weather the storm and to prepare to emerge with strength on the other side:

Build Relationships with Employees
Recruiting is a relationship business. If the relationships with the demand side of the business have gotten quieter, then turn your attention to the supply side! Stay in the game: build more relationships with employees. There are more of them to connect with and they don’t know you as well as your buyers do, so trying to pull this off from your desk is likely not going to work. Get out from behind your email app and Facebook and show up in person at community events. Relationships with people should be personal.

Be a Career Counselor
You’re an HR professional with valuable experience. Get an understanding of employees concerns and give them the benefit of your experience as a professional to help them understand the current climate. You have inside knowledge of the kinds of things that employers value and of the things that make employees valuable. Understand their challenges and concerns and help them to stay focused on the personal development that will help to lend more stability to their professional lives. Listen to what they have to say. Giving the benefit of your experience to the employees is not only beneficial to the employees themselves, but it’s also valuable to employers whose employees you serve. This economy can only benefit from well-informed, level-headed employees that have a continuous source of good counsel.

Invest in their Skills
Nothing builds relationships like giving people something of value. Help people to step up to new skills and to improve the skills they have. Host creative learning and education events to help the supply side of your business to go deeper in their work. Connect them with teachers. Partner with teachers to make investments in the market and leverage the access to the supply-side to deepen your understanding of what makes them tick and what they are concerned with. Build them up!

For Goodness Sake, Don’t Recruit Them!
You’re not doing all this to prepare employees to be poached from their employers once the downturn is over. You’re building relationships with employees because relationships with employers are increasingly scarce. Great relationships with employees that have come about because you’ve made an investment in those employees will turn into great relationships with their employers. And if an employee chooses to move on from an employer, you may have an opportunity to help them find their next position, but under no circumstances should you try to actively harvest them.

When the employment market heats up again, the human resources business people with the best reputation building and sustaining a human resource portfolio – especially in a constrained market – will be in demand.

When this economic slump is over, I’d like to work with recruiters who have not only survived the downturn, but who have thrived. When the slump is over, the specialists are going to be crawling out of the woodwork again after having lain low while the going got tough. While the recruiters who have hibernated through the downturn may be just as good as any others, I’m more confident that those who have learned to thrive under constraints are definitely the ones I want to know more about.

In fact, I not only want to connect with them when all this is over, I want to connect with them right now!

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Productivity, not Bailouts

2009-03-19T19:57:00.002-05:00

More than fifty years ago Toyota learned that it was possible to survive tough economic times by facing them head-on; by dealing with drastic reductions in resources through drastic increases in productivity.

If you have fewer resources to do the work at-hand, and the amount of work can't be reduced, then productivity must increase. Reducing waste is a huge part of increasing productivity, but waste reduction on the scale necessary to dramatically increase productivity doesn't happen without organizational and cultural changes.

People must become inter-disciplinary and cross-functional during periods of lower prosperity; times like right now. We can survive and even thrive in tough times by increasing productivity, and we can increase productivity by reducing the amount of time lost to hand offs between individual specialists in a workflow, and by capitalizing on the increased knowledge that comes from people having a greater understanding of the work going on around them.

Greater inter-disciplinary work leads to greater productivity, and greater inter-disciplinary work means challenging our assertions about our entitlements to our particular comforts with remaining generalists or specialists. Rather than continuing to toil in a paradigm that pits generalism and specialism against each other in a constant, irreconcilable tug of war that itself causes more lost time, we need to extend the reaches of individuals into the disciplines that surround their present areas of operation.

We tend to get a bit lazy in times of greater prosperity. We feel entitled to our comfortable, walled gardens. And from this entitlement we generate ever more waste and slack; waste and slack that isn't much of a concern relative to the prosperity that we enjoy. When the prosperity goes away, we're faced with the inefficiencies inherent in the hand offs between walled gardens that accrete along the workflow.

Becoming inter-disciplinary doesn't merely mean trading specialization for generalization, though. It means purging the entire paradigm that pits specialization and generalization against each other as natural opposites.

Specialization and generalization can only exist in relatively pure forms when we can afford the waste that they cause. The waste comes when we have either specialization or generalization. This either/or model is unsustainable. Sustainable productivity is created by a workforce that is both specialized and generalized within the context of someone's area of operation, with the understanding that a person's area of operation will continue to expand over their career.

Failures to reproduce Toyota's successes in the west were originally blamed on the cultural differences between Japanese and western workers. Western companies ultimately learned that the real reason for the failures was that they had missed the essential and immutable organizational imperative of Toyota's methodology: the creation of a learning organization through the fostering of a learning culture. Western organizations consistently failed to replicate Toyota's successes because they often merely adopted Toyota's processes as process improvement efforts.

The world sits on the cusp of the greatest opportunity we've had to rid ourselves of the antiquated mass production methodologies that we continue to wring ever decreasing returns from. At no time in this workforce generation's life have we needed a revolutionary boost in productivity.

Before we had this massive financial crisis, we had a massive productivity crisis. It has been with for years; looming, ever-present, waiting to be triggered. And here we sit in the midst of it. Nothing is so immoral at this time than the continued protectionism of our petty specializations that do nothing but exasperate the productivity crisis that we find ourselves in. And nothing reinforces the entitlement to these specializations like an organization that has little understanding of how profoundly an organization's productivity can be transformed by meaningfully committing to becoming a learning culture, and bending all procedural and organizational imperatives to that goal.

And once we've made the meaningful changes to our organizations and culture to bring about dramatic and sustainable changes in productivity, those new habits can be sustained through the prosperity that follows. We have an opportunity to learn not only how to thrive during crisis, but also how to bolster our society against the cultural rot that regularly brings us to the brink of productivity ruin.

Whether or not the bailouts will serve us in the long run, productivity certainly will. And the productivity that we need is likely not the productivity that comes in a box with a major software company's logo on it. We need a culture of productivity holding together a society of productive people, empowered by vigilant awareness and knowledge.

It's my great great hope that we look to the revolutionaries of human productivity of our time and seize this opportunity as if our society depended upon it.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Workability

2009-03-12T02:40:00.001-05:00

Software development pop vernacular probably doesn't need another "ility" word. We've got a good collection of the them already: scalability, extensibility, availability, recoverability, among others. Nonetheless...

Mature production industries already recognize a design quality called "workability". Workability is at the heart of a number of software design qualities that individually often deliver a fractured message about what makes good software practices good. Approaching software design and implementation from the perspective of workability helps me stay focused on the higher order objective while simultaneously not loosing track of the principles and practices that permit workability in software development. That higher order objective is producing. Or said differently, it’s production. Or better yet: productivity.

In The Toyota Product Development System, the authors talk about Toyota's concern with workability during digital assembly, a form of virtual reality simulation geared partially to proving that a product's build workflow is practicable:

Use workability DA [Digital Assembly] to study in detail the effects that design changes will have on ergonomic issues involved in assembling a vehicle. By utilizing DA in conjunction with the assembly plant pilot team (hourly workers assigned for two years to work on preparing a new vehicle launch) during the Kentou [prototyping, modeling] period, they can address both current as well as anticipated human factor issues and identify the ergonomically safest and most efficient way to assemble the vehicle.

Workability is the design quality that simply permits a job to be finished some day. Workable software allows us to keep working on a software product without being obstructed by the software that we’ve already added to the product. Without considering workability in design, workers end up painting themselves into corners that they can’t get out of.

We see this all the all the time in software projects whether we recognize it or not. When productivity degrades sharply after the initial parts of a software product are built and assembled, it’s the absence of workability that is at the root of this all-too-common situation.

Workability is a higher order concept that accounts for a myriad of principles, properties, tactics, and counter measures that are constantly in-play when high-functioning teams are at work. In software, workability is the testability, the readability, the maintainability, the SOLID principles, and a host of concerns that are continually being balanced, evaluated, and enacted during design and implementation in the name of defending a software effort from stasis. In fact, those principles are so essential to productivity that they are far from unique to software. They permeate all kinds of production across many industries.

There are a lot of elaborate and expensive tools and ideas in the software industry, but the vast majority of them simply don’t serve workability, and many of them actually obstruct higher orders of productivity. If productivity can’t be sustained over an entire software effort, then that initial productivity really isn’t much in terms of productivity, is it? In fact, productivity that can’t be sustained is arguably not productivity at all, and tools that deliver an initial, rapid productivity spike at the beginning of an effort are likely also contributing to it’s decline.

Sustainable productivity isn’t the result of tools, it’s the result of the collection of design qualities and practices that in aggregate can be thought of as workability. Tools serve workability, but they can’t cause it – people cause it. As long as people are involved in making things, workability is simply how good things get made.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

A Week with a Lean Manufacturer

2009-02-25T06:41:00.002-06:00

I recently spent a week working with the team from MTM Recognition, in Oklahoma City. Raymond Lewallen leads the team, and he hired me to work with MTM's developers on context specification, both for .NET code using mSpec and for the web UI using Selenium and RSpec.

MTM is a Lean manufacturer. The company makes awards, trophies, rings, and a host of recognition materials for businesses, schools, and sports teams and events. Most of the work that MTM does is small batches of products that are frequently custom designed and built for a single batch. Its circumstances are almost a cliché of quintessential Lean.

While I was there, Ray and Troy, one of MTM's developers, took me on a on a tour of the plant. Troy had previously worked on the manufacturing side and knows his way around. He transferred to IT when he wouldn't stop building unofficial shadow IT apps to serve his purposes as a manufacturing worker. Talk about domain expertise!

I had a suspicion before we entered the plant that I had sincerely hoped would play out. The plant didn't disappoint!

Walking into MTM's plant, I couldn't find the trappings of Lean. There were some obvious visual indicators in some of the workspaces, but there were no glaring neon signs announcing Lean. What we saw mostly were signs of people making the products that MTM's customers had ordered. We had to ask to have queues and signal cards pointed out, and the folks at MTM pointed out how Lean was manifested and used.

Lean was just how things were done, but it wasn't the whole point. The point was carrying on the company's tradition of building quality products that took MTM from a one-man-in-a-garage operation to a respected and trusted 100 million dollar company.

When software developers latch on to a new thing, we tend to want it's trappings to be so flagrant and obvious that you can't but trip over them in our work areas or in conversation. Considering It's understandable, but it’s also a bit disconcerting.

After living through a catastrophic failure with Agile that ended with the costly write off of a year's worth of time spent and the dismissal of the entire product team, my concerns over Agile's loss of focus on product, producing, and productivity were powerfully punctuated.

Agile teams can become fascinated on the process to the distraction of the purpose that the process serves. In the case of the aforementioned failed project, it had become so bad that at one point the product owner and I mused that it seems like the team believed that it had become charged with producing Agile process rather than software product. When Agile becomes the whole point rather than merely a means, things can go very wrong.

What really struck me during the tour of the MTM plant was the master crafts people on that MTM has on its staff, working in design, production, and quality assurance. MTM has some very competent and talented people plying their trade in service to MTM's customers. Everything from sculptors, to jewelers, painters, metal workers, and typesetters. It's clear that producing at the pace and quality that MTM has built a reputation for requires people of considerable skill.

There's power in the tools that we use, and as crafts people we can even derive pleasure from our appreciation of fine tools themselves, but we are so susceptible to having our attention lured away from the most important aspect of having them and wielding them: productivity, product, and producing.

I appreciate Lean because it is so focused on product and producing, and on shaping an organizing that is hell-bent on the goal. It makes no apologies for remaining focused on the goal and for bending everything to the service of the goal and sustaining its achievement and its betterment.

Agile can be a disruptive technology. It’s vital as software developers that we remain vigilant against our susceptibility to allowing agile to be disruptive to our software development efforts themselves.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Decade of Agile, Dawn of Lean

2009-02-23T16:58:00.002-06:00

The way that Agile works on a team that is new to Agile and the way that it works on a mature team is necessarily different. The sudden breaking with old habits that happens during new adoption is a radical change. The on-going process of improvement on long-lived, mature Agile teams is usually (but not always) less radical.

Mature Agile teams might still undergo radical improvements now and then, but they will have been steeped in continual change and have developed a taste and facility for change, or at least for the methodology of change. New Agile teams are typically not only not steeped in the protocols and social contracts that they are changing to, but they’re also not usually steeped in change itself.

Becoming used to change is a big part of making a deeper connection to Agile. Habituation to change unfolds over time, and over time it cultivates a vision, a pragmatism, and a facility to change in the people on the team. The deepening of the facility with change will change the protocols and social contracts that are necessary to holding things together while change is under way.

Agile usually starts up in the midst of the organizational and cultural chaos that is the typical status quo of software projects. Presently, software teams have a heritage of insufficient technical abilities compounded by ineffective project management and altogether too-weak product definition.

The circumstances of a project beget the particulars of its methodology. The vast majority of Agile teams are in their initial phase, and most of what we hear about Agile methodologies reflects this particular perspective of Agile development. The fact that most Agile teams are just beginning has a tremendous effect on the content of the Agile dialog, and the frequent repetition of this perspective inevitably reinforces itself.

Scrum is a methodology that is really well-suited to getting started with Agile, and consequently a lot of what is believed about Agile at-large is rooted in Scrum. The downside of the self-reinforcement feedback loop surrounding Scrum is that it tends to drown out the rest of Agile - the technical competencies that permit Scrum to be sustainable beyond the initial Agile boot strap phase.

At the outset, Scrum gets away without the necessary technical competencies because the team and surrounding organization are coping with radical changes in the protocols and social contract. Allowances are made for lower productivity in recognition of the team facing radical changes. But once the bloom is off the rose and real expectations come back into effect, colloquial Scrum efforts can fall apart.

Scrum's tighter feedback and greater adaptability demand engineering techniques that permit these things to happen. The engineering practices are what keeps Scrum from shaking itself apart over the long term and from preventing a new Agile team from regressing Scrum into some variant of waterfall.

Scrum's biggest challenge is the calcification of its own orthodoxy. We end up merely with "Scrum teams" rather than high-functioning software product development teams. We end up with teams who continue to apply practices that are appropriate to the early Agile adoption phases long past the adoption phase. We end up with entirely naive and mechanical software development teams flying the Scum banner, and this is exactly the state of affairs that we were trying to move away from.

Filling a cork board with hundreds of index cards that describe features that we're not going to work on for months is really just an updated version of the hundreds of pages of specification documents that come at the outset of traditional software projects. The index card version is a contemporary manifestation of the same presumptions and thinking that drive waterfall.

We end up with the over-specified product backlog by failing to recognize our unrecognized presumptions. There are lots of unasked questions that remain dangerously-silent in mainstream Agile. It's these presumptions that are hurting software product teams' chances of growing beyond the basics and surviving past a guided adoption.

Without growing beyond the basics, Agile efforts end up collapsing under their own weight when the project heat is turned up for real. Many Agile teams have only been taught how to begin and don't really know how to sustain let alone how to improve while simultaneous getting things done. A team that isn't mining for it's invisible, un-asked questions and presumptions is going to have a hard time establishing the improvement practices that will permit it to sustain beyond bootstrapping.

If advanced agile practice is anything, it's a diamond-tipped drill boring into our unconscious presumptions of what makes software product development work. The deeper this drill goes, the longer we can sustain. As soon as the drill stops, the momentum of our projects' challenges catches up and outpaces our ability to deal with them as easily and as effectively as when we were at the very beginning, when things we simpler.

There's no standing still. If you're not moving forward, you're falling behind. I don't typically find admonitions this strong in Scrum orthodoxy, but I do find it readily in Lean. This might suggest that an inevitable result of years of continuous improvement of Agile practices ultimately leads Agile teams to Lean, and to a forming and shaping of Agile fluency to Lean. I've seen this play out a number of times in my own professional network as agilists with many years of day-to-day agile project experience have begun to see answers to their questions in Lean. Maybe this isn't true for everyone, but it's happening with enough regularity that in the very least it deserves consideration.

We're faced with an uncomfortable question: Is Lean an "advanced form" of Agile development.

If Lean is seen as an advanced form, it inevitably confers a beginner stigma onto teams who aren't doing Lean and can be seen to diminish other forms. The problem here isn't that Lean may or not be an advanced form, but that we have a community of software practitioners who see being a beginner as stigmatic.

Nothing but nothing puts software product development efforts in harm's way like a team that is uncomfortable with standing right where it is. If a team is unwilling to be a beginner, then it is incapable of becoming advanced. Wether or not Lean is an advanced form, we seem to have an aversion to accepting that we do indeed have advanced forms, and that they are often incomprehensible and impenetrable from where we stand right now.

Dave West made a great point about the purpose of a well-stocked backlog in his article on InfoQ entitled Lean and Agile: Marriage Made in Heaven or Oxymoron.

Agile works best when there is a huge product backlog and when there is a large amount of churn in the composition of that backlog. Churn results from conversations about story essence; story priority; feedback from developers about stories not understood; stories that turned out to be easier or harder to develop than expected; stories that had to be refactored to make them more understandable or more tractable to development; and feedback from user acceptance of stories completed.

Eventually, churn diminishes and the product backlog becomes stable. The addition of new stories reduces to a trickle and the prioritization of stories changes only nominally. At this point it becomes even more tempting to consider the product backlog as an inventory.

He makes a really astute observation about Agile that I really like and can identify from some of my project experiences. Dave says that agile is a process "for building a consensual theory of the world: with an artifact being a byproduct - an expression - of that theory". And this, I think, is where Agile and Lean can be seen to part ways, and also to not part ways. This is one of those "radical contradictions" that can be seen on the surface to point to a inconsistencies and incompleteness in Lean thinking that is more likely to be one of its enablers.

Here's a bold statement that is likely to raise the hackles of many a latter-day agilst: There is no inherent need for building a consensual theory of the world in product development. There's no doubt that everyone involved in realizing the product vision has to understand what that vision is, and that churn is a way to get there, but there are approaches to managing that process that aren't predicated upon a "huge product backlog".

There are certain circumstances where churning over a huge product backlog is valuable. And there are yet other circumstances where where such churn is unnecessary, and yet others where it can be downright destructive and a significant contributor to failure. The chosen approach can't be cookie-cutter and mechanical. A project’s context has to drive the specifics of its Agile approach. There is no one, single Agile that can work for all levels of maturity.

Some product development efforts start with a strong product vision and some product development efforts start without a strong product vision. And which ever path the project is on dictates the kind of protocols and social contracts that can go into shaping the process, as well as the process of improving that process.

A huge backlog is often one of the necessary trappings of a team that isn't steeped in the practicable know-how of just-in-time development. The huge backlog theory-proving that Dave talks about may be more than just valuable in the absence of just-in-time fluency, it's likely a necessity and working without it might very well be negligent. Working with a large product backlog helps a team to gel and helps a product vision come together. These are truly valuable things to do, but they come with some cost. If it's a necessary cost, then it should be paid.

Filling a product backlog with stories can be a good exercise in socializing the product vision, but there are other ways to socialize the product vision without risking the waste that a pre-mature backlog often generates. The value of just-in-time is great, but the means of doing it isn't typically readily apparent via Scrum.

Lean offers a lot of experience in doing just-in-time development, but the body of knowledge is quite large. It would be a mistake to believe that one or two books by the Poppendiecks will give you the lay of the Lean land the way that one or two books on Scrum can spell out the entire methodology. That's not a sleight against Scrum. One of Scrum's greatest strengths is its relative lightweight. If Scrum were as voluminous as Lean, the Agile movement may have never made it to the mainstream. Scrum’s relative silence on the technical excellence that allows Scrum to succeed is why more organizations begin with Scrum rather than Extreme Programming, regardless of Extreme Programming’s early prevalence over Scrum.

I value my experience in Scrum because it prepared me for eventual undertakings in Lean. Without the well of enthusiasm for software development that I get from Agile development, I might not have undertaken the sheer volume of study and community engagement that helped me to finally see what Lean is about beyond the endeavors of the Poppendiecks to communicate Lean in digest form to the software world. Returning to the Poppendiecks’ books now, I see that I had missed the significance of much of what I was reading.

I've discovered and cultivated a lot of abilities that I didn't have when I started doing agile on a day-to-day basis in 2001. I'm ten times the software product person that I was then, and Agile is responsible for a much-accelerated growth as a developer, a tech lead, and ultimately a product designer and chief.

I can do things now that I couldn't do then. I fit my methodology to where I am in my understanding and and my ability to execute, lead, and socialize product development. Many of the things I choose to do now are downright heresy to mainstream agile, but I'm not exactly where mainstream agile is right now.

Here's are a few examples of things that I'm intimately-comfortable with now that I had only started entertaining just two years ago:

I don't rely on the embedded (on-site) customer.
I don't do all-hands team estimation a la XP Planning Game.
I don't carry a huge product backlog.

I'm simply not abandoning these practices and leaving gapping holes in the methodology. I'm replacing these practices with practices that are commensurate with my current level of understanding and experience, which are radically different than they were even a couple of years ago.

Toyota recognizes a role called the Chief Engineer. I'm not pompous enough to say that I have the depth of experience to confer such a title to myself, but it's the closest approximation that I've found that describes what I do on software projects. I'm conformable playing a relatively equivalent role on software projects, and I've been playing that role in practice in an ever-increasing capacity for a few years.

The distinct organizational qualities and processes that distinguish Lean from mainstream agile permit me to make a departure from the prescriptions of mainstream Agile and to do my Agile work with what I see as greater effectiveness across the board.

Ultimately, I'm taking on more responsibility for definition and specification. I'm doing this because I'm used to working at a pace that requires less of the variation and oscillation that can shake a mainstream Agile project to pieces at high speeds. I'm synthesizing more inputs and factors faster than what can be done by a committee and I'm leaning on considerable technical competence to convert product vision and product design into work items that I can guide a team through. I’m more open to input from people on the team, and I’m willing to see where things go when people feel strongly about outright dissent. And I’m comfortable with putting my foot down when I see that a point I’m making is not understood due to lack of experience and perspective when I can guide the practices that will eventually feed someone’s understanding.

Dave West commented on optimization, "Lean views software development as a process for moving from conception to product. It wants to optimize that process, albeit in a radically different way and with radically different values than traditional (e.g. Taylorism) attempts at optimization."

Lean does seek optimization, but it doesn't view software development as merely a process. An agile perspective of Lean may be biased to see Lean as a process, but Lean's perspective of Lean, internally consistent with itself, doesn't see it this way. Lean is fundamentally a way to cultivate and sustain a learning organization.

Optimization serves a goal that is outside of the typical purview of Scrum. Optimization drives design quality. Design quality drives productivity, sustainability, adaptability, and ultimately customer satisfaction. The reasons for optimization are found in the technical and engineering excellence underpinnings of Agile development. XP has more to say about this than Scrum because delivering technical excellence is not Scrum's purpose. However, XP practices are often how implementation gets done in a Lean software project.

Dave rightly points out that software development is not like production and that we should be cautious about applying a production methodology to software development, "Lean needs to take off the 'production glasses' and look at Agile and elements of the agile development process from a holistic perspective that includes all seven of the Lean principles".

The Poppendiecks often raise this same point and caution that software development is more like product development than production. Specifically, Lean software development is more like Lean product development, and we can look to resources like Toyota's Lean Product Development System rather than the Toyota Production System for insights and a vision of how Lean product development works.

Regardless of whether Lean is applied to product development or production, Lean principles apply equally to both, and these principles are often seen as conflicting with Agile. Indeed, in some circumstances, Lean is in-conflict with Agile and seems to contradict Agile, but which agile, in which circumstances and on which Agile team at which part of the process of maturation and improvement?

I think it's a mistake to look at the practices of Lean software development and conclude that things are being done in a particular way because of a misunderstanding of and a lack of experience with Agile. Many new Lean practitioners have been in deep Agile immersion since the beginning of the 2000’s. I think that it's specifically due to the broad and deep experience with Agile and in constantly pushing the limits that many people are now finding answers to questions in Lean.

That's not to say that Agile itself has any arbitrary limits, but the majority of the Agile population at this point in time is focused on adoption rather than maturation, and insights on problems found at Agile's quantum barrier can be found in Lean. And exposure to the fifty years of methodological and practical experience in applying and improving Lean arse inevitably having an impact on the way that we've been practicing XP and Scrum.

It's not that the recent spate of Lean exploration and adoption is due to inexperience with Agile. It's happening in spite of tremendous experience with Agile. In fact, some rather deep experiences with Agile seems to be leading people to Lean with a rather predictable regularity - even considering the influence of mere hype and relative novelty. But these people don't appear to be abandoning Agile, but rather adapting Agile to fit within Lean as a means to practice Lean as a software development discipline.

I didn't become interested in Lean because I wasn't interested in Agile. I became interested in Lean because I felt that I had started to hit a glass ceiling in Agile. I was trying to get to the next level and the ready part of the community and it's lore, literature, and protocols were just not helping me get there. I'm not saying that the next level for me wasn't already present in pockets in the Agile community, but it was readily-available and present when I went looking in Lean.

When I really started to put my mind and my effort into Lean, I experienced something very similar to what I had first experienced when my mentor from Bell Labs first introduced me to Extreme Programming ideas, and when I read Kent Beck's Extreme Programming Explained. I experienced again what I think of as the "ugly duckling" syndrome, where all of these doubts about the orthodoxy of what I was doing previously were laid out in black and white as well-known anti-patterns, calling me forward into a deeper realization of my own internal notion of common sense.

At the same time, I was irrevocably thrust into an unpopular movement of software development that took several more years before becoming acceptable and loosing it's pariah stigma. At least the ugly duckling went from being an ugly duckling to a beautiful swan. XP practitioners carved out a place for agile in corporate IT shops, product companies, and conferences amidst some non-trivial resistance and even ridicule at times.

Thanks to the brilliant business savvy of the ScrumMaster Certification program, Agile made deep headway into the mainstream, and XP'ers backed into a much-eased Agile climate by the broader path that Scrum had carved. And because it was just common sense, we laid Scrum down on the solid technical foundations of XP and got back to work making software products and improving ourselves.

Here we are ten years later and those same people are doing exactly what they've always done. They've gone where their common sense points them, and for some people, that means adopting Lean. It's rather ironic that this conclusion is as unpopular with the Agile mainstream now that Agile was to the pre-Agile mainstream then.

As not just a Lean student and practitioner but also as an agile veteran, I find the kind of mechanical orthodoxy that has taken root in the Agile mainstream to be categorically wasteful. But I look forward to the churn and contradictions that will inevitably lead to what comes next.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Complexity isn't Subjective, Our Reactions to it Are

2009-02-16T02:49:00.002-06:00

Software people have significantly-different understandings of what complexity is. Regardless of what software people believe complexity to be, the perception of complexity affects most software developers in the same way: it influences the choices we make about how to build software. What a software developer, development manager, or even a stakeholder believes about complexity might determine whether the project is put in harm’s way.

Complexity isn’t subjective. It’s a hard measure of the number of inputs into a system, the number of moving parts in the system, and the number of decisions that the system makes that lead to different execution paths. But we often use “complexity” to describe what it like getting our heads around something unfamiliar, which can be a purely emotional response.

Two people saying “I don’t want to do it that way, it’s too complex” can have utterly different perspectives of complexity. If so, then they likely have utterly different and likely incompatible approaches to software development. It’s more likely that people who typically use incompatible approaches will not find themselves agreeing on complexity.

Between the start and finish of a project, a lot of things happen. Software is created, and more software is created. Some of that software interacts with other bits of itself. And more software is created and more interconnections are made. The natural increase in code and interconnections during a project while the software gains more features creates real complexity.

With more code, more natural complexity is added to the system. Complexity is going to accrete, the issue is: how fast? And the single most influential factor in how fast complexity increases is the approach we choose based on what we think complexity is. When we don’t make the right decision, we end up with more complexity than we need. Understanding how complexity is created and how it affects software and projects is a essential part of software physics and helps us to understand how to keep projects healthy.

Complexity isn’t an expression of discomfort with approaches that we have no meaningful experience with. Certainly lack of experience can put a project at risk, and some of that risk might manifest from accidental complexity that results from inexperience, but inexperience itself isn’t complexity.

Ultimately, the point of dealing with complexity isn’t the elimination of complexity, it’s the elimination of accidental and unnecessary complexity. Some forms of complexity are desirable and we choose to add some essential complexity in order to improve other desirable design qualities, and even real complexity isn’t the differentiator between software efforts that succeed and software efforts that fail.

When a developer complains of complexity, it's a sign of something that likely needs to be given some attention. It not entirely uncommon, though, to find that what has been called out as complexity isn’t a reflection of the software or the approach itself, but of the programmer’s emotional content. It’s still a matter of concern, but it might not be a technical concern.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Test-Driven Development and Behavior-Driven Development in .NET Workshop in Austin, Tuesday, March 24th

2009-02-10T02:43:00.005-06:00

On Tuesday, March 24th, I'll be teaching a Test-Driven Development and Behavior-Driven Development workshop in Austin, TX. This is a single-day, high-impact workshop geared to creating a deep understanding of TDD and agile programing with minimal time away from your work.

This workshop is about developer productivity. It teaches you how to use test-driven development and lean software development to be more productive in your work while continuously-improving quality and understanding. You'll learn advanced object-oriented techniques in simple and practicable ways that you will remember and apply at work. This workshop isn't just going to teach you how to write unit tests or just how to do test-first programing and test automation. It will show you how to use these techniques properly along side leading software life cycle methods that transform project groups into lean software production teams.

Through test-driven and behavior-driven development, you'll learn which design patterns to use and how to use them. You'll write code that is easier to understand and easier to maintain. You'll write acceptance tests that let you generate documentation that helps your customers become deeply involved with the project, and helps programmers understand unfamiliar areas of a system with much less time wasted deciphering code and tests.

Exercises cover unit-testing, mock objects, test-driven development, behavior-driven development, object-oriented design principles and patterns, and domain-specific languages for testing. Participants will work through testing and design problems individually, with the instructor, and with the group. Students should bring their own laptops installed with Visual Studio 2008 as well as the TortoiseSVN client for Subversion. Tools and materials will be provided on-site and assistance will be provided to students who need help to setup their laptops for the workshop.

Early bird registration is only $240, so sign up now and save more than $50.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Design Flaws, Hernias, and Anemic Quality

2009-02-09T01:49:00.001-06:00

Runtime defects are relatively easy to find. They make undesirable things happen that you can see with your own eyes. Some defects are certainly harder to observe than others. They might show up only in certain uncommon circumstances, or you might need to do some search and discovery with some special tools and instrumentation.

Relative to finding design flaws, the effort required to observe runtime defects is often a walk in the park. And while we put a lot of effort into finding and resolving runtime defects, design flaws can drain more money and value from businesses than the runtime defects that consume the better part of quality assurance efforts and attention. Not to downplay the risks that runtime defects pose to business, but a business that isn't also mitigating the considerable risks posed by design flaws is ensuring that the erosion of the value of software assets will progress exponentially, signaled by sharp, unexpected drops in productivity.

Design flaws are unseen because they manifest specifically as productivity problems and we tend to not track productivity in software development work in meaningful ways, and we rarely reconcile the suboptimal productivity of businesses in general with their business software.

Runtime defects corrupt data, crash applications, cause outages, enable security breaches, process business transactions incorrectly, and lead business people to the wrong conclusions and decisions. Defects are “in your face”. So are design flaws, but if you don't know to be vigilant toward them, you likely won't pay them the decisive attention that they deserve.

Imagine a design flaw as a herniated disk in your spine. The hernia pushes out on the surrounding muscles, bones, and nerves in a way that is not natural to them. This kind of misalignment not only causes problems with it's own structure, but also manifests in structural problems in adjacent systems as well.

For example, a back problem can cause changes in posture and movement to compensate for weakness and pain, changing the way that adjacent and supporting structures and systems align and operate. A spinal problem might manifest as a knee problem. The secondary problem might not only be indicative of the original problem, it might also deteriorate to the point of being as bad or worse than the original problem.

The insidious effect of secondary effects is that solving the original problem may not resolve the secondary problem. The secondary problem might be an indirect result of the original problem, and may continue to deteriorate even once the original problem is resolved. This happens when new habits are formed when learning to compensate for the original flaw.

Imagine a graphical representation of the goodness of a good software design as a series of adjacent alignments. They don't compromise each other's structure or operation.

A design flaw in this system would appear as a hernia - a bulging misalignment in the midst of the surrounding order.

Design flaws often go unnoticed. As work progresses and a system grows, new features and code that are physically or functionally adjacent to a flaw are inevitably shaped to it.

Often this isn’t done purposefully. The shape of the code that we build upon determines the shape of the code that we’re building. Adjacent code will either export order or disorder depending on whether and how much it is flawed. If it exports disorder, then it spreads productivity problems as well.

The most obvious flaws are known to us by their "workarounds". But flaws with workarounds are just those few very obvious flaws. Most flaws don't manifest to our awareness right away, if ever. We might have a spark of insight one day that illuminates the flaw, or we might have an opportunity to work with someone who wields some non-trivial design flaw mitigation skills who recognizes and points out the disorder.

When we finally become aware of a design flaw and if we have the resources to fix it, we have to be careful of what happens to design when we allow ourselves to become overly-fixated on fixing just that part of the problem that we’re looking at.

The goal is to correct the design and to restore productivity, but often we leave behind the secondary effects, either because we haven't become aware of them yet or because of an excessively-narrow focus on the original problem.

This is how we end up with those areas of our systems that aren't right and are difficult to work with but when you ask someone why the design is the way it is they respond with something like, "Dunno. It's always been that way. We don't touch that code. There must be a reason it's like that or else it wouldn't be like that. So we don't change it. It's just one of those things. It's the way software is. You know?"

I know that software often ends up like this, and I also know that it doesn't have to end up like this, and I know how to avoid turning software into these messes, and I know that the ways and means can be taught and learned. Perhaps most importantly, I know why we shouldn't let software degrade into these unknowable messes.

You Can't Afford It

Design flaws are expensive. They cause the same value loss to ripple through our systems that runtime defects do. However, with design flaws it can take you a lot longer to realize that the flaws are there and to locate the wound in the system that value and productivity continue to leak out through.

The longer that design flaws remain in your system, the more secondary flaws are introduced. Those secondary flaws are also the primary causes of their own secondary flaws. This inability to control the density of flaws in a system only gets worse, accelerating exponentially. It’s the principle causes of the need to completely re-write systems that may only be two or three years old.

Having tests for our code only makes it possible to continue to make changes to our systems, but it doesn't say that we can make these changes quickly enough or effectively enough to make these necessary changes feasible and practicable. Code that is awkward to work with has tests that are awkward to work with, doubling the inefficiency of solving the problem.

The simple issue is that you can't afford to have these kinds of problems in your systems. You can't afford design flaws just as you can't afford runtime defects, and frankly, the more design flaws you have, the more likely that you'll have runtime defects, and the more difficult they will be to find and to fix.

We have a rather naive understanding of quality in the software industry and we’re only now starting to understand the physics and economics of software development enough to understand the importance of quality and how to make it happen.

While we continue to have anemic quality that preoccupied with chasing after only a portion of issues in quality, we’ll continue to suffer. Our productivity will be far less than it should be and value will erode much faster than it should. But there are some signs that pockets of the software industry are waking up from the software dark ages and putting quality into play in ways that are much more meaningful than the mere bug-hunting that has dominated software’s concept of “quality” for the longest time.

As we learn to apply the lessons from our own industry as well as from other industries, not only are our abilities to achieve quality improving, but our understanding of the purpose of quality deepens. We are showing some signs of the industrial maturity that suggests that we may yet reach up and reclaim the productivity that we continue to squander to our primordial struggles with software production.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Sunni Brown’s Graphical Recording of My Agile Leadership Talk

2009-02-03T03:18:00.002-06:00

Sunni did a graphical recording of my talk at ProductCamp Austin. The talk was entitled Toyota's Chief Engineer as a Model for Agile Product Development Leadership.

(click the image to view full-size)

Thanks, Sunni! And thanks, ProductCamp Austin!

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Teaching, Symbology, and Intellectual Materialism – The Chasm is a Vacuum

2009-02-01T19:52:00.003-06:00

If you could use a tool and do good works with it, but if you didn't know the name of the tool, would you be less of a craftsperson? How much less? Would your works be less valuable to the people who are served by them?

The world is filled with people who don't understand advanced subjects, but who would like to learn. And the world is blessed with many knowledgeable people who can't understand why the things that they've discovered, learned, and elaborated are not being soaked up by everyone else.

Sadly, the people who are among the first to gain knowledge in a subject often tend to make that knowledge unpalatable to the rest of humanity.

On either side of the issue are people who have fundamentally different reactions to a subject's symbology. Intellectual materialists celebrate and adore a subject's symbology, and the mainstream simply doesn't.

To a mainstream learner, a subject's symbology and pattern language can be a hindrance to learning. This fine point isn't often perceived by those bright and creative people who are quick to recognize, adopt, and deeply-learn an emerging subject, and quick to revel in the accomplishment through the recitation of its symbology within innovator social circles.

When intellectual materialists turn outward from their social circles to readily share their knowledge with the world, the involvement with symbology is often so powerful that they simply can't resist its indulgence. In these moments, the mainstream learner often dismisses many brilliantly-simple and game-changing ideas because they're wrapped up in intimidating symbology, like "Liskov Substitution Principle", "Cyclomatic Complexity", or "Inversion of Control".

These are terms like a lover's whispers to an intellectual materialist. They verily celebrate and decorate the knowledge behind the words, and this celebration of knowledge itself is a deeply-ingrained and vital aspect of intellectual materialism culture. The collection of knowledge and the celebration of knowledge is often as essential as its rightful and meaningful application.

And because the trailblazers aren't often aware of this, they are often only effective as teachers to other early adopters, leaving a chasm between themselves and the mainstream that is a stumble in what should be the continuum of communication along the learning curve.

The Chasm is a Vacuum

The chasm in the technology adoption curve is a bit of a stutter-step in what is otherwise a smoothly-flowing transfer of knowledge from people on the left who are earlier adopters than people on the right.

The word "chasm" describes hiccup in the shape of the population distribution curve, but it doesn't describe the social dynamics. Socially, the "chasm" is a "vacuum".

Intellectual materialists from the left-hand side of the adoption curve spend some amount of time amongst themselves incubating emerging ideas before they venture - either accidentally or purposefully - into the mainstream with the ideas. During the incubation, their insular communication patterns are even further reinforced and institutionalized. The celebration of symbology becomes even deeper ingrained through repetition and elaboration.

When intellectual materialists first meet mainstream learners and reach out through these well-ingrained patterns, the ideas and even their representatives are often met with a resistance that doesn't seem to be commensurate with the goodness of the good news of new understanding. If innovators aren't met with resistance, then they've likely just bumped into a new intellectual materialist, pulling them over to the left-hand side of the curve rather than having transformed knowledge - preparing it for, and transitioning it to the right.

When knowledge is first offered across the cultural seam and is met with resistance, the respective cultures recoil into themselves, leaving behind the vacuum. The vacuum shows up on the graph of the technology adoption curve as that blank spot between early adopters and the early majority known as, "the chasm".

Anxiety and Fear

At the point in the continuum where the people who live in the left-hand side of the curve meet the people in the middle of the curve, a lovely handshake should take place. These two neighbors should forge life-long bonds of respect and recognition building on the near-infinite creative potential of diversity, continually transforming and communicating knowledge across the cultural seam.

Far too often an early adopter will bump into someone from the mainstream who will extend his hand across the seam in greeting and say, "Hi, my name is Joe. Lovely day isn't it?" To which the intellectual materialist replies, "Cyclomatic inversion of Liskov test-driven domains of concern segregation substitution interface... in principle". Which in early adopter land roughly-translates to, "Hi-diddly-ho, neighborino!"

The mainstream learner smiles nervously, backs away from the seam, and accelerates away. He returns later with a sign that reads, "Warning: Irrelevant, Academic Clap Trap Beyond this Point", but finds it unnecessary as the intellectual materialist has returned to his capitol city to participate in the ceremonial Reciting of the Terms of the Sacred Symbols at the dedication of an all-new, top-of-the-line, ultra-modern, Class-A echo chamber that his entire society has decided to retreat into for the next five to ten years.

Mainstream learners are often more receptive to new ideas when they're not introduced with the psychic weight of the subject's symbology. It's not because the symbology is just too much to absorb. It's because the symbology and nomenclature, starting with the grandiosity invested in the new subject's name, usually over-states the complexity of the subject.

Innovators and early adopters give names to things that are celebrations of knowledge. They capture and replay the exaltation of the moment of crystallization of the idea by every intellectual materialist that comes to assimilate the new idea.

The name is the idea's material. The name is more than descriptive, it's the psychic texture of the idea. And none of this makes a lick of difference to someone in the mainstream who doesn't also value an idea's texture, but who just wants to learn how do do something so that he can in fact get something done.

Mainstream learners first try to figure out whether some subject is worth learning before they commit their time to learning. There's a lot of anxiety invested in this filtering. The sheer volume of new things bombarding a mainstream learner's priority filter is literally mind-boggling. The subjects that make it through the priority filter are the ones that seem less intimidating, and frankly, the grandiose symbology that is the mark of intellectual materialism is often very intimidating.

As soon as a student's mind touches anxiety about a subject, we've lost a significant portion of their ability to remain present to the subject. The teacher's first duty is to protect the student from anxiety and fear because these things are the anti-matter of learning and knowledge.

Respect the Learner's Tolerance for Intimidating Symbology

The answer isn't to start over with a new symbology (although it would be nice if some thought were given to the effects of symbology on the vacuum when the symbology is established). As teachers, and as the people whose cultural segment on the adoption curve creates most of the intimidating symbology, we have to be conscious of when and where the use of the symbology is appropriate.

The time at which a subject is being introduced to a mainstream learner is often the least appropriate time to introduce a subject's symbology. It might have no effect, or it might have a positive effect, but there's a strong chance that it will have an altogether negative effect - repelling the learner from the subject and the learning experience.

There's no reason to entertain the risk when any subject can be introduced without introducing the symbology. Doing this means that the psychic weight of the intimidation of the symbology isn't brought into play as an obstruction to the learner's capacity to be open and receptive.

Protecting the learner's cognitive space from the anxiety derived through intimidating symbology is a teacher's scared responsibility. It's akin to a physician's oath to do no harm.

There's usually nothing inherent in the symbology that helps a teacher communicate the subject that the symbology represents. It's the subject that it's important, and the subject isn't its name. A subject's symbology can be introduced later, once the subject has been taught past the point of a learner's inherent initial anxiety with new subjects. Until then, there's no reason why a teacher can't introduce a subject, teach it past the intimidation hump, and then and only then introduce the symbology.

Filling in the Vacuum

Sooner or later, enough knowledge is communicated though the vacuum that it eventually dissipates, and the continuum resumes with normal, expected fluidity. This happens when enough early adopters - be they originally from the left of the chasm or the right - work long enough to bridge the gap. If there were more teachers who understood that the role of a teacher bridging the gap is as translator and transformer rather than symbolist, then the vacuum may never even grow from the width of a humble seam.

The strange and predictable phenomenon of the chasm isn't a necessity of the adoption curve, it's the result of a cognitive impedance mismatch between two sub-cultures that fail to recognize the significance of each other's differences.

Intellectual materialists are so over-stimulated by symbology that their behavior borders on the mania of addiction and obsessive compulsion. The solution to the manifestation of the chasm is to respect the human dispositions that turn seams into vacuums.

If we avoid leading with symbology, we avoid inducing the fear response in the mainstream learners, and this goes a long way toward avoiding inducing the chasm. This is well within the reach of the people on the left-hand side of the curve. It simply requires the same discipline as teachers as they require of their students.

When teaching across the void, teach to empower! When empowered, a subject's symbology won't diminish a learner's capacity to be receptive to the subject, and open to learning.

When empowered, a mainstream learner is just as likely to join the celebration of the subject through the subject's symbology just as intellectual materialists do! That doesn't mean that a mainstream learner will have crossed over the seam and become an intellectual materialist, it means that he has been given a chance to see the beauty and power of a subject from his own cultural perspective.

This shared experience from different cultural perspectives is the diversity that will lead to even more creativity and refinement. It's through this diversity that knowledge becomes even more powerful, with new possibilities and applications emerging that lead to the next big idea. And it's through the practice of this diversity that the sheer waste of time and resources that result from the cultural vacuums can be avoided.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Good Design is Easily-Learned

2009-01-12T19:24:00.004-06:00

The essence of "good design" is it's ability to be absorbed by a human mind. Design is "good" when it can be easily-learned.

The term "testability" refers to how easily the objects in a design can be learned, albeit in a roundabout way. "Testability" is also a catch-all term for good design, and encompasses design qualities that are reflected in the SOLID acronym. Each of the solid principles talks about a specific goodness of software design. The essential, root cause of those goodnesses is the readiness of the design to be turned into knowledge that a person can immediately turn into effort and action.

We have good, simple techniques for getting software to the point where its design is optimally-learnable, and where it's structural qualities aren't compromised. We can create designs that are easy to learn and understand by approaching design from the roundabout course of "testability". The word itself is misleading unless you already know what "testability" means when used to describe design.

Testability isn't a term that describes whether software can be tested. It refers to software that is easily tested. The more "testable" something is, the easier it is to learn.

Use What You Know

If you can't learn something, you won't understand it. If you can't understand something, you can't use it - at least not well enough to avoid creating a money pit. You can't maintain a system that you don't understand - at least not easily. And you can't make changes to your system if you can't understand how the system as a whole will work once the changes are made.

Testability is all about how little effort I need to make to understand how the system works, and what changes I need to make to the system's modules to make a needed change.

To make an object or component easily testable, you have to be able to create one with little fuss, and poke and prod it to see how it works. But if you don't design your software so that it's easy to set up scenarios that help you learn, then it's going to be harder to maintain your system.

For example, if you need to understand logic that might save a new database record in one branch of an if statement, and update an existing row in another branch of the if statement, and if you can set up that scenario to prove that the logic works without also having the set up the database and create sample data and then query the database to verify whether the if statement routes to the right actions under the right conditions, then you've got a testable design. It's also likely that you used the Dependency Inversion Principle and the Interface Segregation Principle to get there (the "D" and "I" letters of the SOLID acronym).

If it's hard to set up an object in order to learn what it does and whether it does it right, then you've probably got a poorly-designed object. This suggests that you can use test code to prove that you've got the right design. And this is what testability means. If I can easily create a scenario that helps me learn what my objects do, then I have also very likely applied SOLID principles - whether I planned to or not.

Even without tests or test-driven development, code that follows SOLID principles is easier to understand and easier to work with. However, if I don't consistently write sample code that concretely and completely demonstrates the desired level of ease that I'm after, I'll consistently over-estimate the ease of setting up an object, and I won't arrive at the best design.

No matter how good I become in object-oriented design, the only proof that a design is good is code that proves concretely that I have achieved the highest level of ease of setting up an object for use. If that object is easy to set up in the example code, it will be easy to set up in the system's functional code, lending ease to the functional code.

The most insidious aspect of poor design is that when two poorly-designed objects are used together, the resulting code is even worse than the sum of the poorly design code in each object. As more poorly designed code is added to the system, more of it will interact, causing an exponential decline in ease of learning, understanding, and using that code.

To Test, or Not

If you choose to ignore testability because you're not interested in jumping on the unit testing bandwagon, then you're falling prey to an unfortunate side-effect of esoteric jargon.

"Testability" means: "Designed well enough that it can be easily tested - if an only if I am one of those people who's into testing." Don't be set back by the jargon. Object-oriented design terminology is often over-the-top at best and rarely self-evident until you've been initiated into the workings of the minds of people who name things.

Despite the jargon, well-designed code has benefits above and beyond testing. You can test code that is poorly-designed using tools like TypeMock (.NET), but you don't get the essential benefits of good design just because you can test. Be careful not to equate "testability" with the mere ability to test something.

Remember, testing is about making observations, and observations are about learning. Using runtime interception to get at the innards of a system that hasn't been designed to readily offer up knowledge of itself is like making an appendectomy incision with a hand grenade.

The necessary use of force on a system to derive understanding from it is one of those tell-tale signs that the system is poorly-designed. And even if you find a way to test an object that is difficult to test, the object will remain more difficult to maintain because the object has not been shaped to have greater ease. A system of objects that are poorly designed will be much more difficult to maintain the the sum of the difficulty of the individual objects.

Design needs to be supple. Supple so that you can easily peel back layers and partitions like rose petals that fit intricately into each other and learn from what's inside. Blasting open a design with a profiler is probably a sign that your system takes its design hints from a chunk of inert rock rather than a living rose - at once intricate as a whole and simple in its components. Regardless of whether you succeed in testing such an inert, rigid design, the essential qualities of design that generate tremendous leaps in productivity and value will not be present.

Make It

Good design is about knowledge. We wrap it up in all kinds of terminology, but in the end it's about making smallish bits of software that are easy to understand on their own, and that fit together exquisitely to make things bigger than themselves, and that can still be as easily understood.

Design so that you can pick up an object, set it up with ease and with little fuss, and see how it works. Design an object so that you can easily introduce it to another object so that both objects are capable of achieving something greater than the sum of their parts without sacrificing an ability to understand either the parts or the system they form.

An object or a system of objects that is easy to understand also very likely reflects SOLID principles, but make sure of this by writing a bit of code that proves it. If you don't constantly endeavor to prove that your code is SOLID, it won't be. No matter how good you are at software development, software is always more complex than what you can hold in your mind at any time.

Write the code that demonstrates the ease of using your objects. If you write this demo code first, you'll have better results because you'll be writing the objects to fulfill your expectations for ease, rather than trying to force-fit ease into your objects later. Once you've over-stepped the window of opportunity to create ease and productive designs, it's often too costly to ever get it back.

It's hard to work with things that are hard to understand. In the extreme, you simply can't work with something you don't understand. Things that are SOLID are easy to understand because they're easy to observe and they're easy to explore. They make sense. They're logical, and rational, and clean.

When you hear someone say "testability" have some sympathy for the poor bugger - he's usually not aware that he's speaking gibberish. He's been through hell and back trying to discover all this OO stuff on his own, and it takes a great toll over the years. Be thankful that there are people who've made the trek to hell who are willing and able to teach. Your aren't required to loose yourself in the wilds on your own, potentially loosing your own ability to communicate with other humans.

If you achieve the ease in software design, you'll find that other desirable qualities will come along for the ride almost incidentally. The qualities that make design easy to learn also make it reusable, adaptable, distributable, and lend the software you already have to changes in architecture, new non-functional requirements, and the necessary changes to the software that result from changes to the competative environment where your business does its work.

I'm a Customer - Meet Me Where I Am

2009-01-01T02:54:00.004-06:00

Today, I had an issue with a software product that I've been evaluating over the past week. I made a mention of it on Twitter, and in fine contemporary form, the vendor was alerted to a mention of the product name in Twitter stream and got in touch with me. The Twitter contact turned into an email conversation conversation and we dug further into the issue I was having from there.

This kind of customer service interaction is increasingly common with the growing ubiquity of lifestream services like Twitter. It's a form of interaction that is much more human-centric than the technology-centric interactions we've been getting from web self-service silos in the past ten years.

Initiating customer service on Twitter doesn't require me to go to a vendor's customer service silo and hunt and peck for that vendor's particular way that they expect me to contact them. Instead, I just stand right where I am and do the lifestream equivalent of saying, "Acme Co's product is giving me fits," and someone from Acme Co gets in touch with me. It makes traditional web self-service seem quite primitive, and compared to the kinds of customer interactions that are being initiated in Twitter, traditional web self-service is indeed primitive.

Unfortunately, the quality of the interaction with the vendor degraded and became more frustrating (to me) as we got further into the exchange.

The software product in question is a text editor. There is a bug in the copy and paste feature of the editor. Text can be copied to the clipboard easily enough, but after positioning the cursor to where I want the text pasted, the text would actually paste in another location in the document, usually some distance beneath the cursor, or even beneath a subsequent paragraph.

It's quite frustrating to work with a text editor that doesn't get this basic interaction pattern down. Upon further explanation it turned out that the feature was coded this way on purpose to enable some other novel form of copy and paste interaction that was unlike anything that I had seen or used in the myriad text editors that I've used over the paste twenty years.

So, I was a bit irate at this point. It seemed to me like the vendor was trying to prove the point of a new interaction pattern for an interaction that is so well-established that it's not only a well-understood standard, but a standard that is an enabler for end users. But what got my customer experience hackles up came next.

The vendor asked me to post my thoughts on this to the online customer service application, Get Satisfaction so that interest from other customers on this issue could be gauged which would indicate to the vendor whether they should change the behavior. There are two mistakes here - a customer service mistake and a product management mistake.

First, the customer service mistake:

I had already provided the feedback to the vendor. In my customer experience, I had taken the time to describe the problem in an email, and now the knowledge is in the hands of the vendor. Because the vendor's customer service channels had not been integrated, the vendor asked me to do the double entry on the problem report. This is not an example of meeting me where I am, it's an example of lack luster customer experience. It's not my responsibility as a customer who is already contending with ill-conceived aspect of the vendor's product to also contend with ill-conceived aspects of the vendor's customer service systems.

Secondly, the vote:

I appreciate the new democratization of just about everything on the web, but if you're a product maker and you're making product decisions by referendum carried out on electronic media then are typically attractive to only a subset of your user population then your shooting yourself in both feet.

As a product designer, you should already know what it is you're building. You should have a strong vision for the product and already have a visceral sense of the customer's expectations, needs, and desired customer experience. If you don't have these, then by all means fall back to voting systems, but if you don't know what it is that you're building, and you're not clear on who you're building it for, then you might consider not building it!

Customer input is vital input to product design, but it's only one form of input input. It should be a clarifying input to a solid product design vision and ability to execute. There's nothing wrong with taking customer input but if it's the primary means of making product design decisions, then it's possible that the strong sense of design that should be within the product team is missing.

Meeting me where I am means meeting me where I am in all stages of customer experience, from my flow through customer service information systems, services, and communication infrastructure to the representation of my expectations in product design through the product designer's visceral sense of me as a customer, and of the things that I value.

Negative Feedback is a Privilege

2009-01-01T01:39:00.005-06:00

Responding to a spate of criticism from Digg customers, founder Kevin Rose shared his sense of the negativity leveled against Digg and its people. He assured his customers that Digg reads their feedback and that their ideas help to shape Digg.

It can be rough to take negative feedback from customers for a product that you're pored a lot of time into, and maybe Kevin is responding with fatigue rather than from unchecked, personal bias, but his response provides a good example of a product management anti-pattern:

"While we don't respond to every comment thread, do know that we do read them and your constructive suggestions do make it into our product roadmap."

While I would certainly hope that all product managers are blessed with constructive feedback, it's not the role of product management or product design to dictate feedback protocol to customers - especially irate customers. A product manager who does dictate protocol is stepping over the lines and allowing his personal biases and sensitivities to color some of the most valuable customer feedback that he can get.

Irate customers are either people with pervasive anger management issues, or people with higher sensitivities to your product experience that cause them to have greater adverse reactions than average customers. Those angry customers who are truly angry people might well be easily dismissed - as long as you know them well enough to know that their anger stems from something unrelated to your product experience. If you can't differentiate between angry people who happen to be your customers and angry customers, then you're in no position to discount angry feedback. And if you're the kind of person who requires all interactions to be positive in nature, then you're in no position to be in a customer-facing role.

You need serious "people skills" to play a customer engagement role, and by "people skills" I don't mean some sterile naivety that trivializes the range of human expression into a narrow band of emotion that includes only the safest and most genial modes. I mean skills in being a person who is comfortable with the complete range of human expression. In a customer-facing role with clear responsibility for why your product behaves the way it does, you're bound to come across a very broad spectrum of human responses as a result of the intersection of your product and real people who've used it.

Those irate customers are the early warning system for your products flaws. They know now about the things that are going to hurt you later. If you chose to project your personal belief that all interaction must be genial, then you're choosing to cut off a vital flow of critical information that you can't afford to ignore.

The products that we create are expressions of ourselves. Like it or not, these expressions affect people emotionally. They might evoke satisfaction or they might evoke anger. We have to be willing to accept that sometimes the product experiences that we create are indeed offensive to our discerning customers. The appropriate reaction to having imposed on a customer with offense even when the customer responds in-kind - is reflection rather than retaliation.

We have no right as product makers to express offense to our customers through our products and we should be prepared to take our lumps when we do so - especially because it will lead us to valuable insights into our products that we may not have yet been privy to. Reflection will train our awareness to be able to more clearly see what our more astute customers see, and will inevitably teach us to detect and avoid flaws during product design.

It's up to us to see the wide spectrum of customer response as inherently constructive, whether we feel the response is positive or negative or even irate and possibly even as offensive as the product is itself. When we put product in the world that brings frustration into the lives of our customers, then we darn well better be equipped to deal with them when they honor us with their input and not seek to dismiss, dismantle, redirect, or trivialize.

As a product manager, Kevin Rose is on the right path with this open dialog with his customers, but he failed to insulate the sanctum of customer relationship from his own exigencies where the niceties of protocol are concerned. He may have aggravated the situation with the already-irate customers, and this could end up isolating him from valuable, actionable, and vital information about his product.

We have to be willing to see the truth of our products. We have to be willing to discipline ourselves to meaningfully engage with customers regardless of the timbre of their feedback. Feedback is a privilege, not a right, and it's certainly not an annoyance - even when it feels annoying, or even downright aggravating.

Productivity: It Comes from Software Design Rather than Software Tools

2008-12-31T01:34:00.004-06:00

There was a time when big challenges in software development were mostly solvable by tools. It was a simpler time, before every desk and every home had a computer, and before most mid-sized companies were creating their own custom software and before many small companies would even consider building customer software.

It was a time that has come and gone, as have most of the tool vendors of the time. The tool vendors that are left standing have one lasting imperative: continue to make the issue of software development productivity an issue of tools.

Software productivity and quality have been suffering for it with a constancy that should have shocked business leaders into action years ago. Something has gone horribly wrong.

A reasonable and rational business response to poor productivity and throughput in software creation and operation should have already come. It should have been swift and decisive in proportion to the hemorrhage of opportunity and value in custom software efforts.

And yet, business leaders sit, passively resigned to what appears to be an intractable cycle - a notion reinforced throughout IT - that under-performance is the inevitable nature of software production and there's nothing that can be done about it. In the ten or fifteen years since the onset of the ubiquitous custom software boom, this sad story has persisted as the blindly-accepted lore that influences so much about what we believe about the ways and means of custom software production.

The reasonable and rational adverse reaction to software woes isn't happening as it should. We are resigned as a society to bad software and software projects that are even worse. We accept the pain that software developers rain down into our lives, whether in the form of custom software, commercial applications and operating systems, or indecipherable web sites.

And yet, the software crisis isn't really a hard problem to solve. The rehabilitation of software development and the reclamation of IT's reputation and credibility starts with the recognition that dramatic shifts took place while decision makers' eyes were necessarily taken off the ball. We're presently trying to solve problems with solutions that worked at one point, but no longer work. We're directing a generation of software developers to think about software development in ways that are out of date not merely by years but by generations.

Here's a short list of heretical ideas that has changed the game for every software developer and software organization that has successfully put them to work:

Design quality is the most important factor influencing productivity in software development
The things that obstruct quality degrade productivity
The reductions in productivity over time that are typical to tool-driven software development are greater than what can be solved by tools
The application of tools to these problems exacerbates the quality problem, creating a vicious cycle that accelerates exponentially
Quality software design is the product of principles and practices, not tools
The typical degradation in a software's quality over time isn't due to the nature of software, it's due to the nature of the approaches we choose to develop and operate software

We still need tools to support our software efforts, but when we return balance to software development, we find that we need fewer elaborate and expensive commercial tools. Looking back, we see most of the commercial tooling that we use as distractions, wasted capital, and excess.

The essential tools that we do need are far less expensive to acquire and to operate than the elaborate commercial tooling offered as solutions to the software problem. In many cases, essential tools are free, open source tools built to fill the vacuum left by entrenched tool vendors who cannot keep pace with the changing conditions and the evolution of software development in the wild. These tools are typically more mature than their commercial counterparts, and are supported by engaged and engaging software professionals, and they are crafted by people who do in fact use these tools on live-fire custom software projects.

It should go without saying that we should be wary of commercial tool makers who use their tools only to make other commercial tools. These tool makers typically have little to no recent relevant experience in the projects that they believe their successive generations of tools will be appropriate to.

Productivity degrades because software becomes harder to change as its complexity increases. Complexity increases naturally as more code is added to a system over time, be it code for brand new features, or for changes to existing code due to improvements, changes in the business, or due to fixing defects. There comes a point (and it comes quickly) when design flaws become so entrenched in software that they can't be resolved affordably.

The whole trick to tactical software design is an exercise in protecting a very volatile investment from erosion. And this means an unwavering vigilance toward design fundamentals and principles.

Deliberate and tactical software design keeps productivity bottlenecks from taking root in software. With bottlenecks, obstructions, and design friction in play, it becomes harder to move forward. Work items progressively take longer to complete, and one day the productivity is so poor that a total re-write is ordered. Typically, the new system is re-written using the exact same approaches that generated the conditions that lead to the need to re-write the system, and the vicious cycle begins anew.

The traditional productivity curve (which is the inverse representation of the traditional software cost curve) is a result of not protecting software from erosion. The exponential degradation of productivity shows the compounded effect of institutionalized negligence of the software development work that is in fact specifically geared to fend off erosion. Namely, continuous, incremental, design quality stabilization and improvement.

To believe that the traditional productivity curve is a natural part of software development is to indulge the same naive presumption that american manufacturers believed to be a natural law of production before Toyota showed that the presumption is tied to a specific production methodology - a methodology that still forms the basis for most software production methodology used today. If you fundamentally change the methodology, you'll change the rules and the equations that govern the productivity curve.

Some software designs are harder to work with than others. Some designs are even more prone to defects. If you arm your software organization with even a basic understanding of the fundamental design principles and the basic practices engendered by them, you would begin to see benefit immediately. Over time you could reshape the productivity curve entirely, creating more value with your investments in custom software, and deriving value for much longer.

But there are no tools that can do this for you. Not even the so-called software design tools are capable of helping you to apply fundamental software design principles. There are analysis tools that might help you understand where trouble spots might be, but they can't create supple design for you. To defer the responsibility of productivity to tools means that the real issues underpinning productivity will not be addressed, and not only will your desired productivity not be achieved, but you'll experience the complete opposite of what you had hoped for.

There will likely always be commercial tools that are used by software developers, but the real, essential need for these tools is far less than the excesses that we see today in vendor-dominated software development communities and cultures. We're on the cusp of a new era of productivity in software development, but it has very little to do with material investments in tools and everything to do with investment in mature, proven design principles and practices, and the readily-available, low-fi, essential tools that support them. And to this mix we add only the essential commercial tools that support our efforts.

The principles, practices, and yes even the necessary tooling to rehabilitate software development and to put an end to the crisis are already here amongst us. Many software developers have already reached out and harnessed them, and more are waking up to the essence of effective software development every day. Armed with the understanding that productivity comes from design, and that design is an intellectual activity that has only a slight dependency on tooling, a growing number of software developers are making huge strides in proving that our assumptions about traditional software development economics add up to little more than superstition.

One day this period in software development history will seem like the dark ages - a time where our primitive approaches to software development delivered commensurately primitive results. We'll look back and scoff at the now obvious mistake of serving the business needs of tool vendors rather than serving the business needs of the businesses we work for. We're under-performing to a shocking extent and hemorrhaging value at an alarming pace. This bleakness ends when we recognize the real source of productivity and reach out and grab it. The productivity we get from tools is merely a distant fallback position from the productivity that we achieve through software design.

To continue to languish at the mere levels of productivity that tools offer is a deeply-disturbing yet deeply-entrenched behavior in software development. As more software developers and organizations wake up to their true potential, these software dark ages can finally be relegated to history and we can move forward into a renaissance. Hopefully we can do it during this generation rather than continuing to be distracted by the endless, well-funded parade of software tool peddlers whose disproportionate success depends entirely on our willingness to remain distracted from our rightful potential.

Nothing Fails Like Success - Why Continuous Improvment is Continuous

2008-12-28T04:18:00.002-06:00

Without the surrounding and supporting, end-to-end learning organization, an encapsulated team - even if it has begun to turn itself into a learning culture - will sooner or later begin to under-perform. It may even degrade to a point of becoming largely ineffective, leading to re-organization or disbanding.

Continuous improvement is the goal of a learning culture. Like the mechanics of a learning culture, the mechanics of continuous improvement isn't an ad hoc series of suggestions from on high on potentially better ways of doing things, or merely random trial and error acted out by workers in place of dealing with higher priorities.

Continuous improvement is a managed process. Improvements are done with the aim of creating systemic optimizations. The terminology from the Toyota Production System is, "optimize across organizations." Making local improvements without considering their impact on the whole system is hacking rather than improvement.

Each improvement is a change. It creates a new set of conditions. It changes the system. At a fundamental level, improvement creates a new system, albeit with a great number of similarities to the previous system.

The conditions created by a previous improvement aren't perfect. They're likely better than the previous conditions, but they are also inevitably the pre-conditions for the next improvement.

Conditions created by an improvement that create the potential for the next improvement typically aren't predictable. We have to make an improvement and then live in the conditions that result, observing them, remaining vigilant for the next improvement, and being watchful for undesirable local optimizations.

Each success creates the next set of conditions that, if not dealt with, can become the next failure. This cycle is why improvement is necessarily continuous.

Learning Culture

2008-12-27T22:17:00.006-06:00

Adopting Lean Product Development of Lean Production is a commitment to turn your organization into a learning organization.

In the Harvard Business Review article, Decoding the DNA of the Toyota Production System (October, 1999), Steven Spear and H. Kent Bowen attribute western car makers' failures to replicate Toyota's successes to not misunderstanding the unspoken essence of the Toyota's nature and the TPS: Toyota is a learning organization.

The authors describe Toyota's organization and culture as a "community of scientists," from the assembly worker to the executive suite. Western manufacturers vainly attributed their inability to apply what they had learned from visits to Toyota City in Japan to cultural differences between Japanese people and western people. The root problem was indeed cultural difference, but likely not cultural differences rooted in national identity or heritage.

Martin Fowler loosely asserts that is lean is agile and vice versa. For the most part, and within the bounds of the essay, the assertion holds up. I tend to see Martin's analysis more specifically as something like, "The set of qualities that comprise Agile are also present in Lean." The inverse of the assertion, "The set of qualities that comprise Lean are also present in Agile," isn't as resonate, and possibly not very accurate - at least not from the perspective of a mainstream understanding of agile as it lives and breathes here and now.

Billy Hollis, a Microsoft community personality, recently complained that pair programming, the most visible approach to cross-training in agile software development, is an "inefficient way of mentoring." In this observation, Billy is absolutely right. Unfortunately, Billy is talking about agile from a non-practitioner's perspective of a vision of agile as it has become in the mainstream, years into the detrimental effects of the monetization of Scrum training on agile practice, culture, and lore, and the outright hostile disinformation efforts of tooling vendors that initially couldn't move fast enough to meet agile's uptake.

Pair programming isn't an effective way of mentoring, but pair programming isn't a whole mentoring practice, regardless of whether the nascent, disproportionate mid-part of the bell curve continues to assert it as such. But then, even outside of agile, and in the software world at large, meaningful mentoring is usually pretty shoddy at best, and mostly absent or just naive. Pair programming could be a component of a mentoring practice, but it's far too limited to be thought of as mentoring in its entirety, and I seriously doubt that experienced, veteran practitioners see it as such.

Scrum thought leaders have always stated that Scrum works when it's approached as an organizational transformation. And while the organizational transformation terminology is often pretentious posturing found in talk bubbles hovering above boutique consultancies, the spirit of the advice is sound. Unfortunately, it's usually this part of Agile adoption efforts that gets disposed of and disregarded first, dooming Agile efforts to the mediocrity of the mere western style process improvement efforts that Steven Spear and H. Kent Bowen called out when reflecting on the Lean Manufacturing adoption failures of the 80's and 90's.

Despite what is apparently a much better recent track record of companies in the west in adopting Lean, Toyota continues to out-perform it's western counterparts - as is evidenced quite dramatically in the effects of the global economic systems failures on car makers. A good part of its success is due to it's community of scientists organization. Toyota is a learning organization built around and to support a learning culture.

While what has become colloquial Agile asks for a few organizational tweaks - embedded customers, team rooms, adaptation, emergence, etc - it rarely goes so far as to insist that remarkable and lasting success is predicated on the forming and support of a learning culture within a learning organization.

A learning organization marches to a different drummer. It's protocols and processes are built to not only accommodate learning, but to support it and enable it.

A manager's mandate in a learning organization is to see to the advancement of the people he is responsible for. This isn't a secondary responsibility in a learning organization, it's a responsibility amongst the organizations highest priorities and imperatives.

You might imagine that the productivity and accounting models for an organization that doesn't let any opportunities for learning slip by may be quite different than your own organization's models. When every failure, accident, and oversight is dealt with as an opportunity for learning - for the betterment and advancement of a worker and the organization - our traditional western obsession with efficiency management would likely work at odds with the imperatives of continuous improvement.

Learning in a learning organization like Toyota isn't predicated on ad-hoc pauses to accommodate expressions of remorse for breaking a build. It's not a trivial commitment to workers to give them at least two weeks of company-paid training or conference attendance each year. Learning at Toyota is rigorous and scientific. And if a manager or leader can't demonstrate the proper way to perform a task to expectations, then that person would likely not be in a leadership position.

A learning organization doesn't just pause for bi-weekly retrospectives. It methodically seeks improvements. It uses scientific method to plan and execute experiments and measure outcomes. It learns from the rigor of experimentation, and it only experiments on work that has already been standardized and harnessed with standard measures.

If a worker in a learning organization has an idea about how to improve his work, his manager or leader has an obligation to teach the worker how to formulate an experiment that proves the workers ideas, capture the lessons learned, and communicate results to the rest of the organization. The manager or leader must not only know the worker's responsibilities, but also must be able to perform them so that he can guide the worker toward observable improvements that allow the worker to perform to new expectations.

I doubt that Billy Hollis had the Toyota DNA in mind when he offered his criticism of pair programming, but it's a good jumping-off point to start thinking about what it really means to have "mentoring" in software development. So far, we're failing quite dramatically in this area.

The Toyota DNA provides a basis for the establishment of not only a solid mentoring practice in software development, but if we adopt as naively into the software industry as western car manufacturers did before us, we should expect the same failures.

To Billy Hollis' astute observations about mentoring, Agile methods indeed don't necessarily provide guidance for "mentoring", but the entire foundational layer of beliefs about producing software and managing production remain seriously flawed and not just a little antiquated and culturally isolated.

There's nothing that I see in the software development industry that suggests that we're even close to realizing the potential and promise of the learning organizations in our midst. Scrum and agile aren't making the case strongly enough and these approaches aren't really coming to the table with the mature organizational disciplines that might make such change possible.

Agile can't be an answer to the mentoring problem. Agile's focus is much to limited - and necessarily and purposefully so. If we are going start the necessary changes at an organizational level that allow us to really take advantage of a culture of scientists, then we should be looking to methodologies that address organizations as a whole, and maybe even that have decades of experience behind them.

Scrum is arguably a whole-organization methodology, but it's not colloquially known or practiced that way. Lean is an organizational methodology, and if it's not introduced and practiced as such, it's prone to lackluster and possibly even detrimental results. As Martin says, you can practice Agile Development and Lean together. However, if you're committed to the Toyota-like results that come from learning organizations, you might find your best guidance from the methodology that Toyota created to shape its own success.

You can practices Lean and Agile together, and if you're a software organization, you probably should. But understand that the extent of your results will inevitably be a reflection of how far Lean has spread throughout your organization, and understand that Agile has more to say about how software is done rather than how accounting, marketing, logistics, and customer service are done.

Optimize across organizations. Do this in a whole learning organization. Situate your software effort in the midst of such an organization and culture. See the holistic system and organization that effectively addresses mentoring; uses the scientific method as a core practice; sees failure as a trigger for improvement; doesn't resort to recrimination in the face of failure but accepts readily accountability; and is driven by customer value first and foremost - no excuses or spin, and no allowance for the petty, self-interested, entitlement and bureaucracy typical of organizations that aren't yet learning organizations.

Think about what your company could do if its people were truly motivated and organized to success above and beyond the expectations of the surrounding, unprepared market. Can your company be the Toyota of it's market? What's holding you back from wholeheartedly applying what we know about systematizing an enculturating a pusuit and achievement of excellence?

Lazy Loading Considered Harmful

2008-12-20T01:32:00.003-06:00

Lazy loading is a common feature in object-relational data access frameworks. It can produce some really nasty side effects though, and can even be considered harmful. The pattern is considered so risky by some commercial vendors that it isn't included in some products.

Consider a class model that has a Customer class, an Order class, and an OrderLine class. In this model, a Customer is associated with many Orders, and an Order is associated with many OrderLines. The same model is reflected in the relational database, where the application data for this model is stored.

An object-relational data framework allows for order data to be retrieved in the form of an Order object rather than an order data row. This is a great benefit to object-oriented programmers since their environments (.NET, Java, etc) are natively object-oriented. They will inevitably be working with Order objects, and Customer objects, and OrderLine objects.

Objects retrieved from an object-relational data access framework with lazy loading can cause unintended queries to be executed against relational database servers, causing performance, scalability, and data integrity problems.

For example, if a programmer writes some program code that retrieves a Customer object, and then makes use of the Customer's reference to Orders, the data access framework will automatically query all of the Customer's Orders. If the customer has done a lot of business with the company in the past, it may have thousands of orders.

An entire application written on an object-relational data access framework that has lazy loading will have quite a lot of these auto-loading relationships set up between objects.

Serving hundreds of users with this application will put stress on database servers that can become unmanageable. In cases like this, it's beneficial to not use lazy loading at all. It's better to write program code that explicitly loads data when its needed rather than have these risky operations lurking in program code that may be hard to find due to their implicit, transparent, and seemingly innocent nature.

Lazy loading can indeed be harmful, and you can see why commercial vendors might not even include such unsafe features in their object-relational data access frameworks. These kinds of problems can lead to increased database server maintenance and operations costs, excess client licenses expenditures, and even costly business continuity problems in when the excess load put on database servers by lazy loading leads to outages.

Up to this point, this article has been a bit of ruse, filled with many misconceptions about object-relational data access that are unfortunately perpetuated by folks who tend to see object-oriented design through invisible relational data-oriented lenses. The argument is often used to dissuade the use of lazy loading, and has indeed been used by vendors to justify the exclusion of lazy loading from object-relational data access frameworks.

The crux of the issue is the assumption that a Customer object would have an association to its list of Order objects. It's likely not a reasonable way to build a class model for these objects. The Customer class would not have an association to its list of Orders.

Database modeling and class modeling have different rules. They are fundamentally different kinds of technologies, and so the fact that the rules are different shouldn't be much of a surprise. However, if you don't stop to wonder if these differences exist, you might just go ahead and shape your objects and their associations the way you shape the database tables and the relationships that you're used to.

A fixed association between a Customer object and its Order objects is an unnatural association. Although you can conceive of the association in real life, it's not an appropriate association for a class model.

Class models, like many kinds of information models, have natural partitions. The Customer class and the Order class are not part of the same partition. Putting a hard link between them, across their partition boundaries, isn't something that you would simply do without putting some consideration into the design, regardless of whether this association is in a database's data model.

Ironically, there are no hard links between relational database tables. The technology doesn't allow for it. Any conception that we have of hard links between the Customer table and the Order table due to a Customer_ID foreign key field in the Order table is merely a trick of the mind. It's a concept. A Customer row has no knowledge of it's Order rows. An Order row has no knowledge of it's Customer row.

The Order row has a copy of the value of a Customer ID in one of its columns, but that isn't a fixed association or hard link to the actual Customer row object in the database server's memory model. That foreign key value can be used to query the Order table to find the Customer's orders, or can be used to query the Customer table to find an Order's customer, but these data structures don't have fixed associations the way that objects in an object model do.

All tables in relational database data models are partitioned. This is just how relational databases work. When we conceive of fixed associations between database tables, we are merely overlaying our conceptual model on a technological model. We can even implement constraint logic in a relational database server to mimic our conceptual model, but none of this useful wizardry changes the fact that the entities in the underlying technological model are naturally partitioned.

Partitioning is a common technique used to reduce the complexity caused by the associations between resources. Every fixed association between two entities will cause a system itself to become increasingly fixed, or rigid, which makes it increasingly difficult to adapt to new requirements and reparations.

Partitions are found at all levels of a system's architecture - from distributed services off in the cloud, all the way down to the relational databases, and even the disk storage systems beneath them.

Pat Helland writes about partitions in his paper on infinitely scalable systems, Life Beyond Transactions. Roger Sessions talks about partitions in his Simple Iterative Partitions process for decreasing complexity in enterprise architecture. You can find this pattern everywhere in software. Once you realize that it exists, you'll see it everywhere.

Eric Evans writes about a particular partitioning pattern called Aggregate in Domain-Driven Design, Tackling Complexity in the Heart of Software. This pattern is useful in guiding the decisions you can make in designing a class model.

The Customer class and the Order class are in separate aggregates. There may be cases where this isn't true, but for the most part it's likely to hold true. Because they are in separate aggregates, I would think twice about establishing a fixed association between them. Without a fixed association between them, I no longer have the lazy loading risk.

I can still get all Order objects for a Customer by specifically querying the database through a data access class written to support Order data access needs for the application. This query is issued from within the higher level business scenario logic that might need a Customer as well as its Orders. In practice, this pattern covers the situations where you might have presumed a need to query the database for a Customer's orders via a Customer object rather than a data access object for Customer.

From within an aggregate, you might make use of lazy loading or eager loading depending on performance analysis and other empirical knowledge. For example, the Order class does have a fixed association between itself and its OrderLines. When an Order object is retrieved, if it is always necessary to retrieve the Order's lines, then the association would be eager loaded on the spot. If not, it would be lazy loaded - deferring the decision to load the related OrderLines till later in the execution of a business transaction if the Order's lines are referenced.

There are cases where you might not put a fixed association between an Order and OrderLines at all. It depends on the context, the amount of data being loaded, and possibly other factors as well. There are no canonical models - only business contexts that are best served with models crafted to built suit the circumstances, regardless of the unbounded predilections for reuse that we sometimes succumb to.

Lazy loading is harmful if you use it in support of naive class modeling. The absence of lazy loading in object-relational applications is just as harmful, leading to infrastructure code bloat in non-infrastructure classes, poor encapsulation, higher complexity, higher coupling, and code that is harder to understand, test, and maintain.

Any trepidation that people usually have with lazy loading is often rooted is misconceptions with object-oriented programming and design. To use lazy loading safely, start by modeling the object-oriented parts of your application according to object-oriented principles, and recognize that there are different principles for object modeling and data modeling, and that this is a good thing rather than something to hide from.

Any tool is considered harmful when used improperly or naively. Or, like my favorite software development quote says:

Every tool is a weapon - if you hold it right
- Ani Difranco